rajpurkarlab / cxr-report-metric Goto Github PK

License: MIT License

Jupyter Notebook 8.45% Python 87.38% Dockerfile 0.42% Shell 1.68% Perl 0.81% Jsonnet 1.27%

cxr-report-metric's Introduction

Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation

Repository referenced in the paper "Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation". This repository provides code for computing metric scores for radiology report evaluation. The metrics are:

BLEU
BERTscore
CheXbert labeler vector similarity
RadGraph entity and relation F1
Composite metric RadCliQ

Prerequisites
Requirements
Usage
License
Citing

Prerequisites

To install the dependencies, run the following command with Python 3.7:

pip install -r requirements.txt

Requirements

Ground Truth and Predicted reports must be arranged in the same order in a column named "report" in two CSV files. The CSVs should also contain a corresponding "study_id" column that contains unique identifies for the reports.

In config.py, set GT_REPORTS and PREDICTED_REPORTS to paths to the CSVs. Set OUT_FILE to the desired path for the output metric scores.

CheXbert

To compute the CheXbert vector similarity (s_emb) metric score, download the CheXbert model checkpoint here. Set CHEXBERT_PATH in config.py to the path to the downloaded checkpoint.

The code for computing the CheXbert metric score is adapted from stanfordmlgroup/CheXbert.

Paper (Accepted to EMNLP 2020): https://arxiv.org/abs/2004.09167.

RadGraph

To compute the RadGraph metric score, download the RadGraph model checkpoint from PhysioNet here. The checkpoint file can be found under the "Files" section at path models/model_checkpoint/. Set RADGRAPH_PATH in config.py to the path to the downloaded checkpoint.

The code for computing the RadGraph metric score is adapted from dwadden/dygiepp. Note: You need to apply for credentialed access to RadGraph on PhysioNet.

Paper (Accepted to NeurIPS 2021): https://arxiv.org/abs/2106.14463.

Usage

from CXRMetric.run_eval import calc_metric
calc_metric(gt_reports, predicted_reports, out_file, use_idf)

License

This repository is made publicly available under the MIT License.

Citing

If you are using this repo, please cite this paper:

@article {Yu2022.08.30.22279318,
	author = {Yu, Feiyang and Endo, Mark and Krishnan, Rayan and Pan, Ian and Tsai, Andy and Reis, Eduardo Pontes and Fonseca, Eduardo Kaiser Ururahy Nunes and Ho Lee, Henrique Min and Abad, Zahra Shakeri Hossein and Ng, Andrew Y. and Langlotz, Curtis P. and Venugopal, Vasantha Kumar and Rajpurkar, Pranav},
	title = {Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation},
	elocation-id = {2022.08.30.22279318},
	year = {2022},
	doi = {10.1101/2022.08.30.22279318},
	publisher = {Cold Spring Harbor Laboratory Press},
	URL = {https://www.medrxiv.org/content/early/2022/08/31/2022.08.30.22279318},
	eprint = {https://www.medrxiv.org/content/early/2022/08/31/2022.08.30.22279318.full.pdf},
	journal = {medRxiv}
}

cxr-report-metric's People

Contributors

Stargazers

Watchers

Forkers

rphlstck jennyziyi-xu akshayswaminathan vuplm2004 deepspeed1

cxr-report-metric's Issues

Support for newer PyTorch versions

This is related to #3.

We are running A6000 GPUs that are not compatible with PyTorch 1.6. Also, our whole codebase is on PyTorch 2.0. We are unable to build any Conda environment that is compatible with the package here. This makes it really cumbersome to use the metrics provided by this repo: we always have to compute metrics in separate processes, and on separate older machines - this is super awkward and time consuming

At the same time, the metrics provided by the package here are super valuable, and we would love to use them more, and in better ways than we can currently

A release of this package that runs with current PyTorch, or at least with a set of more recent PyTorch versions, would be super helpful for our work.
@KathyFeiyang you seem to be most active here on the repo, is that something your team could provide?

Cannot load composite_metric_v1_model

ipdb> composite_metric_v1_model = pickle.load(f)
*** AttributeError: Can't get attribute 'CompositeMetric' on <module 'main'>

Sample of gt_reports and predicted_reports

Could you please give a sample demonstration of gt_reports and predicted_reports?
Is it any X-ray dataset that can be measured please?

fast_bleu Installation

Unable to install the requirements in order to run the program. Any solutions?

(myenv) kinal@Kinals-MacBook-Air CXR-Report-Metric-main % pip install -r requirements.txt
Collecting allennlp==1.1.0 (from -r requirements.txt (line 1))
Using cached allennlp-1.1.0-py3-none-any.whl (485 kB)
Collecting allennlp_models==1.1.0 (from -r requirements.txt (line 2))
Using cached allennlp_models-1.1.0-py3-none-any.whl (322 kB)
Collecting beautifulsoup4==4.8.1 (from -r requirements.txt (line 3))
Using cached beautifulsoup4-4.8.1-py3-none-any.whl (101 kB)
Collecting bert-score==0.3.11 (from -r requirements.txt (line 4))
Using cached bert_score-0.3.11-py3-none-any.whl (60 kB)
Collecting fast-bleu==0.0.90 (from -r requirements.txt (line 5))
Using cached fast-bleu-0.0.90.tar.gz (14 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting lxml (from -r requirements.txt (line 6))
Using cached lxml-4.9.2-cp39-cp39-macosx_10_15_x86_64.whl (4.8 MB)
Collecting numpy==1.18.2 (from -r requirements.txt (line 7))
Using cached numpy-1.18.2.zip (5.4 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [198 lines of output]
Running from numpy source directory.
:461: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-install-tdit1xs0/numpy_e9879434578b4ba18fa5ac06a576db5a/tools/cythonize.py:75: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
required_version = LooseVersion('0.29.14')
/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-install-tdit1xs0/numpy_e9879434578b4ba18fa5ac06a576db5a/tools/cythonize.py:77: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(cython_version) < required_version:
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/_philox.pyx
Processing numpy/random/_bit_generator.pyx
Processing numpy/random/_bounded_integers.pyx.in
Processing numpy/random/_sfc64.pyx
Processing numpy/random/_mt19937.pyx
Processing numpy/random/mtrand.pyx
Processing numpy/random/_generator.pyx
Processing numpy/random/_pcg64.pyx
Processing numpy/random/_common.pyx
Cythonizing sources
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
libraries mkl_rt not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE

  blis_info:
    libraries blis not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  openblas_info:
    libraries openblas not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  atlas_3_10_blas_threads_info:
  Setting PTATLAS=ATLAS
    libraries tatlas not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  atlas_3_10_blas_info:
    libraries satlas not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  atlas_blas_threads_info:
  Setting PTATLAS=ATLAS
    libraries ptf77blas,ptcblas,atlas not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  atlas_blas_info:
    libraries f77blas,cblas,atlas not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  accelerate_info:
    libraries accelerate not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
  Library accelerate was not found. Ignoring
    libraries veclib not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
  Library veclib was not found. Ignoring
    FOUND:
      extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
      extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
      define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
  
    FOUND:
      extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
      extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
      define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
  
  non-existing path in 'numpy/distutils': 'site.cfg'
  lapack_opt_info:
  lapack_mkl_info:
    libraries mkl_rt not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  openblas_lapack_info:
    libraries openblas not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  openblas_clapack_info:
    libraries openblas,lapack not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  flame_info:
    libraries flame not found in ['/Library/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE
  
  atlas_3_10_threads_info:
  Setting PTATLAS=ATLAS
    libraries lapack_atlas not found in /Library/Frameworks/Python.framework/Versions/3.9/lib
    libraries tatlas,tatlas not found in /Library/Frameworks/Python.framework/Versions/3.9/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries tatlas,tatlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib
    libraries tatlas,tatlas not found in /usr/lib
  <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
    NOT AVAILABLE
  
  atlas_3_10_info:
    libraries lapack_atlas not found in /Library/Frameworks/Python.framework/Versions/3.9/lib
    libraries satlas,satlas not found in /Library/Frameworks/Python.framework/Versions/3.9/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries satlas,satlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib
    libraries satlas,satlas not found in /usr/lib
  <class 'numpy.distutils.system_info.atlas_3_10_info'>
    NOT AVAILABLE
  
  atlas_threads_info:
  Setting PTATLAS=ATLAS
    libraries lapack_atlas not found in /Library/Frameworks/Python.framework/Versions/3.9/lib
    libraries ptf77blas,ptcblas,atlas not found in /Library/Frameworks/Python.framework/Versions/3.9/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  <class 'numpy.distutils.system_info.atlas_threads_info'>
    NOT AVAILABLE
  
  atlas_info:
    libraries lapack_atlas not found in /Library/Frameworks/Python.framework/Versions/3.9/lib
    libraries f77blas,cblas,atlas not found in /Library/Frameworks/Python.framework/Versions/3.9/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries f77blas,cblas,atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib
    libraries f77blas,cblas,atlas not found in /usr/lib
  <class 'numpy.distutils.system_info.atlas_info'>
    NOT AVAILABLE
  
    FOUND:
      extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
      extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
      define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
  
  /private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'define_macros'
    warnings.warn(msg)
  running dist_info
  running build_src
  build_src
  building py_modules sources
  creating build
  creating build/src.macosx-10.9-x86_64-3.9
  creating build/src.macosx-10.9-x86_64-3.9/numpy
  creating build/src.macosx-10.9-x86_64-3.9/numpy/distutils
  building library "npymath" sources
  gfortran: warning: this compiler does not support x86 ('-arch' option ignored)
  gfortran: warning: this compiler does not support x86 ('-arch' option ignored)
  gfortran: warning: this compiler does not support x86 ('-arch' option ignored)
  gfortran: warning: this compiler does not support x86 ('-arch' option ignored)
  gfortran: warning: this compiler does not support x86 ('-arch' option ignored)
  gfortran: warning: this compiler does not support x86 ('-arch' option ignored)
  gfortran: warning: this compiler does not support x86 ('-arch' option ignored)
  gfortran: warning: this compiler does not support x86 ('-arch' option ignored)
  Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 380, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 488, in <module>
    File "<string>", line 480, in setup_package
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-install-tdit1xs0/numpy_e9879434578b4ba18fa5ac06a576db5a/numpy/distutils/core.py", line 171, in setup
      return old_setup(**new_attr)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/command/dist_info.py", line 99, in run
      self.egg_info.run()
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-install-tdit1xs0/numpy_e9879434578b4ba18fa5ac06a576db5a/numpy/distutils/command/egg_info.py", line 26, in run
      self.run_command("build_src")
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-build-env-jd0r0z4_/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-install-tdit1xs0/numpy_e9879434578b4ba18fa5ac06a576db5a/numpy/distutils/command/build_src.py", line 146, in run
      self.build_sources()
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-install-tdit1xs0/numpy_e9879434578b4ba18fa5ac06a576db5a/numpy/distutils/command/build_src.py", line 157, in build_sources
      self.build_library_sources(*libname_info)
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-install-tdit1xs0/numpy_e9879434578b4ba18fa5ac06a576db5a/numpy/distutils/command/build_src.py", line 290, in build_library_sources
      sources = self.generate_sources(sources, (lib_name, build_info))
    File "/private/var/folders/pj/4nm1288n4rxd04t1f67hk8cm0000gn/T/pip-install-tdit1xs0/numpy_e9879434578b4ba18fa5ac06a576db5a/numpy/distutils/command/build_src.py", line 380, in generate_sources
      source = func(extension, build_dir)
    File "numpy/core/setup.py", line 661, in get_mathlib_info
      raise RuntimeError("Broken toolchain: cannot link a simple C program")
  RuntimeError: Broken toolchain: cannot link a simple C program
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Issue in finding a csv file.

I am implementing code of XREM paper in which one file uses the "CXR-Report-Metric" whose code look like this below:
"
import pandas as pd
import argparse

#CXR-Metric computes the scores based on 2,192 samples from the mimic-cxr test set that consists of 3,678
#This function selects the respective 2,192 reports
def main(fpath, opath):
df = pd.read_csv(fpath)
test = pd.read_csv('../data/mimic_test_impressions.csv')
gt = pd.read_csv('../data/cxr2_generated_appa.csv')[['dicom_id', 'study_id']] #contains the dicom ids of the 2,192 imgs
gt['report'] = [None] * len(gt)
pred = df['filtered']
pred = pd.concat([test[['dicom_id']], pred], axis = 1)
pred = pred.set_index('dicom_id')
for idx, row in gt.iterrows():
dicom_id, study_id, _ = row
gt['report'][idx] = pred['filtered'][dicom_id]
gt.to_csv(opath, index = False)

if name == 'main':
parser = argparse.ArgumentParser()
parser.add_argument('--fpath', default='../ifcc/example_m2trans_nli.csv')
parser.add_argument('--opath', default='pred.csv')
args = parser.parse_args()
main(args.fpath, args.opath)

I am not sure here what file to pass here in 'gt' variable. Is it ground truth files with labels? If yes then it doesn't contain study_id and dicom_id. Can you explain this what gt files look like any sample of gt file. @KathyFeiyang any lead will be highly appreciated.

Question regarding regression model (RadCliQ)

Hey,
first of all thank you for the great work!
I have a question regarding the regression model aka RadCliQ. If I understood correctly, RadCliQ is a regression model, which predicts the average number of errors a radiologist would assign to a generated report. The RadCliQ input parameters are the BLEU score and the RadGraph score, which are calculated given a generated report and a ground truth report. (Hopefully, I understood this part correctly.)

Now coming to the question: Why did you only provide these two scores to the regression model and not additionally the BERTScore as it appears to have a good correlation to the radiologists error voting as seen in figure 3 of your paper? (https://www.medrxiv.org/content/10.1101/2022.08.30.22279318v1.full.pdf)

pytorch 1.6 is not compatible with nvidia 30 series gpu

I have tried to build the project on 3060 and 3090, and they are not compatible with PyTorch 1.6. pytorch 1.6 requires cudatoolkit=10.2, while 30x can't run on cuda 10.2. I wonder if you can upgrade PyTorch? Or there is any possibility that I can change some code to run on my GPU?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.