This repo includes official implementations of ChatCAD and ChatCAD+
[Nature COMMSENG] ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models by Sheng Wang, Zihao Zhao, Xi Ouyang, Tianming Liu, Qian Wang, Dinggang Shen
(a) Overview of our proposed strategy. The image is processed by various networks to generate diverse outputs, which are then transformed into text descriptions. The descriptions, served as a link between visual and linguistic information, are combined as inputs to a large language model (LLM). With its ability to reason and its knowledge of the medical field, the LLM can provide a condensed report. (b) Interactive explanations and medical advice from ChatCAD.
[IEEE TMI] ChatCAD+: Towards a Reliable and Universal Interactive CAD using LLMs by Zihao Zhao*, Sheng Wang*, Jinchen Gu*, Yitao Zhu*, Lanzhuju Mei, Zixu Zhuang, Zhiming Cui, Qian Wang, Dinggang Shen
Overview of our proposed ChatCAD+ system. (a) For patients seeking a diagnosis, ChatCAD+ generates reliable medical reports based on the input medical image(s) by referring to local report database. (b) Additionally, for any inquiry from patients, ChatCAD+ retrieves related knowledge from online database and lets large language model generate reliable response.
This repository provides the official implementation of some components of ChatCAD+:
- Modality identification
- Chinese version Interactive CAD of Chest X-rays
- LLM-based knowledge retrieval
- An easy-deploy local web ui based on Gradio
- We would like to thank Merck Manual Professional who make all these medical knowledge public, we sorted their website for easier usage: here
- A BART-based model that has the capability to translate chest X-ray reports into Chinese well [link]
- R2GenCMN: r2gcmn_mimic-cxr.pth and annotation.json
- PCAM weights: JFchexpert.pth
- Place annotation.json under ./r2g/ and pre-trained weights under ./weights/
- For template retrieval system, please download MIMIC-CXR reports from official website and organize them into a dictionary, save as report_en_dict.json under the ./
You can either find them from original repository or dowload from Google Drive
- pip install -r requirements.txt
- implement web.py and load your openai api-key
If you find our work useful, please consider giving a star ⭐ and citation.
@article{wang2023chatcad,
title={Chatcad: Interactive computer-aided diagnosis on medical image using large language models},
author={Wang, Sheng and Zhao, Zihao and Ouyang, Xi and Wang, Qian and Shen, Dinggang},
journal={arXiv preprint arXiv:2302.07257},
year={2023}
}
@article{zhao2023chatcad,
title={ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs},
author={Zihao Zhao and Sheng Wang and Jinchen Gu and Yitao Zhu and Lanzhuju Mei and Zixu Zhuang and Zhiming Cui and Qian Wang and Dinggang Shen},
journal={arXiv preprint arXiv:2305.15964},
year={2023},
}
Our implementation (including coming version) is based on the following codebases. We gratefully thank the authors for their wonderful works.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent