Mayan Leavitt • Edan Kinderman

• Summary
• The SGRAF Model
• Data
• Proposed Improvements
• Comparison
• Files and Usage
• How to Run the Code
• References and credits

Our project aimed to improve the matching of two X-ray scans with their fitting radiology report, using the SGRAF image-text matching model as a baseline. To achieve this, we tested various loss functions, architectures, and training methods.

Through our experimentation, we successfully incorporated the second X-ray scan into our models and achieved significantly better results. Our research provides insights into enhancing the accuracy of image-text matching, which can have important implications for medical diagnosis and treatment.

The model extract features from the given image and text, and learn vector-based similarity representations between different areas in them. Then a SAF (Similarity Attention Filtartaion) module processes the vectors alignments using attention mechanisms to identify significant alignments and reduce less meaningful ones. The module outputs a matching score indicating the compatibility between the image and text. For more details see the original article [1].

We used the MIMIC-CXR dataset, which contains studies with a frontal image, a lateral image and a radiology report. In the existing image-text matching models, the lateral image is often not used, even though it contains critical information.

1. Check different loss functions: Bi-directional ranking loss, NT-Xnet [2] and their weighted sum.
2. Train two regular SGRAF models simultaneously – one for each viewpoint, and use learned weights to average the similarity scores.
3. Concatenate the two image types features to obtain one input.
4. Use positional encoding to differentiate between the two viewpoints.

Here is an evaluation of the model's ability to match the image with the correct text. A higher R@K value indicates improved retrieval performance, indicating a better alignment between the image and the corresponding text.

Here is a comparison of the basic models, which trained only on one type of image (frontal or lateral).

Here is a comparison of the "double" models family, which has two encoders for encoding each image type (frontal and lateral). Those models are trained on both image types.

Here is a comparison of the "concatenation" models family, which gets as input a text and a concatenation of the frontal and lateral images. Some of those models trained with positional encoding [4] added to the images.

We can see that using the lateral images improves results as opposed to using frontal data alone. In addition, training two models at once achieves the best performance, but concatenating image features is a cheaper way to combine viewpoints.

You can train a regular SGRAF model on the MIMIC-CXR dataset, using only frontal images, with this script:

opts_xray.py --model_name '../checkpoint/<model_name>' --view 'frontal' --model_num <number> --model_type 'regular_model' --batch_size 64 --num_epochs 40

• Project supervisor: Gefen Dawidowicz. Some of the algorithms were implemented based on her code.
• [1] Z. M. L. Diao, "Similarity Reasoning and Filtration for Image-Text Matching", AAAI conference on artificial intelligence, 2021.
• [2] K. N. H. Chen, “A Simple Framework for Contrastive Learning of Visual Representations” PMLR , pp. 1597-1607, 2020.
• [3] S. M. S. P. G. Ji, “Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment” MLMI , pp. 110-119, 2021.
• [4] S. P. U. J. N. G. K. P. Vaswani, “Attention is all you need” Advances in neural information processing systems, 2017.