Shim values are being extracted as tune-up, standard, and advanced. However, they are not very informative. Consider extracting actual Shim values used. For ex. ShimSetting field in JSON sidecar.
"ShimSetting": [ -3141, -7171, 3588, 510, 25, -150, 52, 0 ],
See more information about associated DICOM tags in dcm2niix

Although not urgent but, how can we make these compliance checks realtime ?

1. We cannot install anything on the main desktop which is connected to the scanner
2. If scanned data is constantly pushed to a destination folder, mrQA can check it against a reference protocol and send a message to adjacent desktop which can be checked by the technician.

Useful resource : YARRA

Given, a report it is very difficult to know where this non-compliant subject/session is stored on disk. The path is required to address the non-compliance for further analysis. Add a hyperlink to each non_compliant subject name, which points to a text file. The text file would contain complete path on disk

Add warnings to the report for different scenarios
Could not calculate majority because of (equal count/ less than 3 subjects)

Per our discussion, mrQA seems to not consider AP/PA pairing in BIDS datasets and then just reports inconsistency in PhaseEncodingDirection somehow, e.g.

Filed an issue so I get alerted when it gets fixed ;)

The user should be able to specify a reference protocol for each modality. It should support multi-echo times and values of each parameter. The vendor should also be specified. A single json file may contain multiple protocols and the software should use them to check compliance for corresponding scans.

The json file can also include enhancements such as taking multiple values for a single parameter, or accepting a range of values, or even specifying the tolerance. This JSON file should read into as an instance of a custom class, which should be used to check compliance.

Create a report in which a user can click on plots

Add a timeline of non-compliance on the report

• Should provide information about when non-compliance occured.
• Useful to understand if the recent sessions were non-compliant or it was years before

We observed that Siemens scans were much more compliant (> 95 %) than GE and Philips (~75 %). As discussed in faculty meeting, it would be much more informative to this issue builds from vendor or there is some hidden issue. It was suggested to check software versions, model information and site name for each scan.

Site names are not present in DICOM tags. But the software version and model name is present. A preliminary analysis on 375 subjects shows

1. Siemens had no issues in software versions. All scans were performed with same software version - syngo MR E11 even though two scanner models were used - Prisma and Prisma-Fit.
2. Philips scans have multiple software versions
   5.3.0.0
   5.3.1.0
   5.3.1.1,
   5.3.0.3
   5.3.0.0
   And two different models were used Achieva dStream, and Ingenia
3. Similarly, for GE there are multiple versions
   27_LX_MR Software release:DV26.0_R01_1725.a,
   27_LX_MR Software release:DV26.0_R02_1810.b,
   27_LX_MR Software release:DV25.1_R01_1617.b,
   25_LX_MR Software release:DV25.0_R02_1549.b,

And there are 2 models - DISCOVERY MR750, Signa Creator

It would be interesting to investigate if these software versions are indeed leading to inconsistencies we observe in parameters.

Keywords like Horizontal/Vertical Audit
Pre/post acquisition check
Criteria - Exact/Range of values for a single parameter

we can even cross-check 3 or more sequences at a time, although this would be based on just one run of those sequences:

for subj, sess, runs, seqs in ds.traverse_vertical_multi(*three_seqs):
    print(f'\n{subj} {sess:3}')
    for rr, ss in zip(runs, seqs):
        print(f'\t{str(ss):>120}\t{rr}')


NDAR_INVVA90J10J 1.3.12.2.1107.5.2.43.167003.30000017062011234526400000025
	                        ABCD-DTI,_SIEMENS,_mosaic,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=4200,TE=89)	1.3.12.2.1107.5.2.43.167003.2017062116561732733026206.0.0.0
	                   ABCD-Diffusion-FM-PA,_SIEMENS,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=12400,TE=89)	1.3.12.2.1107.5.2.43.167003.2017062116542590754325348.0.0.0
	                   ABCD-Diffusion-FM-AP,_SIEMENS,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=12400,TE=89)	1.3.12.2.1107.5.2.43.167003.2017062116554953449925777.0.0.0
NDAR_INVVAGD75XZ 1.3.12.2.1107.5.2.43.166003.30000017070618303923200000004
	                        ABCD-DTI,_SIEMENS,_mosaic,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=4200,TE=89)	1.3.12.2.1107.5.2.43.166003.2017070616245938834992278.0.0.0
	                   ABCD-Diffusion-FM-PA,_SIEMENS,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=12400,TE=89)	1.3.12.2.1107.5.2.43.166003.201707061624039605491420.0.0.0
	                   ABCD-Diffusion-FM-AP,_SIEMENS,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=12400,TE=89)	1.3.12.2.1107.5.2.43.166003.201707061624328940391849.0.0.0
NDAR_INVVA82JDEJ 1.3.12.2.1107.5.2.43.67064.30000017061416254770700000103
	                        ABCD-DTI,_SIEMENS,_mosaic,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=4200,TE=89)	1.3.12.2.1107.5.2.43.67064.2017061815005347256561769.0.0.0
	                   ABCD-Diffusion-FM-PA,_SIEMENS,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=12400,TE=89)	1.3.12.2.1107.5.2.43.67064.2017061814592319229060911.0.0.0
	                   ABCD-Diffusion-FM-AP,_SIEMENS,_original_(baseline_year_1_arm_1)(FA=90,PED=COL,SSEQ=EP,TR=12400,TE=89)	1.3.12.2.1107.5.2.43.67064.2017061815002572536761340.0.0.0

preferably a neuroimaging dataset

Description

We would like to introduce mrQA to our "pipeline" of data acquisition. I wish there was a mode where mrQA could have been used on DICOMs in an incremental fashion. E.g. we collect 1st subject/session, run mrQA which extracts all metadata etc, stores it and produces report on what it can tell up to that point. Then for the next subject/session we point mrQA to prior save extracts and that new acqusition(s) so it could update extracts and the report.

Let me know if you need me to elaborate more on this.

Not quite sure how to name it in addition to "vertical" and "horizontal" (IMHO an easier to grasp something like "cross-{dimension}" would be better so "cross-site" and "cross-subject" correspondingly; and here "cross-sequence"). To do some consistency analysis under assumptions across different sequences within specific exam.
E.g. correspondence (of shims, geometry, etc) of fieldmaps to func/dwi in BIDS "style" based on their assignment using IntendedFor field in side-car files. Also a check that all func/dwi do have fieldmaps if typically there is a fieldmap.

Problem: Just using datatype (e.g. anat or func) is not sufficient. It leads to erroneous compliance reports.
Why:

1. A user may use a custom label to distinguish a different set of parameters for acquiring the same modality. This can be highlighted in filename by the label acq.
2. Similarly, different tasks in fmri may have different set of acquisition parameters

Therefore, it is not quite right to raise alarms for non-compliance, when the changes in parameters are intended,

2. Try if we can also show multiple sites in the same audit figure

Inherit modality class to create multi-echo modalities and normal modalities, will have to change code for compliance accordingly

Currently, the series description for MRI sequences acquired with or without CO2 inhalation is identical. This poses a significant challenge in distinguishing between the two sets of sequences when analyzing the data. How can we accurately identify and separate these sequences with CO2 inhalation and without CO2 inhalation?

Potential directions

• Look for a DICOM tag
• Look for annotations by MR technicians
• Anything in private headers?

So there is a command line tool which satisfies bids-apps interface and then listed on https://bids-apps.neuroimaging.io/apps/ with docker images built and then we would get also absorb it into https://github.com/ReproNim/containers/ datalad dataset.

• mrQA version: pip current version
• Python version: 3.10
• Operating System: Centos 7

Description

Siemens enhanced DICOMs without dcm extension doesn't find DICOMS

What I Did

(qctools) [adamraikes@gpu68 allo_phase2]$ mrqa --data-source $PWD/dicoms/000052/Screening/666e4ef2/ --format dicom --name test     
Traceback (most recent call last):
  File "/home/u26/adamraikes/.conda/envs/qctools/bin/mrqa", line 8, in <module>
    sys.exit(main())
  File "/home/u26/adamraikes/.conda/envs/qctools/lib/python3.10/site-packages/mrQA/cli.py", line 85, in main
    check_compliance(dataset=dataset,
  File "/home/u26/adamraikes/.conda/envs/qctools/lib/python3.10/site-packages/mrQA/project.py", line 63, in check_compliance
    raise DatasetEmptyException
MRdataset.config.DatasetEmptyException: Expected Sidecar DICOM/JSON files in --data_source. Got 0 DICOM/JSON files.

• mrQA version:

"impossible" to figure out -- please add --version:

bids@rolando:/inbox/BIDS/Wager/Wager/1076_spacetop$ ~/.local/bin/mr_proto_compl --version
usage: mr_proto_compl -d DATA_ROOT [-o OUTPUT_DIR] [-s STYLE] [-n NAME] [-h] [-r] [-v] [-ref REFERENCE_PATH] [--strategy STRATEGY] [--include_phantom] [--metadata_root METADATA_ROOT] [-l LOGGING] [--skip SKIP [SKIP ...]]
mr_proto_compl: error: the following arguments are required: -d/--data_root

• Operating System: singularity reproin container

bids@rolando:/inbox/BIDS/Wager/Wager/1076_spacetop$ ~/.local/bin/mr_proto_compl --help | grep -A1 -e '-s STYLE,'
  -s STYLE, --style STYLE
                        type of dataset, one of [xnat|bids|other]

So - no dicom is given as an option at all.

bids@rolando:/inbox/BIDS/Wager/Wager/1076_spacetop$ ~/.local/bin/mr_proto_compl --data_root ./sourcedata/sub-0001/ses-01/ --output_dir .heudiconv/mrQA --style other
/home/bids/singularity_home/.local/lib/python3.9/site-packages/mrQA/cli.py:84: UserWarning: Expected a unique identifier for caching data. Got NoneType. Using a random name. Use --name flag for persistent metadata
  dataset = import_dataset(data_root=args.data_root,
Traceback (most recent call last):
  File "/home/bids/singularity_home/.local/bin/mr_proto_compl", line 8, in <module>
    sys.exit(main())
  File "/home/bids/singularity_home/.local/lib/python3.9/site-packages/mrQA/cli.py", line 84, in main
    dataset = import_dataset(data_root=args.data_root,
  File "/home/bids/singularity_home/.local/lib/python3.9/site-packages/MRdataset/base.py", line 81, in import_dataset
    dataset_class = find_dataset_using_style(style.lower())
  File "/home/bids/singularity_home/.local/lib/python3.9/site-packages/MRdataset/base.py", line 113, in find_dataset_using_style
    dataset_lib = importlib.import_module(dataset_modulename)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'MRdataset.other_dataset'

Nifti files have a small header and they can be checked for compliance. Add it to bids_dataset compliance check

the session id (via SeriesNumber) is changing even for different sequences within the same folder (which I assume are coming from the same session). Slack

can we we use these dicom tags instead.
(0008, 0012) Instance Creation Date
(0008, 0013) Instance Creation Time

A report has the list of non-compliant values against the reference value. In case there are lot of different values, it would be helpful to have a min max value included there itself.

Currently, mrQA compares each sequence from a MRI session against the reference protocol to check for non-compliant parameters. But, if a particular sequence was not acquired in a session, we don't raise any warning about it.

It is very important that any missing sequences for a particular subject in a session should be reported.

This issue makes more sense when a reference protocol is provided.
I feel this would raise much more issues, and there should be a flag to disable it in case, we know that MRI dataset we are working with ,is not research-grade.

• mrQA version: pip installed
• Python version: 3.10
• Operating System: Centos 7

Description

Attempted to run this on mouse DICOMs from a Bruker 70/20 7-Tesla system running Paravision 360 v3.3 (backend software). Outputs include enhanced DICOMs. It appears to look for DICOMs in the right location but then reports no DICOMs

What I Did

(qctools) [adamraikes@gpu68 app_apoe]$ mrqa -d dicoms -f dicom -o mrqa -n compliance -v
2023-10-20 08:11:59,272 - INFO - Created temp file in mrqa
2023-10-20 08:12:24,233 - INFO - Localizer: Skipping /xdisk/adamraikes/app_apoe/dicoms/20230512_114500_apoemouse_APPCA_C1_VM6979_1_11/1/pdata/1/dicom
2023-10-20 08:12:29,049 - INFO - ACR/Phantom: /xdisk/adamraikes/app_apoe/dicoms/20230512_114500_apoemouse_APPCA_C1_VM6979_1_11/5/pdata/1/dicom
DicomDataset compliance is empty.
Traceback (most recent call last):
  File "/home/u26/adamraikes/.conda/envs/qctools/bin/mrqa", line 8, in <module>
    sys.exit(main())
  File "/home/u26/adamraikes/.conda/envs/qctools/lib/python3.10/site-packages/mrQA/cli.py", line 85, in main
    check_compliance(dataset=dataset,
  File "/home/u26/adamraikes/.conda/envs/qctools/lib/python3.10/site-packages/mrQA/project.py", line 63, in check_compliance
    raise DatasetEmptyException
MRdataset.config.DatasetEmptyException: Expected Sidecar DICOM/JSON files in --data_source. Got 0 DICOM/JSON files.

Here's the contents of that folder:

(qctools) [adamraikes@gpu68 app_apoe]$ ls /xdisk/adamraikes/app_apoe/dicoms/20230512_114500_apoemouse_APPCA_C1_VM6979_1_11/5/pdata/1/dicom
4_CIBS_MultishellDWI_EnIm1.dcm

Just as a sanity check:

(qctools) [adamraikes@gpu68 dicom]$ python
Python 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydicom import dcmread
>>> ds = dcmread('4_CIBS_MultishellDWI_EnIm1.dcm')
>>> ds
Dataset.file_meta -------------------------------
(0002, 0000) File Meta Information Group Length  UL: 196
(0002, 0001) File Meta Information Version       OB: b'\x00\x01'
(0002, 0002) Media Storage SOP Class UID         UI: Enhanced MR Image Storage
(0002, 0003) Media Storage SOP Instance UID      UI: 2.16.756.5.5.200.8323328.195705.1683918037.311.3.0
(0002, 0010) Transfer Syntax UID                 UI: Explicit VR Little Endian
(0002, 0012) Implementation Class UID            UI: 1.2.276.0.7230010.3.0.3.6.6
(0002, 0013) Implementation Version Name         SH: 'OFFIS_DCMTK_366'

....

Allowing a range of values for a single parameter
Maybe even more different type of checks
Will be helpful in hierarchical checks (e.g. within modality within session)

Siemens automatically generates motion corrected series for fMRI sequences. We observe that several fMRI sequences are collected together under a new sequence, called mocoseries. As they are different sequences with different protocols, it shows up in the report as non-compliance. We saw that the reference protocol computed is very similar to fmri-ringrewards.

How can we differentiate between raw formats and derivatives? Is there a DICOM flag?

Basic barebones working

see #23 (comment)

See e.g. https://github.com/bids-standard/bids-specification/tree/master/src/schema/rules/checks

Feel welcome to close upon reading.

a useful feature/script would be to produce subsets of the input dataset that are compliant, to enable users who would like to account for it in their analysis in some manner
the data with MRdataset can easily achieve it, and producing clusters of subject/sessions with identical parameter values.. usually no more than 2 or 3 clusters

Reference: Slack

How to decide if the variance in values is expected? Like a modality may have multiple echo times, but it might also be due to non-compliance.

Use echo-numbers for dicom.
Add option for stratification

1. The deviations are because the dataset is aggregated in this manner, not necessarily an issue of non-compliance, but an uninformed user may not know, which may lead to confounds
2. Make a small table on categorizing parameters as critical/important. But, how do we do that?

Create dummy datasets which can be post publicly, and re-run tests

mrQA monitoring produces a full report on the entire dataset. But sometimes for ongoing projects, it might be useful to have a log, for example

• what was the update ?
• Are there new subjects ?
• Was there a new compliance issue?

If we can reconfigure the monitor monitoring module to take into any previous reports into account to put a little log, that would be useful too.
Reference : Slack

Description

Throw the report in text format to the terminal itself — if there is an easy HTML to text export option

Review initial draft

ATM functionality exposed on command line via a collection of scripts

        'console_scripts': [
            'protocol_compliance=mrQA.cli:main',
            'mr_proto_compl=mrQA.cli:main',
            'mrpc_subset=mrQA.run_subset:main'
        ],

which have no common prefix or consistency in naming. Makes it hard to impossible to recall what command I should run for doing mrQA.

Similarly to many other tools (git, datalad, ...) I recommend you to provide a single point of entry mrqa command line tool which would expose those three commands (may be renamed also to become a bit more descriptive).

Creating such interfaces is very easy with click library which we use e.g. for dandi-cli. Here is click documentation on creating such "groups" of CLI: https://click.palletsprojects.com/en/8.1.x/commands/ . See e.g a little "advanced" example to define dandi upload where we also reuse definitions for some options (--instance) https://github.com/dandi/dandi-cli/blob/master/dandi/cli/cmd_upload.py etc.

You could then also benefit from getting shell completion for the script - see https://click.palletsprojects.com/en/8.1.x/shell-completion/#shell-completion and our "helper" to make it easier to activate it: https://github.com/dandi/dandi-cli/blob/master/dandi/cli/cmd_shell_completion.py#L29 .

If MANY subjects are removed, it is better to rerun mrQA

we need to add coil info parameter(s) for protocol compliance checks.

Getting info on this is not easy and varies across vendors unfortunately (see here), but for siemens, it seems to be at 0x0051, 0x100F, so it should be easy for us to add.

In heudiconv, and reproin heuristic in particular we not only convert to BIDS datasets, but also "archive" original DICOMs under the sourcedata/ in mirroring converted to BIDS data hierarchy. See e.g. https://datasets.datalad.org/?dir=/dbic/QA/sourcedata/sub-emmet/ses-20180531/fmap which accompanies nii.gz's in http://datasets.datalad.org/?dir=/dbic/QA/sub-emmet/ses-20180531/fmap .

Since bids analytics are limited to metadata fields extracted, I thought it would have been cool for mrQA to just operate on those original DICOMs which we have. (BTW heudiconv can convert from DICOMs being wrapped in such tarballs -- comes handy) . But I do not think that mrQA is supporting that as part of the dicom style:

bids@rolando:/inbox/BIDS/Wager/Wager/1076_spacetop$ ~/.local/bin/mr_proto_compl --data_root ./sourcedata/sub-0001/ses-01/ --output_dir .heudiconv/mrQA --style dicom
/home/bids/singularity_home/.local/lib/python3.9/site-packages/mrQA/cli.py:84: UserWarning: Expected a unique identifier for caching data. Got NoneType. Using a random name. Use --name flag for persistent metadata
  dataset = import_dataset(data_root=args.data_root,
Traceback (most recent call last):
  File "/home/bids/singularity_home/.local/bin/mr_proto_compl", line 8, in <module>
    sys.exit(main())
  File "/home/bids/singularity_home/.local/lib/python3.9/site-packages/mrQA/cli.py", line 84, in main
    dataset = import_dataset(data_root=args.data_root,
  File "/home/bids/singularity_home/.local/lib/python3.9/site-packages/MRdataset/base.py", line 82, in import_dataset
    dataset = dataset_class(
  File "/home/bids/singularity_home/.local/lib/python3.9/site-packages/MRdataset/dicom_dataset.py", line 70, in __init__
    self.save_dataset()
  File "/home/bids/singularity_home/.local/lib/python3.9/site-packages/MRdataset/base.py", line 351, in save_dataset
    raise EOFError('Dataset is empty!')
EOFError: Dataset is empty!

in large datasets, the checks maybe run multiple times, so it could be great if there is an option to exclude subjects / sessions in the report, when the checks are re-run to reduce info overload in the html reports

perhaps with a json file of IDs

Currently, sessions conducted on different dates have different numbers of MRI sequences acquired for each subject. This inconsistency hinders our ability to reconcile and compare the data effectively, as we need a consistent set of sequences across sessions for accurate analysis.

Note, that the difference in number of sequences might be a part of project requirement. But ensuring consistent number of sequences in each session is important to avoid misinterpretation of protocol compliance reports. If neglected, we may end up comparing protocols for sequences which are not meant to be compatible.

For example, the number of sequences acquired with CO2 inhalation > number of sequences acquired without CO2 inhalation. These differences are important, and we cannot use a blanket algorithm to check for protocol compliance

Complete comments on code to generate sphinx documentation

we need a way to differentiate between raw (EPI) scans and processed data (Motion Corrected by Siemens)... another instance of raw vs. derived data is from ASL / perfusion weighted images also

Add relative links to modality names in the summary table, which will lead to detailed view

As of now, there is a dictionary in MRdataset/config.py which stores the different parameters which are extracted from the DICOM header. Although it is possible to extend the dictionary as per the needs of the user, it would be even better if there is a CLI option to specify what parameters to check compliance for?

• Should we include a separate json/yaml file to read the list of parameters?
• We would need to DICOM tag associated with the particular parameter to read it from the header. What if the user is not aware of the tag? Can we accommodate it?
• How can we incorporate custom tags?

In our projects, we acquire two field maps in each session, one in the Anterior-Posterior (AP) direction and the other in the Posterior-Anterior (PA) direction. Currently, there is no mechanism to confirm whether both field maps were acquired in each session and whether they have opposite PED.

• Add a session-level validation step to verify the acquisition of both field maps for each session.
• Develop a mechanism to check the consistency of PED in the acquired field maps.