This repository facilitates the pre-processing of the CBIS-DDSM mammographic database. It involves downloading the whole database, converting the DICOM images to PNG and parsing the database files. Finally, a custom PyTorch Dataset and Dataloader can be created in a versatile way, covering whole mammograms, patches and segmentation masks of the lesions (see examples below).
First, install the project requirements using
pip install -r requirements.txt
Next, the config.json
file should be edited to specify the download_path
variable.
By default, the database will be downloaded in the root directory, in a new folder CBIS-DDSM
. Alternatively, edit
the file specifying a different dir:
{
"download_path": "< path to download>",
...
}
Next, run setup.py
. The command-line arguments are supported:
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config_file CONFIG_FILE
Path to the configuration file. Default=config.json
-d If used, dcm file will be deleted during conversion, to free up space.However, if download runs again it will need to download the whole dataset again.
The setup.py
script will download the database to the provided path, convert
the images to PNG format and pre-process the database csv files. Note that separate codes for each one of these
processes are provided in the utils
folder.
Datasets are created using the class CBISDDSMDatasetFactory
that provides a verstatile way to filter lesions,
manage their attributes and apply transformations on the corresponding images. A detailed description of the factory
functions is given below. Additionally, the folder examples
provides common cases of dataset creation. Keep in mind that a different copy of config.json
is
provided in this folder, that should point to the same download_path
with the original setup config.
The CBIS-DDSM database provides two subsets for training and testing purposes, respectively.
With CBISDDSMDatasetFactory
either one can be accessed via the .train()
and
.test()
functions or both of them merged via the .train_test_merge()
function:
dataset = CBISDDSMDatasetFactory('./config.json').train()
CBIS-DDSM provides two distinct subsets, concerning lesions of type mass
or calcification
.
With CBISDDSMDatasetFactory
either one or both of them can be accessed via the .add_masses()
and
.add_calcifications()
functions:
dataset = CBISDDSMDatasetFactory('./config.json').train().add_masses()
In CBIS-DDSM, a broad set of attributes is provided for each lesion. The CBISDDSMDatasetFactory
provides the function
.map_attribute_value()
to change a specific value into another one. For example, it is a common case for the
pathology
label BENIGN_WITHOUT_CALLBACK
to be changed to BENIGN
. This can be achieved with the following code:
dataset = CBISDDSMDatasetFactory('./config.json') \
.train() \
.add_masses() \
.map_attribute_value('pathology', {'BENIGN_WITHOUT_CALLBACK': 'BENIGN'})
Additionally, attributes that are not relevant can be dropped via the .drop_attributes()
method:
dataset = CBISDDSMDatasetFactory('./config.json') \
.train() \
.add_masses() \
.drop_attributes("assessment", "breast_density", "subtlety")
By default, the examples above will provide whole mammogram images. However, processing lesion patches is a common case
for mammographic CAD systems. CBISDDSMDatasetFactory
provides two types of patch transforms:
By using the option
.lesion_patches_centered(shape = (1024, 1024))
the factory will provide patches that are centered around each lesion.
The shape
parameter specifies the dimensions of the patch in pixels.
The default size is set to (1024, 1024)
, which is sufficient for all the masses in the dataset.
dataset = CBISDDSMDatasetFactory('./config.json') \
.train() \
.add_masses() \
.lesion_patches_centered()
Please note that there are some cases where the mass is located near the boundary of the mammogram image.
In these cases the patch is adjusted (translated) to contain the mass even if it is not centered.
An example of this option is given in examples/centered_patch_classification_dataset.py
.
By using the option
.lesion_patches_random(shape = (1024, 1024), min_overlap=0.9)
the factory will provide random patches of size shape
, sampled on random locations around the lesion.
The min_overlap
parameter specifies the minimum percentage of overlap that the patch should have with the lesion.
An example of this option is given in examples/random_patch_classification_dataset.py
.
CBISDDSMDatasetFactory
supports the application of PyTorch image transforms on the CBIS-DDSM samples,
both whole images and patches. This is achieved via the function
.add_image_transforms(transform_list, for_train = True, for_val = True)
that accepts a list of transforms. The parameters for_train
and for_val
constrain the application of the
transform to a specific mode (training mode or validation mode). In this way, the preprocessing transforms can be applied
to all the samples, but the augmentation transforms can be applied only for training.
After the dataset creation, the functions .train_mode()
and test_mode()
activate the corresponding configuration.
An example of this option is given in examples/centered_patch_classification_train_val_split.py
.
CBISDDSMDatasetFactory
provides two options for splitting the dataset for training and validation
purposed.
By using the option
.split_train_val(validation_percentage=0.2)
the factory will return a tuple with two distinct datasets, one for training and one for testing.
The parameter validation_percentage
specifies the ratio that will be held out for validation.
An example of this option is given in examples/centered_patch_classification_train_val_split.py
By using the option
.split_cross_validation(k_folds=5)
the factory will return a tuple with k_fold
splits of the dataset in training/validation. For each split, the training
dataset will contain a ratio of (k_fold - 1)/k_fold
of the total samples while the validation set will
contain 1/k_fold
of the total samples. The partitioning is performed in a mutually exclusive fashion, i.e.
a sample is used exactly k_fold
times for validation. An example of this option is given in
examples/centered_patch_classification_crossval.py
.