Git Product home page Git Product logo

dosma's Introduction

DOSMA: Deep Open-Source Medical Image Analysis

License: GPL v3 GitHub Workflow Status codecov Documentation Status

Documentation | Questionnaire | DOSMA Basics Tutorial

DOSMA is an AI-powered Python library for medical image analysis. This includes, but is not limited to:

  • image processing (denoising, super-resolution, registration, segmentation, etc.)
  • quantitative fitting and image analysis
  • anatomical visualization and analysis (patellar tilt, femoral cartilage thickness, etc.)

We hope that this open-source pipeline will be useful for quick anatomy/pathology analysis and will serve as a hub for adding support for analyzing different anatomies and scan sequences.

Installation

DOSMA requires Python 3.6+. The core module depends on numpy, nibabel, nipype, pandas, pydicom, scikit-image, scipy, PyYAML, and tqdm.

Additional AI features can be unlocked by installing tensorflow and keras. To enable built-in registration functionality, download elastix. Details can be found in the setup documentation.

To install DOSMA, run:

pip install dosma

# To install with AI support
pip install dosma[ai]

If you would like to contribute to DOSMA, we recommend you clone the repository and install DOSMA with pip in editable mode.

git clone [email protected]:ad12/DOSMA.git
cd DOSMA
pip install -e '.[dev,docs]'
make dev

To run tests, build documentation and contribute, run

make autoformat test build-docs

Features

Simplified, Efficient I/O

DOSMA provides efficient readers for DICOM and NIfTI formats built on nibabel and pydicom. Multi-slice DICOM data can be loaded in parallel with multiple workers and structured into the appropriate 3D volume(s). For example, multi-echo and dynamic contrast-enhanced (DCE) MRI scans have multiple volumes acquired at different echo times and trigger times, respectively. These can be loaded into multiple volumes with ease:

import dosma as dm

multi_echo_scan = dm.load("/path/to/multi-echo/scan", group_by="EchoNumbers", num_workers=8, verbose=True)
dce_scan = dm.load("/path/to/dce/scan", group_by="TriggerTime")

Data-Embedded Medical Images

DOSMA's MedicalVolume data structure supports array-like operations (arithmetic, slicing, etc.) on medical images while preserving spatial attributes and accompanying metadata. This structure supports NumPy interoperability, intelligent reformatting, fast low-level computations, and native GPU support. For example, given MedicalVolumes mvA and mvB we can do the following:

# Reformat image into Superior->Inferior, Anterior->Posterior, Left->Right directions.
mvA = mvA.reformat(("SI", "AP", "LR"))

# Get and set metadata
study_description = mvA.get_metadata("StudyDescription")
mvA.set_metadata("StudyDescription", "A sample study")

# Perform NumPy operations like you would on image data.
rss = np.sqrt(mvA**2 + mvB**2)

# Move to GPU 0 for CuPy operations
mv_gpu = mvA.to(dosma.Device(0))

# Take slices. Metadata will be sliced appropriately.
mv_subvolume = mvA[10:20, 10:20, 4:6]

Built-in AI Models

DOSMA is built to be a hub for machine/deep learning models. A complete list of models and corresponding publications can be found here. We can use one of the knee segmentation models to segment a MedicalVolume mv and model weights downloaded locally:

from dosma.models import IWOAIOAIUnet2DNormalized

# Reformat such that sagittal plane is last dimension.
mv = mv.reformat(("SI", "AP", "LR"))

# Do segmentation
model = IWOAIOAIUnet2DNormalized(input_shape=mv.shape[:2] + (1,), weights_path=weights)
masks = model.generate_mask(mv)

Parallelizable Operations

DOSMA supports parallelization for compute-heavy operations, like curve fitting and image registration. Image registration is supported thru the elastix/transformix libraries. For example we can use multiple workers to register volumes to a target, and use the registered outputs for per-voxel monoexponential fitting:

# Register images mvA, mvB, mvC to target image mv_tgt in parallel
_, (mvA_reg, mvB_reg, mvC_reg) = dosma.register(
   mv_tgt,
   moving=[mvA, mvB, mvC],
   parameters="/path/to/elastix/registration/file",
   num_workers=3,
   return_volumes=True,
   show_pbar=True,
)

# Perform monoexponential fitting.
def monoexponential(x, a, b):
   return a * np.exp(b*x)

fitter = dosma.CurveFitter(
   monoexponential,
   num_workers=4,
   p0={"a": 1.0, "b": -1/30},
)
popt, r2 = fitter.fit(x=[1, 2, 3, 4], [mv_tgt, mvA_reg, mvB_reg, mvC_reg])
a_fit, b_fit = popt[..., 0], popt[..., 1]

Citation

@inproceedings{desai2019dosma,
  title={DOSMA: A deep-learning, open-source framework for musculoskeletal MRI analysis},
  author={Desai, Arjun D and Barbieri, Marco and Mazzoli, Valentina and Rubin, Elka and Black, Marianne S and Watkins, Lauren E and Gold, Garry E and Hargreaves, Brian A and Chaudhari, Akshay S},
  booktitle={Proc 27th Annual Meeting ISMRM, Montreal},
  pages={1135},
  year={2019}
}

In addition to DOSMA, please also consider citing the work that introduced the method used for analysis.

dosma's People

Contributors

ad12 avatar akshaysc avatar barma7 avatar fsantini avatar gattia avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

dosma's Issues

Activate dosma_env automatically if not the current environment

Is your feature request related to a problem? Please describe.
Activate the dosma_env virtual environment automatically when using the dosma executable.

Describe the solution you'd like
This can be done by checking for the $DEFAULT_CONDA_ENV environment variable, which tracks the name of the active environment in the local terminal window.

Additional context
Note for older versions of conda installed, conda environments may be activated using source activate. Need to find a method for determining when to use source activate vs conda activate.

Allow manual region division

Is your feature request related to a problem? Please describe.
In cases where automatic division of tissues into regions is not what we are looking for, we want to have the support of manual region subdivision. Here, the segmentation can be manually subdivided into regions of interest (deep, superficial, anterior/lateral, etc.)

Describe the solution you'd like
Have support for manual region division where the user can upload a region subdivision, with each region uniquely identifiable by integers.

Bit-precision of NiFTI scans result in different zero-mean, unit std. dev. volumes

Describe the bug
Zero-mean normalization of 32-bit vs 64-bit data results in slightly different estimations of the mean and standard deviation. However, when both volumes are compared pre-normalization using np.all() both volumes are the same (see :meth:test_h5_nifti_same). This slight difference in the image can affect network performance and change the expected segmentations.

As a placeholder, we assert that the dice between the expected and produced segmentations for each tissue must be greater than 99%.

Note that for the following models in DOSMA, no considerable difference in performance on the test example was seen:

  • OAIUnet2D
  • IWOAIOAIUnet2D
  • IWOAIOAIUnet2DNormalized

To Reproduce
See tests/models/test_oaiunet2d.py for more information.

Expected behavior
Segmentation performance should be the same if the volumes are the same.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: MacOS
  • Branch: dev

Upgrade scan registration to use dosma.utils.registration.register()

Upgrade scan intra/inter registration to use dosma.utils.registration.register()

We will add testing to verify consistency between current scan inter/intra registering code and the implementation with the register() function. These tests are to be deleted once they have been verified and scan registration code is upgraded to function

itk error: Too many samples map outside moving image buffer

This error occurs when using a mask

  1. Investigate if it is the automatic mask that is causing the problem (try with manual mask)
  2. If it is automatic mask, see what difference are and if they are attributable to network weights, which are not well trained yet

group_by checking in dicom_io.load() throws an error if EchoNumber is zero indexed

For MESE acquired with SIEMENS EchoNumbers is 0 indexed. In dicom_io.load() the group_by option throws an error when checking if the header entry exists in dicom (line 269).

if is not temp_dicom.get(group_by): raise ...

To solve the issue I suggest changing the conditional check with this:

if temp_dicom.get(group_by) is None: raise ....

Change default setting for `group_by` in the DICOM reader

The group_by argument of DicomReader currently defaults to EchoNumbers, which is not commonly present in DICOM files.

Typically, I would only load one DICOM study and not use grouping at all. I would suggest to default to None and to disable grouping that scenario (i.e., not returning a list of MedicalVolume, but a single instance).

[BUG] Documentation

Describe the bug
Elastix link for downloading is wrong (seems to have changed).

  • Documentation/Installation/Setup/Registration. (1. Download elastix) <= that points to wrong URL.

Also, Elastix 5.0.1 gives errors (certain libraries not found) when using clean install of Ubuntu. Elastix 4.9.0 (the same as is used in the Dosma qMRI Knee Workflow.ipynb) works out of the box on a clean ubuntu install. Maybe it should be suggested to explicitly recommend that version (4.9.0)?

Keras Version?

Is there a reason keras is pinned to <= 2.4.1? Im starting to get install conflicts with this not pretty old version.

make logging directory default to user owned directory

I want to include in dosma in a docker container. All works fine, but the container must be run as root because dosma wants to write to a logfile which is created in a system directory. I would like to run the container as an unprivileged user, so I don't run into ownership problems of the output files.

Could you consider changing the hard coded logfile path (output_dir() in envs.py) to something more easily accessible? Home folder or /tmp?

simple itk functions ? (resampling & registration) [FEATURE]

Is your feature request related to a problem? Please describe.
No problems. Just wanted bring up some thoughts or ideas.

Describe the solution you'd like
Thanks for contributing this library. I quite like the overall library format. I've personally used simpleitk (sitk) quite a bit. But, most of the time I am just using it as a means of importing and exporting data and then do much of my processing in numpy and other python libraries. From that standpoint, this library seems like a great option to represent data in numpy format for interoperability with other libraries.

Anyways, to see if I could transition to this library instead of sitk I went through my code-base and looked at sitk functions that I most heavily relied on. Based on that review, there were two areas I was wondering about - image IO and image resampling.

Resampling
The main function that I don't currently see present in DOSMA but that I know I've used heavily is an image resampling function. Im thinking: 1) resampling image to specified dimension parameters (origin, spacing, volume size) using different methods - nearest neighbour, linear, bspline and 2) resampling within the space (grid) of another image (really, this should just be a special case or helper function related to (1).

Are there any thoughts about incorporating an image resampling function (other than those in Elastix - which I think is itk under the hood)? And if so, what sort of path you were thinking about?

Image IO
For simplicity, I've always saved outputs using .nrrd as it seems to be what was recommended (adopted) by the itk/3D slicer community that I learned/started with. The other format that I've used a bit and is more obscure is par/rec (Philips scanners).

I've used nibable to read par/rec before, and since dosma already uses nibabel to read .nii Im guessing this could relatively easily be added? Im just thinking about reading here, I really dont see a need to save par/rec files. Would this be something you consider adding?

As for .nrrd, it looks like nibabel folks have been meaning to add it for a while but maybe haven't (nipy/nibabel#356). Seems like pynrrd (https://pypi.org/project/pynrrd/) could be a lightweight option that might be the backbone to whatever nibabel uses anyways. Or, sitk can also do this reading and writing, but it is obviously a bigger library that you may not want to include as a dependency.

Those are the two things that I was mostly wondering about when reading through the documentation and examples. Beyond this, I think there are some functions in sitk that could be useful and tend to be fast because they are C++ based. However, Im thinking that the DOMA built-in functions to convert a MedicalVolume to .sitk image format and back would enable users to easily use sitk filters if they had a use case.

Thanks again for contributing!

Anthony

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Rename Dess to qDESS

qdess is the name used to refer to the protocol that is implemented - refactor name

Sort dicom files by creation time

After anonymization, many dicom files are not in the correct order or are not named appropriately. As a result, relying on the order of the dicom files is not sufficient for obtaining the files in the correct order

To solve this, sort by the time the dicom was created

Mapss T2 calculation requires mask[BUG]

Describe the bug
A clear and concise description of what the bug is.

# only calculate for focused region if a mask is available, this speeds up computation
mask = tissue.get_mask()
if mask_path is not None:
mask = (
fio_utils.generic_load(mask_path, expected_num_volumes=1)
if isinstance(mask_path, (str, os.PathLike))
else mask_path
)

The above section of code shows that the tissue class is required for the T2 generation to go through. If the tissue class is not present then .get_mask() will fail (None type does not have the function). However, tissue class is meant to be optional.

Update README

Update README to reflect new structure for using pipeline

Do not enforce upper bound on pydicom version

Per this discussion, pydicom enforces EchoTime as an IS type, which is a derived from the int type, because it is encoded as such in some dicoms. By default this should be a float quantity, but if the dicom metadata enforces its a IS type, pydicom enforces that too.

This explicit type checking was introduced around pydicom==2.1.0. We currently have set the upper bound on the pydicom version to be 2.0.0 to avoid running into this issue with Cones, which is a supported scan in DOSMA. However, this bad practice.

Solution
The solution moving forward is still TBD. The discussion above proposes some workaround of type checking with the downside that the field will no longer be DICOM compliant.

[BUG] dm.register - Elastix throws path error (searching for parameters file)

Describe the bug
When using dm.register sometimes an error is thrown indicating that the transform file is not found:

FileNotFoundError: No such file or directory '['/dataNAS/people/jdemarre/repeat_3T7T/Repeatability/results_April-04-2022/repeat001/3T/9582/elastix_reg_2/moving-0/TransformParameters.0.txt']' for output 'transform' of a Registration interface

If you look inside of the folder where the transform file should be you can find an elastix.log file which includes the printout of what happened during the registration process. In my case, it turns out that there was an error in the registration process - in particular, the error was only being thrown when I passed a segmentation mask to help in the registration process and it was because the segmentation mask was too small:

Description: itk::ERROR: RandomCoordinateSampler(0x4330850): Could not find enough image samples within reasonable time. Probably the mask is too small

Looking under the hood at interregister I noticed that __dilate_mask__ is called on all segmentation masks before they are passed to dm.register:

if target_mask_path:
target_mask_path = self.__dilate_mask__(target_mask_path, out_path)

To fix this problem, I re-implemented the __dilate_mask__ function and called it on the mask and then passed this dilated ask to the registration tool.

def __dilate_mask__(

The Fix
To fix the problem, I created the following version of the __dilate_mask__ function and used it on my segmentation mask before passing the segmentation mask on to dm.register. The main tweak is that it allows the user to specify if they want to receive a path in return or if they want to receive the MedicalVolume itself. Im wondering if this (or something like it) could be implemented as a utility function and then have __dilate_mask__ call this utility function. This way, users can access this as necessary for registration. An option could also be placed in dm.register to dilate a mask before passing it on to the registration process => maybe this should be a default?

from dosma.defaults import preferences
import scipy.ndimage as sni
from dosma.utils import env, io_utils
from dosma.core.med_volume import MedicalVolume
from dosma.core.io import format_io_utils as fio_utils


def dilate_mask(
    mask_path,
    temp_path: str = env.temp_dir(),
    dil_rate: float = preferences.mask_dilation_rate,
    dil_threshold: float = preferences.mask_dilation_threshold,
    return_path: bool=True
):
    """Dilate mask using gaussian blur and write to disk to use with Elastix.
    Args:
        mask_path (str | MedicalVolume): File path for mask or mask to use to use as
            focus points for registration. Mask must be binary.
        temp_path (str): Directory path to store temporary data.
        dil_rate (`float`, optional): Dilation rate (sigma).
            Defaults to ``preferences.mask_dilation_rate``.
        dil_threshold (`float`, optional): Threshold to binarize dilated mask.
            Must be between [0, 1]. Defaults to ``preferences.mask_dilation_threshold``.
        return_path (`bool`, optional): Should a path be returned; if not, return the 
            dilated mask itself. Defaults to ``True``.
    Returns:
        str: File path of dilated mask.
    Raises:
        FileNotFoundError: If `mask_path` not valid file.
        ValueError: If `dil_threshold` not in range [0, 1].
    """

    if dil_threshold < 0 or dil_threshold > 1:
        raise ValueError("'dil_threshold' must be in range [0, 1]")

    if isinstance(mask_path, MedicalVolume):
        mask = mask_path
    elif os.path.isfile(mask_path):
        mask = fio_utils.generic_load(mask_path, expected_num_volumes=1)
    else:
        raise FileNotFoundError("File {} not found".format(mask_path))

    dilated_mask = (
        sni.gaussian_filter(np.asarray(mask.volume, dtype=np.float32), sigma=dil_rate)
        > dil_threshold
    )
    dilated_mask = np.asarray(dilated_mask, dtype=np.int8)
    dilated_mask_volume = MedicalVolume(dilated_mask, affine=mask.affine)
    
    if return_path is False:
        return dilated_mask_volume
    else:
        
        fixed_mask_filepath = os.path.join(io_utils.mkdirs(temp_path), "dilated-mask.nii.gz")
        
        dilated_mask_volume.save_volume(fixed_mask_filepath)

        return fixed_mask_filepath

Desktop (please complete the following information):

  • OS: Ubuntu 20.04 LTS
  • Browser chrome
  • Version 95.0.4638.69

Please also provide the output of the following code:


sys.platform linux
Python 3.8.5 (default, Sep 4 2020, 07:30:14) [GCC 7.3.0]
numpy 1.19.2
dosma 0.1.1 @/dataNAS/people/aagatti/miniconda/envs/dosma-ai/lib/python3.8/site-packages/dosma
h5py 2.10.0
nibabel 3.2.2
nipype 1.7.0
pandas 1.4.1
pydicom 2.2.2
scipy 1.5.2
skimage 0.19.2
tensorflow 2.2.0
keras 2.4.3


Additional context
Add any other context about the problem here.

femoral cartilage segmentation should use total volume whitening

currently, each volume is whitened before preprocessing, but the whitening process is done in the through-plane direction. This is incorrect.

Fix: whitening should be done by taking average/std.dev. of all pixels in volume, not just in through-plane direction

Windows OS support

Is your feature request related to a problem? Please describe.
DOSMA is not fully supported for the Windows OS. Please add additional functionality to DOSMA to allow Windows OS usability.

Describe the solution you'd like

  1. DOSMA support for Windows OS or
  2. Docker container containing OS/file dependencies

Describe alternatives you've considered
Virtual Machines are too bulky and are not seamlessly integrated with existing programs.

[BUG] dosma.register() throws an error when moving_mask is passed to the function

Describe the bug
dosma.register() throws an error when moving_mask is passed to the function either as a string or MedicalVolume.

To Reproduce

import dosma as dm
from dosma.file_constants as flc

reg_info, reg_vol = dm.register(target, moving, flc.ELASTIX_RIGID_INTERREGISTER_PARAMS_FILE, out_path,
return_volumes=True,
target_mask=target_mask_path,
moving_masks=moving_mask_path,
rtype=tuple,
show_pbar=True)

Expected behavior
The function is expected to correctly call Elastix using both fixed (target) and moving masks.

The bug is at line 406 of dosma.register(), and has to do with the fact that we attempt to fill a non existing attribute of the object reg.inputs

Code with bug:
if _use_mask and moving_mask is not None:
reg.inputs.target_mask = os.path.abspath(moving_mask)

Bug is fixed when naming the correct attribute
Solution:
if _use_mask and moving_mask is not None:
reg.inputs.moving_mask = os.path.abspath(moving_mask)

Allow `group_by` and `sort_by` to take a list of tags

For some DICOM datasets I would like to order the slices based on multiple tags. For example, in a dataset with a TemporalPositionIdentifier present, I would like to sort by ImagePositionPatient first, and then using the position identifier. The same applies to group_by too.

Preferably, it would be nice to add support for ascending/descending sorts for each tag too:

# single tag
dm.read(path, sort_by="InstanceNumber")

# multiple tags
dm.read(path, sort_by=["ImagePositionPatient", "TemporalPositionIdentifier"])

# multiple tags + order
dm.read(path, sort_by=[{ "tag": "ImagePositionPatient", "asc": True }, { "tag": "TemporalPositionIdentifier", "asc": False }])

NB: maybe we should consider to set the default sort_by to ImagePositionPatient.

Option to store data in csv instead of xlsx format

Is your feature request related to a problem? Please describe.
xlsx enforces conformance Excel format, but the csv format is more general. Having an option for saving data in csv format would be useful

Bilateral automatic segmentation bug. Segmentation works only for one of the knees

When using automatic segmentation qdess segment with a bilateral knee acquisition. If the volume contains both knees automatic segmentation works only on the first knee. Note that no errors are displayed from terminal, however the output mask has all zeros where the cartilage of the second knee is supposed to be, which is not correct.

As workaround for the current bug is to separate the knees beforehand and to process the two volumes separately.

[BUG] TensorFlow 2.5.0 not supported

Describe the bug
With tensorflow==2.5.0, we get an error message like the one below:

tests/util.py:18: in <module>
    from dosma.cli import SUPPORTED_SCAN_TYPES, parse_args
dosma/cli.py:36: in <module>
    from dosma.models.seg_model import SegModel
dosma/models/__init__.py:1: in <module>
    from dosma.models import oaiunet2d, util
dosma/models/oaiunet2d.py:14: in <module>
    from keras.models import Model
/opt/hostedtoolcache/Python/3.6.13/x64/lib/python3.6/site-packages/keras/__init__.py:20: in <module>
    from . import initializers
/opt/hostedtoolcache/Python/3.6.13/x64/lib/python3.6/site-packages/keras/initializers/__init__.py:124: in <module>
    populate_deserializable_objects()
/opt/hostedtoolcache/Python/3.6.13/x64/lib/python3.6/site-packages/keras/initializers/__init__.py:49: in populate_deserializable_objects
    LOCAL.GENERATED_WITH_V2 = tf.__internal__.tf2.enabled()
E   AttributeError: module 'tensorflow.compat.v2.__internal__' has no attribute 'tf2'

To Reproduce

import dosma

Expected behavior
No error.

Stopgap solution
Install tensorflow and Keras with pip install tensorflow==2.4.1 keras==2.4.3

Potential Fix
As mentioned in this issue, tensorflow>=2.0 should import keras tools using import tensorflow.keras instead of import keras. All files where packages are imported from keras should first check the version for tensorflow and then rename keras. In pseudo-code, this is:

if tensorflow_version >= 2.0:
    import tensorflow.keras as keras

from keras...

Load state for cubequant not working

Bug

Cannot load interregistered files for cubequant

Reproduce

Run two lines separately

1. python -m pipeline -d DIR -s DIR/data cq -fc interregister -ts DIR/data/dess_data/echo1.nii.gz -tm DIR/data/fc/fc.nii.gz
2. python -m pipeline -l DIR/data/cubequant cq -fc -t1_rho

Line 2 results in error Either dicom_path or load_path must be specified

Installation Requirement

Xcode, and subsequently gcc, need to be installed in order for DOSMA's installation to occur properly. Otherwise the libraries fail to install during the installation and must be installed one-by-one.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.