JHU/APL is working to help advance the state of the art in geospatial computer vision by developing public benchmark data sets and open source software. For more information on this and other efforts, please visit JHU/APL.

JHU/APL is supporting the IARPA CORE3D program by providing independent test and evaluation of the performer team solutions for building 3D models based on satellite images and other sources. This is a repository for the metrics being developed to support the program. Performer teams are working with JHU/APL to improve the metrics software and contribute additional metrics that may be used for the program.

Preliminary metrics are described in the following paper:

M. Bosch, A. Leichtman, D. Chilcott, H. Goldberg, M. Brown. “Metric Evaluation Pipeline for 3D Modeling of Urban Scenes”, ISPRS Archives, 2017 pdf.

The following python3 libraries (and their dependencies) are required:

• gdal
• laspy
• matplotlib
• numpy
• scipy
• tk

Alternatively, you can use the provided docker container.

Recommend: use a virtual environment

python3 setup.py install
python3 setup.py install --prefix=$MY_ROOT

If installed

# from command line
core3d-metrics --help
core3d-metrics -c <AOI Configuration>
python3 -m core3dmetrics -c <AOI Configuration>

# in use code:
import core3dmetrics.geometrics as geo
geo.registration.align3d(reference_filename, test_filename)
core3dmetrics.main(['--help"])

If not installed

cd core3dmetrics
python3 run_geometrics.py -c <AOI Configuration> [-o <Output folder>  -r <Reference data folder> -t <Test data folder>]

One of the first steps is to align your dataset to the ground truth. This is performed using pubgeo's ALIGN3D algorithm. The algorithm then calculates metrics for 2D, 3D, and spectral classification against the ground truth.

    usage: core3dmetrics [-h] -c  [-r] [-t] [-o] [--align | --no-align] [--test-ignore] [--save-plots | --skip-save-plots] [--save-aligned]
    core3dmetrics entry point
    optional arguments:
      -h, --help         show this help message and exit
      -c , --config      Configuration file
      -r , --reference   Reference data folder
      -t , --test        Test data folder
      -o , --output      Output folder
      --align            Enable alignment (default)
      --no-align         Disable alignment
      --save-aligned     Save aligned images as geoTIFF (not enabled by default)
      --test-ignore      Enable NoDataValue pixels in test CLS image to be 
                         ignored during evaluation
      --save-plots       Enable saving plots (overrides config file setting)
      --skip-save-plots  Disable saving plots (overrides config file setting)                             

AOI Configuration is a configuration file using python's ConfigParser that is further described in aoi-config.md. This configuration file defines which files to analyze and what to compare against (ground truth). Additionally the config is to toggle various software settings.

python3 -m core3dmetrics -c aoi.config

This command would perform metric analysis on the test dataset provided by the aoi.config file. This analysis will also generate the following files (in place):

• < test dataset >_metrics.json

These files contain the determined metrics for completeness, correctness, f-score, Jaccard Index, Branching Factor, and the Align3d offsets.