• About the project
• Architecture
  • Ingress
  • Interpretation
  • Indexing
• Structure
• How to build the project
• Codestyle and tools recommendations

REMEMBER, YOU HAVE TO USE JAVA VERSION 8

Pipelines for data processing and indexing of biodiversity data

Status: IN PRODUCTION

Vision: Consistent data processing pipelines (parsing, interpretation and quality flagging) for use in GBIF, the Living Atlases project and beyond. Built to scale from laptop to GBIF volumes. Deployable on JVM, Spark, Google Cloud, .

The project provides vanilla JVM-based parsing and interpretation libraries, and pipelines for indexing into SOLR and ElasticSearch, built using Apache Beam.

Apache Beam provides a high level abstraction framework ideal for this purpose with the ability to deploy across target environments (e.g. Spark, local JVM) and with many built in connectors for e.g. HBase, SOLR, ElasticSearch etc.

Ingress is from Darwin Core Archive (zip files of one or more delimited text files) or ABCD Archives (compressed XML) only[1]. During ingress data is converted from its native format and stored as Avro files containing Darwin Core compliant data.

This is depicted below:

Avro is chosen as a storage and interchange format exclusively in this project because a) it is splittable with each split compressed independently, b) it holds the data schema with the data, c) is well supported in the Hadoop ecosystem (e.g. Hive, Spark) and many other tools (e.g. Google Cloud) d) is very robust in serialization and e) reduces boiler plate code thanks to schema to code generators. Darwin Core Archives and JSON for example do not exhibit all these traits.

[1] Other protocols (e.g. DiGIR, TAPIR) are supported by GBIF but are converted by crawling code upstream of this project.

During interpretation the verbatim data is parsed, normalised and tested against quality control procedures.

To enable extensibility data is interpreted into separate avro files where a separate file per category of information is used. Many interpretations such as date / time formatting are common to all deployments, but not all. For example, in the GBIF.org deployment the scientific identification is normalised to the GBIF backbone taxonomy and stored in /interpreted/taxonomy/interpreted*.avro which might not be applicable to everyone. Separating categories allows for extensibility for custom deployments in a reasonably modular fashion.

Interpretation is depicted below:

Note that all pipelines are designed and tested to run with the DirectRunner and the SparkRunner at a minimum. This allows the decision to be taken at runtime to e.g. opt to interpret a small dataset in the local JVM without needing to use cluster resources for small tasks.

It is a design decision to ensure that all the underlying parsers are as reusable as possible for other projects with careful consideration to not bring in dependencies such as Beam or Hadoop.

Initial implementations will be available for both SOLR and for ElasticSearch to allow for evaluation of both at GBIF. During indexing the categories of interpreted information of use are merged and loaded into the search indexes:

Note that GBIF target 10,000 records/sec per node indexing speed (i.e. 100,000 records/sec on current production cluster). This will allow simplified disaster recovery and rapid deployment and of new features.

The project is structured as:

• .buildSrc - Tools for building the project
• docs - Documents related to the project
• examples - Examples of using project API and base classes
  • transform - Transform example demonstrates how to create Apache Beam pipeline, create the new transformation and use it together with GBIF transforms and core classes
  • metrics - The example demonstrates how to create and send Apache Beam SparkRunner metrics to ELK and use the result for Kibana dashboards
• pipelines - Main pipelines module
  • beam-common - Classes and API for using with Apache Beam
  • common - Only static string variables
  • export-gbif-hbase - The pipeline to export the verbatim data from the GBIF HBase tables and save as ExtendedRecord avro files
  • ingest-gbif - Main GBIF pipelines for ingestion of biodiversity data
  • ingest-gbif-standalone - Independent GBIF pipelines for ingestion of biodiversity data
  • ingest-hdfs-table - Pipeline classes for conversion from interpreted formats into one common for HDFS view creation
  • ingest-transforms - Transformations for ingestion of biodiversity data
• sdks - Main module contains common classes, such as data models, data format interpretations, parsers, web services clients etc.
  • core - Main API classes, such as data interpretations, converters, DwCA reader etc.
  • models - Data models represented in Avro binary format, generated from Avro schemas
  • parsers - Data parsers and converters, mainly for internal usage inside of interpretations
  • keygen - The library to generate GBIF identifiers, to support backward compatibility the codebase (with minimum changes) was copied from the occurrence/occurrence-persistence project
• tools - Module for different independent tools
  • archives-converters - Converters from DwCA/DWC 1.0/DWC 1.4/ABCD 1.2/ABCD 2.06 to *.avro format
  • elasticsearch-tools - Tool for creating/deleting/swapping Elasticsearch indexes
  • pipelines-maven-plugin - Maven plugin adds new annotations and interface to avro generated classes

The project uses Apache Maven tool for building. Project contains maven wrapper and script for Linux and MacOS systems, you just can run build.sh script:

./build.sh

or

source build.sh

Please read Apache Maven how-to.

• Use Intellij IDEA Community (or better)
• Use Google Java Format (Please, do not reformat old codebase, only new)
• The project uses Project Lombok, please install Lombok plugin for Intellij IDEA.
• Because the project uses Error-prone you may have issues during the build process from IDEA. To avoid these issues please install the Error-prone compiler integration plugin and build the project using the error-prone java compiler to catch common Java mistakes at compile-time. To use the compiler, go to File → Settings → Compiler → Java Compiler and select Javac with error-prone in the Use compiler box.
• Add a custom parameter to avoid a debugging problem. To use the compiler, go to File → Settings → Compiler → Java Compiler → Additional command line parameters and add -Xep:ParameterName:OFF
• Tests: please follow the conventions of the Maven surefire plugin for unit tests and the ones of the Maven failsafe plugin for integration tests. To run the integration tests just run the verify phase, e.g.: mvn clean verify