irtazaraza / typedb-loader Goto Github PK

If your TypeDB project

• has a lot of data
• and you want/need to focus on schema design, inference, and querying

Use TypeDB Loader to take care of your data migration for you. TypeDB Loader streams data from files and migrates them into TypeDB at scale!

• Data Input:

  • data is streamed to reduce memory requirements
  • supports any tabular data file with your separator of choice (i.e.: csv, tsv, whatever-sv...)
  • supports gzipped files
  • ignores unnecessary columns

• Attribute, Entity, Relation Loading:

  • load required/optional attributes of any TypeDB type (string, boolean, long, double, datetime)
  • load required/optional role players (attribute / entity / relation)
  • load list-like attribute columns as n attributes (recommended procedure until attribute lists are fully supported by TypeDB)
  • load list-like player columns as n players for a relation
  • load entity if not present - if present, either do not write or append attributes

• Appending Attributes to existing things

• Append-Attribute-Or-Insert-Entity for entities

• Data Validation:

  • validate input data rows and log issues for easy diagnosis input data-related issues (i.e. missing attributes/players, invalid characters...)

• Configuration Validation:

  • write your configuration with confidence: warnings will display useful information for fine tuning, errors will let you know what you forgot. All BEFORE the database is touched.

• Performance:

  • parallelized asynchronous writes to TypeDB to make the most of your hardware configuration, optimized with engineers @vaticle

• Stop/Restart (in re-implementation, currently NOT available):

  • tracking of your migration status to stop/restart, or restart after failure

• Basic Column Preprocessing using RegEx's

Create a Loading Configuration (example) and use TypeDB Loader

• as an executable CLI - no coding
• in your own Java project - easy API

To illustrate how to use TypeDB Loader, we will use a slightly extended version of the "phone-calls" example dataset and schema from the TypeDB developer documentation:

The configuration file tells TypeDB Loader what things you want to insert for each of your data files and how to do it.

Here are some example:

• Attribute Examples
• Entity Examples
• Relation Examples
• Nested Relation - Match by Attribute(s) Example
• Nested Relation - Match by Player(s) Example
• Attribute-Player Relation Example
• Custom Migration Order Example

For detailed documentation, please refer to the WIKI.

The config in the phone-calls test is a good starting example of a configuration.

Once your configuration files are complete, you can use TypeDB Loader in one of two ways:

1. As an executable command line interface - no coding required:

./bin/typedbloader load \
                -tdb localhost:1729 \
                -c /path/to/your/config.json \
                -db databaseName \
                -cm

See details here

2. As a dependency in your own Java code:

import com.vaticle.typedb.osi.loader.cli.LoadOptions;
import com.vaticle.typedb.osi.loader.loader.TypeDBLoader;

public class LoadingData {

    public void loadData() {
        String uri = "localhost:1729";
        String config = "path/to/your/config.json";
        String database = "databaseName";

        String[] args = {
                "load",
                "-tdb", uri,
                "-c", config,
                "-db", database,
                "-cm"
        };

        LoadOptions options = LoadOptions.parse(args);
        TypeDBLoader loader = new TypeDBLoader(options);
        loader.load();
    }
}

See details here

A complete tutorial for TypeDB version >= 2.5.0 is in work and will be published.

An example of configuration and usage of TypeDB Loader on real data can be found in the TypeDB Examples.

A complete tutorial for TypeDB (Grakn) version < 2.0 can be found on Medium.

There is an example repository for your convenience.

To connect to TypeDB Cluster, a set of options is provided:

--typedb-cluster=<address:port>
--username=<username>
--password // can be asked for interactively
--tls-enabled
--tls-root-ca=<path/to/CA/cert>

Ranges are [inclusive, inclusive].

• Type DB

Find the Readme for GraMi for grakn < 2.0 here

TypeDB Loader was built @Bayer AG in the Semantic and Knowledge Graph Technology Group with the support of the engineers @Vaticle.

This repository includes software developed at Bayer AG. It is released under the Apache License, Version 2.0.

Icon in banner by Freepik from Flaticon