dlr-sc / prov2neo Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 0.0 112 KB

⬆️ Import W3C PROV documents to Neo4j.

License: MIT License

Python 100.00%

provenance graph-databases neo4j w3c-prov

prov2neo's Introduction

Welcome to `prov2neo`! 👋

prov2neo is a Python library and command line tool that imports W3C PROV documents into Neo4j.

prov2neo enables faster imports than comparable libs such as prov-db-connector with the limitation of being specialized for neo4j.

🏗️ Installation

Clone the project and use pip to install prov2neo locally:

pip install .

Or install the latest release from PyPi:

pip install prov2neo

To install prov2neo with all dependencies for development, use:

pip install .[dev]         # when cloned from GitHub
pip install prov2neo[dev]  # when installing from PyPi

🚀 Usage

prov2neo can be used as a command line script or as a Python lib.

As a Command Line Script

Usage: prov2neo [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

  Connect to a neo4j instance and import/export provenance documents.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  connect  Connect to a neo4j instance.
  export   (Deprecated) Export a provenance document from neo4j.
  import   Import a provenance document into neo4j.

As a Python Lib

from prov2neo import Client, read_doc

doc = read_doc(filepath="examples/horsemeat.json")

# create a client
client = Client()
# connect to your neo4j instance
client.connect(
    username="neo4j",
    password="neo4j",
    dbname="neo4j",
    address="localhost:7687",
    scheme="bolt"
)
# import the document
client.import_doc(doc)

Supported Formats

prov2neo supports all deserialization formats supported by the prov library. Deserialization and with it the import of documents through the command line tool is limited to the following formats:

🤝 Contributing

Contributions and pull requests are welcome!
For major changes, please open an issue first to discuss what you would like to change.

✨ Citable Software

This project is citable and contains a CITATION.cff file!
Please cite the project using the metadata contained in the CITATION.cff if you use prov2neo in your research.

CITATION.cff files are plain text files with human- and machine-readable citation information for software (and datasets).
To find out more about GitHub's support for citation metadata visit here

📝 License

This project is MIT licensed.
Copyright © 2020-2022 German Aerospace Center (DLR) and individual contributors.

prov2neo's People

Contributors

Stargazers

Watchers

prov2neo's Issues

Bug: `neotime.DateTime` no longer supported in `py2neo` > 2021.1.5

When encoding a provenance graph, the following exception is raised:

TypeError: Values of type <class 'neotime.DateTime'> are not supported

py2neo EOL: Port to official bolt driver

neo4j-contrib/py2neo#958 (comment)

ERROR: No matching distribution found for py2neo==2021.2.3

py2neo 2021.2.3 has been deleted and instead uploaded to https://pypi.org/project/py2neo-history/

Allow multiple serialized `PROV` files to be imported in a single command.

The command line tool provided by prov2neo currently supports the import of at most one serialized PROV file at once. To import the files p1.json, p2.json and p3.json, three commands would be necessary.

prov2neo -i p1.json -f json
prov2neo -i p2.json -f json
prov2neo -i p3.json -f json

To improve the ux of prov2neo, we should allow to import all three at once by letting the command line flag -i/--input consume more than one argument. This enables wildcard expansion and should be an overall improvement.

# without wildcard expansion
prov2neo -i p1.json p2.json p3.json -f json

# with wildcard expansion
prov2neo -i p*.json -f json

Steps

Set nargs='+' for the command line flag -i/--input
Update cli.py to loop over the input files in --input
Update README.md with updated usage instructions

py2neo.errors.ClientError: [Procedure.ProcedureNotFound]

I am trying to connect and import a prov file which was created using GitLab2Prov. When running the code given in the documentation, I run into the below error.

Traceback (most recent call last):
File "c:\Users\AKSHAAGA\Documents\prov2neo_test.py", line 12, in
client.connect(
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\prov2neo\client.py", line 96, in connect
self.add_uniqueness_constraints()
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\prov2neo\client.py", line 109, in add_uniqueness_constraints
if "id" not in self.graph_db.schema.get_uniqueness_constraints(label):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\database.py", line 894, in get_uniqueness_constraints
return [k[0] for k in self._get_indexes(label, unique_only=True)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\database.py", line 818, in _get_indexes
result = self.graph.run("CALL db.indexes")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\database.py", line 405, in run
return self.auto().run(cypher, parameters, **kwparameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\database.py", line 992, in run
self.connector.pull(result, -1)
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\client_init.py", line 1434, in pull
cx.pull(result, n=n)
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\client\bolt.py", line 1001, in pull
self._audit(self._transaction)
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\client\bolt.py", line 810, in _audit
task.audit()
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\client\bolt.py", line 1140, in audit
item.audit()
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\client\bolt.py", line 1140, in audit
item.audit()
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\py2neo\client\bolt.py", line 1303, in audit
raise self._failure
py2neo.errors.ClientError: [Procedure.ProcedureNotFound] There is no procedure with the name db.indexes registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.

I am trying to connect to a local neo4j DB which is newly created and thus doesn't contain anything. The format for importing data is json.

I was hoping to just run it on a new instance but it appears that first the DB needs to be setup with some files. What exactly needs to be done as I couldn't find it mentioned in the document for prov2neo.

Any guidance is appreciated. Thanks