API client for Aleph, supports bulk entity and document upload.
Home Page: https://docs.alephdata.org/developers/alephclient
License: MIT License
Command-line client for Aleph. It can be used to bulk import document sets via the API, without direct access to the server. It requires an active API client to perform uploads.
Install using pip.
pip install alephclientRefer to the aleph handbook for an introduction on how to use alephclient,
e.g. to crawl a local file directory, or to stream entities:
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
api class.alephclient crawldir sends the entire directory to Aleph even if most of the content is already processed in a previous crawl. It should be possible to upload only the newer files to Aleph without uploading the rest.
Depending on which library you are using the default branch may be master or main. It's a little confusing and rather annoying to have both master and main used within the org and so we should standardize on main.
Rename the alephclient default branch to main.
When we get a non-200 response we throw a generic exception, but often the response body is actually JSON and contains a meaningful error message. We need to handle and show these.
Following the installation instructions for alephclient on a fresh Ubuntu 20.04 I get the following error:
$ python3 -m venv my-project-env
$ source my-project-env/bin/activate
(my-project-env) $ pip install alephclient
Collecting alephclient
Downloading alephclient-2.0.3-py2.py3-none-any.whl (13 kB)
Collecting pyyaml
Downloading PyYAML-5.3.1.tar.gz (269 kB)
|████████████████████████████████| 269 kB 3.6 MB/s
Collecting banal>=1.0.1
Downloading banal-1.0.1-py2.py3-none-any.whl (5.3 kB)
Collecting requests>=2.21.0
Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
|████████████████████████████████| 61 kB 802 kB/s
Collecting requests-toolbelt>=0.9.1
Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
|████████████████████████████████| 54 kB 4.4 MB/s
Collecting click>=7.0
Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
|████████████████████████████████| 82 kB 1.8 MB/s
Collecting certifi>=2017.4.17
Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
|████████████████████████████████| 156 kB 7.2 MB/s
Collecting idna<3,>=2.5
Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
|████████████████████████████████| 58 kB 6.1 MB/s
Collecting chardet<4,>=3.0.2
Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
|████████████████████████████████| 133 kB 7.1 MB/s
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
|████████████████████████████████| 127 kB 6.7 MB/s
Building wheels for collected packages: pyyaml
Building wheel for pyyaml (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/crito/my-project-env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-obajncoq/pyyaml/setup.py'"'"'; __file__='"'"'/tmp/pip-install-obajncoq/pyyaml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-tzz6gpkm
cwd: /tmp/pip-install-obajncoq/pyyaml/
Complete output (6 lines):
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
ERROR: Failed building wheel for pyyaml
Running setup.py clean for pyyaml
Failed to build pyyaml
Installing collected packages: pyyaml, banal, certifi, idna, chardet, urllib3, requests, requests-toolbelt, click, alephclient
Running setup.py install for pyyaml ... done
Successfully installed alephclient-2.0.3 banal-1.0.1 certifi-2020.6.20 chardet-3.0.4 click-7.1.2 idna-2.10 pyyaml-5.3.1 requests-2.24.0 requests-toolbelt-0.9.1 urllib3-1.25.10
Repeating the same steps and installing wheel before alephclient removes the above error.
$ python3 -m venv alephclient
$ source alephclient/bin/activate
(alephclient) $ pip install wheel
Collecting wheel
Using cached wheel-0.35.1-py2.py3-none-any.whl (33 kB)
Installing collected packages: wheel
Successfully installed wheel-0.35.1
(alephclient) $ pip install alephclient
Collecting alephclient
Using cached alephclient-2.0.3-py2.py3-none-any.whl (13 kB)
Collecting requests>=2.21.0
Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting banal>=1.0.1
Using cached banal-1.0.1-py2.py3-none-any.whl (5.3 kB)
Collecting pyyaml
Using cached PyYAML-5.3.1.tar.gz (269 kB)
Collecting click>=7.0
Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting requests-toolbelt>=0.9.1
Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Building wheels for collected packages: pyyaml
Building wheel for pyyaml (setup.py) ... done
Created wheel for pyyaml: filename=PyYAML-5.3.1-cp38-cp38-linux_x86_64.whl size=44619 sha256=602ab9277e57d0863f63236e7e6d508bb0345316ed722524a16001a980d6038e
Stored in directory: /home/crito/.cache/pip/wheels/13/90/db/290ab3a34f2ef0b5a0f89235dc2d40fea83e77de84ed2dc05c
Successfully built pyyaml
Installing collected packages: certifi, urllib3, chardet, idna, requests, banal, pyyaml, click, requests-toolbelt, alephclient
Successfully installed alephclient-2.0.3 banal-1.0.1 certifi-2020.6.20 chardet-3.0.4 click-7.1.2 idna-2.10 pyyaml-5.3.1 requests-2.24.0 requests-toolbelt-0.9.1 urllib3-1.25.10
But running alephclient results in a runtime error:
(alephclient) $ alephclient -h
Traceback (most recent call last):
File "/home/crito/alephclient/bin/alephclient", line 5, in <module>
from alephclient.cli import cli
File "/home/crito/alephclient/lib/python3.8/site-packages/alephclient/cli.py", line 6, in <module>
from alephclient.api import AlephAPI
File "/home/crito/alephclient/lib/python3.8/site-packages/alephclient/api.py", line 7, in <module>
from six.moves.urllib.parse import urlencode, urljoin
ModuleNotFoundError: No module named 'six'
Manually installing six with a pip install six resolves the issue.
Running crawldir on a directory containing 50,000 files/directories at the same level yields horrible performance. Over an hour just to build the listdir array, then 0.7G of memory. Granted, hard disk is the bottleneck. However we can be a bit more clever and use os.scandir, to read the directory asynchronously. Will submit a PR.
Hi there,
When uploading my ftm dataset to aleph, I use:
ftm store iterate -d NAME_DATASET | alephclient write-entities -f NAME_IN_ALEPH
I also tried the following, both with "casefile"=True and "casefile"=False
api = AlephAPI()
options = {"casefile": False}
collection = api.load_collection_by_foreign_id(my_aleph_dataset, options)
# collection = api.load_collection_by_foreign_id(my_aleph_dataset)
cid = collection.get('id')
# FTM Store
dataset = get_dataset(my_dataset)
api.write_entities(cid, dataset.iterate())
In all cases, the result is an Investigation containing all my entities is created in Aleph UI, but I would rather have a Dataset instance, like my coworkers.
I tried the exact same code on another computer, with the same dataset and same alephclient version, and it creates a dataset instance on their Aleph UI. Can you help me understand how to fix it?
Best,
Seems we've gone Py3 only by accident, but this library should work with 2.x as well
t.ex. "cnr"
I have a dataset that I generate in the FTM format.
Whenever I try to import it with alephclient on a local development server that was just spinned up and brand spanking fresh I get the following error :
alephclient --host="http://localhost:8080" --api-key="xxx" write-entities --infile sirene_unite_legale.ftm.json_part_1.json --foreign-id sirene_unite_legale
WARNING:alephclient.util:Error: Internal server error., back-off: 2.56s
WARNING:alephclient.util:Error: Internal server error., back-off: 4.99s
WARNING:alephclient.util:Error: Internal server error., back-off: 8.68s
WARNING:alephclient.util:Error: Internal server error., back-off: 16.65s
WARNING:alephclient.util:Error: Internal server error., back-off: 32.55s
Error: Internal server error.
this is what seems to be the relevant log in the api:
api-1 | {"es_req_method": "POST", "es_url": "/actionreaction-collection-v1/_search", "es_req_params": {}, "es_req_body": {"query": {"bool": {"should": [{"match_all": {}}], "must": [], "must_not": [], "filter": [{"term": {"foreign_id": "sirene_unite_legale"}}], "minimum_should_match": 1}}, "post_filter": {"bool": {"filter": []}}, "from": 0, "size": 20, "aggregations": {}, "sort": ["_score", {"label.kw": "asc"}], "highlight": {}, "_source": {"excludes": ["text"]}}, "took": 3, "logger": "aleph.util", "timestamp": "2024-07-24 09:10:04.119815", "endpoint": "collections_api.index", "locale": "en", "trace_id": "aa2ad457-a612-4208-9e94-c9e396751587", "method": "GET", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections?filter:foreign_id=sirene_unite_legale", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:04.112026", "path": "/api/2/collections?filter%3Aforeign_id=sirene_unite_legale", "message": "Performed ES request", "severity": "DEBUG"}
api-1 | {"request_logging": true, "logger": "aleph.views.context", "timestamp": "2024-07-24 09:10:04.120958", "endpoint": "collections_api.index", "locale": "en", "trace_id": "aa2ad457-a612-4208-9e94-c9e396751587", "method": "GET", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections?filter:foreign_id=sirene_unite_legale", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 0.00889730453491211, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:04.112026", "status": 200, "path": "/api/2/collections?filter%3Aforeign_id=sirene_unite_legale", "time": "2024-07-24T09:10:04.120928", "message": "Request handled", "severity": "INFO"}
api-1 | {"logger": "werkzeug", "timestamp": "2024-07-24 09:10:04.121518", "endpoint": "collections_api.index", "locale": "en", "trace_id": "aa2ad457-a612-4208-9e94-c9e396751587", "method": "GET", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections?filter:foreign_id=sirene_unite_legale", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 0.00889730453491211, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:04.112026", "status": 200, "path": "/api/2/collections?filter%3Aforeign_id=sirene_unite_legale", "time": "2024-07-24T09:10:04.120928", "message": "172.18.0.7 - - [24/Jul/2024 09:10:04] \"GET /api/2/collections?filter:foreign_id=sirene_unite_legale HTTP/1.1\" 200 -", "severity": "INFO"}
api-1 | {"logger": "aleph", "timestamp": "2024-07-24 09:10:19.254121", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "e2bed91f-6a27-4bb6-98a8-333ea69ba15d", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:08.666188", "path": "/api/2/collections/1/_bulk?", "message": "Exception on /api/2/collections/1/_bulk [POST]", "severity": "ERROR"}
api-1 | {"logger": "aleph.views.base_api", "timestamp": "2024-07-24 09:10:19.254486", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "e2bed91f-6a27-4bb6-98a8-333ea69ba15d", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:08.666188", "path": "/api/2/collections/1/_bulk?", "message": "InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", "severity": "ERROR"}
api-1 | {"request_logging": true, "logger": "aleph.views.context", "timestamp": "2024-07-24 09:10:19.254827", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "e2bed91f-6a27-4bb6-98a8-333ea69ba15d", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 10.588584423065186, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:10:08.666188", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:10:19.254778", "message": "Request handled", "severity": "INFO"}
api-1 | {"logger": "werkzeug", "timestamp": "2024-07-24 09:10:19.255710", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "e2bed91f-6a27-4bb6-98a8-333ea69ba15d", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 10.588584423065186, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:10:08.666188", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:10:19.254778", "message": "172.18.0.7 - - [24/Jul/2024 09:10:19] \"\u001b[35m\u001b[1mPOST /api/2/collections/1/_bulk HTTP/1.1\u001b[0m\" 500 -", "severity": "INFO"}
api-1 | {"logger": "aleph", "timestamp": "2024-07-24 09:10:34.362830", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "95ae4639-cc45-4db4-93d6-794437e541b4", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:23.225461", "path": "/api/2/collections/1/_bulk?", "message": "Exception on /api/2/collections/1/_bulk [POST]", "severity": "ERROR"}
api-1 | {"logger": "aleph.views.base_api", "timestamp": "2024-07-24 09:10:34.363191", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "95ae4639-cc45-4db4-93d6-794437e541b4", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:23.225461", "path": "/api/2/collections/1/_bulk?", "message": "InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", "severity": "ERROR"}
api-1 | {"request_logging": true, "logger": "aleph.views.context", "timestamp": "2024-07-24 09:10:34.363524", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "95ae4639-cc45-4db4-93d6-794437e541b4", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 11.138012409210205, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:10:23.225461", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:10:34.363479", "message": "Request handled", "severity": "INFO"}
api-1 | {"logger": "werkzeug", "timestamp": "2024-07-24 09:10:34.364416", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "95ae4639-cc45-4db4-93d6-794437e541b4", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 11.138012409210205, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:10:23.225461", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:10:34.363479", "message": "172.18.0.7 - - [24/Jul/2024 09:10:34] \"\u001b[35m\u001b[1mPOST /api/2/collections/1/_bulk HTTP/1.1\u001b[0m\" 500 -", "severity": "INFO"}
api-1 | {"logger": "aleph", "timestamp": "2024-07-24 09:10:52.681868", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "a5fbf01b-87cf-42db-9b2c-83e7b739db00", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:40.598841", "path": "/api/2/collections/1/_bulk?", "message": "Exception on /api/2/collections/1/_bulk [POST]", "severity": "ERROR"}
api-1 | {"logger": "aleph.views.base_api", "timestamp": "2024-07-24 09:10:52.682222", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "a5fbf01b-87cf-42db-9b2c-83e7b739db00", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:10:40.598841", "path": "/api/2/collections/1/_bulk?", "message": "InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", "severity": "ERROR"}
api-1 | {"request_logging": true, "logger": "aleph.views.context", "timestamp": "2024-07-24 09:10:52.682523", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "a5fbf01b-87cf-42db-9b2c-83e7b739db00", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 12.083632469177246, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:10:40.598841", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:10:52.682478", "message": "Request handled", "severity": "INFO"}
api-1 | {"logger": "werkzeug", "timestamp": "2024-07-24 09:10:52.684755", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "a5fbf01b-87cf-42db-9b2c-83e7b739db00", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 12.083632469177246, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:10:40.598841", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:10:52.682478", "message": "172.18.0.7 - - [24/Jul/2024 09:10:52] \"\u001b[35m\u001b[1mPOST /api/2/collections/1/_bulk HTTP/1.1\u001b[0m\" 500 -", "severity": "INFO"}
api-1 | {"logger": "aleph", "timestamp": "2024-07-24 09:11:12.146326", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "0e5a5e6a-88fc-4721-ae7e-69e19a2c56c7", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:11:02.556743", "path": "/api/2/collections/1/_bulk?", "message": "Exception on /api/2/collections/1/_bulk [POST]", "severity": "ERROR"}
api-1 | {"logger": "aleph.views.base_api", "timestamp": "2024-07-24 09:11:12.146796", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "0e5a5e6a-88fc-4721-ae7e-69e19a2c56c7", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:11:02.556743", "path": "/api/2/collections/1/_bulk?", "message": "InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", "severity": "ERROR"}
api-1 | {"request_logging": true, "logger": "aleph.views.context", "timestamp": "2024-07-24 09:11:12.147189", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "0e5a5e6a-88fc-4721-ae7e-69e19a2c56c7", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 9.5903959274292, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:11:02.556743", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:11:12.147144", "message": "Request handled", "severity": "INFO"}
api-1 | {"logger": "werkzeug", "timestamp": "2024-07-24 09:11:12.147937", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "0e5a5e6a-88fc-4721-ae7e-69e19a2c56c7", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 9.5903959274292, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:11:02.556743", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:11:12.147144", "message": "172.18.0.7 - - [24/Jul/2024 09:11:12] \"\u001b[35m\u001b[1mPOST /api/2/collections/1/_bulk HTTP/1.1\u001b[0m\" 500 -", "severity": "INFO"}
api-1 | {"logger": "aleph", "timestamp": "2024-07-24 09:11:41.286786", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "6fc2ff6b-237f-45b9-9eca-90a63e5ef4ca", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:11:29.853187", "path": "/api/2/collections/1/_bulk?", "message": "Exception on /api/2/collections/1/_bulk [POST]", "severity": "ERROR"}
api-1 | {"logger": "aleph.views.base_api", "timestamp": "2024-07-24 09:11:41.287160", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 2190, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1486, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py\", line 176, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1484, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.8/dist-packages/flask/app.py\", line 1469, in dispatch_request\n return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n File \"/aleph/aleph/views/collections_api.py\", line 290, in bulk\n for entity_id in bulk_write(\n File \"/aleph/aleph/logic/processing.py\", line 38, in bulk_write\n entity = model.get_proxy(data, cleaned=(not clean))\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/model.py\", line 141, in get_proxy\n return EntityProxy.from_dict(self, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 498, in from_dict\n return cls(model, data, cleaned=cleaned)\n File \"/usr/local/lib/python3.8/dist-packages/followthemoney/proxy.py\", line 53, in __init__\n data = dict(data or {})\nValueError: dictionary update sequence element #0 has length 3; 2 is required", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "6fc2ff6b-237f-45b9-9eca-90a63e5ef4ca", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "referrer": null, "v": "3.17.0", "begin_time": "2024-07-24T09:11:29.853187", "path": "/api/2/collections/1/_bulk?", "message": "InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", "severity": "ERROR"}
api-1 | {"request_logging": true, "logger": "aleph.views.context", "timestamp": "2024-07-24 09:11:41.287502", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "6fc2ff6b-237f-45b9-9eca-90a63e5ef4ca", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 11.434263467788696, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:11:29.853187", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:11:41.287456", "message": "Request handled", "severity": "INFO"}
api-1 | {"logger": "werkzeug", "timestamp": "2024-07-24 09:11:41.288393", "endpoint": "collections_api.bulk", "locale": "en", "trace_id": "6fc2ff6b-237f-45b9-9eca-90a63e5ef4ca", "method": "POST", "ip": "172.18.0.7", "url": "http://api:5000/api/2/collections/1/_bulk", "session_id": "dca02b7d-7362-4b40-b53d-6569b6ced855", "ua": "alephclient/2.4.1", "role_id": 4, "took": 11.434263467788696, "referrer": null, "collection_id": 1, "v": "3.17.0", "begin_time": "2024-07-24T09:11:29.853187", "status": 500, "path": "/api/2/collections/1/_bulk?", "time": "2024-07-24T09:11:41.287456", "message": "172.18.0.7 - - [24/Jul/2024 09:11:41] \"\u001b[35m\u001b[1mPOST /api/2/collections/1/_bulk HTTP/1.1\u001b[0m\" 500 -", "severity": "INFO"}
the code i use to generate the entities is here https://codeberg.org/juke/action-reaction/src/branch/main/sirene/unite_legale/ftm.py
Those now live in servicelayer, so we can get rid of the version included here.
I noticed there's a parameter --schema for stream-entities command, but it seems to always all entities for the given collection as output regardless of what has been provided.
The API documentation doesn't mention this parameters for the _stream function anyway. Is it the client that is outdated in relation to the API, or could it be the case that the Aleph instance or its database I'm trying to reach is misconfigured?
Shall we have the make-list command return just the ID of the created list in some non-verbose mode?
Add an option in write-entities to specify the batch size when sending data to the server.
CD
Dependabot couldn't authenticate with https://pypi.python.org/simple/.
You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.
Does the alephclient require a valid certificate. I'm using self signed certs on my instance and I get the following doing a simple connection.
alephclient crawldir --foreign-id Intel /Intel
What I get back is the following
Error: HTTPSConnectionPool(host='intel.jigsaw-security.com', port=443): Max retries exceeded with url: /api/2/collections?filter%3Aforeign_id=4 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f87951a6750>: Failed to establish a new connection: [Errno 111] Connection refused',))
Golden profiles are the results of manual xref validations. alephclient should be able to download these from collections and add them respectively.
We can call it the .alephignore file
Sometimes when uploading a large number of documents through alephclient crawldir, a few documents may fail to be processed. But there is no easy way to check.
alephclient should be able to match the processed files on Aleph with the current directory structure to check whether the upload fully succeeded or not. Additionally, we should also be able to retry uploading the failed files only.
It would be nice if there'd be a way to configure proxy connection.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.