The Bewai open data api (ODA) is a project for importing large quantities of data from remotely downloaded files, mostly in csv format.
It enables data to be imported into a Postgres database and then retrieved via an Express API.
This project runs on Docker.
The ODA comes with several scripts to create tables, download fresh data files from remote sources and parallel data import into Postgres databse without affecting the availability of the API.
You can either run these commands directly with npm (npm run [...]) or with the ./docker-service [...] script file.
These scripts can be found in the src/utils directory.
commands can be either executed with
npm run [command]or with./docker-service [command]. If you are using the docker stack, you better use./docker-service.
| Command name | Description |
|---|---|
initTables |
Create all tables, their columns and indexes if not existing |
downloadAssets {entity} |
Download fresh data file for the given entity from configured remote source |
importData |
(Interactive) Imports the data for the given entity from a data file in the assets folder. Usually from .csv files. A custom CSV separator can be specified in the script file dataSets configuration object. |
unzipneeds to be installed directly on the machine. Because of some NodeJS limitations with Buffer size, extracting large.zipfiles with a NodeJS library can't be done properly.
In order to run the project, if you're on a Unix-based kernel machine (Mac OS / Linux) you can run commands through the ./docker-service file.
./docker-service initialize./docker-service serve./docker-service start./docker-service stop./docker-service tests
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent