Git Product home page Git Product logo

siren's Introduction

SIREN API

Build Status docker pulls docker image info docker tag

REST API for serving INSEE files v3.

Getting started

To have a working copy of this project, follow the instructions.

Installation

Setup Rust.

Define your environment variables as defined in .env.sample. You can either manually define these environment variables or use a .env file.

Setup a postgresql database (macOS commands).

brew install postgresql
createuser --pwprompt sirene # set password to sirenepw for instance
createdb --owner=sirene sirene

Documentation

Configuration

Recommended configuration for production with docker:

RUST_LOG=sirene=warn
SIRENE_ENV=production
BASE_URL=[Your base URL, needed to update asynchronously]
API_KEY=[Any randomized string, needed to use the HTTP admin endpoint]
DATABASE_URL=postgresql://[USER]:[PASSWORD]@[PG_HOST]:[PG_PORT]/[PG_DATABASE]
DATABASE_POOL_SIZE=100
INSEE_CREDENTIALS=[Base64(consumer-key:consumer-secret)]

How to generate INSEE_CREDENTIALS

This variable is only needed if you want to have the daily updates.

  1. Go to https://api.insee.fr/catalogue/
  2. Create an account or sign in
  3. Create an application on this portal
  4. Subscribe this application to the Sirene - V3 API
  5. Generate a key pair in the application details
  6. Copy the key from the curl example and paste it in .env: Authorization: Basic [INSEE_CREDENTIALS]

CLI

> sirene --help

sirene 2.0.0
Julien Blatecky
Sirene service used to update data in database and serve it through a HTTP REST API

USAGE:
    sirene [OPTIONS] <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --db-folder <db-folder>        Path to the file storage folder for the database, you can set
                                       in environment variable as DB_FOLDER. Could be the same as
                                       FILE_FOLDER if this app and the database are on the same file
                                       system. Files copied by this app inside FILE_FOLDER must be
                                       visible by the database in DB_FOLDER
        --file-folder <file-folder>    Path to the file storage folder for this app, you can set in
                                       environment variable as FILE_FOLDER
        --temp-folder <temp-folder>    Path to the temp folder, you can set in environment variable
                                       as TEMP_FOLDER

SUBCOMMANDS:
    help      Prints this message or the help of the given subcommand(s)
    serve     Serve data from database to /unites_legales/<siren> and /etablissements/<siret>
    update    Update data from CSV source files

> sirene serve --help

sirene-serve
Serve data from database to /unites_legales/<siren> and /etablissements/<siret>

USAGE:
    sirene serve [OPTIONS]

FLAGS:
        --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -k, --api-key <api-key>      API key needed to allow maintenance operation from HTTP, you can
                                 set in environment variable as API_KEY
    -b, --base-url <base-url>    Base URL needed to configure asynchronous polling for updates, you
                                 can set in environment variable as BASE_URL
        --env <environment>      Configure log level, you can set in environment variable as
                                 SIRENE_ENV [possible values: development, staging, production]
    -h, --host <host>            Listen this host, you can set in environment variable as HOST
    -p, --port <port>            Listen this port, you can set in environment variable as PORT

> sirene update --help

sirene-update
Update data from CSV source files

USAGE:
    sirene update [FLAGS] <group-type> [SUBCOMMAND]

ARGS:
    <group-type>    Configure which part will be updated [possible values: unites-legales,
                    etablissements, all]

FLAGS:
        --data-only    Use an existing CSV file already present in FILE_FOLDER and does not delete
                       it
        --force        Force update even if the source data where not updated
    -h, --help         Prints help information
    -V, --version      Prints version information

SUBCOMMANDS:
    clean-file       Clean files from FILE_FOLDER
    download-file    Download file in TEMP_FOLDER
    finish-error     Set a staled update process to error, use only if the process is really
                     stopped
    help             Prints this message or the help of the given subcommand(s)
    insert-data      Load CSV file in database in loader-table from DB_FOLDER
    swap-data        Swap loader-table to production
    sync-insee       Synchronise daily data from INSEE since the last modification
    unzip-file       Unzip file from TEMP_FOLDER, and move it to the FILE_FOLDER

HTTP API

GET /v3/unites_legales/<siren>
GET /v3/etablissements/<siret>

Maintenance

This API is enabled only if you have provided an API_KEY when starting the serve process.

POST /admin/update

{
    api_key: string,
    group_type: "UnitesLegales" | "Etablissements" | "All",
    force: bool,
    asynchronous: bool,
}

If asynchronous is set to true, the update endpoint will immediately return the following:

Status: 202 Accepted
Location: /admin/update/status?api_key=string
Retry-After: 10

[Initial status for the started update]
GET /admin/update/status?api_key=string

If an update is in progress, the status code will be 202, otherwise 200.

POST /admin/update/status/error

{
    api_key: string,
}

Basic usage

Serve:

cargo run serve

Update:

cargo run update all

Help:

cargo run help

Tests

cargo test

Deployment

A docker image is built and a sample docker-compose.yml with its docker folder are usable to test it.

Authors

License

MIT

siren's People

Contributors

dependabot-preview[bot] avatar dependabot[bot] avatar johandufau avatar julien1619 avatar ldouziech avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

siren's Issues

[Suggestion] New endpoints

Hi,

We were really interested in your projet but it lacks some endpoint for us to use it.
We would need to research by RNA or denomination.
As we are not rust developers we can't contribute to the project.
We want to know if you would add this to this project or we will do our own api to match our expectations.

Best regards

Error on UpdatableModel model

Hello,

first of all, thank you for your project!

I tried to set it up but i encounter the following error after
./siren update all
2023-07-31T17:34:06.963032Z ERROR sirene::update: Errored: Error on UpdatableModel model: Unable to run some operations on updatable model (could not open file "/app/data/files/StockUniteLegale_utf8.csv" for reading: No such file or directory).
2023-07-31T17:34:06.965120Z ERROR sirene::update::error: Error on UpdatableModel model: Unable to run some operations on updatable model (could not open file "/app/data/files/StockUniteLegale_utf8.csv" for reading: No such file or directory).

however, the file was created and has data :

/app # tail /app/data/files/StockUniteLegale_utf8.csv
999990526,O,,1983-01-01,,,,,,,,,,NN,,2011-11-29T01:35:23,3,PME,2020,2008-01-01,A,,,TEXAS GULF EXPORT CORPORATION,,,,3220,73.20Z,NAFRev2,00017,,,N
999990542,O,,1984-01-25,,,,,,,,,,NN,,2020-08-25T11:51:54,8,,,2018-09-25,C,,,MILLEIS DIVERSIFICATION,,,,5599,69.10Z,NAFRev2,00022,,,O

what should i do?

thx

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.