The objective with smokeping is to purpose an alternative to the original Smokeping but with recent technos (who still want to play with cgi bins?!) and new advanced functionalities.

The objective in this feature is to monitor the latency of many hosts across the network of workers and add the possibility to create thresholds for handling alerts, trigger actions. Because we are in the 21st century, today we want to have beautiful graphs for populate our televisions in our NOCs, so we want to design a feature that anybody can plug a Grafana or an alternative.

To help us to design the feature to the good way, we need your help for:

• what do you except with a smokeping system, what do you want to make if a metric have trigger a limit (give me your uses cases and suggestions)
• Still do you need RRD png exports? 🤔
• Is better to add one more dependency to the project by adding a new time series database like InfluxDB, using an existing Prometheus or simply use the existing PostgreSQL database and add an extension (e.g. https://github.com/timescale/timescaledb) for having a timeseries optimized engine

Thank be advance for your contribution to this thread

Create the operation page that show the progress of the operation.

If the operation are in pending, or in progress, make an XHR call every 10 seconds and refresh the page data.

UUID and JSON are PostgreSQL specific types. To make Netwark compatible with SQLite (for development/tests purposes) and compatible with MySQL, it's needed to create two new types that convert the UUID as CHAR and JSON as TEXT.

Create a IPv4 and IPv6 calculator available as API endpoint. The route can only receive an IP + CIDR (example: 10.10.0.0/24, 2001:db8:3f::/48).

The API endpoint should seems like: http://netwark/api/v1/ip-calc/10.200.0.0%2f24 (The resource need to be URL encoded. %2f = / character)

The output for IPv4 need to seems like:

{
    "network": "10.200.0.0",
    "netmask": "255.255.255.0",
    "CIDR": 24,
    "first_ip": "10.200.0.1",
    "last_ip": "10.200.0.254",
    "usable_ips": 254,
    "broadcast": "10.200.0.255"
}

For /31 and /32, the result are the same but without the first_ip and last_ip.

For IPv6, the return is quite similar of IPv4 output except is not returning the netmask and the broadcast IP:

{
    "network": " 2001:db8:3f::",
    "CIDR": 48,
    "first_ip": " 2001:db8:3f::1",
    "last_ip": " 2001:db8:3f:ffff:ffff:ffff:ffff:ffff",
    "usable_ips": 1208925819614629174706176,
}

Because clicodromes (french expression to designate GUIs/web interfaces with 3 millions buttons and interactions) are fun but not fast for an ops, we need to create a CLI that permit to:

• create an operation
• list operations with filters
• see the result of an operation (with a follow mode in option)
• use a synchronous tool

The CLI will use the REST API of the webserver. Multiple instances can be configured via a yaml configuration file or an another adapted format.

The CLI should be a new repository.

The API catch a 500 error if the database is not available but it not return the normal 500 error message (it return a traceback in dev env or a waitress error in production env).

Create a cron task that check the life of operations.

Multiple status are available:

• pending
• progress
• done
• error
• timeout

Status explanation:

• If the task is in pending and no entry was added in Operation_result table, put the operation status to timeout.
• If the task is in progress, contains operation_result entries and have not received any update in the last 45 seconds, update the status to done.
• If multiple entries in operation_result are in error, put the entire operation in error.

Description

/!\ Redaction still in progress /!\
The WHOIS part (probably not the definitive name) allows the user to retrieve information about a resource on the internet. In the list of searchable resources:

• IP WHOIS
• Domain WHOIS
• ASN (Autonomous System Number) WHOIS

This part are completely synchronous and get informations from the host server (not through the backend).

IP WHOIS

When an IP is requested, we request theses informations:

• WHOIS result (of course)
• PTR record
• GeoIP results from the MaxMind database.

Domain WHOIS

• WHOIS result (of course)
• ANY result from dig

ASN WHOIS

• WHOIS result (of course)
• PeeringDB informations

Create the mechanism for running a task.

Note: The task only change the status only on start if the status in pending. The task only change it own status on operation_status table.

Process:

• If the operation status is pending, change the status to progress.
• If the status is done, timeout or error, stop the execution of the task.
• Create the entry into operation_result table
• Run the command with the module subprocess
  • If subprocess have not finished, update the updated_at in the database
  • If an error occurs, update the operation_result table and put the stderr in the payload field.
• If the execution is more than 3 minutes, stop the execution and change the status in operation_result table.
• Parse the data in a JSON format and update the operation_result table.

Create a gulp task for copy images folder from assets to dist folder.

Create the API routes for :

• listing all operations
• returning all informations about the specified operation
• create a new operation

GET: /api/v1/operations

Querystring:

• per_page: limit elements per page (default: 50)
• page: page number
• status: filter by operation status
• type: filter by operation type

Return:

[
    {
        "id": "7e81e673-9b64-42ad-ba3f-5e2a74fe9ec7",
        "type": "mtr",
        "target": "1.1.1.1",
        "queues": [
            "lc_paris_eqx_dc2",
            "lc_paris_ild_pa2",
            "lc_ams_eqx_am3"
        ],
        "status": "progress",
        "created_at": "2019-05-01 23:42:23.324324"
    },
   ...
]

GET: /api/v1/operations/{operation_id}

Return:
WIP

POST: /api/v1/operations

Params:

• type: type of the operation (one of: OPERATION_FLAGS)
• target: IP/hostname
• options (optional): options to send to the operation
• queues (optional): list of queues the task will be sent

Return:

{
  "message": "Request created",
  "resource": "{uuid}"
}

Return codes:

• 200: OK
• 400: Bad request
• 500: Internal Error

Tasks

• Create collection_get route
  • Create the pagination system
  • Create the filtering system
• Create get route
• Create post route

Create the page that show the list of operations.

On this page, it's needed to show:

• operation type
• target
• queues
• status
• created_at

Create the database models for operations.

Two tables are needed: one with operation metadata, the second with the result of pushed by backends.

Metadata table: operations

• id: uuid
• type: enum
• destination: string
• payload: json
• queues: array
• status: enum
• created_at: datetime w/ timezone (UTC hour)
• updated_at: datetime w/ timezone (UTC hour)

Note: all fields are mandatory.

Results table: operations_results

• id: uuid
• operation: uuid FK->operations
• worker: string
• queue: string
• status: enum
• payload: json
• created_at: datetime w/ timezone (UTC hour)
• updated_at: datetime w/ timezone (UTC hour)

Note: all fields are mandatory.

The status in this table is related to the status of the task runned by the worker.

Enum type

• ping
• mtr

Enum status

• waiting
• progress
• done
• error
• timeout

Create a update OUI vendor script for updating periodically the table with new data from IEEE.

For running operations and others tools with Netwark around the world, we need to make a Celery backend using RabbitMQ.

The backend should listen two queues:

• General queue (named netwark_common). All non specific tasks are sent through this queue.
• Geographic queue (named lc{location,datacenter,circuit}_). All tasks specific to this location/datacenter/circuit are sent through this queue.

The configuration should be in a YAML file or in the PasteDeploy config (same used for Pyramid).

Some tasks need to be broadcasted to all workers on the queue.

Create a API endpoint for whois a ressource like it's created for the "normal" route. The view route can be reworked to use this endpoint in the future.

Hello,

I've an error when I run docker-compose run --rm webserver poetry run python netwark/bin/update_maxmind_db.py config/production.ini following the install guide.

I've alerdy ran the chown command before.

Output :

Creating network "netwark_default" with the default driver
Creating netwark_rabbit_1 ... done
Creating netwark_db_1     ... done
2019-05-30 00:52:17,027 INFO  [netwark_update_oui_vendor_table:30][MainThread] Downloading new version of City database...
2019-05-30 00:52:20,568 INFO  [netwark_update_oui_vendor_table:40][MainThread] Downloading new version of ASN database...
2019-05-30 00:52:21,337 INFO  [netwark_update_oui_vendor_table:49][MainThread] Uncompress tar archives
GeoLite2-City_20190528/
GeoLite2-City_20190528/LICENSE.txt
GeoLite2-City_20190528/GeoLite2-City.mmdb
GeoLite2-City_20190528/COPYRIGHT.txt
GeoLite2-City_20190528/README.txt
GeoLite2-ASN_20190528/
GeoLite2-ASN_20190528/LICENSE.txt
GeoLite2-ASN_20190528/GeoLite2-ASN.mmdb
GeoLite2-ASN_20190528/COPYRIGHT.txt
2019-05-30 00:52:22,047 INFO  [netwark_update_oui_vendor_table:53][MainThread] Create destination folders if not exists
mkdir: can't create directory '/opt/netwark/config/maxmind_db': Permission denied
mkdir: can't create directory '/opt/netwark/config/maxmind_db': Permission denied
2019-05-30 00:52:22,049 INFO  [netwark_update_oui_vendor_table:57][MainThread] Copy new databases to their destination folders
cp: can't create '/opt/netwark/config/maxmind_db/GeoLite2-City.mmdb': No such file or directory
cp: can't create '/opt/netwark/config/maxmind_db/GeoLite2-ASN.mmdb': No such file or directory
2019-05-30 00:52:22,050 INFO  [netwark_update_oui_vendor_table:62][MainThread] Delete temporary files

If I ignore theses errors the netwark_webserver_1 container wan't to start...

Thanks,

Create the database models for operations.

Two tables are needed: one with operation metadata, the second with the result of pushed by backends.

Metadata table: operations

• id: uuid
• type: enum
• destination: string
• payload: json
• queues: array
• status: enum
• created_at: datetime w/ timezone (UTC hour)
• updated_at: datetime w/ timezone (UTC hour)

Note: all fields are mandatory.

Results table: operations_results

• id: uuid
• operation: uuid FK->operations
• worker: string
• queue: string
• status: enum
• payload: json
• created_at: datetime w/ timezone (UTC hour)
• updated_at: datetime w/ timezone (UTC hour)

Note: all fields are mandatory.

The status in this table is related to the status of the task runned by the worker.

Enum type

• ping
• mtr

Enum status

• waiting
• progress
• done
• error
• timeout

Add content to the overview page like informations about operations