Git Product home page Git Product logo

storehouse's Introduction

Overview

WARNING: As previously announced on our communication channels, the Kytos project will enter the "shutdown" phase on May 31, 2021. After this date, only critical patches (security and core bug fixes) will be accepted, and the project will be in "critical-only" mode for another six months (until November 30, 2021). For more information visit the FAQ at <https://kytos.io/faq>. We'll have eternal gratitude to the entire community of developers and users that made the project so far.

License Build status Code coverage Code-quality score

This NApp is responsible for data persistence, saving and retrieving information. It can be acessed by external agents through the REST API or by other NApps, using the event-based methods.

Each box of data can be saved in a different namespace, with support for nested namespaces.

The NApp is designed to support many options of back-end persistence solutions. Currently it features filesystem operations, but alternatives such as SQL or NoSQL databases will be implemented.

Installing

All of the Kytos Network Applications are located in the NApps online repository. To install this NApp, run:

$ kytos napps install kytos/storehouse

Events

Listened

The NApp listens to events requesting operations. Every event must have a callback function to be executed right after the internal method returns. The signature of the callback function is described with each event.

kytos.storehouse.create

Event requesting to save data to a box in a namespace.

Content

{
    data: <any data to be saved>,
    namespace: <namespace name>,
    callback: <callback function> # To be executed after the method returns.
}

Callback function

def callback_function_name(box, error=False):
    # box: copy of the Box instance stored with data and metadata.
    # error: False when the operation is successful, True otherwise.

kytos.storehouse.retrieve

Event requesting to load data from a box in a namespace.

Content

{
    box_id: <ID of the Box to retrieve data from>,
    namespace: <namespace name>,
    callback: <callback function> # To be executed after the method returns.
}

Callback function

def callback_function_name(box, error=False):
    # box: the retrieved Box instance.
    # error: False when the operation is successful, True otherwise.

kytos.storehouse.list

Event requesting to list all boxes in a namespace.

Content

{
    namespace: <namespace name>,
    callback: <callback function> # To be executed after the method returns.
}

Callback function

def callback_function_name(box_list, error=False):
    # box_list: the retrieved list of Box.box_id.
    # error: False when the operation is successful, True otherwise.

kytos.storehouse.delete

Event requesting to remove a box from a namespace.

Content

{
    box_id: <ID of the Box to be deleted>,
    namespace: <namespace name>,
    callback: <callback function> # To be executed after the method returns.
}

Callback function

def callback_function_name(result, error=False):
    # result: True if the box was deleted, False otherwise .
    # error: False when the operation is successful, True otherwise.

Rest API

You can find a list of the available endpoints and example input/output in the 'REST API' tab in this NApp's webpage in the Kytos NApps Server.

storehouse's People

Contributors

ajoaoff avatar beraldoleal avatar cmagnobarbosa avatar gleybersonandrade avatar hdiogenes avatar josemauro avatar llgoncalves avatar lucasgsfelix avatar macartur avatar niehaus avatar pythrick avatar renanrodrigo avatar viniarck avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar

storehouse's Issues

Support other backend options

For now we support only filesystem operation, but we must implement other backend options, like SQL/NoSQL databases or remote sorage.

Add tests

This NApp has no test implemented.

Several Kytos Errors Found when Creating and Deleting Multiple EVCs

We were running code to test the controller's ability to create and delete multiple EVCs one after another. The code we were using is the same as the one Antonio developed for testing the mef_eline and can be found here: https://github.com/ajoaoff/mef_eline/blob/tests/tests/integration/tests/test1.sh

The errors starting popping up after we had created and deleted a couple hundred EVCs, and then started showing up more recently after the creation of just a few EVCs.

This was originally opened in kytos/kytos#949.

2019-08-15 17:32:19,119 - INFO [kytos/mef_eline] (Thread-126570) Box 0a99ae72cf454386ba5ab7031db228ed was updated.
2019-08-15 17:32:19,170 - INFO [kytos/mef_eline] (Thread-126573) Removing fedbbb81601f4903b54fe2eab5e80d88
2019-08-15 17:32:19,179 - INFO [kytos/mef_eline] (Thread-126572) Box 0a99ae72cf454386ba5ab7031db228ed was updated.
Exception in thread Thread-126560:
Traceback (most recent call last):
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/src/kytos/kytos/core/helpers.py", line 72, in threaded_handler
handler(*args)
File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 311, in event_update
self._execute_callback(event, box, error)
File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 142, in _execute_callback
event.content['callback'](event, data, error)
File "/var/lib/kytos/napps/../napps/kytos/mef_eline/storehouse.py", line 104, in _save_evc_callback
log.info(f'Box {data.box_id} was updated.')
AttributeError: 'bool' object has no attribute 'box_id'
kytos $> 2019-08-15 17:32:15,122 - INFO [kytos/mef_eline] (Thread-126343) Box 0a99ae72cf454386ba5ab7031db228ed was updated.
2019-08-15 17:32:15,192 - INFO [kytos/mef_eline] (Thread-126345) Box 0a99ae72cf454386ba5ab7031db228ed was updated.
2019-08-15 17:32:15,205 - INFO [kytos/mef_eline] (Thread-126349) Removing f1e407a25faa47bbb6e3734ade50ff41
2019-08-15 17:32:15,332 - INFO [kytos/mef_eline] (Thread-126360) Box 0a99ae72cf454386ba5ab7031db228ed was updated.
Exception in thread Thread-126359:
Traceback (most recent call last):
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/src/kytos/kytos/core/helpers.py", line 72, in threaded_handler
handler(*args)
File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 299, in event_update
box = backend.retrieve(namespace, box_id)
File "/var/lib/kytos/napps/../napps/kytos/storehouse/backends/fs.py", line 88, in retrieve
return self._load_from_file(destination)
File "/var/lib/kytos/napps/../napps/kytos/storehouse/backends/fs.py", line 56, in _load_from_file
data = pickle.load(load_file)
EOFError: Ran out of input
Exception in thread Thread-464:
Traceback (most recent call last):
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/tmp/kytos/kytos/core/helpers.py", line 72, in threaded_handler
handler(*args)
File "/tmp/python-kytos/var/lib/kytos/napps/kytos/storehouse/main.py", line 267, in event_retrieve
self._execute_callback(event, box, error)
File "/tmp/python-kytos/var/lib/kytos/napps/kytos/storehouse/main.py", line 134, in _execute_callback
event.content['callback'](event, data, error)
File "/tmp/python-kytos/var/lib/kytos/napps/../napps/kytos/mef_eline/storehouse.py", line 85, in _get_box_callback
log.debug(f'Box {self.box.box_id} was load from storehouse.')
AttributeError: 'bool' object has no attribute 'box_id'
2019-08-16 16:20:38,410 - ERROR [kytos.core.controller] (Thread-460) Exception on /api/kytos/mef_eline/v2/evc/ [POST]
Traceback (most recent call last):
File "/tmp/python-kytos/lib64/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/tmp/python-kytos/lib64/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/tmp/python-kytos/lib64/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/tmp/python-kytos/lib64/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/tmp/python-kytos/lib64/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/tmp/python-kytos/lib64/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/tmp/python-kytos/lib64/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functionsrule.endpoint
File "/tmp/python-kytos/var/lib/kytos/napps/kytos/mef_eline/main.py", line 126, in create_circuit
evc.deploy()
File "/tmp/python-kytos/var/lib/kytos/napps/../napps/kytos/mef_eline/models.py", line 457, in deploy
success = self.deploy_to_backup_path()
File "/tmp/python-kytos/var/lib/kytos/napps/../napps/kytos/mef_eline/models.py", line 427, in deploy_to_backup_path
return self.deploy_to_path()
File "/tmp/python-kytos/var/lib/kytos/napps/../napps/kytos/mef_eline/models.py", line 552, in deploy_to_path
self.sync()
File "/tmp/python-kytos/var/lib/kytos/napps/../napps/kytos/mef_eline/models.py", line 234, in sync
self._storehouse.save_evc(self)
File "/tmp/python-kytos/var/lib/kytos/napps/../napps/kytos/mef_eline/storehouse.py", line 89, in save_evc
self.box.data[evc.id] = evc.as_dict()
AttributeError: 'bool' object has no attribute 'data'
Exception in thread Thread-18977:
Traceback (most recent call last):
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/tmp/kytos/kytos/core/helpers.py", line 72, in threaded_handler
handler(*args)
File "/tmp/python-kytos/var/lib/kytos/napps/kytos/storehouse/main.py", line 267, in event_retrieve
self._execute_callback(event, box, error)
File "/tmp/python-kytos/var/lib/kytos/napps/kytos/storehouse/main.py", line 134, in _execute_callback
event.content['callback'](event, data, error)
File "/tmp/python-kytos/var/lib/kytos/napps/../napps/kytos/mef_eline/storehouse.py", line 85, in _get_box_callback
log.debug(f'Box {self.box.box_id} was load from storehouse.')
AttributeError: 'bool' object has no attribute 'box_id'

Add a search with filters.

The current implementation only returns a list with all the objects in a given namespace. A search method is necessary so the applications using storehouse do not have to retrieve all the objects and then do the filtering.

Storehouse error with brand new Kytos Installation

Command used:
/tmp/kytos/venv/bin/pip3 install -e git+http://github.com/kytos/storehouse#egg=storehouse

The following error happens during initialization and is affecting the mef_eline:

2020-10-30 16:27:47,988 - CRITICAL [kytos.core.controller] (MainThread) NApp initialization failed: kytos/storehouse
Traceback (most recent call last):
  File "/tmp/kytos/venv/src/kytos/kytos/core/controller.py", line 777, in load_napp
    napp = napp_module.Main(controller=self)
  File "/tmp/kytos/venv/src/kytos/kytos/core/napps/base.py", line 192, in __init__
    self.setup()
  File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 112, in setup
    self.create_cache()
  File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 128, in create_cache
    cache = metadata_from_box(box)
  File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 21, in metadata_from_box
    "name": box.name,
  File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 58, in name
    return self._name
AttributeError: 'Box' object has no attribute '_name'

Etcd storehouse backend not working

Hi,

I'm trying to use the the Etcd storehouse backend by the following steps:

  1. Use the Kytos/nightly docker image

  2. After start the container:

apt-get update && apt-get install etcd-server etcd-client && service etcd start

sed -i 's/BACKEND = "filesystem"/BACKEND = "etcd"/g' /var/lib/kytos/napps/kytos/storehouse/settings.py

kytosd -E

Just after starting Kytos we have some errors on the log:

Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:182:  Bad callback function <bound method StoreHouse._get_or_create_a_box_from_list_of_boxes of <napps.kytos.mef_eline.storehouse.StoreHouse object at 0x7faeac31c5c0>>!
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:182:  Bad callback function <bound method Main.request_retrieve_entities of <Main(topology, stopped 140387805136640)>>!
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:183:  'generator' object is not subscriptable
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:183:  'generator' object is not subscriptable
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:182:  Bad callback function <bound method Main.request_retrieve_entities of <Main(topology, stopped 140387805136640)>>!
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:183:  'generator' object is not subscriptable
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:182:  Bad callback function <bound method Main.request_retrieve_entities of <Main(topology, stopped 140387805136640)>>!
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:182:  Bad callback function <bound method StoreHouse._get_or_create_a_box_from_list_of_boxes of <napps.kytos.topology.storehouse.StoreHouse object at 0x7faeb1d62208>>!
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:183:  'generator' object is not subscriptable
Nov 12 21:11:55 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:183:  'generator' object is not subscriptable

If you ignore those erros and go ahead to actually use Kytos ("use Kytos" here means create an EVC in mef_eline napp, but I suppose this will happens for any other usage):

prompt$ curl -s -X POST -d '{"name": "my evc3 100","enabled": true,"dynamic_backup_path":true, "uni_a": { "interface_id": "00:00:00:00:00:00:00:01:1", "tag": {"tag_type":1, "value": 200} }, "uni_z": {"interface_id": "00:00:00:00:00:00:00:02:1", "tag": {"tag_type":1, "value": 200}}}' http://$KYTOS_IP:8181/api/kytos/mef_eline/v2/evc/ -H "Content-Type: application/json"

{"code":500,"description":"The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.","name":"Internal Server Error"}

In the server logs:

Nov 12 21:13:35 d7cc32bbe40d kytos.core.controller:ERROR app:1892:  Exception on /api/kytos/mef_eline/v2/evc/ [POST]
Nov 12 21:13:35 d7cc32bbe40d Traceback (most recent call last):
Nov 12 21:13:35 d7cc32bbe40d File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app
Nov 12 21:13:35 d7cc32bbe40d response = self.full_dispatch_request()
Nov 12 21:13:35 d7cc32bbe40d File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request
Nov 12 21:13:35 d7cc32bbe40d rv = self.handle_user_exception(e)
Nov 12 21:13:35 d7cc32bbe40d File "/usr/local/lib/python3.6/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
Nov 12 21:13:35 d7cc32bbe40d return cors_after_request(app.make_response(f(*args, **kwargs)))
Nov 12 21:13:35 d7cc32bbe40d File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception
Nov 12 21:13:35 d7cc32bbe40d reraise(exc_type, exc_value, tb)
Nov 12 21:13:35 d7cc32bbe40d File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
Nov 12 21:13:35 d7cc32bbe40d raise value
Nov 12 21:13:35 d7cc32bbe40d File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request
Nov 12 21:13:35 d7cc32bbe40d rv = self.dispatch_request()
Nov 12 21:13:35 d7cc32bbe40d File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request
Nov 12 21:13:35 d7cc32bbe40d return self.view_functions[rule.endpoint](**req.view_args)
Nov 12 21:13:35 d7cc32bbe40d File "//var/lib/kytos/napps/kytos/mef_eline/main.py", line 148, in create_circuit
Nov 12 21:13:35 d7cc32bbe40d self.storehouse.save_evc(evc)
Nov 12 21:13:35 d7cc32bbe40d File "//var/lib/kytos/napps/../napps/kytos/mef_eline/storehouse.py", line 89, in save_evc
Nov 12 21:13:35 d7cc32bbe40d self.box.data[evc.id] = evc.as_dict()
Nov 12 21:13:35 d7cc32bbe40d AttributeError: 'NoneType' object has no attribute 'data'
Nov 12 21:13:35 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:182:  Bad callback function <bound method StoreHouse._get_or_create_a_box_from_list_of_boxes of <napps.kytos.mef_eline.storehouse.StoreHouse object at 0x7faeac31c5c0>>!
Nov 12 21:13:35 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:183:  'generator' object is not subscriptable
Nov 12 21:13:35 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:182:  Bad callback function <bound method StoreHouse._get_or_create_a_box_from_list_of_boxes of <napps.kytos.mef_eline.storehouse.StoreHouse object at 0x7faeac31c5c0>>!
Nov 12 21:13:35 d7cc32bbe40d kytos.napps.kytos/storehouse:ERROR main:183:  'generator' object is not subscriptable

I guess the problem starts since this error at the beginning:

kytos.napps.kytos/storehouse:ERROR main:182:  Bad callback function <bound method StoreHouse._get_or_create_a_box_from_list_of_boxes of <napps.kytos.mef_eline.storehouse.StoreHouse object at 0x7faeac31c5c0>

Repeated deprecation message about `name` parameter

We are seeing the following message every few seconds after starting Kytos. Since it doesn't do much for us, it would be ideal to remove it to have a cleaner console environment.

2020-12-08 21:25:39,887 - WARNING [kytos.napps.kytos/storehouse] (Thread-13037) The name parameter will be deprecated soon.

Storehouse does not include a requirements/run.in as the other napps

Storehouse should include a requirements/run.in as the other napps and list as dependency for run:

filelock==3.0.12

Otherwise, when you run kytos/storehouse installed via kytos napps install kytos/storehouse you will get the following error:

2020-06-29 17:54:54,861 - CRITICAL [kytos.core.controller] (MainThread) NApp initialization failed: kytos/storehouse
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kytos/core/controller.py", line 780, in load_napp
    napp = napp_module.Main(controller=self)
  File "/usr/local/lib/python3.6/site-packages/kytos/core/napps/base.py", line 192, in __init__
    self.setup()
  File "//var/lib/kytos/napps/kytos/storehouse/main.py", line 107, in setup
    from napps.kytos.storehouse.backends.fs import FileSystem
  File "//var/lib/kytos/napps/../napps/kytos/storehouse/backends/fs.py", line 10, in <module>
    from filelock import FileLock
ModuleNotFoundError: No module named 'filelock'

Add examples to openapi.yml

As the data in the boxes can be of any type and have any model, we must think on how to put examples of input and output to the OpenAPI file.

Change some INFO messages to DEBUG

There are messages of kytos.storehouse being displayed that should be severity DEBUG but they are as INFO. Some of them are:

2020-05-28 18:48:57,524 - INFO [filelock] (Thread-97) Lock 140001066670456 released on /var/tmp/lock/.var.tmp.kytos.storehouse.kytos.mef_eline.circuits.a1d8a4eff9434c018c10e32494988f51.lock
2020-05-28 18:48:57,532 - INFO [filelock] (Thread-100) Lock 140001036311800 acquired on /var/tmp/lock/.var.tmp.kytos.storehouse.kytos.mef_eline.circuits.a1d8a4eff9434c018c10e32494988f51.lock

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.