rwth-ebc / filip Goto Github PK

Describe the bug
Two tests are failing in the pipeline

To Reproduce
Run the test test_subscriptions and test_cb_subscriptions

Due to Licence issues, we should substitute the fuzzywuzzy package used for unit validation with RapidFuzz.

https://github.com/maxbachmann/RapidFuzz

The usage of RapidFuzz should work in very similar way.

currently the repo contains the specifications as submodules. However, this is not necessary and only makes the repo unnessary large.

Describe the bug
Certain tests leave artifacts in the Fiware contextbroker after their clean up.
The clean uses the .update(action_type="delete") function which only deletes the attributes of the entity if it has some, leaving an entity entry with id an type in the broker.

Solution
Replace in clean up function:
client.update(entities=entities, action_type='delete')
with
for entity in entities: client.delete_entity(entity_id=entity.id, entity_type=entity.type)

This line returns the fiware service path but it needs to return the fiware service

Removing the author and creation date from the module docstrings.
Also deleted files that are no longer needed.

Is your feature request related to a problem? Please describe.
With the new version of the Orion Context Broker notifications via MQTT are supported.
https://fiware-orion.readthedocs.io/en/3.2.0/user/mqtt_notifications/index.html

Describe the solution you'd like
We want to add this functionality in our notifications models.
Furthermore, we need to introduce the deprecated query parameter for skipping the initial notification for backward compatibility. If set the the client will check the version of the context broker first and then return a warning to the user.

Describe alternatives you've considered
None

Extend the ContextEntity model with methods to easily access the existing commands and their additional information (status, info)

Add endpoints /v2/attrs and /v2/attrs/
(this issue is copied from https://git.rwth-aachen.de/EBC/Team_BA/projects/n5geh/tools/n5geh.tools.filip/-/issues/46)

The method post_request does not work, as the http field needs to be in json form:

Current body:
{"provider": {"http": "http://localhost:1234"}, "dataProvided": ...}

Required body:
{"provider": {"http": {"url": "http://localhost:1234" }, ...}

Further this issue will test if existing registrations are correctly removed (or edited) if the corresponding entity is deleted

Currently, units are automatically loaded on import.
This needs to change in order to make FiLiP PyPy ready.
Erase old code fragments.

Is your feature request related to a problem? Please describe.
Currently, the clean up fuction do not check weather the data is realy deleted. Hence, this should be checked instead of a time.sleep()

Describe the solution you'd like
Substitute time.sleep() with get_requests on list level.

Describe alternatives you've considered
None

Describe the bug
The tests for time series leave old subscriptions in orion.
Running multiple test over time will let them pile up.

To Reproduce
Steps to reproduce the behavior:

1. Run tests for time series several times
2. check for subscriptions in the context broker

Expected behavior
Subscriptions should be deleted after testing

Solutions
Use teardown at the end of the test case this will delete old subscriptions even if the test fails

we like to add an mqtt client that can be directly used with the iotagent-ul or iotagent-json.
The client will be placed in the clients.mqtt

Here is the orginal discussion on the module:

https://git.rwth-aachen.de/EBC/EBC_all/fiware/filip/-/issues/45

The client should especially simplify the topic subscription patterns for the mqtt interface.
This is not trivial in FIWARE:

https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/api.md#relationship-between-service-groups-and-devices

And also part of current discussions:

https://github.com/telefonicaid/iotagent-json/issues/585

Currently, unit validation is very strict. Therefore, we want to add hints to the user in order to find the right unit.

• write a try and except block that catches the key error from the unit-tables
• use fuzzywuzzy to generate and provide appropriate error message and hints

Sibling methods to post_entity and post_device.

These methods shall allow the user to readout an entity/device from Fiware, edit the object in any way they want and then publish the made changes (while keeping as much of the current Fiware state) to Fiware.

The PATCH Method introduces a simpler and more compressed interface for the user then the currently used UPDATE.
UPDATE has following issues:

• It does not add new metadata to the fields
• Does only append fields, does not deleted removed ones
• The user does need to know if the entity already exists
• Overwrites current state of fields, even if they were not changed locally in the entity

PATCH allows the user a simpler usage flow.

1. He downloads a few entities
2. Deletes/Adds a few fields
3. Changes values
4. Creates new entities

With Patch, he can now simply call the patch(entity) method for each entity that he has and does not need to differentiate between cases(new entity, field deleted, ...).

Additionally he can provide the entity_state before his editions to the method to only update the fields that he actively changed, thus he does not override the complete Fiware state of the entity, if there is concurrent access.

Describe the solution you'd like
Take the semantic logic that is currently situated in the SENSE fronted and transfer it over into Filip.
Further this logic needs to be transformed to better synergies with Filip entities.

Instead of the currently used instances that are only transformed to Filip entities when saving the state, a wrapper will be written to simply extend the entities with the needed semantic functionalities.

To make the storage and reparsing of the semantic vocabulary much easier the ontology files will be saved inside the pydantic vocabulary object, which can be stored as json.

If this feature is finished a user can provide a semantic structure to Filip by giving ontologies, and use this structure to create models, connect them and model a state.

Describe the bug
Parsing of queries containing special characters in attribute name returns None.
This behavior is caused by the used regular expression rf"^\w(\w*|\.(?=\w))*{op}\w*"

To Reproduce
QueryStatement.parse_str('diffuse-radiation!=\'123\'') returns None

Expected behavior
As described in the Specification, attribute names may include all special characters except control characters, whitespace, &, ?, / and #. Therefore the statement above should return a valid QueryStatement.

Is your feature request related to a problem? Please describe.
When using update_device to push made changes in a device to Fiware, the device in the iota client is correctly updated, but the corresponding entity is not. Here only new attributes are added, but removed attributes are not deleted and changed values are not updated. Further changed device_settings as transport are not taken over to Fiware.

Currently the only reliable way to take over all changes is to delete the device and the corresponding entity and to repost them.

Describe the solution you'd like
To make the interaction here much cleaner, I propose the method patch_device (linked to Issue 21), that takes a device model object and makes all the needed updates for the user to take over all made changes to Fiware. It will internaly delete and repost the device only if the main device settings (endpoint,..) were changed, else it will use the methods update_device and patch_entity to update the stats.

This issue will also provide the tests to ensure the correctness of the functions (and a test for update_device, that is still missing). Further a few simple quality of live methods as does_device_exist are introduced.

Is your feature request related to a problem? Please describe.
If the user creates a device a context entity is automaticaly created. But when deleting a device the entity is not automaticaly removed. Same if the linked entity is removed. This can be quite confussing for the user, as it is behaviour that is not suspected, and can lead to issues, as remenants of objects that are thought to be deleted still contiune existing.

Describe the solution you'd like
A parameter option for the delete methods, that allows the user to clear the whole state of the object with one methode call.
Even if the user does not use this optional parameter, it makes aware that the methode does not normally cleans the whole state.

Describe alternatives you've considered
None

Creating and testing a github action to run all tests automaticaly on a PUSH.

Refactoring all examples to match structure of semantics examples, achieving a coherent style and the ability to parse them all to Jupyter notebooks

The current test landscape leaves a lot untested for the DeviceModels.

New tests need to be created to cover all details in the device model creation process and if the created models can be correctly transferred to Fiware.

Is your feature request related to a problem? Please describe.
Support python 3.9 by dropping support for PyTables.

Describe the solution you'd like
Make tables support optional.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
PyTables/PyTables#823

Describe the bug
Apparently there are still errors when installing filip

To Reproduce
Steps to reproduce the behavior:

1. pip install .
2. pip uninstall filip
   The first does not copy the hdf file properly
   The second one would leave fragments

Expected behavior
Both commands should work

You are doing

logging.basicConfig(level=settings.LOGLEVEL,
                    format='%(asctime)s - FiLiP.%(name)s - %(levelname)s: %('
                           'message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')

in config.py.
This way, you change the settings for all other modules as well.
the LOGLEVL should be set by the user, not you.

What is the issue?

When sending a post_command request to the cb and querying the entity state, the set point status may change to PENDING and will not respond to any further requests.

Proposal

The issue might be related to the cmdexe (see https://github.com/FIWARE/tutorials.IoT-Agent-JSON#southbound-traffic-commands) and the cb expecting to receive an acknowledgement from the device.
As many device might not offer an acknowledgement, this should be optional/settable by the user.
If applied, the cmdexe handling still requires implementation.

Is your feature request related to a problem? Please describe.
Currently the docs are far from perfect. we want to improve them for new users.

Describe the solution you'd like
Refactor the docs and add more examples

• Removing all the now unneeded .env.fillip
• Adding a new .env to be used for local development
• Adapting tests that rely on the additional config files

Is your feature request related to a problem? Please describe.
I'm always frustated when I have to create an issue/feature branch by myself

Describe the solution you'd like
automatic branch creation with: robvanderleek/create-issue-branch

Describe the bug
Currently the StaticDevice metadata field allows the type Optional[Dict[str, Dict]]. But Fiware only allows a very specific Dict structure. Giving any other structure results in a success in Filip, but an uncatched background error in Fiware.

To Reproduce
Steps to reproduce the behavior:

1. Create an ContextDevice containing a StaticDeviceAttribute with the metadata: {"meta": { "test": "1" }}
2. Filip allows this assignment, you can now post the device without error
3. But if you now look in Fiware, no device or corresponding ContextEntity appeared.

Expected behavior
FiLiP tells you, that the assignment of the metadata was not valid, and stops the execution

Fix
Change the type of the StaticDevice metadata field to a BaseModel modeling the allowed structure. The ContextMetadata and NamedContextMetadata modeld in context.py, can probably be reused for this task. The StaticDeviceAttribute metadata field would then have the same structure and logic as the ContextAttribute metadata field.

The used regex:

regex="^((?![?&#/])[\x00-\x7F])*$" # Make it FIWARE-Safe"

does not catch the use of " " in these strings, but FIWARE does not allow spaces in the protected fields.

currently the contribution.md is locating in docs we want to move it on top level

When using Filip for the Cloud Sense Semantic App, each Entity and each Entitiyattribute has an entry that states the IRI this object/attribute is symbolising. This can be done with Datatype.Text, but to better symbolize the meaning of these fields an new Datatype coulb be introduced. Due to the fact that it is not a random char sequence but an concrete identifier of the semantic web.

Is your feature request related to a problem? Please describe.
QuantumLeap API implements a pagination mechanism in order to help clients to retrieve large sets of resources. (https://quantumleap.readthedocs.io/en/latest/user/pagination/)
However, this mechanism has not yet been implemented in the QuantumLeap client (see at https://github.com/RWTH-EBC/FiLiP/blob/development/filip/clients/ngsi_v2/quantumleap.py). Hence, the maximum number of results is limited to 10000 for a single request.

Describe the solution you'd like
The pagination mechanism is somewhat similar to the one already implemented for the context broker, compare https://github.com/RWTH-EBC/FiLiP/blob/development/filip/clients/ngsi_v2/cb.py (and also see https://fiware-orion.readthedocs.io/en/master/user/pagination/index.html). So, the solution should follow a similar approach.
However, the response is different, which results in a few changes, for instance, the json objects cannot be extended but rather the resulting data lists should be extended.
Please find a suggestion below:

def __pagination(*,
                method: PaginationMethod = PaginationMethod.GET,
                url: str,
                headers: Dict,
                limit: Union[PositiveInt, PositiveFloat] = None,
                offset: int = None,
                session: requests.Session = None,
                params: Dict = None,
                data: str = None) -> List[Dict]:
        """
        NGSIv2 implements a pagination mechanism in order to help clients to
        retrieve large sets of resources. This mechanism works for all listing
        operations in the API (e.g. GET /v2/entities, GET /v2/subscriptions,
        POST /v2/op/query, etc.). This function helps getting datasets that are
        larger than the limit for the different GET operations.
        https://fiware-orion.readthedocs.io/en/master/user/pagination/index.html
        Args:
            url: Information about the url, obtained from the original function
            headers: The headers from the original function
            params:
            limit:
        Returns:
            object:
        """
        if limit is None:
            limit = inf
        if limit > 10000:
            params['limit'] = 10000  # maximum items per request
        else:
            params['limit'] = limit
        if offset is None:
            params['offset'] = 0
        else:
            params['offset'] = offset
        additional_offset = 0
        if not session:
            session = requests.Session()
        with session:
            res = session.request(method=method,
                                  url=url,
                                  params=params,
                                  headers=headers,
                                  data=data)
            if res.ok:
                items = res.json()
                # do pagination
                count = int(res.headers['Content-Length'])
                attributes = params['attrs']

                # fresh start
                items['index'] = []
                items['attributes'][0]['values'] = []

                while len(items['index']) < limit and len(items['index']) < count:
                    try:
                        # Establishing the offset from where entities are retrieved
                        for attr in attributes:
                            params['attrs'] = attr
                            params['offset'] = len(items['index']) + additional_offset
                            params['limit'] = min(10000, (limit - len(items['index'])))
                            res = session.request(method=method,
                                                  url=url,
                                                  params=params,
                                                  headers=headers,
                                                  data=data)
                            if res.ok:
                                # items.extend(res.json()) --> this does not work for the given response structure
                                if len(items['attributes']) == len(attributes): # update entries
                                    i = find(lst=items['attributes'],key=attr)
                                    items['attributes'][i]['values'].extend(res.json()['attributes'][0]['values'])
                                elif len(items['attributes']) < len(attributes):
                                    items['attributes'].extend(res.json()['attributes'])
                                else:
                                    logger.error('Inconsistent number of attributes %i', len(items['attributes']))
                            else:
                                res.raise_for_status()
                        items['index'].extend(res.json()['index'])
                    except Exception as e:
                        if '404 Client Error' in e.args[0]:
                            # no records found, skip period
                            limit -= 10000
                            additional_offset += 10000
                        else:
                            print(e)
                    if not (pytz.utc.localize(dateutil.parser.parse(params['toDate']))>
                        dateutil.parser.parse(items['index'][-1])+datetime.timedelta(seconds=1)):
                        break
                logger.debug('Received: %s', items)
                items['entity_id'] = params['entity_id']
                items['entity_type'] = params['entity_type']
                ts = parse_obj_as(TimeSeries, items)
                return ts
            res.raise_for_status()

Describe alternatives you've considered
The obvious alternative is to individually evoke single requests and post-process the results to the desired format.

Additional context
Beside the different response structure, when there are no records within a period, the function is stuck since the request will fail and neither the items nor the limit will be updated. This exception has to be caught. Note that in the above example, the request of several attributes has been considered.

There are some issues with adding complex information (with Enums) to value fields in DeviceAttributes. The here describe issue could also be happening for Context Attribute, this was not yet tested. Below are two approches that I took to save a complex Basemodel in a value field of a StaticDeviceAttribute, both with there own issues.

class DeviceAttribute(Basemodel):
	name: str
    attribute_type: DeviceAttributeType

if we have a DeviceAttribute da, we can export the model in 2 forms:

da.dict()
	=> {'name': 'd1', 'attribute_type': <DeviceAttributeType.lazy: 'lazy'>}
da.json()
	=> {"name": "d1", "attribute_type": "lazy"}

If we add the da export to a values array:

v1 = values.append(da.dict()) 
	=> [{'name': 'd1', 'attribute_type': <DeviceAttributeType.lazy: 'lazy'>}]
v2 = values.append(da.json()) 
	=> ['{"name": "d1", "attribute_type": "lazy"}'] 

If we add the values array to a StaticDeviceAttribute:

attr = iot.StaticDeviceAttribute(
		name="attributeName",
		type=DataType.STRUCTUREDVALUE,
		value= vN,
		entity_name=None,
		entity_type=None,
		reverse=None,
		expression=None,
		metadata=None
	))

With vN = v1:
attr => name='attributeName' ... value={'name': 'attribute_type'}
With vN = v2:
attr => name='attributeName' ... value=['{"name": "d1", "attribute_type": "lazy"}']

The dict() variant gets strangely transformed.
The json() variant does look correct and can be posted without error, but the device containing this attribute and its corresponding context entity will not appear in Fiware.

Currently I can not think of a reason for any of those two.

The Field "expressionLanguage" in Device Model is not correct

• It is states in the description that it is a boolean value, and that it has the possible values legacy or jexl. And is of type ExpressionLangauge.

• It is described as an Optional Field, with the default value of legacy. But it has no default value, as a result the user needs to explicitly state the value of this field for each Device that he creates

  expressionLanguage: Optional[ExpressionLanguage] = Field(
  description="optional boolean value, to set expression language used "
  "to compute expressions, possible values are: "
  "legacy or jexl. When not set or wrongly set, legacy "
  "is used as default value."
  )

Describe the bug
The validator of the DeviceAttribute Model in iot.py, should accept:
Optional[Dict[str, Union[Dict, str]]]

But it throws an error on the values: None, {"test", "testing"}. Both values are correct and can be successfully treated by the post_device methode.

To Reproduce
Steps to reproduce the behavior:

1. Create a new DeviceAttribute object with one of the stated values in the metadata field.

Expected behavior
No Error

Most methods are missing the Raises part in the Docstring commands.

Some are informing about possible Error raises in the description, but not all.

I would suggest adding the Raises tag and detail information to all methods that could raise an error. With this it is directly clear to the user if he needs to handle errors and which types to expect.

Is your feature request related to a problem? Please describe.
Currently, concurrent testing will automatically interfere with each other because the servicepaths are static. Therefore, the tests will all use the same tenants which will result in failing tests, if they try to access the same resources.

Describe the solution you'd like
The idea is to introduce a TestingSettings-class that will check for a local environment for its variables. If a *.env. is not available it will try to use environment variables. These variables should be set in the CI because the CI should not contain a file with information about the used servers. The servicepath should then be either autogenerated or set in the first or second way.

Describe the bug
double definition of paho-mqtt in requirements.txt

Apparently, there is a minor bug in the client log function. It checks for the wrong existence of the response objekt.

if err.response is not None:
    ....output message

Describe the bug
The description of the field states:

"entity_name: the presence of this attribute indicates "
"that the value will not be stored in the original device "
"entity but in a new entity with an ID given by this "
"attribute. The type of this additional entity can be "
"configured with the entity_type attribute. If no type is "
"configured, the device entity type is used instead. "

But if a value other than None is given (see other issue: we need to always state a value, even if we want None) the Attribute does not appear in the context broker.

If the entity_id and type are equal to the device info, it does not appear in the device (this is an edge case, and could be an expected behaviour)
But if some other value is given no entity is created with the given id and type, thus the attribute is not present in the context broker.

To Reproduce
Steps to reproduce the behavior:

1. Create a device object
2. Add some Attribute to it, with entity_name and entitiy_type no null
3. post_device

Expected behavior
A new entity with (entity_name and entity_type) is present in the ContextBroker and holds the attribute

currently, pydantic models are not rendered very well in the material design. Therefore, we want to move back to the readthedocs design

Currently the generate_vocabulary_models always produces a python file with the given path and filename.

To be more flexible with its usage, it should only save the models if both path and filename are given, else it returns the file-content as string.

This opens the possibilities for easier transference of models to a backend server. It is an optional feature which does not impact any currently implemented logic.

Describe the bug
Method update_subscription of filip.clients.ngsi_v2.ContextBrokerClient has following configuration for request serialization:

res = self.patch(
                url=url,
                headers=headers,
                data=subscription.json(exclude={'id'},
                                       exclude_unset=True,
                                       exclude_defaults=True,
                                       exclude_none=True))

If a subscription is created/updated with non default values you can never re-set the default ones because defaults are excluded from serialization.

To Reproduce
Steps to reproduce the behavior:

As an example try to update the status of a subscription:

        with ContextBrokerClient(url='http://localhost:1026',
                                 fiware_header=FiwareHeader(
                                     service='fw-service',
                                     service_path='/testing',
                                 )) \
                as cb_client:

            subscription = cb_client.get_subscription('xyz')
            subscription.status = Status.INACTIVE

            cb_client.update_subscription(subscription)

→ subscription updates status to "inactive"

            subscription.status = Status.ACTIVE
            cb_client.update_subscription(subscription)

→ subscription still has status "inactive"

Environment (please complete the following information):

• OS: Windows 10
• Version 20H2
• Python 3.8

This issue will make the search for existing semantic objects and their loading more flexible by including more of the query parameters that get_entity_list() offers in load_instances_from_fiware().

Prevent that post_subscription (ngsi_v2.cb.py) creates a duplicate entry.

The methode shall now check, if the exact same subscription already exists.
If yes, it makes a log entry and returns the ID of the already created subscription

All semantic instance should hold two variables: name, comment that the user can fill out to keep a better overview of the instances.

This feature is mainly inspired by the needs of the SENSE app, but it can be a useful tool for each instance organistation.
And as it is optional, it does not really disturb if it is not used (only an unused additional field in the CB).