sth_openiot problem about tutorials.historic-context-flume HOT 5 CLOSED

sth_openiot will only be created once the first historic data reading is persisted. The sticking point will be one of the following:

1. Wait for the IoT devices to be provisioned before starting
2. Remember to open the door/switch on the light to start the data flow
3. Ensure that the subscription is working before checking the database.

Firstly ensure you have the latest components:

./services create

Then start-up the mongodb tutorial

./services mongodb

WAIT until the series of {}{}{}{} has completed before continuing (or your devices will register as Thing rather than Motion001, Motion002 etc.

Open the device monitor http://localhost:3000/device/monitor and check that the bell is working before opening the door and switching on the light.

WAIT for the readings to start flowing - then you can start the subscription.

If you check the subscription, you should find that "timesSent" attribute is non-zero;

[
    {
        "id": "5ba9f151be6818f658be8096",
        "description": "Notify Cygnus of all context changes",
        "status": "active",
        "subject": {
            "entities": [
                {
                    "idPattern": ".*"
                }
            ],
            "condition": {
                "attrs": []
            }
        },
        "notification": {
            "timesSent": 388,
            "lastNotification": "2018-09-25T08:58:59.00Z",
            "attrs": [],
            "attrsFormat": "legacy",
            "http": {
                "url": "http://cygnus:5050/notify"
            },
            "lastSuccess": "2018-09-25T08:58:59.00Z"
        },
        "throttling": 5
    }
]

If you view the cygnus logs you should find that data is persisting:

docker logs fiware-cygnus

At that point you can follow the instructions in the tutorial for reading mongodb data.

Note - if you have opened the door before you created the subscription, db["sth_/_Door:001_Door"].find().limit(10) will not return any data. Use:

use sth_openiot
show collections

to see which entities have persisted data.

from tutorials.historic-context-flume.

I've followed your steps, but I still the same problem mentioned, regardless of the chosen DBMS. No DB is create in any DBMS.

If I check the subscription, there is no timeSent or lastNotification.

The log only shows this, over and over:
time=2020-02-15T13:09:06.426Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=run | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[131] : Checking file:/opt/apache-flume/conf/agent.conf for changes time=2020-02-15T13:09:06.470Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=run | msg=com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$PeriodicalNameMappingsReader[205] : [nmi] The configuration has not changed time=2020-02-15T13:09:06.487Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[492] : Batch accumulation time reached, the batch will be processed as it is time=2020-02-15T13:09:06.491Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[492] : Batch accumulation time reached, the batch will be processed as it is time=2020-02-15T13:09:07.169Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=handle | msg=com.telefonica.iot.cygnus.management.ManagementInterface[107] : Management interface request. Method: GET, URI: /v1/version

from tutorials.historic-context-flume.

The tutorial has just been updated to use Cygnus 1.18.0 - please run ./services create to update your components.

Cygnus 1.18.0 fully supports NGSI-v2 subscriptions so the legacy flag is no longer required.

If I check the subscription, there is no timeSent or lastNotification. No DB is create in any DBMS.

This would seem to indicate that the subscription you created has not reached Cygnus. Check the Orion log to see what has happened.

The most likely cause would be a mismatch in the fiware-service Header. The context broker is implicitly multi-tenant, and the the tutorial the devices have been provisioned to be held under openiot, if the fiware-service Header is not passed it will use the default which is / and therefore no entities will be found.

I have updated and re-run the Mongo-DB part of the tutorial, and the updated commands appear to be working with standard NGSI-v2 subscriptions now.

from tutorials.historic-context-flume.

@jason-fox thanks for the reply.

Now I get the following error in the Cygnus log:

org.apache.flume.source.http.HTTPBadRequestException: 'fiware-servicepath' header value does not match the number of notified context responses
        at com.telefonica.iot.cygnus.handlers.NGSIRestHandler.getEvents(NGSIRestHandler.java:366)

and in the Orion log:

srv=openiot | subsrv=<none> | comp=Orion | op=httpRequestSend.cpp[570]:httpRequestSendWithCurl | msg=Notification Successfully Sent to http://cygnus:5051/notify
srv=openiot | subsrv=<none> | comp=Orion | op=httpRequestSend.cpp[583]:httpRequestSendWithCurl | msg=Notification response NOT OK, http code: 400

Steps to reproduce:

• run the command: ./services mongodb
• from the monitor webpage is not possible to issue any commands, so I run the requests from the IoT Agent Tutorial to create the service group, provisioning the sensors (with the default values fiware-service:openiot and fiware-servicepath:/) and the commands to open the door and turn on the light
• now, on the monitor webpage is possible to see the data flowing, but still not possible to send command from the page
• run the requests: "Subscribe to Context Changes" and "Check Subscription is working"
• in the notification section of the response appears "lastSuccessCode": 400
• checking the logs, I've found the errors above
• and the is no database sth_openiot

Can you help?
Thanks!

from tutorials.historic-context-flume.

This error is covered in the README.md

ℹ️ Note: if you see errors of the following form within the Cygnus log:

Received bad request from client.
cygnus | org.apache.flume.source.http.HTTPBadRequestException: 'fiware-servicepath' header
value does not match the number of notified context responses
For versions of Cygnus prior to 1.18.0 an additional attrsFormat=legacy attribute is required in the subscription since
Cygnus currently only accepted notifications in the older NGSI v1 format.

Please ensure that none of your subscriptions are setup using attrsFormat=legacy

from tutorials.historic-context-flume.