Device listings ignore the service and servicepath constraints

With the change from the (service, subservice) key to the (resource, apikey) key, the opportunity arrives for an operation targeting a group in a specific service to be executed by an actor identified in another service (as the only validated piece of data are the fiware-service and fiware-servicepath headers). An error should be raised for those cases when there is a mismatch between the stored service and the one declared in the headers.

This operation currently returns all the information in the db. It should at least provide limit and offset filters to allow the implementation of pagination.

This API is currently accepting the service and subservice as part of the POST body. This should be changed to take them from the headers, as it is done in the Configuraiton API.

Some changes have been accomplished in the device provisioning API:

• The root document must include a single "devices" attribute with an array of devices, instead of a single device per document.
• The "protocol" attribute must be included with each device provisioning in order to identify the kind of protocol in use for the device.
• The "name" attribute must be changed to "deviceId".

IoTAgentTester doesn't currently support string quotes in the command line, so information about an entity or attribute with an space in the name cannot be retrieved.

In order to be more flexible and enable some IoT scenarios (e.g. IoTBroker dealing with NGSI10 and IoTDiscovery dealing with NGSI9 as different components), it would be useful to have the possibility of specifiying different endpoints for NGSI9 and NGSI10 operations (by default, both endpoints will be the same).

The Device Provisioning API should let the user provision a device without the need of having a previously registered type.

Given a device already provisioned, like button_dev_async_c86f5c7b which is in mongoDb entity at devices collection:

{
    "_id" : ObjectId("560a93828d7bcd44799d899c"),
    "protocol" : "TT_BLACKBUTTON",
    "internalAttributes" : [],
    "internalId" : null,
    "registrationId" : "560a9381feb1449905efdfcf",
    "subservice" : "/telepizza_c86f5c7b",
    "service" : "blackbutton",
    "name" : "button_dev_sync_c86f5c7b",
    "type" : "BlackButton",
    "id" : "button_dev_sync_c86f5c7b",
    "creationDate" : ISODate("2015-09-29T13:34:58.050Z"),
    "staticAttributes" : [ 
        {
            "value" : "BBB",
            "name" : "ccid",
            "type" : "string"
        }, 
        {
            "value" : "2345678902",
            "name" : "imei",
            "type" : "string"
        }, 
        {
            "value" : "2987654322",
            "name" : "imsi",
            "type" : "string"
        }, 
        {
            "value" : "synchronous",
            "name" : "interaction_type",
            "type" : "string"
        }, 
        {
            "value" : "blackbutton",
            "name" : "service_id",
            "type" : "string"
        }, 
        {
            "value" : "0",
            "name" : "geolocation",
            "type" : "string"
        }
    ],
    "commands" : [],
    "active" : [ 
        {
            "name" : "internal_id",
            "type" : "string"
        }, 
        {
            "name" : "last_operation",
            "type" : "string"
        }, 
        {
            "name" : "op_status",
            "type" : "string"
        }, 
        {
            "name" : "op_result",
            "type" : "string"
        }, 
        {
            "name" : "op_action",
            "type" : "string"
        }, 
        {
            "name" : "op_extra",
            "type" : "string"
        }, 
        {
            "name" : "sleepcondition",
            "type" : "string"
        }, 
        {
            "name" : "sleeptime",
            "type" : "string"
        }
    ],
    "lazy" : [ 
        {
            "name" : "lazy_op_result",
            "type" : "string"
        }
    ],
    "__v" : 0
}

Trying to remove a device by /iot/devices DELETE method

time=2015-09-29T13:36:03.583Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.MongoDBDeviceRegister | msg=Looking for entity with id [button_dev_async_c86f5c7b].
time=2015-09-29T13:36:03.611Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.MongoDBDeviceRegister | msg=Removing device with id [button_dev_async_c86f5c7b]
time=2015-09-29T13:36:03.614Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.MongoDBDeviceRegister | msg=Entity [button_dev_async_c86f5c7b] not found for removal.
time=2015-09-29T13:36:03.614Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.NorthboundServer | msg=Error [ENTITY_NOT_FOUND] handing request: The entity with the requested id [button_dev_async_c86f5c7b] was not found.

The entity device exists, bug if no entity device was found a 400 error would be enough.

there are differences between Uri calls on both implementations of iota (C++ vs Node)

It happen in the retrieve of services info:

In iota c++ is used to be like:

/iot/services 

but in the Node iota is like:

/iot/agents/default/services 

Looking at: https://github.com/telefonicaid/iotagent-node-lib#iotagentlibupdate
Could you provide any example to pass metadatas over attribute values?

All the requests are currently being sent to the default Context Broker defined in contextBroker.host. It should take the parameters from the Context Broker configuration for each type if existent.

Early in the development of the IoT Agent, the library API was thought to use the combination of deviceId and deviceType as a key to distinguish different devices. This assumption doesn't hold anymore, so it's time to change it.

When the contextBroker returns a response that is being served by different Context Providers, there may be multiple Context Responses with different error codes. The IoT Agent should cope with the different codes.

Service and servicepath attributes of the device provisioning request are mandatory, even now that they are taken from the service headers.

The 'commands' attribute is also mandatory, and it shouldn't.

Provisioning the same device twice currently makes the second definition override the first. This should be changed to ensure that, once a pair (DeviceID, Service) has been provisioned, no other can be provisioned with the same IDs

The features needed to integrate with the IOTA Manager (mainly the on-start autoregistration) were added to the library some time ago, but the full integration with the IOTAM and the Administration Portal has not been tested.

If not present, the Context Provider registration to the Context Broker returns a 400 error. This situation should be prevented and an error raised in the creation of the agent.

Add static attributes to all the entities of the device (always the same information for the device).

Due to a bug in the provisioning API, requests without the lazy attribute raised a TYPE_NOT_FOUND error.

The provision command should send the headers configured in the configIoT object when provisioning a device.

I'm having issues with running tests for the IoT agent library. Here's error description from mocha:

1) MongoDB Group Registry test When a new device group creation request arrives should store the service information from the headers into the DB:
 Uncaught TypeError: Cannot read property 'length' of undefined
  at processResults (/home/marcin/iotagent-node-lib/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1581:31)
  at /home/marcin/iotagent-node-lib/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1619:20
  at /home/marcin/iotagent-node-lib/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1157:7
  at b (domain.js:183:18)
  at /home/marcin/iotagent-node-lib/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1890:9
  at b (domain.js:183:18)
  at Server.Base._callHandler (/home/marcin/iotagent-node-lib/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/base.js:448:41)
  at /home/marcin/iotagent-node-lib/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:481:18
  at MongoReply.parseBody (/home/marcin/iotagent-node-lib/node_modules/mongoose/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
  at null.<anonymous> (/home/marcin/iotagent-node-lib/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:439:20)

My environment is:

• clean install of Ubuntu Server 14.04
• fresh clone of repository
• Mongo 3.0.4 installed as specified here
• node 0.10.25 and npm 1.3.10
• npm install and npm install -g grunt
• grunt test

The problem is also occuring on Windows.

What might be the problem?

Some important pieces of documentation are missing:

• Library configuration.
• How to use the trust-based security.
• How to use de Provisioning API.

It should also be updated to the last version of the library.

Currently, both configuartions differ just in the name of their attributes. This situation should evolve to make both specificaions interchangeable.

The folowing test fails ocasionally:
´´´
NOTE: ACTUAL 0, EXPECTED -1

1. Device provisioning API: List provisioned devices When a request for listing all the devices with a offset of 3 arrives should skip the first 3 devices:

  actual expected

  -10

  at Assertion.prop.(anonymous function) (/Users/mru/github/iotagent-node-lib/node_modules/should/lib/should.js:61:14)
  at Request._callback (/Users/mru/github/iotagent-node-lib/test/unit/listProvisionedDevices-test.js:272:91)
  at Request.self.callback (/Users/mru/github/iotagent-node-lib/node_modules/request/request.js:123:22)

´´´

The general IoT Agent configuration is currently read from a text file (the exception is the per device configuration of the device provisioning API). It should be configurable using the exposed API.

The resource is a unique identifier that may be used for listening in URLs either in the Northbound or the Southbound, so it should be forced to be unique.

Implement the autoregistration of the IOTA in a IOTA Manager, on start.

We are currently using a temporary version of logops that was released temporary, as a link to the Github repository. We should use the standard versions of logops instead.

Currently, just one Configuration is allowed per subservice (as the service/subservice pair is key in the API). The API should accept multiple configuration groups in the POST and more identification information in the PUT operations, using two parameters (apikey and resource) to complete the available interactions.

When an operation relating CB Content is executed and fails with a transport error (HTTP status code != 200 and not known to be a common error), an empty details object is returned. At least information about the status code or the returned error should be provided.

Being optional, as they are, the presence of absence of the types array should not crash the IoT Agent. Either way, if they are mandatory, the error should be catched and presented properly, or if they are be optional, it should not create an error at all.

When a function of the library is executed and the Context Broker answers with a non-200 statusCode, a BAD_REQUEST error is returned, always with code 400 and a standard text. It should return the code indicated in the Orion error and the details returned by the CB.

The test suite seems to fail when using MongoDB 3 instead of using 2.x versions. MongoDB 3 support should be granted to the library (but we should move along with Orion's Mongo version in this case).

The GET operations in the Provisioning API currently return all the information in the device registry without any limits or pagination. It should at least implement the limit and offset parameters.

Currently, the IoT Agent doesn't make any duplicity checks on the names that are provisioned via the Device Provisioning API or the Configuration API. The following checks should be done in each incoming creation or update:

• In a Device provisioning, each attribute name in the active, lazy, static or command category arrays must be unique among all the categories.
• In a Configuration provisioning, each attribute name in the active, lazy, static or command category arrays must be unique among all the categories.
• If a Device belonging to a Configuration is provisioned, and an attribute in any of the attribute categories (e.g: lazy) is defined in that provision, its name should not be present in any other category (e.g: active or command). I.e: a lazy attribute in the device should be able to overwrite a lazy attribute in the Configuration, but should not conflict with attributes in the active, command or static categories.

Add the files needed to integrate Travis CI into the development process.

Currently, the entity type name is an internal key so if a new device group arrives with the same device type it will overwrite de older one. This behavior is wrong, but there is no clear idea about what to use as the universal ID of a device group (service/subservice tuple may be a candidate, but multiple types should appear for a single tuple).

Currently, having a device configuration, device provisioning or configured type without type name will make the IOTAgent crash with a stack trace error. Fix this to avoid crashing, emitting a TYPE_NOT_FOUND 500 error.

Some devices are not being read in the device provisioning API: internalAttributes, timezone, static_attributes.

Currently, commands implementation is not working. Commands should behave like Lazy attributes with state writing in the Context Broker, as defined in the documentation.

An error is raised when running "updatevalue" option from agent console:

IoTAgent> updatevalue l2 Light status/Boolean/false
time=2015-04-20T15:23:59.586Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.NGSIService | msg=Updating device value in the Context Broker at [http://cbroot:1026/v1/updateContext]
time=2015-04-20T15:23:59.586Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.NGSIService | msg=Using the following request:

{
"url": "http://cbroot:1026/v1/updateContext",
"method": "POST",
"json": {
"contextElements": [
{
"type": "Light",
"isPattern": "false",
"id": "l2"
}
],
"updateAction": "APPEND"
},
"headers": {
"fiware-service": "wiseacess",
"fiware-servicepath": "/garden",
"X-Auth-Token": null
}
}

time=2015-04-20T15:24:00.034Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.NGSIService | msg=Value updated successfully
/programs/telefonicaid/iotagent-node-lib/lib/services/ngsiService.js:285
callback(null, body);
^
TypeError: undefined is not a function
at Request._callback (/programs/telefonicaid/iotagent-node-lib/lib/services/ngsiService.js:285:17)
at Request.self.callback (/programs/telefonicaid/iotagent-node-lib/node_modules/request/request.js:123:22)
at Request.emit (events.js:110:17)
at Request. (/programs/telefonicaid/iotagent-node-lib/node_modules/request/request.js:1047:14)
at Request.emit (events.js:129:20)
at IncomingMessage. (/programs/telefonicaid/iotagent-node-lib/node_modules/request/request.js:998:12)
at IncomingMessage.emit (events.js:129:20)
at _stream_readable.js:908:16
at process._tickCallback (node.js:355:11)

There is currently a mechanism to provide internal attributes (for its use in particular details of each protocol) for individual device provisionings, but the Configuration API lacks of the same support. It should provide the same mechanisms.

When new information is sent to the Context Broker, use the service and subservice corresponding to the entity type configured in the ngsi seciton of the IoT Agent configuration, instead of the global one, when available.

An error is raised running "listdevices" option from agent console:

IoTAgent> listdevices
/programs/telefonicaid/iotagent-node-lib/lib/services/deviceRegistryMemory.js:91
callback(null, result);
^
TypeError: undefined is not a function
at Object.listDevices as list
at Object.listDevices (/programs/telefonicaid/iotagent-node-lib/lib/services/ngsiService.js:421:18)
at Object.listDevices as handler
at executeCommander (/programs/telefonicaid/iotagent-node-lib/bin/agentConsole.js:175:30)
at Interface. (/programs/telefonicaid/iotagent-node-lib/bin/agentConsole.js:216:9)
at Interface.emit (events.js:107:17)
at Interface._onLine (readline.js:214:10)
at Interface._line (readline.js:553:8)
at Interface._ttyWrite (readline.js:830:14)
at ReadStream.onkeypress (readline.js:109:10)

Currently, the library handles the Device Provisioning in a fully automatic way, without giving the user code the opportunity to react to the provisioning. In the case of Configuration provisionings, this opportunity is given using an optional user handler for Configuration updates. A similar mechanism should be provided for Device provisioning.

When a query arrives to the Context Provider Server of the IOT Agent, if the query does not specify any attributes, the Agent answers with a response without attribtes. This behavior doesn't comply with the standard: empty queries should return all the attributes in the entity.

There should be a character check for each of the attributes in a Device Provisioning or Configuration creation. The appropriate check should also be discussed with other parties involved (Context Broker, other IoT Agents, and so).

There is currently a slight difference in the NGSI Handlers between the way the information is received (split in different parameters) and the way it should be returned (grouped in an object). Callbacks should offer the option of returning the split information instead of the object. This shouldn't change the signature, as two of the attributes (id and type) are required, so the different uses can be easily distinguished (by number of parameters or type).

The file ngsiService.js is currently too big to be mantainable. It could be easily splitted in at least two files:

• One regarding registrations and device management.
• Another one regarding context operations.

Other tasks that could be performed to improve mantainability:

• Create a RegistryNotAvailable wrapper for the device management functions.
• Create a common function with the sendUpdateValue and sendQueryValue functionalities.
• Extract sendRegistrations request handler to an external function.