edgexfoundry / device-sdk-go Goto Github PK

I built the example service and when trying to run it I get the following:

mhall@mhall-laptop:~/go/src/github.com/edgexfoundry/device-sdk-go/examples/simple/cmd$ ./simple-device 
./simple-device flag redefined: confdir
panic: ./simple-device flag redefined: confdir

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc42007c060, 0x812e00, 0xc420080d70, 0x7ca8b7, 0x7, 0x7d9b25, 0x2d)
	/usr/lib/go-1.10/src/flag/flag.go:810 +0x540
flag.StringVar(0xc420080d70, 0x7ca8b7, 0x7, 0x0, 0x0, 0x7d9b25, 0x2d)
	/usr/lib/go-1.10/src/flag/flag.go:719 +0x8a
main.main()
	/home/mhall/go/src/github.com/edgexfoundry/device-sdk-go/examples/simple/cmd/main.go:40 +0x1ac

Changing the flag from "confdir" to "confDir" (or removing that flag altogether) fixes this issue.

Implement a new private device struct which encapsulates the public Device object imported from edgex-go. This new struct can then include a mutex which can be used to prevent a Device from being removed while operations involving it are in-progress.

Estimated effort: 3d

The SDK currently manages all configuration (local and via the registry). If an author of a device profile wants to add protocol-specific configuration values, this is not possible today. Instead, they would be required handle protocol configuration entirely separate from the existing mechanisms. Fixing this issue would allow a DS develop to leverage the existing configuration mechanisms to support protocol-specific configuration values, perhaps via the use of another level of path in the configuration key naming.

I suggest that we alter this method in two ways.

Firstly that where a command encompasses multiple ResourceOperations these should be handled in a single call. This allows the device implementation to perform optimized multiple reads or writes on hardware which is capable.

Secondly that the method is split into two variants for PUT and GET operations. The parameters required for these two are different, and I feel there is not enough code commonality to warrant implementing both in a single method.

This would result in something like

HandleGetOperation (device *models.Device,
   objects []models.DeviceObject,
   results []CommandResult)

(ValueDescriptor and ResourceObject are held in the CommandResult)

and

HandlePutOperation (device *models.Device,
    objects []models.DeviceObject,
   requests []CommandRequest)

Where CommandRequest is defined similarly to CommandResult but containing a value to set rather than a Result to be filled in.

Implement device resource command chaining.

Command chaining is when a device resource specifies another device resource in it's Resource Operation as opposed to specifying actual device resource commands. See the following comment in command.go for more details:

https://github.com/edgexfoundry/device-sdk-go/blob/master/command.go#L150

Estimated effort: 2-3d

Implement binary encoding (using CBOR) of events & readings.

This first assumes that a new bytes field (vs. the result field which is a string) has been added to the Reading struct, so the CBOR can encode the byes properly.

Estimated effort: 5-7d

Dynamically create provisionwatcher instances based on configuration settings so that device services can implement dynamic discovery.

For more details see the Device Services functional requirements document.

Estimated effort: 3d

Top-level Service methods should be added to allow device service implementations to dynamically update or remove devices. The operations result in changes to Core Metadata in addition to the device service's local Device objects.

A top-level AddDevice Service method already exists.

Estimated effort: 3-4d

Assertions are another attribute in a device resource's value PropertyValue which specify a string value which the result is compared against. If the comparison fails, then the result is set to a string of the form “Assertion failed for device resource: , with value: ”, this also has a side-effect of setting the device operatingstate to DISABLED. In the case of the single device /command endpoint, a 500 status code is also returned. If the all form of the /command endpoint is used an assertion failure will return a 200 status, and it's the responsibilty of the client to check each reading for assertion failures.

As assertions are operations on results after they've been transformed into the string to be passed in the Reading, assertions cannot as currently defined, be applied to binary readings which result in an array of bytes vs. a string result in the Reading.

For more details see the Device Services functional requirements document.

Estimated effort: 2d

The current implementation of the local device profile caching as implemented in profiles.go is based on the original EdgeX Java Device Services SDK. The original code is overly-complicated and has one enormous function called addDevice() which should be broken up. Instead of having a simple map or array of DeviceProfile structs, maps of DeviceObjects and commands are created.

Recently a map of DeviceProfile objects was added to handle automatic importing of device profiles via on-disk YAML files.

This task is to explore simplification of the code, possibly using recently added DeviceProfile map directly instead using the derived command and objects maps. Thread safety should also be reviewed, and unit tests added as part of this task.

Estimated effort: 5-8d

Implement the ping endpoint, and ensure that if the consul/registry is enabled, the ping endpoint is specified when registering as a service.

Estimated effort: 1d

The current SDK only handles querying device readings. This task is to add command actuation (i.e PUT) support.

For more details see the Device Services functional requirements document.

Estimated effort: 5d

Currently there is a bug in getDeviceObjectByName() and always returns an empty DeviceObject.

The current implementation of clients/init.go:

	params := types.EndpointParams{
		UseRegistry: isUG,
		Interval:    15,
	}

The interval should be configurable.

The ProtocolDriver HandleCommands function has a parameter called 'params' which is used to pass command actuation parameters to the protocol specific code. Currently if the 'set/put' form of the command endpoint is called, this parameter is the corresponding JSON passed to the endpoint. It was suggested instead that we should parse this JSON and add a new parms field to the CommandRequest struct to simply the work the ProtocolDriver has to perform.

Implement reading mappings.

Mappings are an attribute in resource operation (see Appendix C - Device Profiles for details) and consist of a static list of string result values that contain a fixed mapping. Ex.

[ “8675309”: “911”, “2112”: “1”, “777”: “666”]

Mapping are applied after assertions are checked, and are the final transformation before readings are created.

For more details see the Device Services functional requirements document.

Note - support for mappings is a nice-to-have for Dehli.

Estimated effort: 2d

The format of log messages are not consistent in the code, so we should identify the format and refine all the log messages

CommandResult currently does not encode floating point values as base64 strings.

For more details see the Device Services functional requirements document.

Estimated effort: 2d

Per the California Service Name design, an EdgeX service should ensure that its dependent services are available before responding to REST endpoints, including ping.

If the registry/consul is specified on startup, then the SDK should query Consul to determine availability of Core Data and Metadata. If the registry is not specified, then the SDK must query the service directly via REST to ensure availability.

Once dependent services have been been verified as available:

• if the registry is being used, then the SDK should register the device service with consul

• the SDK should start responding to REST requests

Priority: High
Estimated effort: 2d

Currently most of the device-sdk-go dependencies pull from head/tip vs. specifying a specific release for each in glide.yaml. The same has been planned for edgex-go.

Note - both the mgo and yaml packages specify versions via gopkg.in.

Origin time in addressable and device service are in microseconds instead of milliseconds

Implement transformations of device readings.

The values that can be read from a device are defined in the PropertyValues of device resources as defined by a DeviceProfile. A PropertyValue may specify optional attributes which are used to transform the readings from devices/sensors. These attributes are defined below and are applied to the native data types (float* or int/uint*) in the following order:

base – a base value which is raised to the power of the reading value
scale – a multiplicative factor applied to the reading
offset – an additive factor applied to the reading

For more details see the Device Services functional requirements document.

Estimated effort: 2d

Implement discovery logic: if discovery endpoint hit, call protocoldriver.scan(), process ScanList, and add new devices that match any provisionwatchers.

For more details see the Device Services functional requirements document.

Estimated effort: 5-7d

https://docs.google.com/document/d/1Zd1-BtnRMmYchh0HbSFGGVjwtfVHH3_MxEORhKOkAbY/edit

The current implementation of clients/init.go:

	for i := 0; i < 50; i++ {
		if common.UseRegistry {
			if checkServiceAvailableByConsul(common.CurrentConfig.Clients[serviceId].Name) == true {
				return nil
			}
		} else {
			if checkServiceAvailableByPing(serviceId) == nil {
				return nil
			}
		}
		time.Sleep(2 * time.Second)
		common.LogCli.Debug(fmt.Sprintf("Checked %d times for %s availibility", i+1, serviceId))
	}

The 50 and 2 should be configurable.

When DeviceProfiles are auto-imported on device service startup, there should be some constraints checking implemented. These constraints should be represented by configuration settings and should include:

• max number of device resources
• max number of resource commands
• max number of resource operations (applies to get & put)

The code in devices.go should be reviewed for thread safety. The current implementation was based on the original EdgeX Java Device Services SDK. This change is dependent on a new internal device struct to be created, so that devices can be locked when operations involving the device are in-progress.

Estimated effort: 2d

Review composition of internal complex data types as used in the device and profile cache. Should arrays/maps use structs or pointers to structs? Currently the device cache use pointers to Device structs in it's map, whereas the profile cache code doesn't use pointers in its commands and objects maps. In theory, none of the code that accesses devices in the local cache should need to modify the devices, so there's no reason for the pointer usage other than the potential optimization.

Estimated effort: 2-3d

Create default schedules & schedule events based on configuration settings on startup.

Estimated effort: 2-3d

Add public Service methods to allow a device service to dynamically add/update or remove DeviceProfile objects. The operations result in changes to Core Metadata in addition to the device service's local DeviceProfile objects.

Estimated effort: 2-3d

A Device interface should be created to allow the mocking of the device cache to allow unit testing of the command REST endpoint (and possibly other code).

Estimated effort: 2-3d

Implement unit tests that cover the device cache logic, which is also responsible to updating Core Metadata with any changes to the local devices.

Estimated effort: 3d

Add support for generic init and disconnect functions when devices are added or removed from the device service. The commands must be valid commands in the corresponding device profile. The functions are defined by the configuration settings InitCmd and RemoveCmd settings, and arguments for both are specified in the InitCmdArgs and RemoveCmdArgs settings.

Note - it might be a good idea to update the setting names to include "Dev" or "Device". Also maybe InitCmd should be AddDeviceCmd to make things more sensical?

Estimated effort: 3-4d

Implement size constraints (w/unit tests) for devices and profiles based on configuration settings. Constraints should include:

• max devices (total & per profile)

• max labels per device

• max device profiles

• max commands per device profile

• max resource operations per device profile command

• max number of device resources

• max number of resource commands

• max number of resource operations (applies to get & put)

Estimated effort: 2-3d

the current design and implementation of internal scheduler mechanism can only run the pre-defined schedule event. To make it can be managed at run time, we could allow callback of schedule event CRUD or periodically re-load all schedule event by service name from Core Metadata

Events to core-data can be sent in another goroutine, like it's done with the notifications in core-data or core-metadata, in other to improve the performance.

Currently the command endpoint only supports specifying a deviceId. Another form of the endpoint replaces a specific device id with the word "all", which instructs the device service to send the given command to all of it's devices.

Note - this form of the command was intended for ease of scheduling periodic readings of all of a device service's devices. If the device service supports multiple device profiles, then the same command needs to be available in all of them.

For more details see the Device Services functional requirements document.

Estimated effort: 3-4d

If the registry/consul is enabled on service startup, try to read initial configuration settings from consul.

In order for this feature to work, the settings for the device service need to be injected into consul via core-config-seed, or some other out-of-band mechanism.

Priority: High
Estimated effort: 5d

Because client api use internal Consul variable. But when startup we have another Consul variable in sdk.

Error log:

Calling service.Start.
Init: useRegistry: true profile:  confDir: ./examples/simple/cmd/res/
Register in Consul...INFO: 2018/09/11 19:37:59 Check edgex-core-metadata service's status by Consul...
INFO: 2018/09/11 19:37:59 Check edgex-core-data service's status by Consul...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12979c3]

goroutine 85 [running]:
github.com/edgexfoundry/device-sdk-go/vendor/github.com/hashicorp/consul/api.(*Client).newRequest(0x0, 0x13e6c99, 0x3, 0x13ead4b, 0x12, 0x0)
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/hashicorp/consul/api/api.go:644 +0x63
github.com/edgexfoundry/device-sdk-go/vendor/github.com/hashicorp/consul/api.(*Agent).Services(0xc000189d90, 0x0, 0x0, 0x0)
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/hashicorp/consul/api/agent.go:245 +0x6d
github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/internal/pkg/consul.GetServiceEndpoint(0xc0000cc540, 0x13, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/internal/pkg/consul/consul.go:85 +0x7b
github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/pkg/clients/types.Endpoint.Monitor(0xc0000cc540, 0x13, 0x13eb304, 0x13, 0x1, 0xc0001e80f0, 0x29, 0xc0001c82a0)
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/pkg/clients/types/endpoint.go:15 +0x1c3
created by github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/pkg/clients/metadata.(*AddressableRestClient).init
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/pkg/clients/metadata/addressable.go:54 +0xe6
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12979c3]

goroutine 91 [running]:
github.com/edgexfoundry/device-sdk-go/vendor/github.com/hashicorp/consul/api.(*Client).newRequest(0x0, 0x13e6c99, 0x3, 0x13ead4b, 0x12, 0x0)
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/hashicorp/consul/api/api.go:644 +0x63
github.com/edgexfoundry/device-sdk-go/vendor/github.com/hashicorp/consul/api.(*Agent).Services(0xc000185d90, 0x0, 0x0, 0x0)
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/hashicorp/consul/api/agent.go:245 +0x6d
github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/internal/pkg/consul.GetServiceEndpoint(0xc0000cc540, 0x13, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/internal/pkg/consul/consul.go:85 +0x7b
github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/pkg/clients/types.Endpoint.Monitor(0xc0000cc540, 0x13, 0x13ec1a6, 0x15, 0x1, 0xc0001e8180, 0x2b, 0xc0001c83c0)
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/pkg/clients/types/endpoint.go:15 +0x1c3
created by github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/pkg/clients/metadata.(*DeviceProfileRestClient).init
	/Users/bruce/go/src/github.com/edgexfoundry/device-sdk-go/vendor/github.com/edgexfoundry/edgex-go/pkg/clients/metadata/device_profile.go:53 +0xe6

Process finished with exit code 2

Implement unit tests for the command endpoint (GET & PUT).

This relies on a few other tasks being completed first, namely ensuring that there are interfaces for needed clients and the caching code so that things can be mocked.

Estimated effort: 5-7d

when I opened this issue, we didn't have complete go doc for each public API. Now, I would like to examine all of them before next release

It was decided at the last F2F, that the one Core Metadata callback function to be supported for Dehli is handling changes to a device's adminState (LOCKED or UNLOCKED).

Estimated effort: 3d