intelsdi-x / snap-plugin-lib-go Goto Github PK

View Code? Open in Web Editor NEW

10.0 25.0 40.0 249 KB

The library for writing plugins in Go for the Snap telemetry framework

Home Page: http://snap-telemetry.io/

License: Apache License 2.0

Go 94.20% Makefile 0.62% Shell 5.18%

snap-plugin-lib-go's Introduction

DISCONTINUATION OF PROJECT.

This project will no longer be maintained by Intel.

This project has been identified as having known security escapes.

Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.

Intel no longer accepts patches to this project.

DISCONTINUATION OF PROJECT

This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.

Snap Plugin Library for Go

This is a library for writing plugins in Go for the Snap telemetry framework.

Notice: Due to standarization which has been made in Logrus project, its import path has been changed from Sirupsen to sirupsen. If you see case-sensitive import collision reporting by glide after updating your plugin to the latest snap-plugin-lib-go, please do the following steps:

change every import of Logrus to the lower case: github./com/sirupsen/logrus,
delete folder vendor/github.com/Sirupsen,
rename references in glide.yaml and glide.lock
clear glide cache (glide -c),
update dependencies (glide update)

More information you can find in Logrus GitHub repo.

Writing a Plugin
- Before Writing a Snap Plugin
Brief Overview of Snap Architecture
Example Plugins
Plugin Flags

Writing a Plugin

Snap has four different plugin types and for instructions on how to write a plugin check out the collector, processor, publisher, and streaming collector plugin docs.

Before writing a Snap plugin:

See if one already exists in the Plugin Catalog
See if someone mentioned it in the plugin wishlist

If you do decide to write a plugin check out the plugin authoring docs and let us know you are working on one!

Brief Overview of Snap Architecture:

Snap is an open and modular telemetry framework designed to simplify the collection, processing and publishing of data through a single HTTP based API. Plugins provide the functionality of collection, processing and publishing and can be loaded/unloaded, upgraded and swapped without requiring a restart of the Snap daemon.

A Snap plugin is a program that responds to a set of well defined gRPC services with parameters and return types specified as protocol buffer messages (see plugin.proto). The Snap daemon handshakes with the plugin over stdout and then communicates over gRPC.

Snap Plugin Go Library Examples:

You will find example plugins that cover the basics for writing collector, processor, publisher, and streaming collector plugins in the examples folder.

Plugin Flags:

For specific details and to see all the options when running, run the plugin with the -help flag. The flag options are:

GLOBAL OPTIONS:
   --config value            config to use in JSON format
   --port value              port GRPC will listen on
   --pprof                   enable pprof
   --tls                     enable TLS
   --cert-path value         necessary to provide when TLS enabled
   --key-path value          necessary to provide when TLS enabled
   --root-cert-paths value   root paths separated by ':'
   --stand-alone             enable stand alone plugin
   --stand-alone-port value  specify http port when stand-alone is set (default: 8181)
   --log-level value         log level - 0:panic 1:fatal 2:error 3:warn 4:info 5:debug (default: 2)
   --required-config         Plugin requires config passed in
   --help, -h                show help
   --version, -v             print the version

Additionally, plugin authors can add custom flags as described here

Custom Config:

Users can provide their own config with the -config flag. If a config is required for the plugin to load, diagnostics will show a warning describing which keys are required and not provided.

When using the -config flag, it expects a parameter in the form of a JSON. This is of the form '{}'. An example config is: -config '{\"key\":\"kelly\", \"spirit-animal\":\"coatimundi\"}'.

Plugin Diagnostics:

Snap plugins using plugin-lib-go can be run independent of Snap to show their current running diagnostics.

Running plugin diagnostics is easy! Simply build the plugin, then run the executable $./build/${GOOS}/${GOARCH}/<plugin binary>.

Diagnostic information includes:

Runtime details
- Plugin version
- RPC type and version
- OS, architecture
- Golang version
Warning if dependencies not met
Config policy (for collector plugins only)
- Warning if config items required and not provided
Collectable metrics (for collector plugins only)
How long it took to run each of these diagnostics

Currently Snap Plugin Diagnostics is only available for collector plugins.

Custom Flags

Plugins authors using snap-plugin-lib-go have the ability to create customized runtime flags. These flags are written using urfave/cli. An example of a custom flag in a plugin can be found in the snap-plugin-collector-rand example.

Flags can be added with the following syntax:

plugin.AddFlag(
		cli.BoolFlag{
			Name:        "required-config",
			Hidden:      false,
			Usage:       "Plugin requires config passed in",
			Destination: &req,
		},
	)

More information about types of cli flags and options for each flag can be found in the documentation for urfave/cli

As always, if you have any questions, please reach out to the Snap team via Slack or by opening an issue in github.

snap-plugin-lib-go's People

Contributors

Stargazers

Watchers

snap-plugin-lib-go's Issues

Facing problems while running the glide up command

Hi!
I am looking into the plugins creation:
I am getting the Errors.
Could anyone please help me with this?

Specifications:
glide version v0.12.3
go version go1.6.2 linux/amd64
System: Ubuntu 16.04

Thanks

Plugin won't load on Windows10

When I load the example "rand" plugin it fails to load with the following error. It should be noted that Go plugins written with the old plugin lib do not exhibit this behavior.

Possibly related: golang/go#5615

Return int ZERO is not enough

Standard alone function StartCollector/StartPublisher/StartProcessor returns only int zero otherwise it panics. It would be nice if it can:

It returns the started plugin info if it succeeded.
It returns an error instead of panic so that the calling program may be able to recover it.
it's very hard to write a mock test for such standalone functions.

Replace plugin.ConfigPolicy struct with interface

Currently plugin.ConfigPolicy is an exported type. As a constructor is required to initialize it correctly I believe that:

plugin.ConfigPolicy should not be exported
an interface should be defined and depended upon, eg:

type ConfigPolicy interface {
    AddNewIntRule([]string, string, bool, ...integerRuleOpt) error
    AddNewBoolRule([]string, string, bool, ...boolRuleOpt) error
    AddNewFloatRule([]string, string, bool, ...floatRuleOpt) error
    AddNewStringRule([]string, string, bool, ...stringRuleOpt) error
}

Such an approach would force users to call NewConfigPolicy instead of using a literal to instantiate plugin.ConfigPolicy. Using literal causes panics. Keeping the interface name same as the struct name is at the moment should simplify refactoring.

Missing library files to run plugin examples

When trying to make the example collector binary available to load by snapd:

sjhan-mac02:collector sjhan$ go run main.go

Output

../../v1/plugin/rpc/plugin.pb.go:54:8: cannot find package "github.com/golang/protobuf/proto" in any of:
    /Users/sjhan/.gvm/gos/go1.6/src/github.com/golang/protobuf/proto (from $GOROOT)
    /Users/sjhan/Documents/testing-snap-v1/src/github.com/golang/protobuf/proto (from $GOPATH)
../../v1/plugin/rpc/plugin.pb.go:59:2: cannot find package "golang.org/x/net/context" in any of:
    /Users/sjhan/.gvm/gos/go1.6/src/golang.org/x/net/context (from $GOROOT)
    /Users/sjhan/Documents/testing-snap-v1/src/golang.org/x/net/context (from $GOPATH)
../../v1/plugin/rpc/plugin.pb.go:60:2: cannot find package "google.golang.org/grpc" in any of:
    /Users/sjhan/.gvm/gos/go1.6/src/google.golang.org/grpc (from $GOROOT)
    /Users/sjhan/Documents/testing-snap-v1/src/google.golang.org/grpc (from $GOPATH)

When trying to do "glide update", got:

sjhan-mac02:collector sjhan$ glide up
[WARN]  The name listed in the config file (github.com/intelsdi-x/snap-plugin-lib-go) does not match the current location (examples/collector)
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for google.golang.org/grpc.
[INFO]  --> Fetching updates for github.com/gopherjs/gopherjs.
[INFO]  --> Fetching updates for github.com/smartystreets/goconvey.
[INFO]  --> Fetching updates for github.com/golang/protobuf.
[INFO]  --> Fetching updates for github.com/jtolds/gls.
[INFO]  --> Fetching updates for github.com/smartystreets/assertions.
[INFO]  --> Fetching updates for golang.org/x/net.
[INFO]  --> Setting version for github.com/jtolds/gls to 8ddce2a84170772b95dd5d576c48d517b22cac63.
[INFO]  --> Setting version for github.com/gopherjs/gopherjs to 4b53e1bddba0e2f734514aeb6c02db652f4c6fe8.
[INFO]  --> Setting version for github.com/smartystreets/goconvey to 995f5b2e021c69b8b028ba6d0b05c1dd500783db.
[INFO]  --> Setting version for github.com/golang/protobuf to 888eb0692c857ec880338addf316bd662d5e630e.
[INFO]  --> Setting version for github.com/smartystreets/assertions to 443d812296a84445c202c085f19e18fc238f8250.
[INFO]  --> Setting version for golang.org/x/net to 154d9f9ea81208afed560f4cf27b4860c8ed1904.
[INFO]  --> Setting version for google.golang.org/grpc to 0032a855ba5c8a3c8e0d71c2deef354b70af1584.
[INFO]  Resolving imports
[ERROR] Error scanning github.com/intelsdi-x/snap-plugin-lib-go/v1/plugin: open /Users/sjhan/Documents/testing-snap-v1/src/github.com/intelsdi-x/snap-plugin-lib-go/examples/collector/v1/plugin: no such file or directory
[ERROR] This error means the referenced package was not found.
[ERROR] Missing file or directory errors usually occur when multiple packages
[ERROR] share a common dependency and the first reference encountered by the scanner
[ERROR] sets the version to one that does not contain a subpackage needed required
[ERROR] by another package that uses the shared dependency. Try setting a
[ERROR] version in your glide.yaml that works for all packages that share this
[ERROR] dependency.
[ERROR] Failed to retrieve a list of dependencies: Error resolving imports
sjhan-mac02:collector sjhan$

Use Snap core.Separator for Namespace.Key()

Use core.Separator as delimiter in Namespace.Key().
https://github.com/intelsdi-x/snap-plugin-lib-go/blob/master/v1/plugin/metric.go#L206-L210

Maybe we can have a function func ParseKey() Namespace that reverse the process.

EDIT: or remove it. @IRCody

Graceful shutdown of gRPC

We should shutdown gRPC server gracefully rather than forcefully exiting.

removal of dot separator

In line with #1240 from snap, the removal of the dot separator would also be good to do in the lib so that we don't have unnecessary restrictions on namespaces.

I think this happens here and is probably not the hugest change, though it depends on the work being done as part of the linked snap issue first.

Add dependency management

Either godeps/glide/etc. Needed to keep the grpc/protobuf libraries in sync.

setting Timestamp and LastAdvertisedTime if they haven’t been set by the plugin author

Currently Protobuf/GRPC defaults un-set time to a default value. For example:

LastAdvertisedTime:<sec:-62135596800 nsec:452794115 >

It seems that defaulting to time.Now() is desired.

GRPC ignores metric config

Description:
Plugins using GRPC aren't passing metric config even if it's specified.
In toProtoMetric function metric's config isn't passed along.
This behaviour differs from plugins using deprecated plugin lib, in which there is a possibility to pass metric's config.

Steps to reproduce:

Load mock1 plugin from snap/plugin/collector
Load passthru and passthru-grpc plugins from snap/plugin/processor
Load test-file-publisher plugin from snap-plugin-lib-go/examples
Create two tasks from following task files:

# task-grpc.yml
---
max-failures: 10
schedule:
  interval: 10s
  type: simple
version: 1
workflow:
  collect:
    metrics:
      "/intel/mock/foo": {}
    config:
      "/intel/mock/foo":
        password: test
        testkey: testval
    process:
    - plugin_name: passthru-grpc
      publish:
      - plugin_name: test-file-publisher
        config:
          file: "/tmp/snap_mock_passthru-grpc.log"

# task-deprecated.yml
---
max-failures: 10
schedule:
  interval: 10s
  type: simple
version: 1
workflow:
  collect:
    metrics:
      "/intel/mock/foo": {}
    config:
      "/intel/mock/foo":
        password: test
        testkey: testval
    process:
    - plugin_name: passthru
      publish:
      - plugin_name: test-file-publisher
        config:
          file: "/tmp/snap_mock_passthru.log"

Compare output in /tmp/snap_mock_passthru-grpc.log and /tmp/snap_mock_passthru.log

Expected outcome:
Both tasks producing the same output.

Actual outcome:
Metric config present only in /tmp/snap_mock_passthru.log - passthru-grpc stripped config.

bool data type is not supported for "toProtoMetric" function

bool data is valid as either metric data or ConfigMap (BoolMap). By supplying bool data type to either metric or config, func toProtoMetric caused panic.

Integer and Float configs were empty

Please see these two test cases. Data should be equal but it's not. As it picked up the random data.

https://github.com/intelsdi-x/snap-plugin-lib-go/pull/17/files#diff-a2de8e2b3330febc44bbd52d92ff4eeaR51

https://github.com/intelsdi-x/snap-plugin-lib-go/pull/17/files#diff-a2de8e2b3330febc44bbd52d92ff4eeaR67

Provide Init() once func behavior needed by plugins requiring initialization

Some plugins require initialization based on params passed in configuration, so they usually have func init() which should be executed only once.

It would be nice to have a support from plugin-lib side to provide Once func behavior ready to use.

The idea is to provide Init(cfg plugin.ConfigType) for plugins to make easier implementation of some kind of initialization for plugins which require that.

Init() should be optional;
a) to keep compatibility with existing plugins - they should still work even they do not call Init()
b) some plugins do not require initialization, so there is no sense to call Init for them.

As a result, plugin developer might use ready-to-use Init() available in snap-plugin-lib-go instead of implement own function and provide mechanism to assure that its execution happens only once.

snap-plugin-lib-go seems to be the best candidate to provide such enhancement as the first one of existing snap-plugin-libs because most of available plugins are written in Go.

Adding this enhancement to the rest of libs (snap-plugin-lip-cpp and snap-plugin-lib-py) can be done later as the next steps depending on request or interest of having that.

Plugin diagnostic not passing default config values to run CollectMetrics

What happens:

I used diagnostic mode for snap-plugin-collector-processes plugin and noticed that CollectMetrics returns error config item not found because of lack of providing proc_path config items. Notice that config policy says that it not required and default value is set. However, the default value is not passed in metrics config. That is the reason why CollectMetrics() fails in diagnostic mode.

$ snap-plugin-collector-processes
Runtime Details:
    PluginName: processes, Version: 8
    RPC Type: gRPC, RPC Version: 1
    Operating system: linux
    Architecture: amd64
    Go version: go1.7.3
printRuntimeDetails took 28.646µs

Config Policy:
NAMESPACE                KEY             TYPE            REQUIRED        DEFAULT         MINIMUM         MAXIMUM
intel.procfs.processes   proc_path       string          false           /proc
printConfigPolicy took 127.332µs

Metric catalog will be updated to include:
    Namespace: /intel/procfs/processes/state/wakekill
    Namespace: /intel/procfs/processes/process/*/*/ps_disk_octets_rchar
    Namespace: /intel/procfs/processes/process/*/all/ps_disk_octets_rchar
    Namespace: /intel/procfs/processes/process/*/*/ps_disk_octets_wchar
    Namespace: /intel/procfs/processes/process/*/all/ps_disk_octets_wchar
    Namespace: /intel/procfs/processes/process/*/*/ps_cmdline
    Namespace: /intel/procfs/processes/state/running
    Namespace: /intel/procfs/processes/state/waiting
    Namespace: /intel/procfs/processes/state/tracing
    Namespace: /intel/procfs/processes/state/dead
    Namespace: /intel/procfs/processes/state/waking
    Namespace: /intel/procfs/processes/process/*/*/ps_code
    Namespace: /intel/procfs/processes/process/*/all/ps_code
    Namespace: /intel/procfs/processes/process/*/*/ps_stacksize
    Namespace: /intel/procfs/processes/process/*/all/ps_stacksize
    Namespace: /intel/procfs/processes/process/*/*/ps_cputime_user
    Namespace: /intel/procfs/processes/process/*/all/ps_cputime_user
    Namespace: /intel/procfs/processes/process/*/*/ps_disk_ops_syscw
    Namespace: /intel/procfs/processes/process/*/all/ps_disk_ops_syscw
    Namespace: /intel/procfs/processes/state/zombie
    Namespace: /intel/procfs/processes/state/stopped
    Namespace: /intel/procfs/processes/process/*/*/ps_rss
    Namespace: /intel/procfs/processes/process/*/all/ps_rss
    Namespace: /intel/procfs/processes/process/*/*/ps_data
    Namespace: /intel/procfs/processes/process/*/all/ps_data
    Namespace: /intel/procfs/processes/process/*/*/ps_cputime_system
    Namespace: /intel/procfs/processes/process/*/all/ps_cputime_system
    Namespace: /intel/procfs/processes/process/*/*/ps_disk_ops_syscr
    Namespace: /intel/procfs/processes/process/*/all/ps_disk_ops_syscr
    Namespace: /intel/procfs/processes/state/sleeping
    Namespace: /intel/procfs/processes/process/*/*/ps_vm
    Namespace: /intel/procfs/processes/process/*/all/ps_vm
    Namespace: /intel/procfs/processes/process/*/*/ps_pagefaults_min
    Namespace: /intel/procfs/processes/process/*/all/ps_pagefaults_min
    Namespace: /intel/procfs/processes/process/*/*/ps_pagefaults_maj
    Namespace: /intel/procfs/processes/process/*/all/ps_pagefaults_maj
    Namespace: /intel/procfs/processes/process/*/ps_count
    Namespace: /intel/procfs/processes/state/parked
printMetricTypes took 730.096µs

printCollectMetrics took 4.797µs

showDiagnostics took 924.901µs

ERRO[0000] ! Error in the call to CollectMetrics. Please ensure your config contains any required fields mentioned in the error below.
 config item not found  _block=StartCollector

What is expected:

Plugin diagnostic should pass defaults config params to run CollectMetrics, or merge it with provided one. So, making a diangostic for snap-plugin-collector-processes is expected to be successful even no configs provided

Plugin changes required

This ticket is not to report a bug or an issue. Here are a few changes that plugins need:

processor and publisher interface changed
method receiver changed from the pointer receiver to the value receiver
return changed from a pointer to a value
metric version is int64

The publish response object (ErrReply) should be used to communicate errors

Today errors that occur are not sent back to the framework in the response object.

err := p.plugin.Publish(metrics, cfg)
if err != nil {
	return &rpc.ErrReply{}, err
}

Consistent logging

This library needs a consistent logging mechanism. Currently I believe it's mostly just printing plainly to stdout via fmt.Println(). This issue is to track the formalization of what the expectations are regarding logging.

Make namespace type public

As a plugin developer, I want to be able to manage namespaces on the plugin side.

plugin.namespace should be plugin.Namespace.

https://github.com/intelsdi-x/snap-plugin-lib-go/blob/master/v1/plugin/metric.go#L32

A graceful error handling for unsupported metric data types would be great

When a metric has an unsupported data type, the program was panic. A graceful error handling would be nice especially when there are many good data.

Example input metric

                  Metric{
                Namespace:          NewNamespace("a", "b", "1"),
                Version:                 8,
                Timestamp:          time.Now(),
                Data:                     make(chan bool),
                Tags:                     map[string]string{"label": "abc"},
                Description:        "desc",
                lastAdvertisedTime: time.Now(),
            }

Should we use grpc 1.0.1-GA?

Due to the version incompatible, test got the following error:

github.com/intelsdi-x/snap-plugin-lib-go/v1/plugin/rpc

v1/plugin/rpc/plugin.pb.go:682: undefined: grpc.SupportPackageIsVersion2

Add doc for standalone plugin

Adding a doc to explain how a standalone plugin is supported in Snap would be great.

Default config values are not applied

When I implement this kind of config policy in my plugin:

// GetConfigPolicy returns the config policy
func (p *Plugin) GetConfigPolicy() (plugin.ConfigPolicy, error) {
    policy := plugin.NewConfigPolicy()

    policy.AddNewIntRule([]string{""}, "myint", false, plugin.SetDefaultInt(1), plugin.SetMinInt(1))
    policy.AddNewStringRule([]string{""}, "mystring", false, plugin.SetDefaultString("plop"))
    return *policy, nil
}

Tested on a processor:

DEBU[2016-10-26T16:50:13-07:00] starting processor job                        _module=scheduler-job block=run job-type=processor plugin-config=map[] plugin-name=statistics plugin-version=-1
DEBU[2016-10-26T16:50:13-07:00] plugin selected                               _module=control-routing block=select hitcount=0 index=processor:statistics:v2:id7 pool size=1 strategy=least-recently-used
ERRO[2016-10-26T16:50:13-07:00] error with processor job                      _module=scheduler-job block=run error=rpc error: code = 2 desc = config item not found: "statistics" job-type=processor plugin-config=map[] plugin-name=statistics plugin-version=-1
WARN[2016-10-26T16:50:13-07:00] Process job failed                            _block=submit-process-job _module=scheduler-workflow parent-node-type=collector process-name=statistics process-version=-1 task-id=670c1916-1205-4422-b0f1-5a2c8163fef5 task-name=Task-670c1916-1205-4422-b0f1-5a2c8163fef5
DEBU[2016-10-26T16:50:13-07:00] Batch submission complete                     _block=work-jobs _module=scheduler-workflow count-process-nodes=1 count-publish-nodes=0 parent-node-type=collector task-id=670c1916-1205-4422-b0f1-5a2c8163fef5 task-name=Task-670c1916-1205-4422-b0f1-5a2c8163fef5
WARN[2016-10-26T16:50:13-07:00] Task failed                                   _block=spin _module=scheduler-task consecutive failure limit=2 consecutive failures=1 error=rpc error: code = 2 desc = config item not found: "statistics" task-id=670c1916-1205-4422-b0f1-5a2c8163fef5 task-name=Task-670c1916-1205-4422-b0f1-5a2c8163fef5

Allow grpc Server options to be passed by plugin author

See #1238. Need a way to pass grpc.ServerOptions to allow plugin authors to have some control over these defaults. Not sure if we'd want to expose all of them or a subset.

Difficult to correlate issue getting started with plugin library

I started by cloning the example/collector/ repo as a baseline for my new collector. Once I built an empty plugin, I wanted to ensure it compiled and ran: go build github.go main.go where both .go files are in the main package for now. It has a messy output:

# google.golang.org/grpc/transport
../../../google.golang.org/grpc/transport/handler_server.go:209: undefined: http2.TrailerPrefix
../../../google.golang.org/grpc/transport/http2_client.go:877: undefined: http2.MetaHeadersFrame
../../../google.golang.org/grpc/transport/http2_client.go:999: undefined: http2.MetaHeadersFrame
../../../google.golang.org/grpc/transport/http2_server.go:154: undefined: http2.MetaHeadersFrame
../../../google.golang.org/grpc/transport/http2_server.go:314: undefined: http2.MetaHeadersFrame
../../../google.golang.org/grpc/transport/http_util.go:382: f.fr.ReadMetaHeaders undefined (type *http2.Framer has no field or method ReadMetaHeaders)
../../../google.golang.org/grpc/transport/http_util.go:512: f.fr.ErrorDetail undefined (type *http2.Framer has no field or method ErrorDetail)

I couldn't find a reason for the failure until I tried to see if I somehow forgot to download the plugin library itself and got the same error:

go get github.com/intelsdi-x/snap-plugin-lib-go/v1/plugin
# google.golang.org/grpc/transport
../../../google.golang.org/grpc/transport/handler_server.go:209: undefined: http2.TrailerPrefix
../../../google.golang.org/grpc/transport/http2_client.go:877: undefined: http2.MetaHeadersFrame
../../../google.golang.org/grpc/transport/http2_client.go:999: undefined: http2.MetaHeadersFrame
../../../google.golang.org/grpc/transport/http2_server.go:154: undefined: http2.MetaHeadersFrame
../../../google.golang.org/grpc/transport/http2_server.go:314: undefined: http2.MetaHeadersFrame
../../../google.golang.org/grpc/transport/http_util.go:382: f.fr.ReadMetaHeaders undefined (type *http2.Framer has no field or method ReadMetaHeaders)
../../../google.golang.org/grpc/transport/http_util.go:512: f.fr.ErrorDetail undefined (type *http2.Framer has no field or method ErrorDetail)

Then I remembered we have a very specific version of gRPC needed for the library, so I went to the plugin and installed:

cd $GOPATH/src/github.com/intelsdi-x/snap-plugin-lib-go/v1/plugin
glide install 
[INFO]	Lock file (glide.lock) does not exist. Performing update.
... lots of stuff

Now my error issues make more sense:

go build github.go main.go
# command-line-arguments
./github.go:15: undefined: plugin.CollectorPlugin
./main.go:32: cannot use GithubCollector literal (type GithubCollector) as type plugin.Collector in argument to plugin.StartCollector:
	GithubCollector does not implement plugin.Collector (CollectMetrics method has pointer receiver)

I write out this issue because I imagine others will run into the same missed step. @IRCody did point out that this step is documented in the example/README.md, but I bypassed it for the example/collector/README.md. It'd be good to reconcile this gap to some degree. For example, could we be prescriptive about how they get a skeleton repository setup for a plugin?

Error loading exemplary streaming plugin

Error loading exemplary streaming plugin.
To reproduce:

build exemplary streaming plugin
load plugin

Current behavior:

Error loading plugin:
rpc error: code = 12 desc = unknown service rpc.Collector

Expected behavior: plugin loads without errors

Rename GetMetricTypes

Rename GetMetricTypes was named that way because we use plugin.MetricType in the old library. Naming this method GetMetrics or GetMetricList would make more sense.

AFAIK, 2 collector plugins would be affected by this change: "mock1-grpc" (Snap repo) and "rand" (this repo).

Add a TagDynamicElements method for Namespace

A lot of publisher (opentsdb, influxdb, warp10) remove dynamic elements from the namespace and put it in tags instead.

Is it something we want in publishers?

No

Then we should create a processor for doing that.

Yes

Then we should have a method in this plugin lib to make plugin developper experience easier.

func (ns Namespace) TagDynamicElements() (Namespace, map[string]string)

Remove metric.String()

This method Metric.Namespace.String() should be updated now Snap PR 1252 is merged.

Implement fmt.Stringer interface in plugin.Namespace

Method Strings() []string is implemented on plugin.Namespace at the moment. It would be great to have method String() string implemented too - it should return string representation of the namespace.

Similar function exists on core.Namespace in Snap.

Unable to test plugins with TLS channel on Windows

Description
When starting an example plugin (rand) on Windows with TLS enabled, error prevents plugin from working. Plugin complains that system root certificates are unavailable on Windows.

Steps to reproduce

build example plugin for windows

cd examples/snap-plugin-collector-rand
GOOS=windows GOARCH=386 go build

launch example plugin with TLS enabled

snap-plugin-collector-rand.exe "{\"CertPath\":\"libtest-srv.crt\",\"KeyPath\":\"libtest-srv.key\"}

Observed outcome
Plugin throws an error:
panic: unable to read root CAs: crypto/x509: system root pool is not available on Windows

Expected outcome
Plugin starts without issues.

Remarks
I understand that golang doesn't have complete support for root certificates on Windows, according to issue golang/go#16736.
It would be better to add support for explicitly specified root certificates for all plugins based on this library, e.g.: RootCertPaths: \"libtest-CA.crt;clients-use-this.crt\".

[Important] Compatibility issues with grpc

When I started writing my first plugin I ran into lots of issues to just get started, using this library.

It's not compatible with the latest version of grpc.

This is the error in question when running go get to fetch this library (snap-plugin-lib-go).

../../intelsdi-x/snap-plugin-lib-go/v1/plugin/rpc/plugin.pb.go:823: undefined: grpc.SupportPackageIsVersion3

See https://travis-ci.org/inteleon/snap-plugin-collector-newrelic/builds/223473231 for more information.

I was forced to clone the grpc repository manually and then check out a version that works, more specifically version 1.0.3 which was released the 18th of October 2016.

Empty Unit field on CollectMetrics function

The []plugin.Metric received on CollectMetrics function doesn't have the Unit field as was defined on GetMetricTypes.

I tried the following change in the rand collector. This only defines a string unit for all the metrics and then will print the content in CollectMetrics.

diff --git a/examples/collector/rand/rand.go b/examples/collector/rand/rand.go
index ceaf876..ddd0628 100644
--- a/examples/collector/rand/rand.go
+++ b/examples/collector/rand/rand.go
@@ -70,6 +70,7 @@ The output is the collected metrics as plugin.Metric and an error.
 func (RandCollector) CollectMetrics(mts []plugin.Metric) ([]plugin.Metric, error) {
        metrics := []plugin.Metric{}
        for idx, mt := range mts {
+               fmt.Printf("The Unit for metric %s is %s\n", mt.Namespace, mt.Unit)
                mts[idx].Timestamp = time.Now()
                if val, err := mt.Config.GetBool("testbool"); err == nil && val {
                        continue
@@ -119,6 +120,7 @@ func (RandCollector) GetMetricTypes(cfg plugin.Config) ([]plugin.Metric, error)
                metric := plugin.Metric{
                        Namespace: plugin.NewNamespace("random", val),
                        Version:   1,
+                       Unit:      "string",
                }
                metrics = append(metrics, metric)
        }

The output that I get is the following:

DEBU[2017-02-20T21:03:39-06:00] The Unit for metric [{random  } {integer  }] is   _module=plugin-exec io=stdout plugin=test-rand-collector
DEBU[2017-02-20T21:03:39-06:00] The Unit for metric [{random  } {float  }] is   _module=plugin-exec io=stdout plugin=test-rand-collector
DEBU[2017-02-20T21:03:39-06:00] The Unit for metric [{random  } {string  }] is   _module=plugin-exec io=stdout plugin=test-rand-collector

Should string unit be printed at this point?

Thanks.

plugin.ConfigPolicy should have exported fields (or interface) to access the policy for testing

Currently unit testing a plugin's GetConfigPolicy is nearly impossible, other than checking for lack of a errors. The problem is that plugin.ConfigPolicy has an interface to ADD configuration policies, but no way to READ back those APIs. What I would like to do in a unit test is:

    Convey("GetConfigPolicy", func() {
        policy, err := publisher.GetConfigPolicy()

        Convey("should not return an error", func() {
            So(err, ShouldBeNil)
        })

        // Unfortunately GetConfigPolicy is not further testable. The problem is
        // that the ConfigPolicy has no public APIs to read the policy, only to
        // set the policy.  So, while we would like to test that the policy
        // returned by GetConfigPolicy is valid, we cannot.
        Convey("Configuration item foo",  func () {
                // We cannot do this because stringRules (and intRules, etc) are private
                actualConfigItem, err := policy.stringRules[""].Rules["foo"]
                Convey("should exist", func () {
                    SkipSo(err, ShouldBeNil)
                })

                Convey("should be required", func () {
                    SkipSo(actualConfigItem.Required, ShouldBeTrue)
                })
            })
    })

Return public errors instead of inline Errorf

E.g. when I get the config of a String https://github.com/intelsdi-x/snap-plugin-lib-go/blob/master/v1/plugin/config.go#L46

I want to be able to test what error I encounter:

myString, err := plugin.GetString("mykey")
if err == plugin.NotAString {
    // ...
}

Error while building the example plugins.

Hi!
while building the plug-ins I am getting this error. I guess

this is because of packages.
So do we need to install the packages separately.
If that is the case, then could you please tell me how to install those packages.

Thanks,
Praveen

Proposal: apply default config values on the plugin side

We still need to send rules, but without the default value. Instead we can apply default value when receiving metrics from gRPC, if required config are empty.

Pros:

Less data through gRPC.
Would avoid creating spaghetti code to solve #41 easily.

Cons:

requires to change all libraries.

plugin's string ConfigPolicy should allow a regular expression constraint

Currently I can constrain the values of int/float configuration policies (e.g., set min/max), but I would like to specify a regular expression to string configuration policies. If the string config value does not match the regular expression, snap should reject the task specification, something like:

 policy := plugin.NewConfigPolicy()
 policy.AddNewStringRule(
    []string{},
    "url",
    true,
    plugin.SetRegEx("^(http[s]:...?$"), // Ignore my specific example of a reg ex
 )

Mobile use case

Can i run a plugin on mobile to collect mobile OS stats and send to a snap server ?
from what i saw in the architecture the system relies in a snap daemon, and communicates with the plugin over stdout. this will not work on mobile.

thanks..

Miss metric Desc after conversion

The supplied config info was missing after calling either "fromProtoMetric" or "toProtoMetric".

For example:

Input metric:

             m := &pb.Metric{
            Namespace:          ns,
            Timestamp:          &pb.Time{Sec: int64(10), Nsec: int64(9)},
            LastAdvertisedTime: &pb.Time{Sec: int64(10), Nsec: int64(9)},
            Version:            int64(i),
            Tags:               map[string]string{"x": "123"},
            Data:               &pb.Metric_Int64Data{Int64Data: 1},
            Unit:               "int64",
            Config:             &pb.ConfigMap{StringMap: map[string]string{"xyz": "123"}},
        }

calling : tm := fromProtoMetric(m)
tm won't have supplied config info. The output was:

{/a/b/c 9 map[] 9 map[tag9:value9] 2016-08-17 09:41:36.900529877 -0700 PDT int  {0 900529877 0x8cad20}}

Same for function "ToProtoMetric". For example:

 Expected: 'Namespace:<Value:"val7" Description:"desc7" Name:"name7" > Version:7 Config:<StringMap:<key:"xyz" value:"123" > > LastAdvertisedTime:<sec:10 nsec:9 > Tags:<key:"x" value:"123" > Timestamp:<sec:10 nsec:9 > Unit:"int32" int32_data:2 '
  Actual:   'Namespace:<Value:"val7" Description:"desc7" Name:"name7" > Version:7 LastAdvertisedTime:<sec:10 nsec:9 > Tags:<key:"x" value:"123" > Timestamp:<sec:10 nsec:9 > Unit:"int32" int32_data: