flowbased / fbp-protocol Goto Github PK

Now noflo-ui uses a per-runtime-type hard-coded list of languages. However, in some cases this may depend on the installation. For example:

• NoFlo may or may not support CoffeeScript depending on whether the CoffeeScript compiler is present
• MsgFlo supports lots of languages, but only if their "runners" are installed

So, runtime:runtime message could return an array of supported languages.

Needed for noflo/noflo-ui#206

https://www.asyncapi.com is a standard for making messaging specs. Would provide nicer tooling around the FBP Protocol.

program.command is a method name in commander.js

fbp-init --name chix-runtime --collection test
Stored the following configuration to /home/ubuntu/workspace/fbp-config.json

name: chix-runtime
command: function (name, desc) {
  var args = name.split(/ +/);
  var cmd = new Command(args.shift());

  if (desc) {
    cmd.description(desc);
    this.executables = true;
    this._execs[cmd._name] = true;
  }

  this.commands.push(cmd);
  cmd.parseExpectedArgs(args);
  cmd.parent = this;

  if (desc) return this;
  return cmd;
}
host: localhost
port: 8080
collection: test

To test, run `fbp-test` from this directory

Easiest fix will be to change the name for the --command option.

run or exec maybe?

or else use a different cli tool.

update:
I guess command could still be used, just beware if no command option is passed the value will not be empty.

Hi,

it would be great if I could add my own metadata to a component and to each port.

This would be useful if for instance I want to choose a graphical representation of the data carried in a component in the UI.

Example:
.... description:"...",
metadata: '{ "viewer":"griddata", .... }

The UI would do a component:list over the fbp protocol, get this metadata 'viewer' and draw a UI element which shows the components/processes data in a 2d grid.

The metadata should carry a JSON string which can be set completely free by the user.

At the moment I put a JSON string with metadata in the description field. But this is only a temporary solution.

Is this possible ?

Thanks,

Josef

I think this would be much more clear. Any reason not to?

Now the generated HTML contains mixed up input/output messages with duplicated keys, and some properties missing

Every other message in the network protocol contains graph information, allowing one runtime to have multiple networks/graphs.

I think such messages should optionally also include "node", which should be a valid node identifier in "graph".

Migrated from noflo/noflo#164

Hi,

I think we need a command which lists all graphs which are set up in the runtime.

This will be needed for a UI which connects to the runtime through the fbp protocol. It doesn't know anything about formerly dynamically created graphs in the beginning.

The command graph:list should return an array of strings with the names of the graphs (ids?).

This should be pretty easy to implement.

Best regards,

Josef

Currently https://flowbased.github.io/fbp-protocol/ looks a bit basic...

Version 5.5.0 of semver was just published.

This version is covered by your current version range and after updating it in your project the build failed.

semver is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

Currently "values" on "inport" of component:component message just takes an array of values. I think it instead should take an array of { "id": "my-id", "name": "MyHumanId" }.
The "id" would be what is stored as an IIP, and "name" what is shown in UI. id could potentially be an integer value.
This is also more extensible in case there are other things we have not thought about.

Migrated from noflo/noflo#173

As a user of a UI that is compatible with fbp-protocol such as noflo-ui, I want UI widgets with finer grained controls to make my life easier.

I propose that we expand the definition of our port types to allow json schema documents so that developers can use this to provide a better UX for users.

We currently support enums and defaults.

For starters, I would like to add things like:

• minimum / maximum values for numeric types
• prefix and/or suffix around an input widget (e.g.: data size: [<input>] MB)
• display precision for float types
• "soft" max and min values for numerics, for setting initial slider bounds which can be expanded up to the "hard" min/max

As well as the ability to reinterpret a base datatype with a more specific widget. For example:

• file / folder browser for strings
• date picker for strings

Here is an example sub-document for an integer port:

{
  "id": "input_size",
  "required": true,
  "addressable": true,
  "type": 
  {
    "$schema": "http://json-schema.org/schema#",
    "title": "input size",
    "description": "input size in megabytes",
    "type": "integer",
    "default": 10,
    "minimum": 0,
    "maximum": 100000,
    "softMaximum": 1000,
    "labelSuffix": "MB"
  }
}

Notes:

• "title" is the display label. can be different from the "id"
• in the case that "type" is a json-schema doc, some keys that would normally be siblings of "type" will be contained within it: "values", "description", and "default"

Other advantages:

• gives us the ability to describe compound types. For example, it's very common to use object type, but unfortunately, since the UI knows nothing of an object's schema, it can't provide a proper UI or validation. With json-schema, runtimes can provide more information for these compound types and noflo-ui can build appropriate UIs.
• could also be used to validate IIPs on the UI side

Related issues:

• noflo/noflo-ui/issues/309
• noflo/noflo-ui/issues/482
• noflo/noflo-ui/issues/371

To get out of this, we should make the schemas the canonical single-source-of-truth for all protocol description. Currently this information is in https://github.com/noflo/noflo.github.io/blob/master/documentation/_posts/2013-07-09-protocol.md
The human-readable documentation page should be generated from the JSON schemas

If we can use graphs as components we should be able to specify an icon and description for when the graph is displayed in the component library of noflo-ui or anywhere else component description is used.

I've been digging around in the code to try and implement this but can't find the right hooks to add this functionality. If anyone could give me some pointers I'd be glad to implement this myself and send a pull request.

Migrated from noflo/noflo#196

When thinking about the ability for one graph to load another graph within it (i.e. subgraphs), it seems to me that the storage of such graphs can take 3 forms:

1. graphs published and consumed as components: often defined as code, and thus not editable within a graph editor UI.
2. external graph references: graphs saved externally to the parent graph (i.e. a separate "project"). the same graph may be referenced by many other graphs. unlike #1, these are the product of a graph editor (stored as .json or .fbp), rather than code. it may or may not be editable from the parent graph editor, depending on the abilities of the UI
3. embedded graphs: saved within the parent graph. can only be referenced indirectly by other graphs via its parent. editable from the parent graph editor.

fbp-protocol already supports #1 (in reality it requires no additional support as these subgraphs can simply be treated as components). This ticket is for #3. EDIT: and #2

Below is a bare-bones example of a command sequence that creates a graph within a graph:

protocol: graph
command: clear
payload:
    id: myGraph

---
protocol: graph
command: addnode
payload:
    id: myNode
    graph: myGraph

---
protocol: graph
command: clear
payload:
    id: mySubGraph
    parent: myGraph

---
protocol: graph
command: addnode
payload:
    id: mySubNode
    graph: myGraph/mySubGraph

---
protocol: graph
command: addinport
payload:
    public: IN
    node: mySubNode
    port: IN
    graph: myGraph/mySubGraph

This adds 2 new concepts:

• sub-graphs may be identified within any graph attribute using /, as in parentGraph/childGraph
• graph clear gets an optional parent attribute

I chose to add the parent attribute because I wanted this action to be explicit. An alternative would be to use the parent/child notation for the graph clear, like this:

protocol: graph
command: clear
payload:
    id: myGraph/mySubGraph

I'm open to suggestions and opinions.

fbp-spec exercises the FBP protocol implementation of a runtime quite a lot, and is generally useful, so we could include it here.

A testcase for core/Repeat (like
https://github.com/flowbased/fbp-spec/blob/master/examples/simple-passing.yaml)
but covering all JSON types would be a really good start.

It can be trivally integrated with Mocha like this https://github.com/flowbased/fbp-spec/blob/master/spec/mocha.coffee

Running "exec:fbp_test" (exec) task
>> '.' is not recognized as an internal or external command,
>> operable program or batch file.

https://ci.appveyor.com/project/bergie/noflo-runtime-websocket/build/1.0.165#L146

It is not neccearily that the programming language/format of tests are the same as the code (for which we have language key). This makes it impossible to know if tests are for instance fbp-spec YAML/JSON, JavaScript, C - and to validate the contents.

So should have a sepate key for testslanguage (or some other name).

Currently secrets are transmitted in message payload. We should instead shape the message so that we have something like:

{
  "protocol": "graph",
  "command": "clear",
  "payload": {...},
  "secret": "xx"
}

Likely we have discrepancies right now as nothing has really checked compliance before.

Clients/tools

• noflo-ui
• fbp-spec
• flowtrace

Runtimes (only listing known maintained ones)

• MicroFlo
• NoFlo
• MsgFlo
• imgflo
• pflow

On the client-side, like noflo-ui / fbp-spec etc, I think that flowbased/fbp-protocol-client#50 is the way to go. This will also give some tools for checking that the various runtimes are doing OK, and maybe provide some debugging help if things behave wierdly

Hi!

I'm trying to implement the FBP protocol by using fbp-test and implementing the required commands one by one.

What I have now is this:

--> {"protocol":"runtime","command":"getruntime","payload":{}}
<-- {"protocol":"runtime","command":"runtime","payload":{"id":"30a1eb22-a999-11ea-a909-93c8afee90c7","label":"deno-fbp","version":"0.1","type":"deno-fbp","capabilities":["network:control","network:data","network:persist","network:status","protocol:component","protocol:graph","protocol:network","protocol:runtime"]}}
--> {"protocol":"graph","command":"clear","payload":{"baseDir":"/home/tom/Code/code9/deno-fbp/test/node_modules/fbp-protocol","id":"foo","main":true,"name":"Foo graph"}}
<-- {"protocol":"graph","command":"clear","payload":{"id":"foo","main":false}}
--> {"protocol":"graph","command":"addedge","payload":{"src":{"node":"Repeat1","port":"out"},"tgt":{"node":"Drop1","port":"in"},"metadata":{"route":5},"graph":"foo"}}
graph.addEdge
graph.addEdge:graph {
  id: "foo",
  nodes: {},
  edges: {},
  initial: {},
  components: [],
  status: { running: false, started: false, debugging: false, startTime: null }
}
TypeError: Cannot read properties of undefined (reading 'ports')
    at addEdge (file:///home/tom/Code/code9/deno-fbp/src/domain/graph.ts:156:33)
    at addedge (file:///home/tom/Code/code9/deno-fbp/src/runtime/protocols/graph.ts:174:39)
    at WebSocketAcceptedClient.<anonymous> (file:///home/tom/Code/code9/deno-fbp/src/runtime/runtime.ts:51:35)
    at WebSocketAcceptedClient.emit (https://deno.land/[email protected]/node/events.ts:134:20)
    at WebSocketAcceptedClient.open (https://deno.land/x/[email protected]/lib/websocket.ts:92:16)

As you can see graph/addedge is sent to the server but the graph has 0 nodes at this point.

1. Should I throw/return an Error event?
2. Shouldn't fbp-test send two graph/addnode events first, before connecting them? When connecting Repeat1:out to in:Drop1 I do not know which components the nodes are so I can not create the nodes when the graph/addedge event happens.

Looking through the schema files I realized that these are quite a few problems and shortcomings with the current schema files:

• The schema files actually contain many schemas: one for each message type under "output" and "input". While it might make it easier to edit and read, there seems to be some expectation in the json-schema spec and community that there is only one schema per document. For example, each schema doc is expected to have an "id" attribute specifying where the file can be downloaded on the web. more here. One-file-per-schema also makes it easier when working with command-line tools which seem to have this expectation.

• we can improve the schemas by using the $ref keyword to avoid duplication. Note that this can be used to refer to other objects in the same doc { "$ref": "#/definitions/address" } or in another doc { "$ref": "definitions.json#/address" }.

• no $schema keyword: it's good form to indicate what version of the draft is being used

• many required properties are not indicated as such (there are none in component.json). easy enough to cross reference properties with "optional" in their description.

• the "required" keyword must be a list of properties and can only exist for compound schemas. e.g. this is bad:

  "changenode": {
    "description": "Change the metadata associated to a node in the graph",
    "properties": {
      "id": {
        "type": "string",
        "description": "identifier for the node",
        "required": true
      },
      "metadata": {
        "type": "object",
        "description": "structure of key-value pairs for node metadata",
        "required": true
      },
      "graph": {
        "type": "string",
        "description": "graph the action targets",
        "required": true
      }
    }
  }

  this is good:

  "changenode": {
    "description": "Change the metadata associated to a node in the graph",
    "properties": {
      "id": {
        "type": "string",
        "description": "identifier for the node",
      },
      "metadata": {
        "type": "object",
        "description": "structure of key-value pairs for node metadata",
      },
      "graph": {
        "type": "string",
        "description": "graph the action targets",
      }
    },
    "required": ["id", "metadata", "graph"]
  }

• nested schemas don't indicate "type": "object". it's implicit, but should probably be there.

• "outPorts" and "inPorts" should indicate that they are an array of objects

• by default, extra keywords in the document are allowed. I think we should set "additionalProperties": false for all schemas to prevent typos from slipping through without error.

I tried using one of our schemas for validation (after extracting it from its parent doc) and got an error ("schema/json/component.json#/properties/name: true is not a valid "required", must be a array."), so I think technically our schemas are not valid as they are. we should definitely be running validation of our schema files in the tests for this repo

This ticket should be done in conjunction with converting fbp-test to use the schemas.

Some resources for learning more about json-schema:

• http://spacetelescope.github.io/understanding-json-schema/index.html
• https://brandur.org/elegant-apis
• https://github.com/brandur/json_schema
• http://www.jsonschemavalidator.net/

The tests validate two aspects of messages sent from the runtime: their structure and their values. We should replace all the structural tests with validation against the schema files. This looks to be the most popular json-schema validator for node: https://www.npmjs.com/package/jsonschema

Requires #13

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

• Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

Your Greenkeeper Bot 🌴

In many places e.g.
https://github.com/noflo/noflo-runtime-base/blob/master/src/protocol/Runtime.coffee#L44-51
I see component:setsource, but it is missing in https://github.com/flowbased/fbp-protocol/blob/master/schema/json/component.json

It is not clear, by reading the FBP protocol, what is required that runtimes send to clients in the Graph Protocol.

For example, by reading the spec, I understood that the runtime is expected to react to a removenode with a succession of changeedge, removeedge, changenode and finally removenode. What is the rationale behind deleting the metadata of an edge before removing it? I believe noflo-ui takes care of deleting dependent edges/IIPs before removing a node; why is that not expected of every client, requiring the runtime to handle only simple cases? Or assume the runtime takes care of all dependent resources in a removenode, requiring it to reply with a single removenode?

I was also confused with the response to changing the metadata of edges or nodes.

I noticed another related issue, a bit more severe: when "removing an IIP", the spec indicates the runtime "should provide response that iip was removed". The message it expects is a removeinitial command but contains a src field, which is not accepted in removeinitial messages.

The rationale behind the requests and responses/events should be clarified in the spec, so people implementing runtimes have a clear idea of how to handle those cases. My personal opinion is that request message types should not be used also as "notification event" message types, as the contents differ in some cases and drawing that line that would result in a clearer protocol. However, anything that is defined a bit more formally would be great for me! :)

One of the most confusing aspects of the protocol description is that it does not clearly specify which messages are requests, and if so, what type of message should be sent in response. It would be good to identify whether something is a request, response, or both, and have hyperlinks to jump between.

changenode, changeedge and so on which allow changing metadata are nice but should not be treated as mandatory. The metadata is not required for building/running functional graphs, it is merely informational.

When connected to a runtime, it would be useful to know the runtime's project metadata, including:

• namespace: Project namespace. Should be used as the prefix of component names
• repository: A URL for the source-code respository of the running project. Preferably git

This would allow clients to know which components in the library listing are part of the project, and which are from dependencies. Related to noflo/noflo-ui#695