plafer / tissue.jl Goto Github PK

We want an error message system as the one in Rust. For every possible error by the user, we should output an error message which explains what went wrong concisely, as well as an error code such as E1032, which users can use in a call to Tissue.describe_error(:E1032) in the REPL, and get a verbose description of the errors, common mistakes related to it, examples on how to fix it, etc.

This will be especially useful when writing macros, to help the user use the correct syntax.

It would be great to have a GUI that lets you define your graph topology, as well as all your calculator types. So you would add nodes, annotate them with their type (e.g. FaceDetectionCalculator), add edges between the nodes, and the GUI would create a bunch of files with

1. the @graph definition
2. all the calculator types (perhaps one file per calculator type)
3. an empty Tissue.process(c::TheCalculatortype, ...) function definition, ready to be filled in by the user

It could simultaneously be used as a graph visualization tool: launch it with a file that has a @graph definition in it, and it will visualize the graph for you, on which you can make edits.

It would be nice to be able to define a subgraph to be used as just another node in graphs, similar to how they do it in mediapipe. However, in mediapipe, a subgraph and a graph have the same interface. We'll need to make them different because we don't expose input and output streams coming in/out of graphs.

The idea would be that you can define a subgraph with a similar language to @graph, and be able to use it just as you'd use any other calculator. close(MySubgraph) calls close(calculator) on all the calculators in the subgraph. Similarly with open(calculator) (#10) when we have it.

Defining a subgraph would look like:

using Tissue

struct DinkyCalculator end
Tissue.process(c::DinkyCalculator, input_stream) = 42

struct MergerCalculator end
Tissue.process(c::MergerCalculator, in1, in2) = in1 + in2

@subgraph MySubgraph begin
    @inputstreams subgraph_input1, subgraph_input2

    @calculator c1 = DinkyCalculator()
    @calculator c2 = DinkyCalculator()
    @calculator c3 = MergerCalculator()
    
    @bindstreams c1 (input_stream = subgraph_input1)
    @bindstreams c2 (input_stream = subgraph_input2)
    @bindstreams c3 (in1 = c1) (in2 = c2)

    # Sets the output stream of MySubgraph as the output stream of c3
    @outputstream c3
end

Using that subgraph in a graph would look like:

using Tissue

struct SourceCalculator end
Tissue.process(c::SourceCalculator) = 42

struct PassthroughCalculator end
Tissue.process(c::PassthroughCalculator, input) = input

@graph MyGraph begin
    @calculator source = SourceCalculator()
    @calculator p1 = PassthroughCalculator()
    @calculator p2 = PassthroughCalculator()
    @calculator subgraph = MySubgraph()

    @bindstreams p1 (input = source)
    @bindstreams p2 (input = source)
    @bindstreams subgraph (subgraph_input1 = p1) (subgraph_input2 = p2)
end

After graphs are parameterized (#4), we can reuse that same idea with subgraphs to allow parameters to be passed down to internal calculators.

The current flow limiter is currently an open loop controller. Based on how fast calculators a running, we pull new packet from the source calculator. We should "close the loop" by actually monitoring the channels in the graph to make sure they're not overflowing. This could technically happen, for example if there are sudden changes in how fast a certain calculator runs (e.g. the network drops and the calculator is waiting for the connection to come back up).

A graph can be in 3 different states: READY, RUNNING, or STOPPING.

1. when a graph is just created, its state is READY
2. after start() is called, the graph moves in RUNNING state
3. when stop() is called, or the source returns nothing, the graph moves in STOPPING state, until wait_until_done() is called and returns, after which it goes back into READY state

Note that wait_until_done() also does cleanup, and must be called before starting the graph again.

We can also expose a get_state(graph) which returns the current state, if ever someone needs it.

Preferably we implement #10 before this one.

Most likely not all applications will want to run as fast as possible. The user should be able to specify a desired_fps parameter somewhere in @graph.

We need to share the insight that if packets are treated as immutable, and no global state is used, then a graph's outputs given a set of inputs will always be the same (assuming of course no random number is used insight a calculator).

For example, this example might give different results on different runs because global state is used.

using Tissue;
x = 0

struct Source
    count
    Source() = new(0)
end
function Tissue.process(c::Source)
    c.count += 1
    return c.count > 50 ? nothing : 1
end

struct Adder end
function Tissue.process(c::Adder, num) 
    global x += 1
    return num
end

struct Multer end
function Tissue.process(c::Multer, num) 
    global x *= 2
    return num
end

struct Barrier end
function Tissue.process(c::Barrier, num1, num2) 
    global x
    x
end

@graph NonReproducibleGraph begin
    @calculator source = Source()
    @calculator adder = Adder()
    @calculator multer = Multer()
    @calculator barrier = Barrier()

    @bindstreams adder (num = source)
    @bindstreams multer (num = source)
    @bindstreams barrier (num1 = adder) (num2 = multer)
end

The example for using mutable is very similar: Source sends a mutable struct object with an Atomic{Int} field, and adder and multer modify that field similar to how they modify global x in this example.

Currently, if a graph is stoped and wait with wait_until_done , it cannot be started again, because close() will have been called on all calculators. We should then

1. introduce an open(calculator) function which we call in start(graph) for all calculators.
2. Modify the Getting Started documentation to use it.

This would allow users to turn off the graph in certain situations (e.g. during off-hours) without needing to completely shut down their application.

Version: 0.2.0
Julia version: 1.6.2
To reproduce, failing test:

using Tissue
using Test
using Base.Threads: @spawn

flag_touched = false

struct Source end
Tissue.process(s::Source) = 42

struct Dropper end
Tissue.process(c::Dropper, in_num) = nothing

struct Touch end
function Tissue.process(c::Touch, in_num) 
    global flag_touched
    flag_touched = true
end

@graph G begin
    @calculator source = Source()
    @calculator dropper = Dropper()
    @calculator toucher = Touch()

    @bindstreams dropper (in_num = source)
    @bindstreams toucher (in_num = dropper)
end

g = G()
start(g)

sleep(0.5)

@test flag_touched == true

This is because we use that name internally. We should switch to gensym'ed names instead, or var names.

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

There is currently no way to dynamically change the parameters of calculators in a graph (e.g. pass as command line argument). The user should be able to name parameters in @graph, and use them in the body.

A potential syntax would be:

@graph MyGraph(param1, param2) begin
    @calculator c = SomeCalculator(@param param1, @param param2)
    ...
end

We might also want optional and/or keyword arguments too. And multiple constructors.

It would be great to have a tracer that traces (record + timestamp) all the packets that traveled through the graph. We would then have a visualizer where you see the actual packets going through the graph.

This would be great to get a feel for how your graph runs: which calculators are taking longer that others? Is some calculator being slow only on some types of inputs? It would probably be very useful in debugging a graph, say if it unsuspectedly hangs, or goes slower than expected.

There could also be a "timeline view", say, at the bottom, that shows you the more traditional way that traces are visualized (similar to how MediaPipe's tracer is visualizing it).

Main concepts: documents things like the flow limiter, immutability (#16), and whatever else needs a deep dive.

Design decisions: Document why we made certain decisions (e.g. only one output stream per calculator, graphs don't have input/output streams, etc.)

Some code might need to be run in the main thread (e.g. all UI operations in QT).

The idea is that if it takes 10ms to generate a packet, then we should start generating the packet 10ms in advance so that the data is ready for when the graph is ready to process it.

To implement this:

1. We currently don't gather runtime statistics for the generator calculator. We need to do that. This might also remove some complexity in the code, where the generator calculator wouldn't be as much of a special case anymore.
2. in evaluate_packet_period(graph), we should subtract the source calculator's exec_time from the new source period before setting it.

The initial flow estimate is very much an underestimate, and the graph takes a few seconds to get to its limiting fps, depending on how we tune the alpha parameter. The main reason for that underestimate is probably that the initial estimate takes into account compilation time.

We should rethink how to do this so that the graph gets to its limiting fps as fast and robustly (i.e. don't overflow the channels in any case) as possible. New design should also fix #3.