flojoy-ai / studio Goto Github PK

flowchart is too zoomed-in on initial load:

can it look less imposing like this, please:

In the debug tab, the end node is often excessively far to the right of the other nodes:

I would expect the DEBUG node positions to mirror the SCRIPT node positions

In this state, before selecting a node input parameter from the dropdown, I cannot input any text and there are no errors visible to the user.

This behavior is different from other widgets like sliders and knobs, where I can adjust the widget even before I've assigned it to a node input parameter.

The user should be able to input text before assigning that text to a node input.

While running the following app I encountered that CONSTANT node is not getting current parameter value and processing default parameter on @flojoy decorator.

Run script Result:

it does not seem that the flowchart program is removed from the app state when it is deleted

steps to reproduce:

1. shift select the entire flowchart
2. click "delete"
3. click "run" - the previous app which was deleted will be sent to the server to run

expected behavior: a notification or alert that there is no program to send to the server. please add at least one node first

If you go to the LOGS tab, it is impossible to then go back to the CTRLS tab or the SCRIPT tab:

In addition, the LOGS tab could have a much more useful UI. It was originally a quick and hacky dev tool for me to help debug which nodes were failing or not behaving as expected.

I'd propose renaming this tab "DEBUG" and moving it to the left side of the nav along with SCRIPT and CTRLS. Instead of the current accordion layout with collapsible <details> for each node, a better UX might be a static, un-editable replica of the flow chart, where each node is the output plot. This would be a more intuitive view IMO on where the failure is occurring. I could imagine building up this static flowchart debugger overtime to include other information, like the size and shape of vectors being passed between nodes.

reactflow.dev has example of a node with a plot on their home page:

edit toggle should only show in CTRL tab

Latest FIGMA mocks are here:

https://www.figma.com/file/REC9JoYvxLGmvNiKUL5MYq/Flojoy_app?node-id=0%3A1

I'd like to redo the knob css as brushed metal:

https://codepen.io/oobleck/pen/MYBvGa

https://simurai.com/lab/2011/08/21/brushed-metal

https://freesvg.org/1538113580

Looks 🔥

To build a Python developer user community, we'll want it to be as simple as possible to contribute new nodes.

One way to do this is ensuring that the entire dev experience is in Python, as most scientists, mech engineers, and electrical engineers may know a little Python, but are unlikely to know Web dev also.

I propose doing this by allowing manifest specification a function's doc string, then creating a flojoy_sync.sh script that:

1. searches for new or modified scripts in the PYTHON/FUNCTIONS folder (using git)
2. reads the doc string of a new or modified script if it exists
3. attempts to transpile the docstring information to the relevant JSON manifests

If the Python user wishes, they can then create a pull request on flojoy-desktop that add this new node to the platform. Ideally, we can grow the platform with open-source scientific Python contributors in this way.

We'll base flojoy docstrings on the numpy/pandas docstring standard. However, we'' also have to add some flojoy-specific conventions on top of this standard.

I propose these convention additions to flojoy function docstrings:

Inputs
-------
1 : vector           [node input is 1 vector]

Inputs
------
2 : vector           [node input is 2 vectors]

Inputs
------
N : vector           [node input is N vectors]

Inputs
------
1 : matrix           [node input is 1 matrix (eg, an image)]

Inputs
-------
0                    [node does not take an input - eg LINSPACE or a random number generator]


Inputs
------
1 : matrix           [node input is 1 matrix (eg, an image)]

Inputs
------
1 : scalar           [node can take a single scalar, matrix, or vector as input]
1 : matrix
1 : vector

Inputs
------
1 : matrix           [node takes a matrix and 2 vectors as input]
2 : vector

...

etc...

Example

import numpy as np
from scipy import signal
import flojoy

@flojoy
def WAVEFORM(**kwargs):
    """The waveform node generates a waveform

    Parameters [parameters transpiled into PARAMETERS_MANIFEST.js]
    ----------
    frequency : float, default=3.14
                Waveform frequncy in radians
    offset : float, default=0
             Waveform offset along the y-axis
    amplitude : float, default=1
             Waveform amplitude
    waveform : str, {'sine', 'square', 'triangle', 'sawtooth'}
               Options: Sine, Square, Triangle, or Sawtooth

    # Flojoy-specific docstring parameters

    Inputs [transpiled to COMMANDS_MANIFEST for wiring into frontend node UI]
    --------
    1 : vector

    Outputs [transpiled to COMMANDS_MANIFEST for wiring into frontend node UI]
    ---------
    1 : vector

    Commands [keys transpiled into COMMANDS_MANIFEST]
    -----------
    NAME : Waveform
    KEY : WAVEFORM
    TYPE : GENERATOR
    """

    valid_waveforms = ["sine", "square", "triangle", "sawtooth"]

    params = {'frequency': 3.14, 'offset': 0, 'amplitude': 1, 'waveform': 'sine'}

    WF = params['waveform']
    A = params['amplitude']
    F = params['frequency']
    Y0 = params['offset']

    if waveform not in valid_waveforms:
        waveform = valid_waveforms[0]

    if WF == 'sine':
        y = Y0 + A * np.sin(np.radians(2 * np.pi * F) * x)
    elif WF == 'square':
        y = Y0 + A * signal.square(2 * np.pi * F * x / 10)
    elif WF == 'triangle':
        y = Y0 + A * signal.sawtooth(2 * np.pi * F * x / 10, 0.5)
    elif WF == 'sawtooth':
        y = Y0 + A * signal.sawtooth(2 * np.pi * F / 10 * x)

    return {'x0':x, 'y0':y}

I noticed that I cannot input text into the new text input controller.

Checking the console, I see this error on lines 114 and 127 of ControlTabsView.tsx

TypeError: undefined is not an object (evaluating 'ctrl.param.nodeId')

Can be found here:

https://www.figma.com/file/F3X0IwHNyyhWt5DWpHwojH/flojoy_chart_icon?node-id=0%3A1

after running a script, when i click on a node, the node info modal does not display its chart (output) and reports that it has not been run yet

To replicate, just create an Output Plot and minimize the RGL container size:

For RGL with plots, maybe we just set a min width and height if that's possible. Or, could set overflow to hidden. Whatever's easiest.

is still much different from

we should drop the emojis and on smaller window widths use CSS to shorten the menu item names:

• visual python script -> script
• ctrl panel -> ctrls
• add python function -> add
• run script -> run
• load file -> load
• save file -> save

can pull the favicon file from docs.flojoy.io

23:01:18 Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 256)
23:01:18 flojoy: FUNCTIONS.GENERATORS.LINSPACE.LINSPACE(ctrls={'LINSPACE_Linspace_start': {'functionName': 'LINSPACE', 'param': 'start', ...) (JOB_ID_5)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/joyflo/flojoy_python.py", line 229, in wrapper
    pm = get_parameter_manifest()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/joyflo/flojoy_python.py", line 144, in get_parameter_manifest
    f = open(root + '/src/data/manifests-latest.json')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jackparmer/Desktop/projects/daq-labs/flojoy//src/data/manifests-latest.json'

23:01:18 Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 256)
23:01:18 flojoy: FUNCTIONS.GENERATORS.LINSPACE.LINSPACE(ctrls={'LINSPACE_Linspace_start': {'functionName': 'LINSPACE', 'param': 'start', ...) (JOB_ID_5)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/joyflo/flojoy_python.py", line 229, in wrapper
    pm = get_parameter_manifest()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/joyflo/flojoy_python.py", line 144, in get_parameter_manifest
    f = open(root + '/src/data/manifests-latest.json')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jackparmer/Desktop/projects/daq-labs/flojoy//src/data/manifests-latest.json'

23:01:18 Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 256)
23:01:18 flojoy: FUNCTIONS.GENERATORS.LINSPACE.LINSPACE(ctrls={'LINSPACE_Linspace_start': {'functionName': 'LINSPACE', 'param': 'start', ...) (JOB_ID_5)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/joyflo/flojoy_python.py", line 229, in wrapper
    pm = get_parameter_manifest()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/joyflo/flojoy_python.py", line 144, in get_parameter_manifest
    f = open(root + '/src/data/manifests-latest.json')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jackparmer/Desktop/projects/daq-labs/flojoy//src/data/manifests-latest.json'

23:01:18 Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 256)
23:01:18 flojoy: FUNCTIONS.GENERATORS.LINSPACE.LINSPACE(ctrls={'LINSPACE_Linspace_start': {'functionName': 'LINSPACE', 'param': 'start', ...) (JOB_ID_5)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/joyflo/flojoy_python.py", line 229, in wrapper
    pm = get_parameter_manifest()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/joyflo/flojoy_python.py", line 144, in get_parameter_manifest
    f = open(root + '/src/data/manifests-latest.json')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jackparmer/Desktop/projects/daq-labs/flojoy//src/data/manifests-latest.json'

should use PurePath or os.path.join() instead of + string operator in f = open(root + '/src/data/manifests-latest.json'). This would prevent the spurious '//'

There’s an issue I had encountered couple of days ago that, when watch.py requeues a deferred job back, rq worker actually starts processing it, even if its dependencies are still not processed. Pasted a link to the code below.

https://github.com/flojoy-io/flojoy-desktop/blob/develop/PYTHON/WATCH/watch.py#L147

Make sure to pull in the latest of the develop branch, this line helps checking for a job’s status for longer duration so that Watch.py doesn’t try to requeue a deferred job.

Possibly with WebSocket implementation this issue will go away.

Top navbar style was removed in this commit 49a3126 for fixing linting issues.

Increase sine wave node default frequency, so that several periods show up by default (currently only 1/4 of a sine wave period shows currently)

running this flowchart:

i get this output in the debug tab:

If I save a flow chart then reload it, all of the nodes appear stacked on top of each other like this:

Here - (https://github.com/flojoy-io/flojoy-desktop/blob/main/mac_start_up.sh#L61) - it's not enough to test whether the flojoy.yaml exists unfortunately. If it does exist, we have to test whether the project directory has changed.

For example, my flojoy.yaml looks like this:

PATH: /Users/jackparmer/Desktop/projects/daq-labs/flojoy/

But if I install flojoy in a new folder, this PATH needs to be updated to reflect the location of the flojoy project that is being run.

FILE=$HOME/.flojoy/flojoy.yaml
if test -f "$FILE"; then
    echo "$FILE exists."
else
   mkdir $HOME/.flojoy && touch $HOME/.flojoy/flojoy.yaml
   echo "PATH: $CWD" > $HOME/.flojoy/flojoy.yaml
   echo "directory ~/.flojoy/flojoy.yaml does not exists. Creating new directory with yaml file."
fi

typescript-eslint for JS and Flake8 for Python?

this would be a good simple app to try to recreate in flojoy and compare the experience:

https://github.com/elyra-ai/examples/tree/main/pipelines/dax_noaa_weather_data

Steps

• Requirements finalization (need to review the modification with @jackparmer)
• UI Design
• Implementation

Anyone have a Windows computer out there?

At a minimum, RQ does not work on Windows (see
https://stackoverflow.com/questions/55612559/queue-background-tasks-in-python-application-on-windows)

Perhaps this can be fixed with rq-win, but I imagine there will be several other wrinkles to iron out

The value of this node:

should be editable here in the control panel:

Currently at the startup of the whole app, when it posts a job to the backend, backend returns 200 OK, but the jobs for some reason don't get run with RQ workers. Most likely jobs don't get enqueued.

If possible try to reproduce the issue and identify the root cause and resolve that.

Also, we should have a mechanism to test if all the jobs are enqueued successfully. If they were not, then it should retry (total 3 tries) after certain period of time (use exponential backoff + jitter).

If after multiple retries, it still fails, then the scheduler API should return 5xx status with some details on the exception.

Currently, in develop branch, I faced an issue in changing the value of a node parameter in the knob. If you change its value from 0 to 15 it will start to blink between 0 and 15. I'll try to add a GIF for better understanding later.

The app looks a lot better with some margins, like this:

Rather than full-bleed, like this:

I would like the initial app state to have some margins on its initial load

We need to provide a guarantee on how long the script execution flow take. So on a certain machine (ex. GitHub Action) it will check if the different steps of the flow are within some threshold of time.

So for example, it will check:

1. When user hits the play button, the Django server receives the HTTP POST request within certain threshold milliseconds.
2. The flojoy-watch rq worker receives the job from Django server within a certain time
3. etc.

The test will run the same execution flow multiple times to get the P99 of these timings, and will fail when the timings are larger than their corresponding thresholds.

the items Undo and Delete do now work in the Edit menu - we should just delete it

Undo was always buggy and it's easier to Delete simply by pressing the Delete key

Here's the app:

Here's the CTRL panel output:

I would expect this to be a line that looks like this:

This is what I see when I click on the LINSPACE node in the flowchart.

When I click on the SCATTER node in the flowchart, the program crashes:

Here's the console output:

Along with Plots, this React component should be another Output widget option in the CTRL tab:

https://github.com/nachovigilante/react-7-segment-display

when I delete the preloaded function generator flowchart, and try to create a new node on the blank flowchart canvas by clicking "Add", nothing happens

when changing the frequency with the slider, the visualization output does not update:

here is the flowchart:

I cloned and tried to run the latest main branch with sh mac_startup.sh. I received this error:

I think it makes sense, if possible, to port all errors to the first terminal tab. This way, it will be simpler for community users to copy/paste error output for reporting.

Am I supposed to be installing with docker now? Can we please create and INSTALL.md with instructions for how to do this, if so?

Add plotly chart types:

• line chart (scatter with mode set to lines)
• 3d surface plot (surface - https://plotly.com/javascript/reference/surface/#surface)
• 3d scatter plot (https://plotly.com/javascript/3d-scatter-plots/)
• bar chart (bar - https://plotly.com/javascript/bar-charts/)
• Big Number (indicator with mode set to number -https://plotly.com/javascript/indicator/)

For the function generator, it's hard to see the different waveform types clearly unless the chart is a line chart (in plotly speak, a line chart is type scatter with mode set to lines - https://plotly.com/javascript/reference/scatter/#scatter-mode)

This will allow us to track flojoy downloads and avoid annoying local import errors and duplications

See this:
#26 (comment)

A @flojoy decorator should abstract away the internals of plugging a numpy or scipy routine into a flojoy visual program.

Here's a usage example in pseudo-code:

import numpy as np
import flojoy as fj

@flojoy
def numpy_add(a b):
    return np.add()

The @flojoy decorator itself (from the open-source flojoy Python library published on pip would then look something like this:

def flojoy(func):
    xy = fj.get_previous_job_outputs()

    # Do some type checking and sizing of xy, etc...

    return func(x, y)

Test visual script (script text file attached):

Control panel output:

I'd expect the output here to be a plot with a single point where y=4.

If I change the output node to one of the constants, I get the same result:

Here, I'd expect the plot to be a single point where y=2.

If I delete the flowchart and reload it from the same file, all of the nodes are superimposed (perhaps their position info on the canvas is not be saved):

If I run this app, I get the same plot output as before:

flojoy (1).txt
: