bcliang / dash-extendable-graph Goto Github PK

The doc page is a little misleading saying that updateData contains data:

... in a format matching figure['data']

Specifically, I naively thought I could include trace name there and it would figure out the correct trace to append data to (or create new):

dict(x=[1], y=[1], name='Trace 1')

That works on initial load (because it just creates a new plot) however on subsequent updates it fails because it assumes trace indices are simply linearly increasing, and also because passing name to extendTraces throws an error.

It would be very convenient if that was handled automatically, namely:

• If name is present, check if it matches any of existing traces.
  • If it does, that's the trace to extend (and also name should be removed from the data passed to extendTraces).
  • If it does not, add as a new trace.

Background

#35 requested support for a "sorted" extend
#39 asked about updating data within a trace

Unfortunately, the plotly.js graphing library doesn't support this type of "data update". In both cases, the feature could be implemented on the callback side provided there is/was a way to replace (or just delete) an existing trace.

Imagined use-case within a callback:

1. Grab existing trace data in callback:

#either 
[..Input('graph', 'figure')]
#or
[..State('graph', 'figure')]

2. Make updates/modifications to the trace in question:

..
# add new values to x,y and sort by x
data_unsorted = zip(data[0]['x'].extend(new_x_values), data[0]['y'].extend(new_y_values))
data_sorted = sorted(data_unsorted, key=lambda x: x[0])
# map results to independent lists to be used in the callback output
x_new, y_new = map(list, zip(*data_sorted))

3. Do something like

• delete the existing trace, add a new one to the figure (replace a trace)
• delete the existing trace, add a new one with the same trace index (update existing trace)

In the above case, it may be better to go with update in order to support preserving the trace order for the figure (for future indexing)

Considerations

• Could implement support for this within the extendData callback, and possibly rename it to a more general name e.g. updateData. Pros: It already supports specifying the trace index. In this implementation, I suppose there would be an additional input allowing the user to specify whether to extend, replace, or just delete a trace. Cons: complexity increases
• Could implement separate props the explicitly implements delete, replace, and/or update functions. Pros: simpler to test, easier to implement each prop on its own. Cons: More difficult on the dash-side to perform required operations (e.g. if logic flows in such a way that some conditions require update, others require extend, and still others require delete)
• While we're at it, what about implementing a prependData prop / adding it to the extendData prop? Plotly.prependTraces() exists, and this is a special case for updating a trace. I'm sure there's a use case, though no one has explicitly requested it.
• To get a better sense of actual use case(s), prototype something similar within a PR made to dash-core-components

Requires

• Decide on which use case(s) to support via new/existing props.
• Ensure that figure.data contains the latest trace information. I'm not sure but it's possible that when using the extendData callback that figure.data isn't immediately updated.
• Add functionality for Plotly.deleteTraces()

Adding support for trace_order in the extendData property provides explicit instruction on how the component should handle the new data:

• redraw figure
• extend existing traces
• add new traces

Alternative to #6

extendData property should expect to receive an object {update, indices, maxPoints}

Hi, I am testing your code and is working with 'extendData' which is fine if I want to add more points to the same graph, my question is if there is a way that I could use your code to update the data instead of appending new data since I have been doing an "updating" in my graph each time creating a new figure which is time-consuming, so your code looks really good for what I want to do. So, to summarize, is there a way where I could use your code to just update a graph?

• upgrade serialize-javascript to version >=2.1.1 (GHSA-h9rv-jmmf-4pgx)
• component responsiveness to match other components plotly/dash-core-components#706
• plotly.js ==> 1.52.2

since this is still a single-component package I will continue to ignore the async loading features being added into components.

Currently, this component expects data to be provided as a dict. However, many users instead generate traces using Plotly's build-in graph_objs. For example:

Currently, dash-extendable-graph supports:

...
@app.callback(Output('sensor-signal', 'extendData'),
              [Input('store-sensor-data', 'modified_timestamp')],
              [State('store-sensor-data', 'data')])
def extend_scatter_plot(data_updated, data):
    payload = [dict(x=data['x1'], y=data['y1']), 
               ... #additional traces
               dict(x=data['xn'], y=data['yn'])]
    return payload

But users may prefer not to rely on creating their own dictionaries for each trace that should be updated. For the above example, a user may prefer to generate a trace object via plotly.graph_objs.Scatter

@app.callback(Output('sensor-signal', 'extendData'),
              [Input('store-sensor-data', 'modified_timestamp')],
              [State('store-sensor-data', 'data')])
def extend_scatter_plot(data_updated, data):
    payload = [
        plotly.graph_objs.Scatter(
            x=data['x'],
            y=data['y'])
    ]
    return payload

In the simple example, it doesn't appear to make much difference. However, it could be significantly easier for users if they, for example, use many different trace types with different data-containing keys.

Considerations

• How to handle non-data-containing keys? For example styling and legend information (e.g. line, marker, mode, visible, name)
• How to handle non-data graph_objs? (e.g. Layout)

When a figure contains subplots of the form

figure={
  'data': [
    {
      'name': 'spec',
      'type': 'line',
      'xaxis': 'x1',
      'yaxis': 'y1',
      'y': spec[0],
    },
    {
      'name': 'waterfall',
      'type': 'heatmap',
      'xaxis': 'x1',
      'yaxis': 'y2',
      'z': spec,
    }
  ]

where spec is a 2D numpy array the data is not extensible. For the above example I want to use extendData to update the y-values of the first subplot and the z-values of the second subplot. The two subplots share an x-axis but I don't think that's relevant.

Passing extendData an array of these two dicts results in a wrong number of arguments error. Is this supported and I'm getting the syntax wrong?

The following code produces undefined is not a constructor (evaluating 'new window.ResizeObserver(_this.resizeHandler)') error in safari and firefox browsers, however it works normally in google chrome.

import dash
import dash_core_components as dcc
import dash_extendable_graph as deg
import dash_html_components as html
import plotly.express as px
import plotly.graph_objects as go
from dash.dependencies import Input, Output, State
from dash.exceptions import PreventUpdate


if __name__ == "__main__":
    app = dash.Dash()
    layout = [deg.ExtendableGraph(id="graph", figure=px.scatter(),
                                  style={"height": "75vh"}),
              dcc.Interval(id="ms-beat",
                           interval=1000,
                           n_intervals=0)
              ]
    app.layout = html.Div(children=layout)
    flag = False

    @app.callback(Output("graph", "figure"),
                  Input("ms-beat", "n_intervals"),
                  State("graph", "figure"))
    def add_level_lines(n_intervals, figure):
        global flag
        if flag:
            raise PreventUpdate
        flag = True
        figure = go.Figure(figure)
        figure.add_hline(y=24, line_dash="dot", annotation_text="label")
        figure.update_layout(autosize=True)
        return figure

    app.run_server(debug=True)
    # app.run_server()

exception details:

undefined is not a constructor (evaluating 'new window.ResizeObserver(_this.resizeHandler)')

(This error originated from the built-in JavaScript code that runs Dash apps. Click to see the full stack trace or open your browser's console.)
ResizeDetector

constructClassInstance@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:13015:28

updateClassComponent@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:17235:29

callCallback@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:182:21

dispatchEvent@[native code]

invokeGuardedCallbackDev@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:231:31

invokeGuardedCallback@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:286:38

beginWork$1@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:23338:30

performUnitOfWork@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:22292:25

workLoopSync@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:22265:41

performSyncWorkOnRoot@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:21891:23

performSyncWorkOnRoot@[native code]

http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:11224:34

unstable_runWithPriority@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:2685:26

flushSyncCallbackQueueImpl@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:11219:26

flushSyncCallbackQueue@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:11207:31

flushPassiveEffectsImpl@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:23018:27

unstable_runWithPriority@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:2685:26

http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:22834:32

workLoop@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:2629:44

flushWork@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:2584:26

performWorkUntilDeadline@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/[email protected]_9_1m1617985068.14.0.js:2196:50

expanding on #6, figures may have multiple trace types. plotly.extendTraces requires that every key be populated:

dict(
    x=[[x1_1,... x1_n], [x2_1..., x2_n], ..., [xn_1, ..., xn_n]], 
    y=[[y1_1,... y1_n], [y2_1,... y2_n], ... [yn_1, ... yn_n]], 
    ...
)

will fail with the following input:

dict(
    x=[[x1_1,... x1_n], None, ..., [xn_1, ..., xn_n]], 
    y=[[y1_1,... y1_n], [y2_1,... y2_n], ... [yn_1, ... yn_n]], 
    ...
)

In 0.2.0 of the component, I implemented a pluck function to grab trace keys from the array of objects, and generate an object with array of arrays to fit the plotly.extendTraces api.

const pluck = (arr, key) => arr.map(o => o[key]);

However, if using trace objects that have different keys, the solution will fail. Instead, consider building out a per-trace object call to extendTraces.

I just filed a issue against plotly to add a way to extend a plot in a sorted way (not append but insert sorted) (plotly/plotly.js#4408)

now I though it may also possible (and probably faster implemented) to do in from the graph component. is it possible to hold a copy or reference to the plot's data and on extend insert the new data in a sorted way and afterwards call Plotly.animate()? is this feasible?

Any plans on updating this to the latest version of plotly.js? It's had quite a few fixes and performance improvements since the currently bundled 1.58.4.

Separately but related, it'd be really nice if there was an option to load this with un-minified version of plotly - to make for a more pleasant debugging.

Will be supported in dcc.Graph > 0.47.next.

ExtendableGraph['figure'] = dict(
    data = [],
    layout = {},
    frames = []
)

Sync with latest dash-core-components to support the loading states component.

boilerplate provides an example of test run using pytest. Let's implement tests for the following:

ExtendData input callback for a new ExtendableGraph object (no props) should revert to using this.plot()

• check layout is default
• check figure is default
• check number of traces matches len(extendData)

ExtendData input callback (n-count traces) for an ExtendableGraph object with existing (n-count) traces should extend traces as expected.

• check len(figure.data[0...n].[x, y])

ExtendData input callback (n-count traces) for an ExtendableGraph object with existing (m-count, m>n) traces should extend-then-append traces.

• check len(figure.data[n+1...m].[x, y])

Current behavior:
In the integration test suite (and within usage.py, the extendData and prependData callbacks are triggered on a >= 100ms periodic interval. All tests pass.

However, if the interval is reduced there can be situations where a new callback may be triggered while an existing callback is still being executed (large dataset, slow compute). In this situation, the latest previous changes are discarded and a new state is created with the new information. Other possible triggers: asynchronous data updates when using an asynchronous/websocket server framework (dash-devices, dash-django).

The end result is that data can be lost in the ExtendableGraph object (traces are miss some updates).

Expected behavior:
The component should properly extend (or prepend) all data (no skips). I think the "eventually-consistent" model is the right approach.

Proposed Solution
dcc.Graph implements an array for extend / prepend props, allowing the "final" callback to handle multiple callbacks within a single execution. There is an issue in dash-core-components suggesting that the implementation is flaky.

Additional investigation:

• how should we handle simultaneous callback from prependData and extendData?
• how should we handle simultaneous callback from figure.data and extendData ?

Dash 1.5+ implemented lazy loading on the dash-core-components suite. Let's update ExtendableGraph to match.

Should trigger

• integration tests using dash[testing]
• python pycodestyle/PEP8 linting
• javascript ES6 linting

For now, I still consider unit testing unnecessary, instead relying on the PlotlyJS release process for that.

devDependencies on lodash and mem
CVE-2019-10744
WS-2018-0236

doesn't impact prod.

For users using clientside callbacks to trigger changes to the dash-extendable-graph Figure, the current code in master will fail.

Cause: To mimic the responsive behavior of the core Graph component, the plotly.js figure has been moved inside a parent div to support the use of react-resize-detector. Because the component ID is tied to the outer div, calls using the plotly.js library fail (e.g. Plotly.relayout('my-extendable-graph-id', {layout object..});

Solution: Need to move the key / id declarations into the child div.

Additional: Add new integration test to make sure this doesn't break moving forward.

Use case: len(extendData) > figure.data
More traces inputted into this property than exist in the figure. What is the expected behavior?

• Do nothing, ignore the data in the new property. Throw an error
• Treat this as new data. Throw out the existing and use this.plot (e.g. Plotly.react()) to redraw the figures
• Extend traces that exist, and then Plotly.addTraces() for the remaining. We will have to use a default trace style for new traces.

Use case: len(extendData) < figure.data
There are fewer traces inputted into the property than exist in the figure.

• Do nothing, ignore the data in the new property. Throw an error.
• Assume the first n- traces should be extended. The remaining traces should be populated with empty trace (no extended data)

As reported in plotly/dash-core-components#603

Relevant commits from plot/dash-core-components:

• excess ids and graph cleanup fix impl. in PR 604
• Additional changes to doc (PR 608)

8/25: No resolution on the infinite loop on relayout

• eslint-utils 1.3.x has a vulnerability allowing execution of arbitrary code that was fixed in v1.4.1

eslint-utils is part of the eslint package.. relevant dependencies to update

"babel-eslint": "^10.0.3",
"eslint": "^6.2.2",
"eslint-config-prettier": "^6.1.0",
"eslint-plugin-import": "^2.18.2",
"eslint-plugin-react": "^7.14.3",