contrib-workflow-dag's People
Contributors
Stargazers
Watchers
contrib-workflow-dag's Issues
/ray/workflow/step_function.py", line 133, in options else self._step_options.catch_exceptions, AttributeError: 'NoneType' object has no attribute 'catch_exceptions'
/ray/workflow/step_function.py", line 133, in options
else self._step_options.catch_exceptions,
AttributeError: 'NoneType' object has no attribute 'catch_exceptions'
add option to "update" node?
updating a node might be useful:
- For DataNode, this will make it a truly data "holder" node, which gives user the option to switch to a different dataset and re-run the graph.
- For FunctionNode, this gives the option to update the operation (i.e. function) without having to re-build the graph. For example, there might be a feature extraction node somewhere in the middle of the graph, by updating the node to different kinds of feature extraction and re-run the graph, we can evaluate which feature exaction performs best.
This requires careful thought as it makes Node "mutable".
TODO:
- Add "update" methods in Node class. e.g. update execute() and get_name()
- update should be done on dag level, i.e. dag.update(node, xxx)
- when node is update, we need to clear self._node_output in DAG for the target node and all later nodes.
- add methods to get ALL pre nodes and post nodes (add recursive flag to existing method)
- add methods to reset node outputs.
Add StepFunctionNode
Currently there is FunctionNode with decorator to decorate a function to Node.
But a StepFunction -> Node conversion is missing, and it is important because:
- step functions might be already there, and we don't always have raw function.
- for nested step functions. we cannot simply nest nodes. so nested step functions -> node that contains the nested functions.
TODO:
add a StepFunctionNode.
how to run?
can not get result:
│16:14:48 08.19 INFO node.py:94]: running this node │
│2022-08-19 16:14:49,910 INFO services.py:1470 -- View the Ray dashboard at http://127.0.0.1:8265 │
│2022-08-19 16:14:51,042 INFO workflow_access.py:394 -- Initializing workflow manager... │
│2022-08-19 16:14:51,953 INFO execution.py:50 -- Workflow job created. [id="c8e8f110-8c70-45e5-9682-6006be0a4eca.1660896891.953237057"]. Type: FUNCTION. │
│(WorkflowManagementActor pid=9303) 2022-08-19 16:14:51,968 INFO workflow_access.py:189 -- run_or_resume: c8e8f110-8c70-45e5-9682-6006be0a4eca.1660896891.953237057, __main__.multiply,ObjectRef(df742f7ba0bfa464fffffffffffff│
│fffffffffff0100000001000000) │
│(WorkflowManagementActor pid=9303) 2022-08-19 16:14:51,968 INFO workflow_access.py:200 -- Workflow job [id=c8e8f110-8c70-45e5-9682-6006be0a4eca.1660896891.953237057] started. │
│(WorkflowManagementActor pid=9303) 2022-08-19 16:14:52,056 INFO workflow_access.py:382 -- Workflow job [id=c8e8f110-8c70-45e5-9682-6006be0a4eca.1660896891.953237057] succeeded. │
│(_workflow_step_executor_remote pid=9286) 2022-08-19 16:14:52,044 INFO step_executor.py:350 -- Step status [RUNNING] [c8e8f110-8c70-45e5-9682-6006be0a4eca.1660896891.953237057@__main__.minus] │
│(_workflow_step_executor_remote pid=9286) 2022-08-19 16:14:52,046 INFO step_executor.py:523 -- Step status [SUCCESSFUL] [c8e8f110-8c70-45e5-9682-6006be0a4eca.1660896891.953237057@__main__.minus] │
│(_workflow_step_executor_remote pid=9289) 2022-08-19 16:14:52,053 INFO step_executor.py:350 -- Step status [RUNNING] [c8e8f110-8c70-45e5-9682-6006be0a4eca.1660896891.953237057@__main__.multiply] │
│(_workflow_step_executor_remote pid=9289) 2022-08-19 16:14:52,055 INFO step_executor.py:523 -- Step status [SUCCESSFUL] [c8e8f110-8c70-45e5-9682-6006be0a4eca.1660896891.953237057@__main__.multiply]
nothing returns, and log I printed all empty data, how to. debug?
Add DAG.sequential
Sequential workflow is common, and it is helpful to add a convenient api to make it simple.
For example, for a DAG of a -> b -> c -> d -> e
Currently we have:
dag.add_edge(a, b, 0)
dag.add_edge(b, c, 0)
dag.add_edge(c, d, 0)
dag.add_edge(d, e, 0)
new api would be:
dag.sequential([a,b,c,d,e])
To do so, items to be done:
- make default value 0 to add_edge's in_arg_mapping parameter
- create sequential api which under the hood invoke multiple add_edge
add tests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.