Comments (6)
From #4246
Me:
Presumably we could figure out what the matched tasks or family members will be, and put those in the future hold list.
The suggestion of expanding families is orthogonal and was previously rejected due to issues with multiple flows and graph branching.
I've kinda forgotten that conversation, sorry Oliver. But maybe we made the wrong call (or you can correct me again).
It is possible of course, but would require more thought, for consistency it should apply to all of the "task glob" commands, but we would need to think about how that would interact with cylc reset, there may be a housekeeping problem with graph branching (if an action is run against a task which isn't run, then we will accumulate messages causing a slow memory leak unless housekept), and if expanding families, why not cycles?
We could, initially at least, only implement this for families. A family name is simply than a glob, and less risky. There's no wildcard involved, the members are well-defined.
Ideally we could do it with globs too, but there is as clear use case (as raised by Tom) for family hold, and we could implement that first.
The other issue you raise there also applies to individual tasks, right? I can hold a future task, by ID, that might not ever appear in the pool because of future graph branching. If we don't make housekeeping possible somehow, that would cause a memory leak (the list of future tasks to hold), but any such leak should be small and inconsequential - it would only grow as a result of a small number of manual intervention commands.
from cylc-flow.
I've kinda forgotten that conversation, sorry Oliver. But maybe we made the wrong call
Possibly, I'm not particularly happy with the status quo which is definitely a backwards step from Cylc 7 so happy to re-open this one. For reference the list of commands which currently "glob" for tasks are:
- hold
- kill
- pause
- ping
- poll
- release
- remove
- set_outputs
- show
- trigger
We would need to make sure any proposed functionality could be adapted for all of these. Note, currently only the "trigger" command will auto-spawn a task.
Off the top of my head, the issues with doing this are:
- Should probably have some form of housekeeping for operations.
- Could couple to the runahead limit?
- Operations apply to all selected tasks irrespective of whether they would be run or not (according to graph logic).
- Perhaps irrelevant providing we have housekeeping?
- Operations applied to one flow could linger and affect a subsequent flows.
- Housekeeping might help to narrow down the window of opportunity for this.
- Alternatively we could add the
--flow
argument to these commands and default as--flow=all
. If we expand thatall
into the corresponding flow numbers at the time the command is processed then we could limit the scope of the operation so that subsequent flows are not affected.
- Require some form of visibility for requested operations which are waiting for tasks to be spawned before taking effect.
- No idea on this one.
from cylc-flow.
IMO, we should come up with a general policy for all these commands rather than working around the limitations in each command individually e.g. #5677.
from cylc-flow.
The following issues all concern the limitation of multi-task commands where globs/families are only expanded in the context of the task pool.
If we are going to address this we should come up with a consistent approach which works for the relevant commands so we can implement and document it consistently rather than hashing out the details for each command separately.
This is also somewhat related to:
Although this issue will not close those by itself as this is a different problem related to display and selection.
from cylc-flow.
IMO, we should come up with a general policy for all these commands rather than working around the limitations in each command individually e.g. #5677.
If we are going to address this we should come up with a consistent approach which works for the relevant commands so we can implement and document it consistently rather than hashing out the details for each command separately.
If possible, but issues typically first come to light with individual commands - particular when reported by users as is the case with all of these - and need to be documented (Issues posted, I mean) as they arise. We can certainly consolidate and triage later, if appropriate.
And in the context of this particular issue, "hold" is kind of a special case - we can already hold future tasks individually, which means maintaining a list of future tasks that will be held if/when they get spawned. The problem is simply that we need to be able to hold future tasks collectively - by family name or pattern match.
I can't see much commonality with the other commands you've listed above, in terms of teeing up future behaviour.
Note, currently only the "trigger" command will auto-spawn a task.
And set-outputs
(soon to be set
).
from cylc-flow.
Superseded by #5827
from cylc-flow.
Related Issues (20)
- vr: multi workflow support
- graph parser: absolute dependency offset on the RHS HOT 5
- tdef: graph parents - absolute dependencies ignored HOT 1
- Wrapper script sets `PATH` when running `cylc hub` which breaks version selection for `cylc play` in UI HOT 1
- Double-reload if command in job script HOT 4
- cylc lint - site/workflow specific rules? HOT 4
- `cylc clean` timeout traceback
- examples: add more example workflows
- Command UUID should be supplied by the client.
- cylc config shows unhelpful information for xtriggers
- flaky test: tests/unit/test_flow_mgr.py::test_all
- optional outputs: tasks which complete outputs on multiple tries marked as incomplete HOT 3
- cylc message: three or more outputs cause problems HOT 3
- Better output formatting for task-listing commands
- Script (and similar) sections strip out everything after # HOT 1
- Clearing execution time limit does not work HOT 3
- VIP: Fails if both `--workflow-name` & `--run-name` are set
- resolvers: reduce the impact of to_snake_case HOT 2
- Don't record satisfied clock triggers in the DB HOT 1
- Start paused or held, then manually trigger individual tasks HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cylc-flow.