Comments (4)
@Future-Outlier this is already being addressed, will assign accordingly. Thanks for the initiative!
from flyte.
🤖
Hello @hamersaw!
I'm Dosu, a bot designed to assist you with bugs, answer your queries, and help you journey through the world of open-source contribution. While waiting for a human maintainer, I'm here to make sure your issue doesn't go unnoticed.
I'm taking a quick look at your issue regarding the ActiveNodeExecutions
and ActiveTaskExecutions
metrics. We'll delve into it in a moment. Thanks for your patience!
from flyte.
One proposal, is to move these metrics to FlytePropeller so they can more accurately reflect actual execution status. A relatively high-level design is as follows:
- Add
NodeExecutionCount
andTaskExecutionCount
values to theExecutionContext
through the ControlFlow struct and increment these accordingly as FlytePropeller progresses through DAG execution (entrypoint here). - Maintain an in-memory mapping of
execution_id
to node and task execution counts that is updated as FlytePropeller evaluations executions. This update will in-turn update the prometheus gauge metric. - Use a separate go routine to periodically iterate over the in-memory mapping and remove executions that have been manually deleted (decrementing the gauge metric).
With this solution the metric under FlytePropeller restarts will be eventually consistent. Although, as the workflow reeval duration is, by default, 30s
it should converge within that timeframe.
from flyte.
@hamersaw Are you working on this issue?
If not, can I try it?
from flyte.
Related Issues (20)
- [Core feature] Flytekit should support using output with `Non-Any` type as the input with `Any` type. HOT 5
- [BUG] Retriability of timeouts appears inconsistent HOT 2
- [BUG] `PanderaTransformer::to_python_value()` seems to be returning an incorrect type HOT 2
- [BUG] flytectl upgrade is broken after moving to the monorepo HOT 2
- [BUG] Pin fsspec<2024.5.0 HOT 2
- [BUG] Namespace creation fails with default pod template HOT 5
- [BUG] flytectl demo start fails with "Error: malformed version" HOT 2
- [Docs] Clarify PodTemplate restrictions and behavior HOT 2
- [Docs] Prevent using mutable default arguments in flytesnacks HOT 1
- [Core feature] Replace `os.path` with `pathlib` for flytekit HOT 1
- Obfuscate sensitive data in TaskConfig HOT 4
- [BUG] Fix non thread safe token cache behavior HOT 1
- [Core feature] Flyteadmin SMPT email publisher HOT 1
- [BUG] rshift '>>' operator doesn't work properly with remoteEntities HOT 2
- [Core feature] Allow flytectl to set a targetExecutionCluster HOT 1
- [BUG] Boolean values within pydantic base model being casted to scalar value HOT 1
- [Housekeeping] Files used in `data_types_and_io.normalize_csv_file` and `data_types_and_io.download_and_normalize_csv_files` are no longer accessible HOT 6
- [Core feature] Default task resource behavior should apply for node level overrides HOT 2
- [Core feature] Update/register multiple launch plans with different inputs HOT 1
- [BUG] (Kubeflow) PyTorchPlugin sets Replicas to 0 casuing infinite loop HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flyte.