Comments (6)
According to https://docs.aws.amazon.com/batch/latest/userguide/using_awslogs.html#create_awslogs_logdriver_options
the default is aws/batch/job
Moreover, check the usage here: #1767 (comment)
We supply try/batch/custom/demo
i.e. without the /
in beginning and it worked as shown in #1767 (comment)
from metaflow.
@stathva Did you have issues with running flows with what we have here i.e. aws/batch/job
without the /
in beginning?
from metaflow.
@stathva Did you have issues with running flows with what we have here i.e. aws/batch/job without the / in beginning?
@madhur-ob yes, we deploy Metflow Wofklows as StepFunctions, our workflow is using the @batch
decorator to define the container image and step resource requirements but it doesn't specify a log group.
The jobs failed with the following error:
"Reason": "CannotStartContainerError: Error response from daemon: failed to initialise logging driver: failed to create Cloudwatch log stream: InvalidParameterException: Log groups starting with AWS/ are reserved for AWS.",
So the problem is not missing /
in general like in the try/batch/custom/demo
example but using aws/*
since its reserved.
from metaflow.
Aah, I think the fix is to not include it at all I guess...let me remove it, that way, it will consider aws/batch/job
by default..
from metaflow.
This happened to me now for unknown reasons. The "fix" is to assign log_options=["awslogs-group:/aws/batch/job"] to every batch step in the step functions i create. Using metaflow 2.11.6.
from metaflow.
@simonhs87 Can you please try with 2.11.9
? There was one more fix made in #1786
from metaflow.
Related Issues (20)
- micromamba complain from @pypi_base about package incompatibility in intel macos HOT 13
- argo-workflows create --only-json doesn't export sensor configuration
- Argo Workflows Overrides Image in @kubernetes annotation. HOT 2
- Errors resuming a workflow HOT 2
- S3 External Client Option Ignored by s3op
- metaflow failed to resume a flow HOT 1
- `MetaflowData.__getattr__` should raise `AttributeError` on failure
- Request: Allow shadowing in metaflow_extensions HOT 5
- Metadata request (/flows/<flowname>) failed (code 502) HOT 8
- METAFLOW_S3_ENDPOINT_URL env variable not working in 2.12.17
- GPU constraint of 0 added to Kubernetes step spec, making it unsatisfiable
- Error on argo-workflows create HOT 3
- migrating databases HOT 1
- Hearbeat daemon creates a run out of production namespace HOT 2
- Prevent idle computer from sleeping while flow is running HOT 7
- AttributeError in Runner async_run and async_resume
- Buggy signal handler impl in CommandManager and SubprocessManager
- Set memory limit to equal memory request?
- Abstracting away SubcommandManager's signal handling
- Argo heartbeat-task-daemon using default KSA instead of METAFLOW_KUBERNETES_SERVICE_ACCOUNT
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metaflow.