Comments (7)
chrisgorgo
commented on August 20, 2024
We have a few more jobs hanging indefinitely due to lack of this constraint. They were hard to debug because cloud watch logs seemed to be purged.
from openneuro.
nellh
commented on August 20, 2024
@chrisfilo CloudWatch should keep logs until the compute environment is recreated. Maybe these jobs never started? Logs are not created until it enters the running state. I have seen this with some jobs that hung the host container but not recently.
from openneuro.
chrisgorgo
commented on August 20, 2024
Here's an example of a recent job that I terminated because it run too long
https://console.aws.amazon.com/batch/home?region=us-east-1#/jobs/queue/arn:aws:batch:us-east-1:488777458602:job-queue~2Fopenneuro-prod/state/FAILED/job/e996ae5a-cab2-405a-99f0-941da4e7bedc/queue/arn:aws:batch:us-east-1:488777458602:job-queue~2Fopenneuro-prod/state/FAILED
The longs do not seem to be present in cloud watch, but maybe I am missing
something.
…On Thu, Sep 7, 2017 at 9:28 AM, Nell Hardcastle ***@***.***> wrote:
@chrisfilo <https://github.com/chrisfilo> CloudWatch should keep logs
until the compute environment is recreated. Maybe these jobs never started?
Logs are not created until it enters the running state. I have seen this
with some jobs that hung the host container but not recently.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAOkp0SRwu62kWzolrO0397AJ-IA79wgks5sgBmogaJpZM4PK_nK>
.
from openneuro.
nellh
commented on August 20, 2024
@chrisfilo Ah, I see what's going on. When Amazon changed the CloudWatch log names, they didn't fix the Batch UI, so that button in Batch is currently broken. You can find them but it's a pain, you have to search for something like FMRIPREP/default/<taskARN> which is not a value you can see without making API calls.
from openneuro.
chrisgorgo
commented on August 20, 2024
Ok - thanks for the clarification!
…On Thu, Sep 7, 2017 at 9:45 AM, Nell Hardcastle ***@***.***> wrote:
@chrisfilo <https://github.com/chrisfilo> Ah, I see what's going on. When
Amazon changed the CloudWatch log names, they didn't fix the Batch UI, so
that button in Batch is currently broken. You can find them but it's a
pain, you have to search for something like FMRIPREP/default/ which is not
a value you can see without making API calls.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAOkp6V_2lfMsBKZUOQzVYRjGD0ZJABRks5sgB2qgaJpZM4PK_nK>
.
from openneuro.
chrisgorgo
commented on August 20, 2024
One of our apps is running into a deadlock issue and runs forever. It would be great to increase the priority of this feature.
from openneuro.
chrisgorgo
commented on August 20, 2024
Having the major performance bottle neck solved I would like to propose prioritizing this issue:
- we are seeing users confused by jobs running for days
- those jobs are also consuming resources
Since bids-core performance has improved maybe this issue #109 could be replaced by the 48h cutoff one in the current sprint. @JohnKael
from openneuro.
Related Issues (20)
- datalad unable to get latest data from ds003745 HOT 2
- Snapshot generation failure: ds005115 HOT 1
- DOI minting failure: ds005166 HOT 1
- SyntaxError: Invalid or unexpected token (again -#2902?) HOT 3
- Uploading new file through web interface creates new dataset
- Unable to download dataset with openneuro-cli
- Login through ORCID returns to entry page with "Sign in" button
- Upload to ds005250 failing HOT 12
- Annex object requests should throw an error when made to the wrong worker
- fetching and pushing through git HOT 5
- pre-receive hook is not tolerant for an initial commit when branch is not even there
- `datalad get` failures on subset of files in ds005250
- Unable to install openneuro cli HOT 3
- Add NIRS and Motion modalities to search
- ezBIDS transfer failing HOT 7
- ORCID deny workflow error message not shown
- ORCID suggested improvements for login workflow
- Draft file view can fail with reasonable committed files HOT 1
- uploading ds005295 using openneuro CLI HOT 4
- bids validation hanging on ds005123? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openneuro.