Comments (5)
Yea this issue was already mentioned here: #54
I need to fix that but had not the time. If you can add a reconnect methodology then I will def accept a PR. If I get some time I'll fix that up but currently it is an open issue.
Feel free to close this issue when you are done.
from kubernetesjoboperator.
Hi, this looks like a kind of timeout.
Note that the restart policy should be never.
What is your system? Also, this seems like a timeout which is known to happen on Amazon kubernetes. See the other open conversation.
Can you run this locally with docker desktop kubernetes or some other local cluster?
Can you try this with 1 completion? And 1 fails?
Can you try running the example in the repo? Or the test in the repo?
I can only get to this mid next week. Apologies about that.
from kubernetesjoboperator.
Hi, @LamaAni,
Tks for reply...
I've changed the RestartPolicy to Never and the problem keeps happening.
I've tried to run with 1 competitions and doesn't work either.
Sorry but I don't know what your meant about running the example in the repo... Sorry I'm new on this. Could you explain me how to do it?
In meanwhile I'll setup and test localy as you suggested.
Tks
from kubernetesjoboperator.
Can you please test the example here:
from airflow import DAG
from airflow_kubernetes_job_operator.kubernetes_job_operator import KubernetesJobOperator
from airflow.utils.dates import days_ago
default_args = {
"owner": "tester",
"start_date": days_ago(2),
"retries": 0,
}
dag = DAG(
"job-operator-simple-test",
default_args=default_args,
description="Test base job operator",
schedule_interval=None,
)
KubernetesJobOperator(
task_id="very-simple-job",
dag=dag,
image="ubuntu",
command=[
"bash",
"-c",
"echo start; sleep 5; echo end",
],
)
The above should make a very fast image run - check if the timeout is the issue. If that passes, just increase the sleep time in there and you would get a test of the timeout limit on the cluster.
I forced in the code the restart policy to Never - but the yaml
should be correct anyways - this is since airflow should control the restarts. Completions should works, but in this case I would set that in airflow rather then the job... it would give you more proper logs.
The error sometimes comes from a timeout on the open connection with the Kubernets cluster (connection timeout forced by server) - hence testing locally may shed light on the issue.
from kubernetesjoboperator.
I've just test your proposed dag and the task end successfully when the sleep value was 5 seconds. After increasing to 300 seconds, the task ends with Connection reset by peer.
I think the issue is related to a know issue on AKS:
https://learn.microsoft.com/en-us/answers/questions/467642/no-response-from-api-for-long-running-jobs-in-aci
Since this problem is critical to the delivery of a project. We had to create our own custom operator to submit Kubernetes jobs. During the creation and testing process we came across a similar problem when we tried to stream read the logs generated by Kubernetes using the official python library for k8s.
We got around this problem by abandoning the log flow strategy and starting to make periodic requests.
from kubernetesjoboperator.
Related Issues (20)
- Duplicate pod logs HOT 4
- BUG: Delete pods after job completion HOT 6
- Connection reset by peer on long job run HOT 14
- Get pod logs when multiple containers are used HOT 7
- Uses cases and testimonials HOT 3
- Executing each task without restarting pods for every task HOT 7
- Is KubernetesJobOperator compatible with Apache Airflow >= v2.0? HOT 10
- FEATURE: Delete all completed tasks, regardless the final result of the task (successful or failed). HOT 24
- Investigate warnings HOT 3
- Secrets not found HOT 1
- FEATURE: Refresh aws token HOT 13
- Error after upgrading to airflow 2.3.3 HOT 2
- Cannot create resource "jobs" HOT 1
- No logs from pod HOT 3
- Any specific setup on kubernetes? HOT 22
- Running task logs is incomplete HOT 12
- FEATURE: need 'templates_dict' to ensure proper rendering of time macros. HOT 2
- FEATURE: TaskFlow / Decorated Task HOT 1
- Delete the job due to an error triggered by chunk.decode("utf8"). HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetesjoboperator.