Comments (10)
This error occurred for me when the permissions for the private key file were not sufficiently strict.
It was solved by running a command like chmod 600 id_rsa
inside the container.
For reference, the working configuration has an SSHHook created with
from airflow.contrib.hooks import SSHHook
ssh_hook = SSHHook(conn_id='ssh_conn')
where the connection ssh_conn
has properties:
conn_id='ssh_conn'
conn_type='SSH'
host='172.17.0.1'
login='username'
port=2222
extra='{"key_file": "/usr/local/airflow/.ssh/id_rsa", "no_host_key_check": true}'
from docker-airflow.
Hi,
Did you test a ssh connection inside container (outside airflow dag) ?
from docker-airflow.
Hi @puckel ,
The ssh connection was tested inside the container.
We have tried this method as we don't know how to make airflow docker container communicate with external hosts. For example to execute some hdfs or spark functions.
from docker-airflow.
(And it required creating a new docker container based on puckel/docker-airflow
that installs the package openssh-client
.)
from docker-airflow.
@expz thank you for your answer. But there is something that still unclear. do you generate the id_rsa key inside the container (worker?scheduler?webserver? or all?) or you generate it on the host machine and you define a volume in the docker-compose-CeleryExecutor file for each service to the id_rsa key that is found on the host machine ?
from docker-airflow.
Generate id_rsa on the host and mount the .ssh directory as a volume inside the container. This will guarantee persistence of the key. Otherwise the key will disappear if you destroy and remake the container.
from docker-airflow.
@expz docker is on a host with ip @ 1.2.3.4 and the host that i want my docker conatiner to coneect to is my hadoop cluster with ip @ 1.2.3.5.. when i use you suggested ip @ 172.17.0.1 it will connect to the docker0 but the docker it self is not connected somehow to the 2sn host. Did i miss something? how to make docker conect to the seconf host without install it there?
from docker-airflow.
@bdrjonny - did you resolve this? I am seeing the same error.
I am trying to use SSHExecuteOperator https://github.com/OpenGov/airflow/blob/master/airflow/contrib/operators/ssh_execute_operator.py#L74
and I create a task like :
self.arg_dev_task = SSHExecuteOperator(
task_id="arg_dev_task",
bash_command='sh /hostname/directory/test.sh',
ssh_hook=sshHook,
dag=dag)
but I get an error:
TemplateNotFound: sh /hostname/directory/test.sh
any idea whats happening? I am trying to execute a shell script on a remote host
(P.S: I have already created a SSH connection for the remote host in the airflow UI and I am passing that connection id in the ssh_hook)
when I pass bash_command = 'echo $hostname' then it gives me an error Failed to create remote temp file
from docker-airflow.
@sid88in @bdrjonny Did anyone found the solution.I have similar problem.
It says "Failed to create remote temp file"
from docker-airflow.
may not be directly related, just wasted five minutes on this myself, so will restate for others. known_hosts
make sure not to miss the extra json that @expz listed for disabling known host check. without that ( or a valid entry in known_hosts
) airflow will fail with the remote file error - it is stuck on the known hosts prompt.
if checking the remote key is required, you will need to request/accept the key as part of provisioning or persist the known_hosts
file with your SSH key.
this can sneak by manual testing, as you might test the connection, accept the prompt "confirming" that it works, retry in airflow again confirming that it works, and then lose the known hosts file on next spin-up.
this is not specific to docker-airflow
- it applies to any airflow environment that gets rebuilt with any frequency. for long-lasting environments, after the first success this is a non-issue, since known_hosts
will never need to be regenerated.
from docker-airflow.
Related Issues (20)
- Latest Image is not working on for mac M1 Chip HOT 3
- Airflow - DAG Jobs fails but didn't show any logs
- REDIS_HOST not being used in redis URL
- Celery worker container should wait for already running jobs to complete before shutting down on worker TERM signal
- Some workers seem to have died and gunicorn did not restart them as expected HOT 3
- Issue with Installing PyICU package - build fails HOT 1
- Error fetching Celery task state HOT 1
- flower start fail HOT 1
- How to keep Airflow Variables when restarting the docker image? HOT 2
- Update Airflow Version to 2.1.0 HOT 2
- Is there any case of running a py file?
- Airflow 2.0 attempt by Generally Unstable HOT 1
- sqlalchemy.ext.declarative.clsregistry HOT 1
- Launching tensorflow-gpu inside docker-airflow
- ModuleNotFoundError: No module named 'wtforms.compat' HOT 5
- No s3 conn type for connection! HOT 1
- Multiple vulnerabilities on airflow image. HOT 1
- /usr/bin/env: ‘bash\r’: No such file or directory HOT 1
- some workers seem to have died and gunicorn did not restart them as expected
- Detect and copy files in a shared windows folder from under the airflow docker container.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-airflow.