Bug Description
I faced multiple issues trying to enable sync from a private repository.
- Specifying the private key during the charm deployment doesn't work, the private key file is not created. After resetting and setting again to the same value, the file is created.
- git-sync command fails because the known_hosts file path doesn't exist.
Now the following issues were encountered after fixing the above with juju ssh
workarounds (all commands listed below).
- Remote server's SSH key is not auto-accepted/ignored by git-sync called by the charm and the sync action fails.
- Private SSH key has incorrect permissions 0644 instead of 0600 (or even more restrictive).
Additionally, there's no validation of the private key, after setting the option value using =$(cat id_ecdsa)
results in a file without a newline at the end, using the =@id_ecdsa
syntax works. Because of that, the sync action may also fail as it will report that the key file has an invalid format.
To Reproduce
- Deploy:
juju deploy cos-configuration-k8s --config git_repo=git+ssh://[email protected]/~redacted/+git/redacted --config git_branch=main --config git_depth=1 --config git_ssh_key="$(cat redacted.key)" cos-configuration
- Try to sync:
juju run-action cos-configuration/0 sync-now --wait
...
log:
- 2023-07-07 07:24:46 +0000 UTC Calling git-sync with --one-time...
- '2023-07-07 07:24:46 +0000 UTC ERROR: can''t configure SSH: can''t access SSH
key: stat /run/cos-config-ssh-key.priv: no such file or directory'
message: 'Sync error: Exited with code 1.'
- Reset the config option and set it to the same value again.
- Attempt to sync again, the key is there now but the action fails again:
juju run-action cos-configuration/0 sync-now --wait
...
log:
- 2023-07-07 07:29:59 +0000 UTC Calling git-sync with --one-time...
- '2023-07-07 07:29:59 +0000 UTC ERROR: can''t configure SSH: can''t access SSH
known_hosts: stat /etc/git-secret/known_hosts: no such file or directory'
message: 'Sync error: Exited with code 1.'
- Created empty known_hosts file manually in an attempt to work this around:
juju ssh --container git-sync cos-configuration/0 mkdir /etc/git-secret/
juju ssh --container git-sync cos-configuration/0 ls -l /etc/git-secret/known_hosts
- Synced again, this time it failed on
Host key verification failed.
juju run-action cos-configuration/0 sync-now --wait
...
log:
- 2023-07-07 07:40:05 +0000 UTC Calling git-sync with --one-time...
- 2023-07-07 07:40:05 +0000 UTC I0707 07:40:05.187060 146 main.go:473] "level"=0
"msg"="starting up" "pid"=146 "args"=["/git-sync","--repo","git+ssh://[email protected]/~redacted/+git/redacted","--branch","main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--ssh","--ssh-key-file","/run/cos-config-ssh-key.priv","--one-time"]
- 2023-07-07 07:40:05 +0000 UTC I0707 07:40:05.187213 146 main.go:923] "level"=0
"msg"="cloning repo" "origin"="git+ssh://[email protected]/~redacted/+git/redacted"
"path"="/git"
- '2023-07-07 07:40:05 +0000 UTC E0707 07:40:05.303971 146 main.go:525] "msg"="too
many failures, aborting" "error"="Run(git clone -v --no-checkout -b main --depth
1 git+ssh://[email protected]/~redacted/+git/redacted /git):
exit status 128: { stdout: "", stderr: "Cloning into ''/git''...\nHost key verification
failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you
have the correct access rights\nand the repository exists.\n" }" "failCount"=0'
message: 'Sync error: Exited with code 1.'
- I assumed git-sync prompts to accept the remote key so I ran the same command via
juju ssh
:
juju ssh --container git-sync cos-configuration/0 "/git-sync --repo git+ssh://[email protected]/~redacted/+git/redacted --branch main --rev HEAD --depth 1 --root /git --dest repo --ssh --ssh-key-file /run/cos-config-ssh-key.priv --one-time"
It did and I typed yes
:
The authenticity of host 'git.launchpad.net (185.125.188.44)' can't be established.
RSA key fingerprint is SHA256:UNOzlP66WpDuEo34Wgs8mewypV0UzqHLsIFoqwe8dYo.
Are you sure you want to continue connecting (yes/no)? yes
It failed again after that:
E0707 07:44:58.716917 181 main.go:525] "msg"="too many failures, aborting" "error"="Run(git clone -v --no-checkout -b main --depth 1 git+ssh://[email protected]/~redacted/+git/redacted /git): exit status 128: { stdout: "", stderr: "Cloning into '/git'...\nWarning: Permanently added 'git.launchpad.net,185.125.188.44' (RSA) to the list of known hosts.\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: UNPROTECTED PRIVATE KEY FILE! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nPermissions 0644 for '/run/cos-config-ssh-key.priv' are too open.\r\nIt is required that your private key files are NOT accessible by others.\r\nThis private key will be ignored.\r\nLoad key \"/run/cos-config-ssh-key.priv\": bad permissions\r\[email protected]: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n" }" "failCount"=0
The private key file created by the charm has incorrect permissions 0644 instead of 0600.
- I changed permissions manually using
juju ssh
to work this around:
juju ssh --container git-sync cos-configuration/0 chmod 0600 /run/cos-config-ssh-key.priv
- Tried the sync again and it finally worked:
juju run-action cos-configuration/0 sync-now --wait
unit-cos-configuration-0:
UnitId: cos-configuration/0
id: "46"
log:
- 2023-07-07 07:49:13 +0000 UTC Calling git-sync with --one-time...
- '2023-07-07 07:49:15 +0000 UTC Warning: I0707 07:49:13.172126 270 main.go:473]
"level"=0 "msg"="starting up" "pid"=270 "args"=["/git-sync","--repo","git+ssh://[email protected]/~redacted/+git/redacted","--branch","main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--ssh","--ssh-key-file","/run/cos-config-ssh-key.priv","--one-time"]'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:13.172225 270 main.go:923]
"level"=0 "msg"="cloning repo" "origin"="git+ssh://[email protected]/~redacted/+git/redacted"
"path"="/git"'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:13.886433 270 main.go:737]
"level"=0 "msg"="syncing git" "rev"="HEAD" "hash"="3fecae3005ecf7eb23fd7e748b9999eb6ead91cb"'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:14.839919 270 main.go:772]
"level"=0 "msg"="adding worktree" "path"="/git/3fecae3005ecf7eb23fd7e748b9999eb6ead91cb"
"branch"="origin/main"'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:14.845755 270 main.go:833]
"level"=0 "msg"="reset worktree to hash" "path"="/git/3fecae3005ecf7eb23fd7e748b9999eb6ead91cb"
"hash"="3fecae3005ecf7eb23fd7e748b9999eb6ead91cb"'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:14.845783 270 main.go:838]
"level"=0 "msg"="updating submodules"'
results:
git-sync-stdout: ""
status: completed
timing:
completed: 2023-07-07 07:49:19 +0000 UTC
enqueued: 2023-07-07 07:49:12 +0000 UTC
started: 2023-07-07 07:49:12 +0000 UTC
Environment
COS Lite on top of microk8s, charm from latest/edge
.
Relevant log output
Included in steps to reproduce.
Additional context
No response