SSH host key errors with v0.23.0 #378
Comments
|
hi @jmriebold, could you share more information about this, like the contents of the |
|
I encounter the same issue here with Github and I'm using I refreshed my |
|
@jmriebold is it just IAC which is failing like this, or do you observe the same issue in source-controller as well? |
|
@ViBiOh IAC can't use |
The reconciliation problem only occurs on ImageAutomation, here an example of logs |
|
@aryan9600, yes, like @ViBiOh it's only the IAC which is failing. All other pods are working as expected. Our known_hosts file hasn't changed for about 2 years (nor have the server's keys, to be clear), so that seems unlikely to be the issue, especially since the source-controller and other pods are working fine. |
|
I am having the same issues with IAC, worked yesterday before my upgrade to 0.31.0. Interestingly I am only seeing this on the IAC, but if I switch the git repo over to |
|
I enabled debug logging on the source controller and the debug logs simply echoed the error logs. As a workaround I simply reverted the IAC to |
|
I can't reproduce this with ECDSA, here is my secret: |
|
To reproduce this we'll need the details on how you've create the SSH key, I guess it wasn't generated with the Flux CLI. |
|
I just created my key a couple of days ago for this so I used: ssh-keygen -t ecdsa -a 100 -f priv_key -C flux@example.com -b 521I generated the known_hosts with identity: identity.pub: |
|
I have the same kind of SSH key (ed25519), generated manually and pushed into the git repository secret. |
|
So, I have two First one is in namespace ---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: infra
namespace: monitoring
spec:
[ ... ]
sourceRef:
kind: GitRepository
name: infra
namespace: flux-system
update:
path: flux/k3s/monitoring/
strategy: Setters
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: infra
namespace: flux-system
spec:
gitImplementation: go-git
interval: 120m
ref:
branch: main
secretRef:
name: github-sshAnother automation with all things in the namespace ---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: flux-goweb
namespace: default
spec:
[ ... ]
sourceRef:
kind: GitRepository
name: flux-goweb
namespace: default
update:
path: ./infra
strategy: Setters
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: flux-goweb
namespace: default
spec:
gitImplementation: go-git
interval: 120m
ref:
branch: main
secretRef:
name: github-sshI've removed extra details from the |
Previously, KnownKey.Matches() accepted a SHA256 hasher as an argument, which could lead to unintended bugs when calling it in a loop. This eliminates that, by intializing a new hasher itself instead of relying on the caller for the same. Enables us to fix a regression in the source-controller: fluxcd/image-automation-controller#378 Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Previously, KnownKey.Matches() accepted a SHA256 hasher as an argument, which could lead to unintended bugs when calling it in a loop. This eliminates that, by initializing a new hasher itself instead of relying on the caller for the same. Enables us to fix a regression in the source-controller: fluxcd/image-automation-controller#378 Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Previously, KnownKey.Matches() accepted a SHA256 hasher as an argument, which could lead to unintended bugs when calling it in a loop. This eliminates that, by initializing a new hasher itself instead of relying on the caller for the same. Enables us to fix a regression in the source-controller: fluxcd/image-automation-controller#378 Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Earlier, host key verification could potentially fail if there were multiple entries in the known_hosts file and if the intended encryption algorithm wasn't the first entry. This happened because we used the same hasher object to compute the sum of all the public keys present in the known_hosts file, which led to invalid hashes, resulting in a mismatch when compared with the hash of the advertised public key. This is fixed, by not creating the hasher ourselves and instead delegating that to the function actually doing the matching, ensuring that a new hasher is used for each comparison. Regression introduced in v0.25.0 and reported in fluxcd/image-automation-controller#378 Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Earlier, host key verification could potentially fail if there were multiple entries in the known_hosts file and if the intended encryption algorithm wasn't the first entry. This happened because we used the same hasher object to compute the sum of all the public keys present in the known_hosts file, which led to invalid hashes, resulting in a mismatch when compared with the hash of the advertised public key. This is fixed, by not creating the hasher ourselves and instead delegating that to the function actually doing the matching, ensuring that a new hasher is used for each comparison. Regression introduced in v0.25.0 and reported in fluxcd/image-automation-controller#378 Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Earlier, host key verification could potentially fail if there were multiple entries in the known_hosts file and if the intended encryption algorithm wasn't the first entry. This happened because we used the same hasher object to compute the sum of all the public keys present in the known_hosts file, which led to invalid hashes, resulting in a mismatch when compared with the hash of the advertised public key. This is fixed, by not creating the hasher ourselves and instead delegating that to the function actually doing the matching, ensuring that a new hasher is used for each comparison. Regression introduced in v0.25.0 and reported in fluxcd/image-automation-controller#378 Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
|
We have just released a new image with the fix: |
|
Hello 👋 I've tested it and all automations are resolving now 👍 Thank you for the quick fix ;) |
|
Thanks all! |
Updated to Flux v0.31.0 today (from v0.30.2) and as soon as there was something to commit, the image-automation-controller immediately started throwing errors about the SSH host key failing verification. For example:
{"level":"error","ts":"2022-06-07T02:38:41.735Z","logger":"controller.imageupdateautomation","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"flux-system","namespace":"flux-system","error":"unable to fetch-connect to remote 'ssh://git@git.company.com/repo': ssh: handshake failed: hostkey could not be verified"}For context, our Flux repo is configured to use SSH, and we supply a known_hosts file along with the SSH key in our flux-system secret. In addition, we've been running the image-automation-controller (along with the others) with
EXPERIMENTAL_GIT_TRANSPORT=true, so I'm a little surprised that things broke with this update instead of back when we enabled this functionality.The text was updated successfully, but these errors were encountered: