I believe I am seeing this too. I have this annotation set on a K8s job. When that job gets replaced, it happens twice therefore the job runs twice even though it succeeds the first time.

Having done a little more testing, I have found that the job gets recreated twice when something else has the annotation: argocd.argoproj.io/sync-wave: "-1"

To test this out I created an app that installs a custom helm chart. That helm chart creates two things, a configmap and a job. The job runs a hello world container.

If I have no annotations, ArgoCD can not update the job. To accommodate this I add the annotation argocd.argoproj.io/sync-options: Replace=true,Force=true to the job. Everything works as expected if I change the image tag for that job via a parameter override and the job gets replaced once.

If however I add the annotation argocd.argoproj.io/sync-wave: "-1" to the configmap, then when I update the jobs image tag, ArgoCD will replace the job twice causing it to run twice.

Here are some additional observations:

• If I set the configmap's annotation to argocd.argoproj.io/sync-wave: "0", things work as expected.
• If I set the job to have annotation to argocd.argoproj.io/sync-wave: "1", the job gets replaced twice.

I am running ArgoCD version v2.12.3+6b9cd82

I am observing exactly the same issue.

Contrary to @mmclane I did not find that setting same argocd.argoproj.io/sync-wave on all resources in the application resolves this. What I observe instead is following:

1. If a Job has argocd.argoproj.io/sync-options: Force=true,Replace=true and argocd.argoproj.io/sync-wave: 1 , and there are other resources with argocd.argoproj.io/sync-wave: 0 in the Application, then 2 Job objects are created with the same name, in about 3 seconds interval of each other. Each of these Jobs spawns a duplicate Pod and the first one is immediately set to the Terminating state.
2. If a Job has argocd.argoproj.io/sync-options: Force=true,Replace=true and argocd.argoproj.io/sync-wave: 0 basically same thing is happening. Only difference is that interval between the Jobs is shorter (1 second or maybe less), so it's harder to observe.

This is how it looks in kubectl get pods:

I would also expect that setting ServerSideApply=false flag on an Application or Resource level should fix this (since Argo would internally use kubectl instead of kubernetes library). But ServerSideApply=false seems to have no effect on this bug.

@rafal-jan any chance for having your PR merged?

@rafal-jan or @mmclane did you find any workaround to this issue?

I am running ArgoCD version v2.14.11+8283115