Upgrade of flux marks all helmreleases as unreconciled

[brujoand]

Describe the bug

When I upgrade the flux version in my cluster by running a new bootstrap every helmrelease will be unreconciled until manually reconciled.

Steps to reproduce

1. Upgrade flux version by rerunning the bootstrap command
2. Wait for it to complete successfully
3. Notice that all helmreleases are now unreconciled
4. Write bash script to iterate through all helmreleasese an ask flux to reconcile them

Expected behavior

All helmreleases which where reconciled before the bootstrap should stay reconciled after the bootstrap.

Screenshots and recordings

No response

OS / Distro

Ubuntu 20.04

Flux version

0.16.2, just upgraded from 0.16.1

Flux check

► checking prerequisites
✔ kubectl 1.21.1 >=1.18.0-0
✔ Kubernetes 1.20.5 >=1.16.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.11.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.13.3
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.15.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.15.4
✔ all checks passed

Git provider

Github

Container Registry provider

No response

Additional context

This has happened for a while now after upgrades, and I can't find anything interesting in the logs. But it might help if I had some pointers on what to look for.

Code of Conduct

• I agree to follow this project's Code of Conduct

[hiddeco]

Can you notice a difference in the Status of a HelmRelease object post-upgrade?

[brujoand]

The status just changes to 'namespace/release' is not ready

[nvanheuverzwijn]

I noticed that if the helm-controller is heavily used, the liveness check may fail and will restart the pod. This may be what is happening.
@brujoand I suggest that you check if the helm-controller pod has restarted. If it has, this may be why you see weird behavior i.e. chart stuck in a deployed state, all helmrelease being unreconciled, etc.
Hope you found out what was going on :)

[stefanprodan]

Fixed by fluxcd/helm-controller#738