Retry on leader lease renewal failure

[davidjumani]

Description

After a container has become a leader, any Kube API server unavailability results in the container crashing. This happens as the leader is unable to renew the lease. This is by design as outlined here

if we allowed gloo to continue to function as a leader during kube apiserver outage, we risk having two leaders in other failure modes. we should remove the panic and allow gloo to continue to serve last-known xds as a follower (effectively having two followers until kube apiserver recovers). this idea is similar to the role xds relay could play for gloo edge

However, crashing the gloo pods can also lead to an outage during scaling as the gateway-proxy pods that come up cannot fetch any configs and all routes result in 404s.

Now instead of crashing, the gloo pod falls back to a follower. This prevents an outage and any other pod can take over as leader if possible
This is fine as a leader only writes reports / statuses over here and here. On any failure, the pod becomes a follower and if elected back as a leader will continue to write reports.

Code changes

• Introduce a new Reset method on the identity implementation that allows an identity to fall back to a follower

CI changes

• Kind clusters that run in CI now do not come up with their own CNI as kindnet does not support custom network policies.
  Instead, cilium is installed as CNI as we need to test Kube API server unavailability

Context

Kube API unavailability results in a gloo container crash
When leader election fails, gloo crashes
Design Doc

Interesting decisions

Testing steps

Kube2e tests to verify the following :

• It can recover from a Kube APIServer unavailability
• It falls back to a follower
• Another leader can get elected

Notes for reviewers

Be sure to verify intended behavior by ...

Please proofread comments on ...

This is a complex PR and may require a huddle to discuss ...

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

BOT NOTES:
resolves #8107

[github-actions]

Visit the preview URL for this PR (updated for commit 7cf3c2b):

https://gloo-edge--pr9563-dont-crash-on-failed-6s4x9piv.web.app

_{(expires Tue, 25 Jun 2024 00:48:42 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

[ashishb-solo]

thanks @davidjumani - i've read it over a few times and have a few questions. i think a little more documentation (especially around the go funcs) would be helpful. also, a major question regarding whether we still have the regular behavior if we're not configured to recover from failure.

happy to talk in person if that would be easier

ps: i haven't had a chance to look at the test yet

[sam-heilbron]

looking good! I mainly focused on how we're testing for now, and once that is updated, I'm happy to take a closer look at the implementation itself

[ashishb-solo]

looks good to me. one question on the comment from Sam regarding whether the goroutine is safe. if so, happy to ignore the nits to approve

[davidjumani]

kick bulldozer