Ingress controller should react to node scale down event from autoscaler

[freehan]

Cluster autoscaler adds the ToBeDeletedByClusterAutoscaler taint on the candidate node. Then it goes on to evict pods from the nodes. After everything settles, it will delete the node.

Ingress controller should observe the taint and react by removing instance from instance group so that connection draining is triggered. This help avoid new connection to be arrive at the removing nodes which causes 502s.

[retpolanne]

Hi, I would like to grab this issue, if no one is working on it already.

[retpolanne]

@freehan I might be looking at it wrong, but I think the problem is not on ingress-gce itself.

Over here, on GetNodeConditionPredicate (which is used to filter nodes during sync), all Unschedulable nodes are marked out of the filter.

However, over here the node is only tainted as NoSchedule.

IMO the autoscaler should also mark the node as Unschedulable. I'm writing a few tests on the autoscaler for that.

[freehan]

Thanks for the PR!

[freehan]

Yes. As you pointed out, the problem is the handshake between autoscaler and ingress-gce (or any other loadbalancer controller). This required a better design for synchronization between cluster node life cycle and load balancer controller.

But for now, as a stop gap, we just need to watch autoscaler's taint and react.

[philpearl]

Would really like this fix in GKE - can anyone comment on how long it will take before it's available?

[freehan]

We will cherry pick this into 1.6 branch. And it will follow the GKE release pipeline and possibly available in newer version of 1.13.8+