PD needs stop uncontrollable expandsion

[benmaoer]

Bug Report

What version of Kubernetes are you using?
v1.16

What version of TiDB Operator are you using?

/ # /usr/local/bin/tidb-scheduler --version
TiDB Operator Version: version.Info{GitVersion:"v1.0.6", GitCommit:"982720cd563ece6dbebfc4c579b17fa66a93c550", GitTreeState:"clean", BuildDate:"2019-12-27T16:53:44Z", GoVersion:"go1.13", Compiler:"gc", Platform:"linux/amd64"}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
not about pvc

What's the status of the TiDB cluster pods?

What did you do?
When the cluster was initialized, the PD had 3 pods, demo-pd-{0,1,2}. After demo-pd-1 failed and was unrecoverable, one by one, the PD cluster expanded to 7 nodes.

What did you expect to see?
Expand 1 PD pod to replace crash one.

What did you see instead?
PD cluster expanded to 7 nodes.

[DanielZhangQD]

@benmaoer Thanks for reporting this issue!
When one PD Pod is down, after 5 minutes by default, tidb-operator will create a new pod to mitigate the failure.
We have already limited the number of Pods triggered by failover for tidb and tikv, we will add the limit for PD in future release. cc @weekface

[weekface]

Expand 1 PD pod to replace crash one.

The failover feature should be refactor with AdvancedStatefulSet to replace failed pod in place. This feature may be targeted at v1.2.x

[DanielZhangQD]

This issue will be fixed in v1.1.0-rc.2 with PR #2191.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days