scheduling performance drop with a large amount of pods existing

[elinx]

What happened:
deploying 3k pods in a cluster with 8k existing pods costs much more time than in an empty cluster, like a 40% performance drop.

What you expected to happen:
same scheduling time or less than a 5% performance drop.

How to reproduce it (as minimally and precisely as possible):

1. deploy 8k pods in a cluster
2. deploy another 3k pods
3. statistics the scheduling time

Anything else we need to know?:

Environment:

• Volcano Version: 1.6.0
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others: