[feature] Add support for podGroup number limits for one queue

[jiaxuanzhou]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature
/kind feature

What happened:
For scenarios of BigData and ML, the batch jobs often summited in queues, we need to provide the capacity to control the number of running pod groups in queues with limited hardware resources.
What you expected to happen:
Add a parameter in Queue API:

type QueueSpec struct {
	Weight int32 `json:"weight,omitempty" protobuf:"bytes,1,opt,name=weight"`
         //  PodGroupNumber defines the max number of running podGroup in one queue, default to infinite.
        PodGroupNumber `json:"podgroupnumber,omitempty" protobuf:"bytes,1,opt,name=podgroupnumber"`
}

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[jiaxuanzhou]

/kind feature

[k82cn]

/sig scheduling
/milestone v0.3
/assign

[hex108]

@k82cn I could start to work on it using admission webhook if you have not started.

[k82cn]

/assign hex108

I'm not working on that, please go ahead :)

[k82cn]

/sig apps

[jiaxuanzhou]

@hex108 it is perfect if you also consider the resource quota of one podGroup queue, default to be infinite.

type QueueSpec struct {
	Weight int32 `json:"weight,omitempty" protobuf:"bytes,1,opt,name=weight"`
         //  PodGroupNumber defines the max number of running podGroup in one queue, default to infinite.
        PodGroupNumber `json:"podgroupnumber,omitempty" protobuf:"bytes,1,opt,name=podgroupnumber"`
        ResourceQuota *Resource `json:"resourcequota,omitempty`
}

[hex108]

@jiaxuanzhou There is no resource quota for PodGroup, is there any use case for it?

[jiaxuanzhou]

@jiaxuanzhou There is no resource quota for PodGroup, is there any use case for it?
the resource quota is for the queue of the pgs
in prod, the users are teams to run the jobs, but the resources can not be shared fairly even we use the namespace queue
so we can restrict them by claiming the resource limits for the queues in a manual way.

[hex108]

@jiaxuanzhou You might need something like minResource that mentioned at #813 (comment)?

[jiaxuanzhou]

@hex108 actually , I need maxResource for one queue

[hex108]

@hex108 actually , I need maxResource for one queue

Thanks. Get it now! We'll add it. cc @k82cn

[jiaxuanzhou]

@hex108 thx, 👍

[jiaxuanzhou]

/reopen

[k8s-ci-robot]

@jiaxuanzhou: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k82cn]

@jiaxuanzhou , what's the purpose of maxResources? are you trying to avoid too many pending pods? If so, "delay pod creation" maybe helpful.

[jiaxuanzhou]

@jiaxuanzhou , what's the purpose of maxResources? are you trying to avoid too many pending pods? If so, "delay pod creation" maybe helpful.

maxResource defines the threshold of the resource for one queue, once the threshold reached, the subsequent jobs will still stay in pending status even the idle resource of the cluster can satisfy them.

[k82cn]

maxResource defines the threshold of the resource for one queue, once the threshold reached, the subsequent jobs will still stay in pending status even the idle resource of the cluster can satisfy them.

Similar to Quota but for Job and Queue, right? Admission controller + Queue controller maybe better :)

[jiaxuanzhou]

maxResource defines the threshold of the resource for one queue, once the threshold reached, the subsequent jobs will still stay in pending status even the idle resource of the cluster can satisfy them.

Similar to Quota but for Job and Queue, right? Admission controller + Queue controller maybe better :)

yeah, Quota for queue.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.