-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Trigger the workloads eviction on admission check rejection. #1562
Trigger the workloads eviction on admission check rejection. #1562
Conversation
✅ Deploy Preview for kubernetes-sigs-kueue canceled.
|
/cc @alculquicondor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a release note if this change makes a user-facing impact (which seems to be the case)
/approve |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/hold cancel
/approve cancel |
d7c44c3
to
89c647e
Compare
cqName, cqOk := r.queues.ClusterQueueForWorkload(&wl) | ||
if cqOk { | ||
if updated, err := r.reconcileSyncAdmissionChecks(ctx, &wl, cqName); updated || err != nil { | ||
return ctrl.Result{}, err | ||
} | ||
} | ||
|
||
if workload.SyncAdmittedCondition(&wl) { | ||
// If the workload is not admitted update its admitted condition if necessary. | ||
if !workload.IsAdmitted(&wl) && workload.SyncAdmittedCondition(&wl) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this change needed, or just a performance optimization? If needed, does the order matter?
Also, is it correct, I'm asking because if the workload was already admitted, then (IIUC) we don't enter the branch, we also skip if workload.HasQuotaReservation(&wl) {
, then we enter if rejectedChecks := workload.GetRejectedChecks(&wl); len(rejectedChecks) > 0 {
. So we would mark the workload as finished.
I think the idea was to go via eviction rather than marking the workload as finished? Can you explain what is the code path leading to workload eviction in that scenario now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is needed, and the order is important since SyncAdmittedCondition
can update the Admitted
cind. in the wl.
If the workload is admitted, it will skip this and just check if the eviction is needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I checked that the integration test passes when the check !workload.IsAdmitted(&wl)
is removed. Can we add / extend the integration test to demonstrate its importance?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
(If we don't keep this the Admitted condition is cleared before the eviction starts)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(If we don't keep this the Admitted condition is cleared before the eviction starts)
Why does that matter?
To be clear I'm fine with having Admitted=True and Evicted=True, I'm also fine with clearing the Admitted before adding Evicted=True. Trying to understand what is the difference, to avoid unnecessary code complications.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
comment updated.
To be clear I'm fine with having Admitted=True and Evicted=True, I'm also fine with clearing the Admitted before adding Evicted=True. Trying to understand what is the difference, to avoid unnecessary code complications.
It's more logical in my opinion and is the same flow as in case of premonition based eviction.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sgtm, I buy the argument of consistent flow with preemption-based eviction
c42c52e
to
598ba28
Compare
598ba28
to
683d38c
Compare
/lgtm |
LGTM label has been added. Git tree hash: fd03579ded1ca0a3f9f48e31766aaf8683edb151
|
9edd7f2
to
b10409e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, I think we also need a release note, I suggest "Fix the bug that a job would continue execution if the admission check state is changed to Rejected during execution."
Co-authored-by: Michał Woźniak <mimowo@users.noreply.github.com>
done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
LGTM label has been added. Git tree hash: dba641e2cb63a7d9e8df71013d46316a634496cc
|
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alculquicondor, trasc The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/release-note-edit
|
What type of PR is this?
/kind bug
What this PR does / why we need it:
Allow the workload reconciler to trigger the eviction of a workload that has
Rejected
admission checks and isAdmitted
, before making it asFinished
.Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?