quota enforcement for cores not reported in plan results

[henrikjohansen]

Nomad version

Nomad v1.6.2+ent
BuildDate 2023-09-13T17:11:57Z
Revision a93af01

Issue

I was initially a little stumped by #18875 and now I know why :) Normally when you exceed your quota you get a nice little warning like :

$ nomad plan test.hcl
+ Job: "test"
+ Task Group: "test" (1 create)
  + Task: "test" (forces create)

Scheduler dry-run:
- WARNING: Failed to place all allocations.
  Task Group "test" (failed to place 1 allocation):
    * Quota limit hit "cpu exhausted (4000 needed > 3000 limit)"

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 test.hcl

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

... or ...

$ nomad job run -check-index 0 test.hcl
==> 2023-10-26T14:35:13+02:00: Monitoring evaluation "04e5ad72"
    2023-10-26T14:35:13+02:00: Evaluation triggered by job "test2"
    2023-10-26T14:35:14+02:00: Evaluation within deployment: "8fa290f0"
    2023-10-26T14:35:14+02:00: Evaluation status changed: "pending" -> "complete"
==> 2023-10-26T14:35:14+02:00: Evaluation "04e5ad72" finished with status "complete" but failed to place all allocations:
    2023-10-26T14:35:14+02:00: Task Group "test" (failed to place 1 allocation):
      * Quota limit hit "cpu exhausted (4000 needed > 3000 limit)"
    2023-10-26T14:35:14+02:00: Evaluation "94a1f752" waiting for additional capacity to place remainder
==> 2023-10-26T14:35:14+02:00: Monitoring deployment "8fa290f0"
  ⠙ Deployment "8fa290f0" in progress...
    
    2023-10-26T14:35:14+02:00
    ID          = 8fa290f0
    Job ID      = test2
    Job Version = 0
    Status      = running
    Description = Deployment is running
    
    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    test        1        0       0        0          N/A

When you are using cores (instead of cpu) in the resource stanza of your jobfile it fails without warning ...

$ nomad plan test.hcl
+ Job: "test3"
+ Task Group: "test" (1 create)
  + Task: "test" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 test.hcl

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

$ nomad run test.hcl
==> 2023-10-26T14:43:57+02:00: Monitoring evaluation "4a816cfa"
   2023-10-26T14:43:57+02:00: Evaluation triggered by job "test3"
   2023-10-26T14:43:58+02:00: Evaluation within deployment: "46b7c96a"
   2023-10-26T14:43:58+02:00: Evaluation status changed: "pending" -> "failed"
==> 2023-10-26T14:43:58+02:00: Evaluation "4a816cfa" finished with status "failed"
==> 2023-10-26T14:43:58+02:00: Monitoring deployment "46b7c96a"
 ⠙ Deployment "46b7c96a" in progress...
   2023-10-26T14:43:58+02:00: Error fetching deployment: Unexpected response code: 404 (deployment not found)

Expected Result

nomad plan and nomad job run should inform the operator of the quota violation regardless of which CPU resource definition they use in their jobfile ...

[tgross]

Hi @henrikjohansen! Unfortunately as it turns out, quota is currently unenforced for cores.

The API silently accepts cores because the structure we're accepting is a Region that includes cores, but they're not actually considered by quotas. This doesn't appear to be documented, and the API will accept them in a region limit spec, so this makes for a nasty gotcha for ENT customers expecting quotas to work as intended.

One challenge of implementing enforcement here is how we decide to calculate quota enforcement of cores vs cpu. But at the very least we could check a quota for reserved cores, and that'll need to account for the new NUMA-aware scheduling feature we're adding in Nomad 1.7.

I'll make sure this is getting surfaced to the team to address. (Internal ref: https://github.com/hashicorp/nomad-enterprise/issues/1065)

[henrikjohansen]

Hi @tgross 👋 It seems that cores actually do contribute to the quota and that they get enforced?

$ nomad quota status default-quota
Name        = default-quota
Description = Limit the shared default namespace
Limits      = 1

Quota Limits
Region  CPU Usage  Memory Usage  Memory Max Usage  Variables Usage
global  0 / 3000   0 / 1000      0 / 1000          0 / 1000

Now, running a small job with cores = 1 contributes to the quota ...

$ nomad quota status default-quota
Name        = default-quota
Description = Limit the shared default namespace
Limits      = 1

Quota Limits
Region  CPU Usage    Memory Usage  Memory Max Usage  Variables Usage
global  2793 / 3000  256 / 1000    256 / 1000        0 / 1000

Running a second job with cores = 1 fails (because we are over the quota) :

$ nomad run test2.hcl
==> 2023-10-26T16:23:03+02:00: Monitoring evaluation "fa58e0c0"
    2023-10-26T16:23:03+02:00: Evaluation triggered by job "test2"
    2023-10-26T16:23:04+02:00: Evaluation within deployment: "e80cb126"
    2023-10-26T16:23:04+02:00: Evaluation status changed: "pending" -> "failed"
==> 2023-10-26T16:23:04+02:00: Evaluation "fa58e0c0" finished with status "failed"
==> 2023-10-26T16:23:04+02:00: Monitoring deployment "e80cb126"
  ⠙ Deployment "e80cb126" in progress...
    2023-10-26T16:23:04+02:00: Error fetching deployment: Unexpected response code: 404 (deployment not found)

This issue is that it fails without warning the operator .....

Looking at the logs :

    2023-10-26T16:23:47.017+0200 [DEBUG] worker: submitted plan for evaluation: worker_id=a551bf0f-547e-b717-854a-d70439dfacd7 eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42
    2023-10-26T16:23:47.017+0200 [DEBUG] worker: refreshing state: worker_id=a551bf0f-547e-b717-854a-d70439dfacd7 refresh_index=23 eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42
    2023-10-26T16:23:47.017+0200 [DEBUG] worker.service_sched: refresh forced: eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42 job_id=test2 namespace=default worker_id=a551bf0f-547e-b717-854a-d70439dfacd7
    2023-10-26T16:23:47.017+0200 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42 job_id=test2 namespace=default worker_id=a551bf0f-547e-b717-854a-d70439dfacd7
  results=
  | Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) (disconnect 0) (reconnect 0)
  | Created Deployment: "36d1b8dc-ca43-1314-6778-4d8e00abfabb"
  | Desired Changes for "test": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)
  
    2023-10-26T16:23:47.017+0200 [DEBUG] nomad: plan for evaluation exceeds quota limit. Forcing state refresh: eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42 refresh_index=23
    2023-10-26T16:23:47.017+0200 [DEBUG] worker: submitted plan for evaluation: worker_id=a551bf0f-547e-b717-854a-d70439dfacd7 eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42
    2023-10-26T16:23:47.017+0200 [DEBUG] worker: refreshing state: worker_id=a551bf0f-547e-b717-854a-d70439dfacd7 refresh_index=23 eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42
    2023-10-26T16:23:47.017+0200 [DEBUG] worker.service_sched: refresh forced: eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42 job_id=test2 namespace=default worker_id=a551bf0f-547e-b717-854a-d70439dfacd7
    2023-10-26T16:23:47.018+0200 [DEBUG] worker: created evaluation: worker_id=a551bf0f-547e-b717-854a-d70439dfacd7 eval="<Eval \"668ed15c-f7a4-52cd-d030-8920ff014fa7\" JobID: \"test2\" Namespace: \"default\">"
    2023-10-26T16:23:47.018+0200 [DEBUG] worker.service_sched: setting eval status: eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42 job_id=test2 namespace=default worker_id=a551bf0f-547e-b717-854a-d70439dfacd7 status=failed
    2023-10-26T16:23:47.018+0200 [DEBUG] worker: updated evaluation: worker_id=a551bf0f-547e-b717-854a-d70439dfacd7 eval="<Eval \"3b1a36a0-4261-ca2a-4483-c7b0f5bffd42\" JobID: \"test2\" Namespace: \"default\">"
    2023-10-26T16:23:47.018+0200 [DEBUG] worker: ack evaluation: worker_id=a551bf0f-547e-b717-854a-d70439dfacd7 eval_id=3b1a36a0-4261-ca2a-4483-c7b0f5bffd42 type=service namespace=default job_id=test2 node_id="" triggered_by=job-register
    2023-10-26T16:23:48.018+0200 [DEBUG] http: request complete: method=GET path=/v1/evaluation/3b1a36a0-4261-ca2a-4483-c7b0f5bffd42?namespace=default duration="728.191µs"
    2023-10-26T16:23:48.019+0200 [DEBUG] http: request complete: method=GET path=/v1/evaluation/3b1a36a0-4261-ca2a-4483-c7b0f5bffd42/allocations?namespace=default duration="180.695µs"
    2023-10-26T16:23:48.020+0200 [DEBUG] http: request failed: method=GET path="/v1/deployment/36d1b8dc-ca43-1314-6778-4d8e00abfabb?namespace=default&stale=" error="deployment not found" code=404
    2023-10-26T16:23:48.020+0200 [DEBUG] http: request complete: method=GET path="/v1/deployment/36d1b8dc-ca43-1314-6778-4d8e00abfabb?namespace=default&stale=" duration="767.098µs"

[tgross]

Thanks for the clarification @henrikjohansen

[jrasell]

Hi @henrikjohansen and thanks again for the detail. I am looking into this and have a local reproduction which I will detail below for future readers.

Run using a Nomad Enterprise agent with the -dev flag.

$ nomad version
Nomad v1.8.1-dev+ent
BuildDate 2024-06-05T18:09:01Z
Revision 2d9e904d5f09c9cb12041bd8a81cfd87f3739b08

Create the quota spec detailed below using nomad quota apply <filename> and then apply it to the default namespace using nomad namespace apply -quota default default.

name = "default"

limit {
  region = "global"
  region_limit {
    cpu    = 1000
    memory = 1000
  }
}

Plan and run the job specification detailed below; notice the plan returns without error and the run errors and no allocations are placed.

job "example" {

  group "cache" {
    network {
      port "db" {
        to = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image          = "redis:7"
        ports          = ["db"]
        auth_soft_fail = true
      }

      identity {
        env  = true
        file = true
      }

      resources {
        memory = 256
        cores  = 1
      }
    }
  }
}