Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubernetes 1.22 compatibility #1665

Closed
d4l3k opened this issue Aug 10, 2021 · 9 comments
Closed

kubernetes 1.22 compatibility #1665

d4l3k opened this issue Aug 10, 2021 · 9 comments
Assignees
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Milestone

Comments

@d4l3k
Copy link

d4l3k commented Aug 10, 2021

What happened:

The latest release 1.3.0 (and master) appear to be incompatible with Kubernetes 1.22.

What you expected to happen:

It's able to launch jobs

How to reproduce it (as minimally and precisely as possible):

$ minikube start --kubernetes-version v1.22.0
$ kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.3.0/installer/volcano-development.yaml
$ kubectl logs --namespace volcano-system pod/volcano-scheduler-5665cdc4d9-r82qt
W0809 23:58:27.561845       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0809 23:58:27.580859       1 reflector.go:127] volcano.sh/volcano/pkg/scheduler/cache/cache.go:440: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
I0809 23:58:27.591897       1 event_handlers.go:199] Added pod <kube-system/kube-apiserver-minikube> into cache.
I0809 23:58:27.591935       1 event_handlers.go:199] Added pod <kube-system/kube-controller-manager-minikube> into cache.
I0809 23:58:27.591949       1 event_handlers.go:199] Added pod <kube-system/etcd-minikube> into cache.
I0809 23:58:27.591959       1 event_handlers.go:199] Added pod <kube-system/storage-provisioner> into cache.
I0809 23:58:27.591965       1 event_handlers.go:199] Added pod <kube-system/kube-proxy-ghmr6> into cache.
I0809 23:58:27.591972       1 event_handlers.go:199] Added pod <volcano-system/volcano-admission-5bb77cd5b7-jn448> into cache.
I0809 23:58:27.591979       1 event_handlers.go:199] Added pod <kube-system/kube-scheduler-minikube> into cache.
I0809 23:58:27.591990       1 event_handlers.go:199] Added pod <volcano-system/volcano-admission-init--1-j24lq> into cache.
I0809 23:58:27.591998       1 event_handlers.go:199] Added pod <volcano-system/volcano-scheduler-5665cdc4d9-r82qt> into cache.
I0809 23:58:27.592005       1 event_handlers.go:199] Added pod <volcano-system/volcano-controllers-5bbcc9c49f-xffv7> into cache.
I0809 23:58:27.592013       1 event_handlers.go:199] Added pod <kube-system/coredns-78fcd69978-lrwvz> into cache.
E0809 23:58:28.924696       1 reflector.go:127] volcano.sh/volcano/pkg/scheduler/cache/cache.go:440: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E0809 23:58:31.023793       1 reflector.go:127] volcano.sh/volcano/pkg/scheduler/cache/cache.go:440: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E0809 23:58:35.884044       1 reflector.go:127] volcano.sh/volcano/pkg/scheduler/cache/cache.go:440: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource

Anything else we need to know?:

Seems like v1beta1 version of PriorityClass is gone and instead needs to use v1. https://kubernetes.io/docs/reference/using-api/deprecation-guide/#priorityclass-v122

Environment:

  • Volcano Version: 1.3.0
  • Kubernetes version (use kubectl version): 1.22
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@d4l3k d4l3k added the kind/bug Categorizes issue or PR as related to a bug. label Aug 10, 2021
@k82cn
Copy link
Member

k82cn commented Aug 10, 2021

@Thor-wl , please help on that :)

@Thor-wl Thor-wl self-assigned this Aug 10, 2021
@wpeng102
Copy link
Member

scheduling.k8s.io/v1beta1 PriorityClass is deprecated in v1.14+, unavailable in v1.22+; use scheduling.k8s.io/v1 PriorityClass

https://kubernetes.io/docs/reference/using-api/deprecation-guide/#priorityclass-v122

@Thor-wl
Copy link
Contributor

Thor-wl commented Aug 13, 2021

We'll support k8s 1.20, 1.21, 1,22 after v1.4 released. @d4l3k Thanks for your advice.

@k82cn
Copy link
Member

k82cn commented Aug 17, 2021

how about the older version?

@d4l3k
Copy link
Author

d4l3k commented Nov 2, 2021

@Thor-wl is 1.22 supported? v1.4.0 has been released but the repro steps above still don't work.

tristanr@tristanr-arch2 ~> kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml
namespace/volcano-system created
namespace/volcano-monitoring created
serviceaccount/volcano-admission created
configmap/volcano-admission-configmap created
clusterrole.rbac.authorization.k8s.io/volcano-admission created
clusterrolebinding.rbac.authorization.k8s.io/volcano-admission-role created
service/volcano-admission-service created
deployment.apps/volcano-admission created
job.batch/volcano-admission-init created
customresourcedefinition.apiextensions.k8s.io/jobs.batch.volcano.sh created
customresourcedefinition.apiextensions.k8s.io/commands.bus.volcano.sh created
serviceaccount/volcano-controllers created
clusterrole.rbac.authorization.k8s.io/volcano-controllers created
clusterrolebinding.rbac.authorization.k8s.io/volcano-controllers-role created
deployment.apps/volcano-controllers created
serviceaccount/volcano-scheduler created
configmap/volcano-scheduler-configmap created
clusterrole.rbac.authorization.k8s.io/volcano-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/volcano-scheduler-role created
service/volcano-scheduler-service created
deployment.apps/volcano-scheduler created
customresourcedefinition.apiextensions.k8s.io/podgroups.scheduling.volcano.sh created
customresourcedefinition.apiextensions.k8s.io/queues.scheduling.volcano.sh created
customresourcedefinition.apiextensions.k8s.io/numatopologies.nodeinfo.volcano.sh created

tristanr@tristanr-arch2 ~> kubectl get pods -n volcano-system
NAME                                   READY   STATUS      RESTARTS   AGE
volcano-admission-f5f4fb5b6-spspp      1/1     Running     0          4m51s
volcano-admission-init--1-2nr5p        0/1     Completed   0          4m52s
volcano-controllers-5bbcc9c49f-zllwh   1/1     Running     0          4m51s
volcano-scheduler-5665cdc4d9-c7wnd     1/1     Running     0          4m51s

tristanr@tristanr-arch2 ~> kubectl logs --namespace volcano-system pod/volcano-scheduler-5665cdc4d9-c7wnd
2021/11/02 21:30:47 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
W1102 21:30:47.709996       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E1102 21:30:47.742901       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
I1102 21:30:47.766739       1 event_handlers.go:210] Added pod <volcano-system/volcano-scheduler-5665cdc4d9-c7wnd> into cache.
I1102 21:30:47.766801       1 event_handlers.go:210] Added pod <kube-system/coredns-78fcd69978-qskxq> into cache.
I1102 21:30:47.766823       1 event_handlers.go:210] Added pod <kube-system/kube-apiserver-minikube> into cache.
I1102 21:30:47.766841       1 event_handlers.go:210] Added pod <kube-system/kube-scheduler-minikube> into cache.
I1102 21:30:47.766860       1 event_handlers.go:210] Added pod <kube-system/etcd-minikube> into cache.
I1102 21:30:47.766878       1 event_handlers.go:210] Added pod <volcano-system/volcano-admission-init--1-2nr5p> into cache.
I1102 21:30:47.766942       1 event_handlers.go:210] Added pod <kube-system/kube-proxy-cqfh8> into cache.
I1102 21:30:47.766993       1 event_handlers.go:210] Added pod <kube-system/kube-controller-manager-minikube> into cache.
I1102 21:30:47.767021       1 event_handlers.go:210] Added pod <kube-system/storage-provisioner> into cache.
I1102 21:30:47.767048       1 event_handlers.go:210] Added pod <volcano-system/volcano-controllers-5bbcc9c49f-zllwh> into cache.
I1102 21:30:47.767077       1 event_handlers.go:210] Added pod <volcano-system/volcano-admission-f5f4fb5b6-spspp> into cache.
E1102 21:30:49.038325       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:30:51.471580       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:30:57.286107       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:31:05.070295       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:31:22.758295       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:31:56.521138       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:32:40.629298       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:33:19.158683       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:33:57.984681       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:34:48.390669       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:35:24.957685       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource

tristanr@tristanr-arch2 ~> vcctl job list
Name               Creation       Phase       JobType     Replicas    Min   Pending   Running   Succeeded   Failed    Unknown     RetryCount
sh-ts9rlsfqjtl4d   2021-11-02                 Batch       1           0     0         0         0           0         0           0         

tristanr@tristanr-arch2 ~> kubectl get job.batch.volcano.sh/sh-ts9rlsfqjtl4d -oyaml
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  creationTimestamp: "2021-11-02T21:31:38Z"
  generation: 1
  name: sh-ts9rlsfqjtl4d
  namespace: default
  resourceVersion: "666"
  uid: 7db95feb-de6d-4ee5-a0ca-1578b2067db5
spec:
  maxRetry: 0
  plugins:
    env: []
    svc: []
  queue: default
  schedulerName: volcano
  tasks:
  - name: sh-0
    replicas: 1
    template:
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
        labels:
          torchx.pytorch.org/app-name: sh
          torchx.pytorch.org/replica-id: "0"
          torchx.pytorch.org/role-index: "0"
          torchx.pytorch.org/role-name: sh
          torchx.pytorch.org/version: 0.1.1dev0
      spec:
        containers:
        - command:
          - sh
          - -c
          - env
          env: []
          image: alpine:latest
          name: sh-0
          ports: []
          resources:
            limits: {}
            requests: {}
        restartPolicy: Never

@Thor-wl
Copy link
Contributor

Thor-wl commented Nov 3, 2021

@Thor-wl is 1.22 supported? v1.4.0 has been released but the repro steps above still don't work.

tristanr@tristanr-arch2 ~> kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml
namespace/volcano-system created
namespace/volcano-monitoring created
serviceaccount/volcano-admission created
configmap/volcano-admission-configmap created
clusterrole.rbac.authorization.k8s.io/volcano-admission created
clusterrolebinding.rbac.authorization.k8s.io/volcano-admission-role created
service/volcano-admission-service created
deployment.apps/volcano-admission created
job.batch/volcano-admission-init created
customresourcedefinition.apiextensions.k8s.io/jobs.batch.volcano.sh created
customresourcedefinition.apiextensions.k8s.io/commands.bus.volcano.sh created
serviceaccount/volcano-controllers created
clusterrole.rbac.authorization.k8s.io/volcano-controllers created
clusterrolebinding.rbac.authorization.k8s.io/volcano-controllers-role created
deployment.apps/volcano-controllers created
serviceaccount/volcano-scheduler created
configmap/volcano-scheduler-configmap created
clusterrole.rbac.authorization.k8s.io/volcano-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/volcano-scheduler-role created
service/volcano-scheduler-service created
deployment.apps/volcano-scheduler created
customresourcedefinition.apiextensions.k8s.io/podgroups.scheduling.volcano.sh created
customresourcedefinition.apiextensions.k8s.io/queues.scheduling.volcano.sh created
customresourcedefinition.apiextensions.k8s.io/numatopologies.nodeinfo.volcano.sh created

tristanr@tristanr-arch2 ~> kubectl get pods -n volcano-system
NAME                                   READY   STATUS      RESTARTS   AGE
volcano-admission-f5f4fb5b6-spspp      1/1     Running     0          4m51s
volcano-admission-init--1-2nr5p        0/1     Completed   0          4m52s
volcano-controllers-5bbcc9c49f-zllwh   1/1     Running     0          4m51s
volcano-scheduler-5665cdc4d9-c7wnd     1/1     Running     0          4m51s

tristanr@tristanr-arch2 ~> kubectl logs --namespace volcano-system pod/volcano-scheduler-5665cdc4d9-c7wnd
2021/11/02 21:30:47 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
W1102 21:30:47.709996       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E1102 21:30:47.742901       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
I1102 21:30:47.766739       1 event_handlers.go:210] Added pod <volcano-system/volcano-scheduler-5665cdc4d9-c7wnd> into cache.
I1102 21:30:47.766801       1 event_handlers.go:210] Added pod <kube-system/coredns-78fcd69978-qskxq> into cache.
I1102 21:30:47.766823       1 event_handlers.go:210] Added pod <kube-system/kube-apiserver-minikube> into cache.
I1102 21:30:47.766841       1 event_handlers.go:210] Added pod <kube-system/kube-scheduler-minikube> into cache.
I1102 21:30:47.766860       1 event_handlers.go:210] Added pod <kube-system/etcd-minikube> into cache.
I1102 21:30:47.766878       1 event_handlers.go:210] Added pod <volcano-system/volcano-admission-init--1-2nr5p> into cache.
I1102 21:30:47.766942       1 event_handlers.go:210] Added pod <kube-system/kube-proxy-cqfh8> into cache.
I1102 21:30:47.766993       1 event_handlers.go:210] Added pod <kube-system/kube-controller-manager-minikube> into cache.
I1102 21:30:47.767021       1 event_handlers.go:210] Added pod <kube-system/storage-provisioner> into cache.
I1102 21:30:47.767048       1 event_handlers.go:210] Added pod <volcano-system/volcano-controllers-5bbcc9c49f-zllwh> into cache.
I1102 21:30:47.767077       1 event_handlers.go:210] Added pod <volcano-system/volcano-admission-f5f4fb5b6-spspp> into cache.
E1102 21:30:49.038325       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:30:51.471580       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:30:57.286107       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:31:05.070295       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:31:22.758295       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:31:56.521138       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:32:40.629298       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:33:19.158683       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:33:57.984681       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:34:48.390669       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource
E1102 21:35:24.957685       1 reflector.go:127] github.com/volcano-sh/volcano/pkg/scheduler/cache/cache.go:552: Failed to watch *v1beta1.PriorityClass: failed to list *v1beta1.PriorityClass: the server could not find the requested resource

tristanr@tristanr-arch2 ~> vcctl job list
Name               Creation       Phase       JobType     Replicas    Min   Pending   Running   Succeeded   Failed    Unknown     RetryCount
sh-ts9rlsfqjtl4d   2021-11-02                 Batch       1           0     0         0         0           0         0           0         

tristanr@tristanr-arch2 ~> kubectl get job.batch.volcano.sh/sh-ts9rlsfqjtl4d -oyaml
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  creationTimestamp: "2021-11-02T21:31:38Z"
  generation: 1
  name: sh-ts9rlsfqjtl4d
  namespace: default
  resourceVersion: "666"
  uid: 7db95feb-de6d-4ee5-a0ca-1578b2067db5
spec:
  maxRetry: 0
  plugins:
    env: []
    svc: []
  queue: default
  schedulerName: volcano
  tasks:
  - name: sh-0
    replicas: 1
    template:
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
        labels:
          torchx.pytorch.org/app-name: sh
          torchx.pytorch.org/replica-id: "0"
          torchx.pytorch.org/role-index: "0"
          torchx.pytorch.org/role-name: sh
          torchx.pytorch.org/version: 0.1.1dev0
      spec:
        containers:
        - command:
          - sh
          - -c
          - env
          env: []
          image: alpine:latest
          name: sh-0
          ports: []
          resources:
            limits: {}
            requests: {}
        restartPolicy: Never

Not yet. I'm working on hierarchical queue recently. Will do that later.

@k82cn k82cn added good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Nov 3, 2021
@william-wang william-wang added this to the v1.5 milestone Dec 24, 2021
@stale
Copy link

stale bot commented Mar 25, 2022

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

@stale stale bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 25, 2022
@Thor-wl Thor-wl added kind/feature Categorizes issue or PR as related to a new feature. and removed help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. kind/bug Categorizes issue or PR as related to a bug. labels Apr 6, 2022
@Thor-wl
Copy link
Contributor

Thor-wl commented Apr 6, 2022

/close

@volcano-sh-bot
Copy link
Contributor

@Thor-wl: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

No branches or pull requests

6 participants