RBAC to launch as different service account

[dmerrick]

We have a web service that launches Argo jobs. We have it working with Couler but are having trouble with Hera. I think we're having a similar issue to this, but the solution isn't the same.

Basically our webservice runs as a deployment in example as service account example. We have an argo-workflows service account in namespace example that has all of the necessary RBAC permissions.

Unfortunately, when we run get_sa_token() (as suggested in your examples) we run into an unauthorized error. The full error looks like this:

get_token(service_account='argo-workflows', namespace='example')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/lib/python3.7/site-packages/reverie/infrastructure/argo.py", line 499, in get_token
    secret_name = v1.read_namespaced_service_account(service_account, namespace).secrets[0].name
  File "/venv/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 24660, in read_namespaced_service_account
    return self.read_namespaced_service_account_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/venv/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 24761, in read_namespaced_service_account_with_http_info
    collection_formats=collection_formats)
  File "/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
    _preload_content, _request_timeout, _host)
  File "/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
    _request_timeout=_request_timeout)
  File "/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 377, in request
    headers=headers)
  File "/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 244, in GET
    query_params=query_params)
  File "/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6f2689ee-604a-4207-a0c5-cd7e0080ebfb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 25 Apr 2022 16:38:56 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

I take this to mean that the example:example service account is unable to access the secret for the argo-workflows:example service account. What would we need to do to allow the deployment service account to launch jobs as the argo-workflows service account?

Thanks!

[flaviuvadan]

Hey @dmerrick! 👋🏻 Something like launch as different service account makes me think of K8S impersonation. In this case you may need to change the role bound to sa/example to include the impersonate verb. Here are some guides I found in a hasty search:

• https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation
• https://clux.dev/post/2019-03-31-impersonating-kube-accounts/

My hunch is that Couler works on the deployment because the sa/example has create permission on its bound role. Couler does not talk to the Argo server (unless this changed recently, of course) and it creates the workflow object directly so the deployment may end up using that create role.

[flaviuvadan]

@dmerrick did the above help, by any chance?

[dmerrick]

Unfortunately not, but it was still helpful.

We ended up converting the Workflow to YAML and submitting it as a k8s object instead of going through the Argo API