Example fails with minikube

[KukumavMozolo]

I tried the following example with minikube as the backend. Everything is set up according to this guide:
https://argoproj.github.io/argo-workflows/quick-start/

`from hera.task import Task
from hera.workflow import Workflow
from hera.workflow_service import WorkflowService

def say(message: str):
"""
This can be anything as long as the Docker image satisfies the dependencies. You can import anything Python
that is in your container e.g torch, tensorflow, scipy, biopython, etc - just provide an image to the task!
"""
print(message)

token = "sometoken"

ws = WorkflowService(host='https://localhost:2746', verify_ssl=False, token=token,namespace="argo")
w = Workflow('my-workflow', ws)
t = Task('say', say, [{'message': 'Hello, world!'}])
w.add_task(t)
w.submit(namespace='argo')`

I am getting the following error message:
argo.workflows.client.exceptions.ApiException: (401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Trailer': 'Grpc-Trailer-Content-Type', 'Date': 'Mon, 17 Jan 2022 12:01:05 GMT', 'Transfer-Encoding': 'chunked'}) HTTP response body: {"code":16,"message":"Unauthorized"}

I am able to execute the basic examples with couler.

[flaviuvadan]

Hi @KukumavMozolo! Thanks for submitting the issue 🙂 I believe Couler works because it uses your k8s config on local. In the case you presented, the k8s config file is likely pointed to the minikube cluster, so couler is able to create your workflow. The kube config file is loaded here and the workflow is not submitted through the Argo Server. Rather, it is created as a namespaced custom object directly in the cluster (minikube in this case). You can see that here. So, the auth part is set because of the k8s config file, and the workflow is created as a K8S object that's picked up by the Workflow Controller and run.

By comparison, Hera talks to the Argo Server, which is able to perform authentication checks itself. So, you might actually need to obtain a service account token first, as illustrated in this example. Any chance you have a moment to try that one and, perhaps, respond to this thread with the results? Would love to help get you going!

[KukumavMozolo]

Hi @flaviuvadan, thank you for your quick response.
I tried your suggested code, it failed, but I believe I need to set up "argo-server" as service account. Do you know how that would tie in with the argo quick-start guide, where I had to create this role? kubectl create clusterrolebinding YOURNAME-cluster-admin-binding --clusterrole=cluster-admin --user=YOUREMAIL@gmail.com

Error message: "status":"Failure","message":"workflows "argo" is forbidden: User "system:serviceaccount:argo:argo-server" cannot create resource "workflows" in API group "" at the cluster scope","reason":"Forbidden","details":{"name":"argo","kind":"workflows"},"code":403}"

Ahh, sorry I put the URL from the k8s config, but after I exchanged it to https://localhost:2746 it worked!!!

As a side node, does hera allow to manipulate Docker image pull policy like https://github.com/couler-proj/couler/blob/cd10d33fc587cf405275635afabf4eb7a8ee9e00/couler/core/constants.py#L37

[flaviuvadan]

Ahh, sorry I put the URL from the k8s config, but after I exchanged it to https://localhost:2746 it worked!!!

Oh, perfect! I am so happy to hear this works for you! 🙂 I actually encourage you, if you'd like to do this and have the time of course, to add a mini tutorial on how to run Hera + Minikube! I think it would be a phenomenal addition! 🚀

As a side node, does hera allow to manipulate Docker image pull policy

Not yet, but it sure can! The V1alpha1ScriptTemplate contains an image_pull_policy field, which can definitely be set 🙂 Would you like to submit an issue requesting this? It would be a very nice first issue!

[KukumavMozolo]

Not yet, but it sure can! The V1alpha1ScriptTemplate contains an image_pull_policy field, which can definitely be set slightly_smiling_face Would you like to submit an issue requesting this? It would be a very nice first issue!

If I arrive at a stage where I feel like I understand what is going on, I could do that.

The service account used here is called 'argo-server', it works but its not the one I created, and I can't seem to find any documentation about this. Do you know where it's coming from? Or how do I create one because the one created in the quick-start guide does not work.

[flaviuvadan]

The service account used here is called 'argo-server', it works but its not the one I created, and I can't seem to find any documentation about this. Do you know where it's coming from?

That is an example service account named argo-server and does not have any particular significance 🙂 in your cluster, for instance, if you have a K8S service account called argo, then the example would use argo! The one from the quick start guide of Argo might not have all the necessary permissions. I will try my best to write a very short overview of what I think is happening.

Synopsis

There are several K8S objects at play:

• service account (SA)
• role
• role binding (RB)
• deployment

Each of these objects is used in a very specific context:

• service account (SA) - an identifier account that can be bound to a deployment. This SA dictates the identifier of a deployment so when a deployment performs specific actions, it uses the permissions/roles bound to the respective SA
• role - a role dictates the ability to perform a specific action in a cluster. In this case, creation of custom resource definitions such as Workflow
• role binding (RB) - this is the mechanism of binding a role to a specific service account. If there exists some role (kubectl get roles) that says "this role provides the ability to create workflows in namespace X", the RB makes the association between an SA and a specific role
• deployment - the definition for a pod. This has characteristics such as a service account, as covered by the SA section above. The SA on the deployment associates an identifier with the deployment, the identifier is associated with multiple roles via role bindings.

When a client submits a workflow through Hera, Hera talks to the Argo server. The Argo server is a deployment, which has an SA associated with it, which has specific roles, constructed through RBs. When the server, or the workflow controller, talks to the K8S API to create some specific K8S objects, such as a Workflow (CRD), the kubelet checks whether the SA associated with that client/deployment has the right roles in the cluster. In your case, the error says that the argo-server SA does not have some specific role. So, you might have to manually create the role binding. You can find the roles in the cluster via kubectl get roles and the bindings via kubectl get rolebindings. Then you can get the service accounts via kubectl get sa. You can manually edit the role binding of the Argo server SA and add the necessary roles via kubectl edit rolebindings [whatever-argo-binding]. I think that might be sufficient to make the deployment have sufficient permissions for creating workflows.

I hope this short synopsis helps solidify understanding of some K8S concepts at play here! 🙂

[flaviuvadan]

@KukumavMozolo, circling back to this - did the above help? Should we add some docs on runnings Hera + MiniKube? Could we close this issue? 🙂

[flaviuvadan]

Closing this for now. Feel free to reopen if there's more to discuss! 🙂

[EricBoix]

Hello @flaviuvadan and sorry for the bother again,
If we look at the code you refer to k8s_sa.py on line 39 (which is pretty dated code but I wasn't able find any recent equivalent), it assumes that at least a secret will be encoutered with the expression

v1.read_namespaced_service_account(self.service_account, self.namespace).secrets[0]...

But (by default) if you

• start a minikube server (minikube start)
• install and start an argo server (e.g. with kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-workflows/master/manifests/quick-start-postgres.yaml)
• possibly define some argo namespace (e.g. with kubectl create ns argo)
• expose the argo-server (e.g. with kubectl -n argo port-forward service/argo-server 2746:2746 &)
• eventually explore the available service accounts through the minikube UI (with minikube dashboard)

then neither of the available service accounts (argo-server and argo) have secrets defined. Which makes the k8s_sa.py fail when trying to retrieve the secret_name.

Is there some Hera on minikube installation guide (even sketchy) or examples documenting how to install argo + Hera on minikube and then get the tutorial examples to run ?