Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(artifacts): support ephemeral credentials for S3. Fixes #5446 #12467

Merged
merged 1 commit into from
Sep 14, 2024
Merged

feat(artifacts): support ephemeral credentials for S3. Fixes #5446 #12467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions api/jsonschema/schema.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions api/openapi-spec/swagger.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

200 changes: 171 additions & 29 deletions docs/configure-artifact-repository.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,16 +75,20 @@ artifacts:

## Configuring AWS S3

Create your bucket and access keys for the bucket. AWS access keys have the same
permissions as the user they are associated with. In particular, you cannot
create access keys with reduced scope. If you want to limit the permissions for
an access key, you will need to create a user with just the permissions you want
to associate with the access key. Otherwise, you can just create an access key
using your existing user account.
First, create a bucket:

```bash
$ export mybucket=bucket249
$ cat > policy.json <<EOF
mybucket=my-bucket-name
aws s3 mb s3://mybucket [--region xxx]
```

### AWS S3 IAM Access

Next, create a policy file.
You will attach this in one of the sections below based on your chosen authentication method.

```bash
cat > policy.json <<EOF
{
"Version":"2012-10-17",
"Statement":[
Expand All @@ -107,43 +111,181 @@ $ cat > policy.json <<EOF
]
}
EOF
$ aws s3 mb s3://$mybucket [--region xxx]
$ aws iam create-user --user-name $mybucket-user
$ aws iam put-user-policy --user-name $mybucket-user --policy-name $mybucket-policy --policy-document file://policy.json
$ aws iam create-access-key --user-name $mybucket-user > access-key.json
```

If you do not have Artifact Garbage Collection configured, you should remove `s3:DeleteObject` from the list of Actions above.
If you do not have [Artifact Garbage Collection](walk-through/artifacts.md#artifact-garbage-collection) configured, you should remove `s3:DeleteObject` from the list of Actions above.

NOTE: if you want argo to figure out which region your buckets belong in, you
must additionally set the following statement policy. Otherwise, you must
specify a bucket region in your workflow configuration.
<!-- markdownlint-disable MD046 -- allow indentation within the admonition -->

```json
{
"Effect":"Allow",
"Action":[
"s3:GetBucketLocation"
],
"Resource":"arn:aws:s3:::*"
}
...
```
!!! Note "Region discovery"

Argo can discover the region of your buckets with the additional policy below.
Without this, you must specify the region in your artifact configuration.

```json
{
"Effect":"Allow",
"Action":[
"s3:GetBucketLocation"
],
"Resource":"arn:aws:s3:::*"
}
...
```

<!-- markdownlint-enable MD046 -->

### AWS S3 IRSA
#### AWS S3 IRSA

IAM Roles for Service Accounts (IRSA) is the recommended Kubernetes native mechanism to authenticate to S3.
If you are using EKS, follow [the IRSA setup guide](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
If not, follow the [Pod Identity Webhook self-hosted setup guide](https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md).

With the bucket and policy as described above, create an IAM role and add the policy:

```bash
aws iam create-role --role-name $mybucket-role
aws iam put-role-policy --role-name $mybucket-user --policy-name $mybucket-policy --policy-document file://policy.json
```

If you wish to use S3 IRSA instead of passing in an `accessKey` and `secretKey`, you need to annotate the service account of both the running workflow (in order to save logs/artifacts) and the argo-server pod (in order to retrieve the logs/artifacts).
Attach this IAM role to a service account with an annotation:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::012345678901:role/mybucket
eks.amazonaws.com/role-arn: arn:aws:iam::012345678901:role/mybucket-role
name: myserviceaccount
namespace: mynamespace
```

Use the service account in a workflow:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
serviceAccountName: myserviceaccount
```

#### AWS S3 with IAM Access Keys

!!! Note "Least privilege user"
To reduce the privileges of an access key, create a user with only the necessary permissions and no more.

With the bucket and policy described above, create an IAM user and add the policy:

```bash
aws iam create-user --user-name $mybucket-user
aws iam put-user-policy --user-name $mybucket-user --policy-name $mybucket-policy --policy-document file://policy.json
aws iam create-access-key --user-name $mybucket-user > access-key.json
```

Configure an artifact with the access keys:

```yaml
artifacts:
- name: my-output-artifact
path: /my-output-artifact
s3:
endpoint: s3.amazonaws.com
bucket: my-s3-bucket
key: path/in/bucket/my-output-artifact.tgz
# The following fields are secret selectors.
# They reference the k8s secret named 'my-s3-credentials'.
# This secret is expected to have the keys 'accessKey' and 'secretKey',
# containing the base64 encoded credentials to the bucket.
accessKeySecret:
name: my-s3-credentials
key: accessKey
secretKeySecret:
name: my-s3-credentials
key: secretKey
```

#### AWS S3 with IAM Assume Role

> v3.6 and after

hittingray marked this conversation as resolved.
Show resolved Hide resolved
You can use an [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) for temporary access.

With the bucket and policy described above, create an IAM role and add the policy:

```bash
aws iam create-role --role-name $mybucket-role
aws iam put-role-policy --role-name $mybucket-user --policy-name $mybucket-policy --policy-document file://policy.json
```

Retrieve the role credentials:

```bash
aws sts assume-role --role-arn arn:aws:iam::012345678901:role/$mybucket-role
```

Configure an artifact with the credentials:

```yaml
artifacts:
- name: my-output-artifact
path: /my-output-artifact
s3:
endpoint: s3.amazonaws.com
bucket: my-s3-bucket
key: path/in/bucket/my-output-artifact.tgz
# The following fields are secret selectors.
# They reference the k8s secret named 'my-s3-credentials'.
# This secret is expected to have the keys 'accessKey', 'secretKey', and 'sessionToken',
# containing the base64 encoded credentials to the bucket.
accessKeySecret:
hittingray marked this conversation as resolved.
Show resolved Hide resolved
name: my-s3-credentials
key: accessKey
secretKeySecret:
name: my-s3-credentials
key: secretKey
sessionTokenSecret:
name: my-s3-credentials
key: sessionToken
```

hittingray marked this conversation as resolved.
Show resolved Hide resolved
!!! Note "Temporary"
IAM role credentials are temporary, so you must refresh them periodically via an external mechanism.

### AWS S3 with S3 Access Grants

> v3.6 and after

hittingray marked this conversation as resolved.
Show resolved Hide resolved
You can use [S3 Access Grants](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-grants.html) for temporary, reduced scope access.
Follow the [AWS guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-grants-get-started.html) to set this up in your AWS account and retrieve access grant credentials.

Configure an artifact with the access grant credentials:

```yaml
artifacts:
- name: my-output-artifact
path: /my-output-artifact
s3:
endpoint: s3.amazonaws.com
bucket: my-s3-bucket
key: path/in/bucket/my-output-artifact.tgz
# The following fields are secret selectors.
# They reference the k8s secret named 'my-s3-credentials'.
# This secret is expected to have the keys 'accessKey', 'secretKey', and 'sessionToken',
# containing the base64 encoded credentials to the bucket.
accessKeySecret:
name: my-s3-credentials
key: accessKey
secretKeySecret:
name: my-s3-credentials
key: secretKey
sessionTokenSecret:
name: my-s3-credentials
key: sessionToken
```

!!! Note "Temporary"
S3 Access Grants are temporary, so you must refresh them periodically via an external mechanism.

## Configuring GCS (Google Cloud Storage)

Create a bucket from the GCP Console
Expand Down
1 change: 1 addition & 0 deletions docs/executor_swagger.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3140,6 +3140,7 @@ cause implementors to also use a fixed point implementation.
| region | string| `string` | | | Region contains the optional bucket region | |
| roleARN | string| `string` | | | RoleARN is the Amazon Resource Name (ARN) of the role to assume. | |
| secretKeySecret | [SecretKeySelector](#secret-key-selector)| `SecretKeySelector` | | | | |
| sessionTokenSecret | [SecretKeySelector](#secret-key-selector)| `SecretKeySelector` | | | | |
| useSDKCreds | boolean| `bool` | | | UseSDKCreds tells the driver to figure out credentials based on sdk defaults. | |


Expand Down
2 changes: 2 additions & 0 deletions docs/fields.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3527,6 +3527,7 @@ S3Artifact is the location of an S3 artifact
|`region`|`string`|Region contains the optional bucket region|
|`roleARN`|`string`|RoleARN is the Amazon Resource Name (ARN) of the role to assume.|
|`secretKeySecret`|[`SecretKeySelector`](#secretkeyselector)|SecretKeySecret is the secret selector to the bucket's secret key|
|`sessionTokenSecret`|[`SecretKeySelector`](#secretkeyselector)|SessionTokenSecret is used for ephemeral credentials like an IAM assume role or S3 access grant|
|`useSDKCreds`|`boolean`|UseSDKCreds tells the driver to figure out credentials based on sdk defaults.|

## ValueFrom
Expand Down Expand Up @@ -4248,6 +4249,7 @@ S3ArtifactRepository defines the controller configuration for an S3 artifact rep
|`region`|`string`|Region contains the optional bucket region|
|`roleARN`|`string`|RoleARN is the Amazon Resource Name (ARN) of the role to assume.|
|`secretKeySecret`|[`SecretKeySelector`](#secretkeyselector)|SecretKeySecret is the secret selector to the bucket's secret key|
|`sessionTokenSecret`|[`SecretKeySelector`](#secretkeyselector)|SessionTokenSecret is used for ephemeral credentials like an IAM assume role or S3 access grant|
|`useSDKCreds`|`boolean`|UseSDKCreds tells the driver to figure out credentials based on sdk defaults.|

## MutexHolding
Expand Down
32 changes: 16 additions & 16 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ require (
github.com/aliyun/aliyun-oss-go-sdk v3.0.2+incompatible
github.com/aliyun/credentials-go v1.3.2
github.com/argoproj/argo-events v1.9.1
github.com/argoproj/pkg v0.13.7-0.20240208112602-3bb8fe9a0527
github.com/argoproj/pkg v0.13.7-0.20240704113442-a69fd34a8117
github.com/blushft/go-diagrams v0.0.0-20201006005127-c78c821223d9
github.com/colinmarc/hdfs/v2 v2.4.0
github.com/coreos/go-oidc/v3 v3.9.0
Expand Down Expand Up @@ -46,7 +46,7 @@ require (
github.com/sirupsen/logrus v1.9.3
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966
github.com/soheilhy/cmux v0.1.5
github.com/spf13/cobra v1.8.0
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.18.2
github.com/stretchr/testify v1.9.0
Expand Down Expand Up @@ -87,8 +87,8 @@ require (
dario.cat/mergo v1.0.0 // indirect
github.com/TylerBrock/colorjson v0.0.0-20200706003622-8a50f05110d2 // indirect
github.com/alibabacloud-go/debug v0.0.0-20190504072949-9472017b5c68 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.4 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.3 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.2 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/cloudflare/circl v1.3.7 // indirect
github.com/containerd/stargz-snapshotter/estargz v0.14.3 // indirect
Expand Down Expand Up @@ -165,25 +165,25 @@ require (
github.com/ajg/form v1.5.1 // indirect
github.com/andybalholm/brotli v1.0.4 // indirect
github.com/awalterschulze/gographviz v0.0.0-20200901124122-0eecad45bd71 // indirect
github.com/aws/aws-sdk-go-v2 v1.24.1 // indirect
github.com/aws/aws-sdk-go-v2/config v1.26.6 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.16.16 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.11 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.7.3 // indirect
github.com/aws/aws-sdk-go-v2 v1.30.1 // indirect
github.com/aws/aws-sdk-go-v2/config v1.27.23 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.24 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.9 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.13 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.13 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
github.com/aws/aws-sdk-go-v2/service/ecr v1.17.8 // indirect
github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.13.8 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.10 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.18.7 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.26.7 // indirect
github.com/aws/smithy-go v1.19.0 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.15 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.22.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.30.1 // indirect
github.com/aws/smithy-go v1.20.3 // indirect
github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20220706184558-ce46abcd012b // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/chai2010/gettext-go v1.0.2 // indirect
github.com/chrismellard/docker-credential-acr-env v0.0.0-20220327082430-c57b701bfc08 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.4 // indirect
github.com/creack/pty v1.1.21
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/daviddengcn/go-colortext v1.0.0 // indirect
Expand Down
Loading
Loading