Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Flytekit] Envd builder with extra copy commands #2774

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

mao3267
Copy link
Contributor

@mao3267 mao3267 commented Sep 27, 2024

Tracking issue

Related to #2715, after adding support for extra copy commands in the default builder, we also want to extend this support to the envd builder.

Why are the changes needed?

When creating images with ImageSpec on a remote cluster, we may need to copy additional files or directories required during the container build process. To address this, we aim to support extra copy commands in ImageSpec for copying files and directories into the container’s /root directory.

What changes were proposed in this pull request?

  1. Now we support ImageSpec with copy commands to be built by the envd builder.

Example Usage:
input.txt

test for document example

build_image.py

from flytekit.image_spec import ImageSpec
from flytekit import task, workflow

image_spec = ImageSpec(
    name="envd-copy-cmds",
    registry="localhost:30000",
   # envd is the default builder for imageSpec
    apt_packages=["git", "gh"],
    copy=["local_test/input.txt"],
    env={"GH_TOKEN": "<YOUR_GITHUB_TOKEN>"},
    commands=[
        "git clone https://github.com/flyteorg/flytekit.git", 
        "cd flytekit", 
        "gh pr checkout 2715", 
        "pip install -e ."
    ]
)

@task(container_image=image_spec)
def my_task() -> str:
    with open("/root/local_test/input.txt", "r") as f:
        return f.read()


@workflow
def my_wf() -> str:
    return my_task()

How was this patch tested?

  1. Run the example above with remote execution.
  2. Added tests in plugins/flytekit-envd/tests/test_image_spec.py to verify the correctness of the envd configuration and its ability to be built.

Setup process

git clone https://github.com/flyteorg/flytekit.git
gh checkout pr 2774
make setup && pip install -e .

Screenshots

Screenshots of running pyflyte run --remote build_image.py my_wf.

  • Remote Execution

image

  • Terminal
    The entire building process is too lengthy; therefore, only the start and the end of the build process are displayed here.

image

image

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

#2715

Docs link

Signed-off-by: mao3267 <chenvincent610@gmail.com>
Signed-off-by: mao3267 <chenvincent610@gmail.com>
@mao3267 mao3267 changed the title [WIP] [Flytekit] Envd builder with extra copy commands [Flytekit] Envd builder with extra copy commands Sep 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant