RFC: Execution Environments

[hone]

Readable

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[jabrown85]

Can you add an example of a full project.toml that is used for producing a test image and a production image?

[hone]

Just for Project Descriptor?

[samj1912]

I'm having a difficulty trying to grasp how environment is being used by different toml files. @hone could you please provide examples of places where you imagine it being used and how that information will be leveraged by buildpacks/platform?

[jabrown85]

Can we perhaps get a conceptual example of what a buildpack author might do with this new execution environment information?

I have in my head a language buildpack author may...

• install test group dependencies
• create ruby-tests process / ruby-tests-verbose process
• Skip cleaning up things they might clean up otherwise for production images
• Set env vars to test mode operation (RAILS_ENV or similar)

[loewenstein]

Frankly, I don't get this proposal aligned with my mental model of what buildpacks do and how they help.

For me, container images are just another software artefact nowadays produced to ship software. Like a while back one was to produce JAR and/or WAR files that got delivered and finally deployed into some execution environment.
Buildpacks are a great tool to create those, but I don't see how or why I would create a test container that was separate from the production ones. I would want my tests to validate exactly the artefact that I am about to deliver. Otherwise, how do I know I checked the right thing.

From my POV buildpacks should focus on that task, creating the container image, and leave the other CI tasks to specialized tools.

Please help me to adjust my mental model, why's this added complexity in the buildpack spec worth it?

[jabrown85]

Please help me to adjust my mental model, why's this added complexity in the buildpack spec worth it?

I'm not the author but I can speak for my own mental model.

One use case is from the app developer side. Today they can pack build <img> to get a production artifact.

If we imagine the resulting image is ruby + nodejs, the developer has to setup both of those environments locally to develop and AGAIN on say GitHub Actions. The way those installations happen can differ. The GitHub action may install a different minor version of node or ruby to run the tests for instance.

Setting up CI with buildpacks could make this experience easier. pack test would use the same buildpacks and therefore test a more prod-like experience as the buildpack would install the same versions. This is especially important if your buildpack has options like DO_THIS_GARBAGE_COLLECTION_SETTING that you would have to know and replicate in your CI environment.

Another future-proofing thing here may be around future ARM support. Building and running an ARM test image via pack test seems tractable.

[natalieparellano]

How about "mode"?

[joshwlewis]

I like "context", but that's just as overloaded. How about "purpose = test" or "intent = test"?

[natalieparellano]

Do you mean [[order.group]]?

[natalieparellano]

Isn't this a lifecycle concern? i.e., when processing a group within an order, should it skip buildpacks that do not declare exec-env matching the desired env?

[hone]

When talking to @jabrown85, I thought the Platform/Builder provide the order.toml to lifecycle?

[hone]

From WG, lifecycle will decide which buildpacks to run based on the execution environment being passed along by the platform.

[natalieparellano]

Is this ever something we would want to spec? Having a consistent way for buildpacks to e.g., dump test output could help ensure portability across platforms.

[hone]

I could see us wanting to do that, but I wasn't sure how much we wanted to impose standards there.

[natalieparellano]

Are there any downsides to doing this? It would be more flexible and possibly avoid some duplication within orders (we could keep all as a special value)

[natalieparellano]

It would be nice to see the pack flow as well

[natalieparellano]

Should we add anything to the app image to designate it as being built for a particular environment? To avoid users accidentally deploying a test image in production...

I could see folks wanting to use the same tag when re-building a test image for production, in order to use previously cached dependencies.

[hone]

@natalieparellano that's a good question. Do you think cached dependencies should be shared b/t different execution environments?

[hone]

Is a label the best place to designate the execution environment?

[kanaksinghal]

I have a use-case for this.
In our company we use a variety of languages/stack and I wanted to write a generic CI pipeline where buildpacks can help us build the container image by detecting the stack but then to run unit test, I need to detect the stack again myself and run specific commands for nodejs/go/java etc.
In a typical CI pipeline we would do unit-test, run sonar quality checks, and build container image.

[jama22]

Really excited for this proposal, and I think there's a lot of interesting ways to make use of it.

In the context of a self-hosted PaaS
I've seen a common dynamic where the "operator" needs to establish fine grained controls over what the final containers look like. These configurations are also prone to change as the artifact itself moves into different environments (service connectors, certificates, environment variables, access to secrets managers, etc.)

On the developer side, most folks don't ever see production-like environments, so being able to debug their applications in production-like environments becomes incredibly difficult. In the past, I've seen some of these per-environment configurations being managed through GitOps and config files. This gives the user some control over managing the movement of the workload through the system, but even still they may not be able to build that container and interact with it directly

Moving the environment configuration opens up a lot of possibilities in this use case. I could image an operator encoding their per-environment configurations into the builder, and giving the developer the ability to simulate how their application will change in behavior across configurations.

For CI/CD
I think there have been some interesting ideas already discussed above and in the proposal. But I think a common use case that i can come up with is if you're building a library with buildpacks, you'll want to test on multiple OS-distros and architecture combinations (e.g. x86 vs win vs arm, ubuntu vs debian vs rhel). Those env configurations are commonly configured in CI itself or possibly externalized to some other config. Moving them back into the buildpack may provide for an improved UX

Changing what "prod-like" means
I think there's a pretty reasonable use case here where operators may want to give developers the ability to change what "prod-like" means. For example, the GCP buildpacks has GOOGLE_DEVMODE that can be enabled for faster changes. I can see it being used as a developer-oriented configuration while maintaining guardrails in prod-like envs

[jama22]

Similar to devmode, I just ran across live reloading in the Paketo python buildpacks https://paketo.io/docs/howto/python/#using-bp_live_reload_enabled

[joshwlewis]

What would consume this new key? If a platform or user ran pack build -e CNB_EXEC_ENV=test my-app:test, a buildpack might contribute new launch entries flagged for test, but how would a platform or user determine which of the launch entries were all or test? My guess is at the very least, we'd want this key to show up in the io.buildpacks.build.metadata process-types label data?

[cz4rny]

Two questions:

1. How would one extract artifacts from the test env? Running unit tests, or any other static-analysis tools produces outputs separate from the end OCI image. Currently pack performs the build in a tmp directory, which is cleaned out at the end. How would one access the results of all the checks done in the dev env?

2. How would I ensure, that the code produced in another pack built, to the production environment is tested? Having the CNB_EXEC_ENV set could introduce a completely different set of layers and hence a different output. So I would test the source code and the binary produced from the test environment, but that doesn't mean I've tested the exact same binary, source code, and artifacts produced in the production env.

[jabrown85]

Two questions:

1. How would one extract artifacts from the test env? Running unit tests, or any other static-analysis tools produces outputs separate from the end OCI image. Currently pack performs the build in a tmp directory, which is cleaned out at the end. How would one access the results of all the checks done in the dev env?

This RFC aims to only produce a test image artifact. A test platform would then execute the image and collect the results (TAP/json/etc).

2. How would I ensure, that the code produced in another pack built, to the production environment is tested? Having the CNB_EXEC_ENV set could introduce a completely different set of layers and hence a different output. So I would test the source code and the binary produced from the test environment, but that doesn't mean I've tested the exact same binary, source code, and artifacts produced in the production env.

There would be no such guarantee. Production images vary quite a bit from stack to stack and there is no one size fits all solution for testing. For a practical example, a ruby or node app will often have test environment specific dependencies that should not make their way into the final production images. Another example is a simple go app on a scratch/tiny stack. The go tooling itself, go test, should not be packaged into the final production image.

Thinking out loud, a test pipeline could grab the SBOM/digests/shasum of the things it cares about and compare them to the production image that was built.

[cz4rny]

Right, so right now the image produced is a self-executing app built, whether it's npm, or go or whatever.

The test image would not execute anything but would contain the app with all of the, keeping to the npm example, dev dependencies. Maybe it's even built. And the pipeline would use that image, run its tools on top of that (static analysis, test, etc.), grab the produced outputs, and publish them.

[jabrown85]

The way I understand this is the produced image would have test processes contributed by buildpacks.

A golang buildpack would create a layer that contains go testing tools and maybe even run a go build ./... during the build phase so you can fail early. The go buildpack could contribute test process to launch.toml that runs go test ./.... That would be the default process launched when the image is executed in a testing pipeline. As you said, the pipeline could also choose to run static analysis as well as any other process on the image (maybe lint in this example) and capture the results.

[keskad]

Hi,

I wanted to comment the concept with a use-case. So, I don't see the point in creating a production images in environment X, while testing in environment Y. That's not consistent way of doing CD pipelines (current behavior) 🙂

I give buildpacks to teams, they set e.g. BP_NODE_VERSION 16.1 and it will execute with Node 16.1, while in tests it will still execute with hardcoded Node 16.0 because I have to maintain additional images. Using buildpacks in this scenario is loosing sense for me 🙂 More consistent way is to just use the Dockerfile with base image used in tests, because it has consistent Maven and Java version inside.

Due to nature of buildpacks - preparing the environment (by installing required tooling, setting up those tools) - I think the test phase is even a neccesary scenario to implement a valid CD pipeline in order to gain reproducible environment everytime. The tests needs to be running on something that is CLOSE TO PRODUCTION 🙂

So I see that build phase even as neccessary to implement a valid CD pipeline, so I like this proposal.

The case is similar to the principle, where we should not build artifact twice if promoting to next environment e.g. dev -> prod, but just reuse the artifact. There we should use the same build parameters, same tooling.

Its difficult to maitain e.g. Node 14, Node 15, Java 8, Java 11, and 15 other base images just for tests, while giving teams a possibility to set a Node or Java version in buildpacks.

A testing phase in my opinion would benefit in:

• not maintaing the base images internally just for running tests (in bigger organizations its a huge effort)
• have a consistent way of setting up both test & prod tooling (versions and parameters - e.g. team sets Node v16.1, then it is working in both test & prod)
• forget about testing command for each project (buildpack test phase would know a testing command for each technology just as it knows how to run the build)