Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimal project descriptor #32

Closed
wants to merge 21 commits into from
Closed
Changes from 16 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
0da24aa
Created an RFC for project descriptor
jkutner Jun 12, 2019
18b7fe2
Change includes/excludes to generate an error when both are present
jkutner Jul 24, 2019
da7af56
change images.name -> images.id
hone Jul 24, 2019
e3f0552
add stacks into [[buildpacks]]
hone Jul 24, 2019
fadee81
inline buildpack definition + stack
hone Jul 24, 2019
0cc209e
Added full schema to project descriptor
jkutner Sep 4, 2019
110c995
Remove path from project descriptor
jkutner Sep 4, 2019
86353dc
Clean up after forking from old RFC
jkutner Sep 26, 2019
3821b17
Updated based on discussion in the working group
jkutner Oct 4, 2019
684f133
Added definition around how images.id is used with multiple image tags
jkutner Oct 22, 2019
db9ccec
Made changes to project.toml RFC based on feedback from WG
jkutner Oct 30, 2019
b500dda
Added a minimal project descriptor proposal
jkutner Dec 5, 2019
fe0a1be
Updates based on WG feedback
jkutner Dec 5, 2019
92f2c06
Clean up some mistakes in minimal proj descriptor rfc
jkutner Dec 11, 2019
f3d2836
Add build, buildpacks, env to project descriptor
jkutner Dec 11, 2019
87ddbde
remove required from buildpack.id
jkutner Dec 13, 2019
077a1e4
add description about licenses
hone Dec 17, 2019
0d862de
Add description of project id and name
jkutner Dec 17, 2019
ab63bdb
Loosen requirements on project.toml license type and uri
jkutner Jan 8, 2020
2cabc8d
Fix a type in license spelling
jkutner Jan 8, 2020
f047502
Add build table to list of keys excluded from label
jkutner Jan 16, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
205 changes: 205 additions & 0 deletions text/0000-project-descriptor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
# Meta
[meta]: #meta
- Name: Minimal Project Descriptor
- Start Date: 2019-06-11
- CNB Pull Request: (leave blank)
- CNB Issue: (leave blank)
- Supersedes: https://github.com/buildpack/rfcs/pull/25

# Summary
[summary]: #summary

This is a proposal for a new file, `project.toml`, containing configuration for apps, services, functions, and buildpacks.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While .toml is a great readable format, I have concerns that picking .toml might add more friction to users coming from the container world, which seems to have converged towards using .yaml. In my opinion, this is a data point that should be factored in when picking a format.

As a middle ground, maybe this proposal could say that .yaml and .json format are also accepted, but .toml is preferred (in the end, they all represent structured data, and .yaml is even a superset of .json).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should stick to toml. My reasoning is that all other parts of the buildpack spec use yaml, and as a platform maintainer, it would be quite frustrating to have to support other formats both in the platform itself and in support requests coming in from users.

The name project.toml also has fewer (no?) conflicts with other projects (I see that references to project.yaml in OpenStack and ArgoCD).

The json format doesn't allow for comments, which is why it wasn't picked.


# Motivation
[motivation]: #motivation

This proposal is meant to address all of the following issues:

* Build metadata (i.e. information about the code being built)
* [Ignoring files](https://github.com/buildpack/pack/issues/210)

It also provides a foundation for solving the problems defined in the [Application descriptor issue](https://github.com/buildpack/spec/issues/44).

# What it is
[what-it-is]: #what-it-is

Terminology:

* **project**: a repository containing source code for an app, service, function, buildpack or a monorepo containing any combination of those.
* **image**: the output of a running a buildpack(s) against a project

The target personas for this proposal is buildpack users who need to enrich or configure buildpack execution. The elements in `project.toml` support many different use cases including these top level tables:

- `[project]`: (optional) defines configuration for a project
- `[metadata]`: (optional) metadata about the repository

Here is an overview of the complete schema:

```toml
[project]
id = "<string>"
name = "<string>"
jkutner marked this conversation as resolved.
Show resolved Hide resolved
version = "<string>"
authors = ["<string>"]
documentation-url = "<url>"
source-url = "<url>"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

include and exclude are used by the build process. How are the other fields here intended to be used? (Are they turned into image annotations?) Should there be builder and run-image fields?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're not specifying any behavior associated with the other fields.

I think any option on pack (including those you mentioned) is a good candidate for this descriptor. We batted around many different schemas, but are now trying to land on a very minimal subset of what was originally proposed.


[[project.licenses]]
type = "<string>"
jkutner marked this conversation as resolved.
Show resolved Hide resolved
uri = "<uri>"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nebhale for the non SPDX use case, do people usually point to a URI or just include a file to the LICENSE on disk locally?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of the URI is typically handled orthogonally from whether there's an SPDX value or not. The customers that have asked us for this don't so much want a link to the boiler-plate of the license (the SPDX takes care of that in most systems), but rather proof that the project uses that license. So you'll see:

[[metadata.dependencies.licenses]]
type = "GPL-2.0 WITH Classpath-exception-2.0"
uri  = "https://openjdk.java.net/legal/gplv2+ce.html"


[build]
include = ["<string>"]
exclude = ["<string>"]

[[build.buildpacks]]
id = "<string>"
version = "<string>"
uri = "<string>"

[[build.env]]
name = "<string>"
value = "<string>"

[metadata]
# additional arbitrary keys allowed
```

The following sections describe each part of the schema in detail.

## `[project]`

The top-level `[project]` table may contain configuration about the repository, including `id` and `version`, but also metadata about how it is authored, documented, and version controlled.

The `project.id`

```toml
[project]
id = "<string>"
name = "<string>"
version = "<string>"
```

## `[build.include]` and `[build.exclude]`

A list of files to include in the build (while excluding everything else):

```toml
[build]
include = [
"cmd/",
"go.mod",
"go.sum",
"*.go"
]
```

A list of files to exclude from the build (while including everything else)

```toml
[build]
exclude = [
"spec/"
]
```

The `.gitignore` pattern is used in both cases. The `exclude` and `include` keys are mutually exclusive, and if both are present the Lifecycle will error out. These lists apply to both buildpacks built with `pack create-package` and apps built with `pack build`.

Any files that are excluded (either via `include` or `exclude`) will be excluded before the build (i.e. not only exluded from the final image).

If both `exclude` and `include` are defined, the build process will error out.

## `[[build.buildpacks]]`

The build table may contain an array of buildpacks. The schema for this table is:

```toml
[[build.buildpacks]]
id = "<buildpack ID (optional)>"
version = "<buildpack version (optional default=latest)>"
uri = "<url or path to the buildpack (optional default=urn:buildpack:<id>)"
```

This defines the buildpacks that a platform should use on the repo.

Either an `id` or a `uri` are required, but not both. If `uri` is provided, `version` is not allowed.

## `[[build.env]]`

Used to set environment variables at build time, for example:

```toml
[[build.env]]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would launch.env be viable to be added as well? Thinking about environment variables that you want when your application launches that are outside of buildpacks (or override those provided by the buildpacks). Can be added later if we still need to discuss it further.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think launch.env is a bit more contentious, and we should discuss it in a separate update RFC.

name = "JAVA_OPTS"
value = "-Xmx1g"
```

## `[metadata]`

This table includes a some defined keys, but additional keys are not validated. It can be used to add platform specific metadata. For example:

```toml
[metadata.heroku]
pipeline = "foobar"
```

# How it Works
[how-it-works]: #how-it-works

The `project.toml` contents will be read by the platform and/or the lifecycle. The launch image will contain labels for the data in all fields except `include` and `exclude`.
jkutner marked this conversation as resolved.
Show resolved Hide resolved

## Example: Basic app

This is a complete `project.toml` for a simple app:

```toml
[project]
id = "io.buildpacks.my-app"
version = "0.1"
```

No other configuration is required.

## Example: Codified Buildpacks

Given an app with a `project.toml`, the lifecycle will read the `build.buildpacks` and generate an ephemeral buildpack group in the lifecycle. Only the buildpacks listed in `build.buildpacks` will be run (no other groups will be run). For example, an app might contain:

```toml
[[build.buildpacks]]
id = "io.buildpacks/java"
version = "1.0"

[[build.buildpacks]]
id = "io.buildpacks/nodejs"
version = "1.0"
```

These entries override any defaults in the builder image. If, for example, the project code contains a `Gemfile` and the `heroku/buildpacks` builder image is used, this will override the default buildpack groups, which would normally detect and run the `heroku/ruby` buildpack.

This is similar to running `pack build --buildpack io.buildpacks/java,io.buildpacks/nodejs`.

# Drawbacks
[drawbacks]: #drawbacks

- The use of the filename `project.toml` does not strongly indicate that it can be used with buildpacks
- Conflicts with `Project.toml` from Julia

# Alternatives
[alternatives]: #alternatives

- Name the file `package.toml` and keep the schema described herein.
- Overload `buildpack.toml` as described in [PR #2](https://github.com/buildpack/rfcs/pull/3)

# Prior Art
[prior-art]: #prior-art

- [`manifest.json`](https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html)
- [`app.json`](https://devcenter.heroku.com/articles/app-json-schema)
- [`.buildpacks`](https://github.com/heroku/heroku-buildpack-multi)
- [heroku-buildpack-multi-procfile](https://github.com/heroku/heroku-buildpack-multi-procfile)

# Unresolved Questions
[unresolved-questions]: #unresolved-questions

- How do we support `[[images]]` in the future?