Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Implement crate publishing on CI #12768

Merged
merged 21 commits into from
Dec 7, 2022

Conversation

joao-paulo-parity
Copy link
Contributor

@joao-paulo-parity joao-paulo-parity commented Nov 23, 2022

related to https://github.com/paritytech/release-engineering/issues/129 and https://github.com/paritytech/release-engineering/issues/122

See https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2107373 for an example.

Note that, although the publishing job is set up for master, publishing to crates.io won't be enabled quite yet because some parts of the publishing script are disabled for now (paritytech/releng-scripts#20). The idea is to try the script out for some time while only publishing to the local instance, and after we see that it works fine, the parts of the script which publishes to crates.io will be re-enabled gradually. We also need to resolve https://github.com/paritytech/release-engineering/issues/132 before that.

Some notes for reviewers

  • The publishing automation strips away [dev-dependencies] before publishing crates, thus some dependencies have been copied from [dev-dependencies] to [dependencies] as optional entries so that cargo can find them while verifying the packages' features before publishing.
    Note also that their [dependencies] entries don't include a version field in their import declaration because it's not relevant for them.
  • The CI job's time to completion should be improved substantially after rusty-cachier kicks in. rusty-cachier 's cache will only be created after the job runs on master.

close #12678
close https://github.com/paritytech/release-engineering/issues/122
close https://github.com/paritytech/release-engineering/issues/129

@joao-paulo-parity joao-paulo-parity changed the title [WIP] Try out crate releases on CI [WIP] Check crate publishing on CI Nov 23, 2022
@joao-paulo-parity joao-paulo-parity added A2-insubstantial Pull request requires no code review (e.g., a sub-repository hash update). B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit. labels Nov 23, 2022
@joao-paulo-parity joao-paulo-parity changed the title [WIP] Check crate publishing on CI Check crate publishing on CI Dec 1, 2022
@joao-paulo-parity joao-paulo-parity marked this pull request as ready for review December 1, 2022 17:17
@joao-paulo-parity joao-paulo-parity changed the title Check crate publishing on CI Implement crate publishing on CI Dec 2, 2022
ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions
@joao-paulo-parity
Copy link
Contributor Author

bot merge

@paritytech-processbot
Copy link

Waiting for commit status.

@bkchr
Copy link
Member

bkchr commented Dec 11, 2022

  • The publishing automation strips away [dev-dependencies] before publishing crates, thus some dependencies have been copied from [dev-dependencies] to [dependencies] as optional entries so that cargo can find them while verifying the packages' features before publishing.
    Note also that their [dependencies] entries don't include a version field in their import declaration because it's not relevant for them.

Okay this is completely wrong. I would be very happy that we don't do these things nor that we approve these kind of things. The underlying issue here being that we enable features for dev dependencies, which is wrong. To solve this issue properly, you need to remove the enabling of features from dev dependencies.

Next time please reach out to the devs before doing this.

@bkchr
Copy link
Member

bkchr commented Dec 11, 2022

This fixes the crates: #12899

@joao-paulo-parity
Copy link
Contributor Author

Okay this is completely wrong. I would be very happy that we don't do these things nor that we approve these kind of things. The underlying issue here being that we enable features for dev dependencies, which is wrong. To solve this issue properly, you need to remove the enabling of features from dev dependencies.

The change was made to work around how the code was set up at the time. I couldn't intuit that the code was "wrong" before since it was presumably approved by the devs the way it was. Glad to know that there is a proper fix in #12899.

@bkchr
Copy link
Member

bkchr commented Dec 12, 2022

Yes I know that the initial "wrong" cargo tomls weren't added by you. However, it failing in your process you could have asked in Substrate channel or similar on these things!

@joao-paulo-parity
Copy link
Contributor Author

@bkchr definitely, I'll keep it in mind for next time. Thanks for the tip.

@bkchr
Copy link
Member

bkchr commented Dec 12, 2022

Thank you @joao-paulo-parity :)

ltfschoen pushed a commit to ltfschoen/substrate that referenced this pull request Feb 22, 2023
* implement crate publishing from CI

* fix indentation

* use resource_group for job exclusivity

ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions

* correct publish = false

* Remove YAML anchors as GitLab's `extends:` doesn't need it

* Temporarily force cache upload for the new jobs

* Revert `RUSTY_CACHIER_FORCE_UPLOAD`

* pin libp2p-tcp=0.37.0 for sc-telemetry

* Revert "pin libp2p-tcp=0.37.0 for sc-telemetry"

This reverts commit 29146bf.

* always collect generated crates

* increase timeout for publish-crates-template

* Force upload the new job cache again

* Revert "Force upload the new job cache again"

This reverts commit 5a5feee.

* reformat

* improve timeout explanation

* s/usual/average

Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
ark0f pushed a commit to gear-tech/substrate that referenced this pull request Feb 27, 2023
* implement crate publishing from CI

* fix indentation

* use resource_group for job exclusivity

ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions

* correct publish = false

* Remove YAML anchors as GitLab's `extends:` doesn't need it

* Temporarily force cache upload for the new jobs

* Revert `RUSTY_CACHIER_FORCE_UPLOAD`

* pin libp2p-tcp=0.37.0 for sc-telemetry

* Revert "pin libp2p-tcp=0.37.0 for sc-telemetry"

This reverts commit 29146bf.

* always collect generated crates

* increase timeout for publish-crates-template

* Force upload the new job cache again

* Revert "Force upload the new job cache again"

This reverts commit 5a5feee.

* reformat

* improve timeout explanation

* s/usual/average

Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A2-insubstantial Pull request requires no code review (e.g., a sub-repository hash update). B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Annotate crates not meant to be published with publish = false
7 participants