Node release workflow #378

acalcutt · 2022-07-30T01:15:14Z

This is an attempt to create a node release workflow for #314

It is made to publish to github releases and the npm registry. It is based on what I am using for @acalcutt/maplibre-gl-native and https://github.com/acalcutt/maplibre-gl-native/releases. It is based on some ideas and conversation from @mnutt in #217

What this needs:
'Environment Secrets' set in github for 'NODE_PRE_GYP_GITHUB_TOKEN' and 'NPM_TOKEN'

NODE_PRE_GYP_GITHUB_TOKEN (needed to publish to github)

Within GitHub, create a new authorization:

go to Settings -> Developer settings
click [Personal access tokens](https://github.com/settings/tokens)
click Generate new token
Select public_repo and repo_deployment
Generate Token
copy the key that's generated and set NODE_PRE_GYP_GITHUB_TOKEN environment variable to it. Within your command prompt:
SET NODE_PRE_GYP_GITHUB_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

NPM_TOKEN (needed to publish to the maplibre npmjs registry)
This needs a maplibre npmjs token which has publish rights...like created here https://docs.npmjs.com/creating-and-viewing-access-tokens

Other Notes

I had to make two changes to node-pre-gyp-github . The first change made it target the 'main' branch instead of 'master'. The second makes the name include 'node-' to fit in with the other maplibre releases (so the name would be like 'node-v5.0.0' instead of 'v5.0.0').
Because there is no automatic versioning on this project, the version in package.json must be changed before running the node-release workflow.
I removed 'macos-11.0' and 'ubuntu-18.04' after making the node matrix because it was just to many jobs running. also 'ubuntu-18.04' had trouble building on node 18 due to dependencies not available in that OS. I was planning to use the newer MacOS OS, but was asked to use the older one for compatibility (the older os version would work on the newer od, but not the other way around)

This reverts commit fe87e4a.

error: A correctly formatted GitHub repository.url was not found within package.json

This reverts commit 50efb32.

ntadej · 2022-09-11T11:31:50Z

OK, we'll have to figure out how to bypass this. Do you have reviews disabled in JS or how do you do it there?

birkskyum · 2022-09-11T11:32:25Z

Review is not disabled in JS. We do manually update the version and Changelog in a PR, and hit release. It's maybe slower, but it feels very controlled and we don't publish releases often enough for it become annoying.

birkskyum · 2022-09-11T11:33:53Z

something like this:
maplibre/maplibre-gl-js#1556

acalcutt · 2022-09-11T11:54:52Z

I tested enabling branch protection in my branch and saw the same thing. It looks like the second option here works, specifying a token with permission on the checkout https://stackoverflow.com/a/63733988

Should I submit the change to this PR or make a new one?

birkskyum · 2022-09-11T11:57:20Z

Because this is merged already, I think a new one is necessary.

ntadej · 2022-09-11T11:59:00Z

What you could also think of is to make an action to make a PR with the changelog updates and version bump and then one to do final processing when the tag is made. This could also work.

acalcutt · 2022-09-11T12:08:03Z

OK I added #466

For the tag, that gets created by node-pre-gyp-github which is already using a git token. we will have to see if that has the same issue

acalcutt · 2022-09-11T12:24:53Z

Hmm, that fix didn't seem to work here. wondering CR_PAT doesn't have the permission needed

birkskyum · 2022-09-11T12:28:52Z

The PR is not, but the tag is created automatically in the JS repo in the release workflow.

acalcutt · 2022-09-11T12:33:23Z

In my repo I am using the NODE_PRE_GYP_GITHUB_TOKEN secret for this, which seems to be working for me. That was created like this, with public_repo and repo_deployment permissions

Within GitHub, create a new authorization:

go to Settings -> Developer settings
click [Personal access tokens](https://github.com/settings/tokens)
click Generate new token
Select public_repo and repo_deployment
Generate Token
copy the key that's generated and set NODE_PRE_GYP_GITHUB_TOKEN environment variable to it. Within your command prompt:
SET NODE_PRE_GYP_GITHUB_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Though that token is a personal token and I am the administrator of those repos, so that could make a difference.

birkskyum · 2022-09-11T13:30:57Z

I've adding my PAT (generated as you described) as a NODE_PRE_GYP_GITHUB_TOKEN in the repo. In principle that could allow skipping the branch protection. @acalcutt are you able to use the key from the script, and is public_repo and repo_deployment enough? @ntadej , what's your view on this approach? ... the strategy seems a bit dangerous to me.

acalcutt · 2022-09-11T14:00:28Z

This is what I am using in my branch, it is what the node-pre-gyp-github readme says to use for permissions and it seemed to work.

These are the only two checkbox checked in permissions

I just did a test run of this in my fork and it worked
https://github.com/acalcutt/maplibre-gl-native/actions/runs/3031922268

birkskyum · 2022-09-11T14:03:40Z

Maybe swapping secrets.CR_PAT with the PAT-key I just added "secrets.NODE_PRE_GYP_GITHUB_TOKEN" will resolve it then, because it has the same checkboxes marked.

wipfli · 2022-09-12T05:39:46Z

When adding new secrets, please write down what they do in #312. We could also start adding some documentation about the build processes and secrets used in this repo...

wipfli · 2022-09-12T05:41:43Z

@birkskyum as a Governing Board member you have access to our Bitwarden key manager. Do you think it would make sense to store clear-text passwords and keys there as well? I am asking because maybe it is enough if we just document well how the secrets were generated and who has access to them...

birkskyum · 2022-09-13T14:31:36Z

The release workflow is running now.

acalcutt · 2022-09-13T20:25:42Z

It looks like it all published correctly. One thing I notice though is it shows that it was published by birkskyum not github-actions. Most likely because we used your token for the version bump checkout/push

Also, I know you we asking about the security of this. I was thinking about that and I wonder what would could happen if someone moved that into a pr ci script. could they affect something real?

Maybe I should make NODE_PRE_GYP_GITHUB_TOKEN and NPM_TOKEN some type of enterable field so it's not part of the workflow? idk

ntadej · 2022-09-13T20:28:57Z

Maybe we should figure out if there's an action for that so we do not do it manually. It may help.

birkskyum · 2022-09-13T20:31:08Z

This release workflow works quite well for gl-js to npm.
https://github.com/maplibre/maplibre-gl-js/blob/main/.github/workflows/release.yml

I wouldn't mind taking the dangerous admin-push permissions away from the workflow, so we manually had to merge the updated version in package.json / package-lock.json / CHANGELOG.md before it publishes.

acalcutt · 2022-09-13T20:57:58Z

That looks like it also uses secrets.GITHUB_TOKEN and secrets.NPM_ORG_TOKEN, so possibly the same concern?

The PR ci workflow runs automatically with whatever workflow someone uploads.... so my thinking would be what happens if someone maliciously threw those secrets in a ci?

birkskyum · 2022-09-13T21:06:12Z

In that workflow, GITHUB_TOKEN is used to make a tag and a release, not push a commit, but if it has the same permissions I guess your concern is the same. I don't know what permissions were set for the GITHUB_TOKEN when it was created.

ntadej · 2022-09-13T21:11:24Z

Indeed, but something in the current workflow made you Birk the releasing user, not github-actions. Maybe GITHUB_TOKEN is enough and can replace NODE_PRE_GYP_GITHUB_TOKEN?

acalcutt · 2022-09-13T21:11:35Z

The permission we gave NODE_PRE_GYP_GITHUB_TOKEN we so node-pre-gyp-github could release the binaries. It just happended to work for that push too

ntadej · 2022-09-13T21:12:58Z

Also note that in the Qt release workflow I added

    permissions:
      contents: write

to the job. Not sure where I copied it from but this may add more permissions to GITHUB_TOKEN.

birkskyum · 2022-09-13T21:13:39Z

Also, regarding actions automatically running on a PR - it doesn't start workflows automatically for first-time contributors, but it's hardly a guarantee for anything.

acalcutt · 2022-09-13T21:34:43Z

On the plus side I tried testing it in https://github.com/acalcutt/tileserver-gl/tree/maplibre_pre and it seems to work as expected.

acalcutt added 30 commits April 24, 2022 20:33

Update package.json

ab01fc8

try to get publish working

af985ce

Update node-ci.yml

3bc4391

Update node-ci.yml

103921e

Update package.json

753eedf

Update package.json

bd2bf52

Update package.json

e8d6ac7

Try github publish

e4b3374

fix missing comma

a416a0d

Update package-lock.json

187217b

change publish order

47e250f

remove old node-pre-gyp

fe87e4a

Revert "remove old node-pre-gyp"

577144b

This reverts commit fe87e4a.

change to NODE_PRE_GYP_GITHUB_TOKEN

50efb32

change url format to try and fix error

93be6a9

error: A correctly formatted GitHub repository.url was not found within package.json

Revert "change to NODE_PRE_GYP_GITHUB_TOKEN"

f30e932

This reverts commit 50efb32.

put back NODE_PRE_GYP_GITHUB_TOKEN

1135485

remove other builds for now, remove matrix.os

5f111f5

fix user?

62f5b55

change remote_path

c09aa6b

Update package.json

f0874ab

use node-pre-gyp-github that uses main instead of master

69a64fa

update version

2915d1c

fix "remate_path", which gets used as "tag_name"

ed7a06d

try updating links

2740339

put back macos-11.0

1995cc5

Update version to test new release

22da555

set to false to see if this works

4219597

put back host url

49f867d

try to make npm only publish once

1ce1d17

acalcutt mentioned this pull request Sep 11, 2022

add node release checkout token #466

Merged

acalcutt mentioned this pull request Sep 11, 2022

Update token to NODE_PRE_GYP_GITHUB_TOKEN #467

Merged

birkskyum mentioned this pull request Sep 13, 2022

Publish @maplibre/maplibre-gl-native npm module #314

Closed

birkskyum mentioned this pull request Sep 21, 2022

Increase minimum node version to 14 #495

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Node release workflow #378

Node release workflow #378

acalcutt commented Jul 30, 2022 •

edited

Loading

ntadej commented Sep 11, 2022

birkskyum commented Sep 11, 2022 •

edited

Loading

birkskyum commented Sep 11, 2022

acalcutt commented Sep 11, 2022

birkskyum commented Sep 11, 2022 •

edited

Loading

ntadej commented Sep 11, 2022

acalcutt commented Sep 11, 2022

acalcutt commented Sep 11, 2022 •

edited

Loading

birkskyum commented Sep 11, 2022 •

edited

Loading

acalcutt commented Sep 11, 2022

birkskyum commented Sep 11, 2022 •

edited

Loading

acalcutt commented Sep 11, 2022

birkskyum commented Sep 11, 2022 •

edited

Loading

wipfli commented Sep 12, 2022

wipfli commented Sep 12, 2022

birkskyum commented Sep 13, 2022

acalcutt commented Sep 13, 2022

ntadej commented Sep 13, 2022

birkskyum commented Sep 13, 2022 •

edited

Loading

acalcutt commented Sep 13, 2022

birkskyum commented Sep 13, 2022 •

edited

Loading

ntadej commented Sep 13, 2022

acalcutt commented Sep 13, 2022

ntadej commented Sep 13, 2022

birkskyum commented Sep 13, 2022

acalcutt commented Sep 13, 2022

Node release workflow #378

Node release workflow #378

Conversation

acalcutt commented Jul 30, 2022 • edited Loading

ntadej commented Sep 11, 2022

birkskyum commented Sep 11, 2022 • edited Loading

birkskyum commented Sep 11, 2022

acalcutt commented Sep 11, 2022

birkskyum commented Sep 11, 2022 • edited Loading

ntadej commented Sep 11, 2022

acalcutt commented Sep 11, 2022

acalcutt commented Sep 11, 2022 • edited Loading

birkskyum commented Sep 11, 2022 • edited Loading

acalcutt commented Sep 11, 2022

birkskyum commented Sep 11, 2022 • edited Loading

acalcutt commented Sep 11, 2022

birkskyum commented Sep 11, 2022 • edited Loading

wipfli commented Sep 12, 2022

wipfli commented Sep 12, 2022

birkskyum commented Sep 13, 2022

acalcutt commented Sep 13, 2022

ntadej commented Sep 13, 2022

birkskyum commented Sep 13, 2022 • edited Loading

acalcutt commented Sep 13, 2022

birkskyum commented Sep 13, 2022 • edited Loading

ntadej commented Sep 13, 2022

acalcutt commented Sep 13, 2022

ntadej commented Sep 13, 2022

birkskyum commented Sep 13, 2022

acalcutt commented Sep 13, 2022

acalcutt commented Jul 30, 2022 •

edited

Loading

birkskyum commented Sep 11, 2022 •

edited

Loading

birkskyum commented Sep 11, 2022 •

edited

Loading

acalcutt commented Sep 11, 2022 •

edited

Loading

birkskyum commented Sep 11, 2022 •

edited

Loading

birkskyum commented Sep 11, 2022 •

edited

Loading

birkskyum commented Sep 11, 2022 •

edited

Loading

birkskyum commented Sep 13, 2022 •

edited

Loading

birkskyum commented Sep 13, 2022 •

edited

Loading