Skip to content

Commit

Permalink
Merge pull request #3212 from swcurran/managing-doc-site
Browse files Browse the repository at this point in the history 
Document the documentation site generation process
  • Loading branch information
@swcurran
swcurran committed Sep 3, 2024
2 parents 8e3cdec + 8ce7da9 commit e8d89e8
Show file tree
Hide file tree
Showing 5 changed files with 194 additions and 91 deletions.
135 changes: 135 additions & 0 deletions Managing-ACA-Py-Doc-Site.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Managing the ACA-Py Documentation Site

The ACA-Py documentation site is a [MkDocs Material] site generated from the
Markdown files in this repository. Whenever the `main` branch is updated or a
release branch is (possibly temporarily) created, the [publish-docs] GitHub
Action is fired, generating and publishing the documentation for the
updated/created branch. The generation process generates the static set of HTML
pages for the version in a folder in the `gh-pages` branch of this repo. The
static pages for each (other than the `main` branch) version are not updated
after creation. From time to time, some "extra" maintenance on the versions are
needed and this document describes those activities.

[MkDocs Material]: https://squidfunk.github.io/mkdocs-material/
[publish-docs]: https://github.com/hyperledger/aries-cloudagent-python/blob/main/.github/workflows/publish-docs.yml

## Generation Process

The generation process includes the following steps as part of the GitHub Action
and mkdocs configuration.

When the GitHub Action fires, it runs a container that carries out the following steps:

- Checks out the triggering branch, either `main` or `docs-v<version>` (e.g `docs-v1.0.0`).
- Runs the script [scripts/prepmkdocs.sh], which moves and updates some of the
markdown files so that they fit into the generated site. See the comments in
the scripts for details about the copying and editing done via the script. In
general, the copying of files is to put markdown files in the root folder into
the `docs` folder, and to update links that need to be changed to work on the
generated site. This allows us to have links working using the GitHub UI and
on the generated site.
- Invokes the mkdocs extension `mike` that generates the mkdocs HTML pages and
then captures and commits them into the `gh-pages` branch of the repository.
It also adds (if needed) a reference to the new version in the site's
"versions" dropdown, enabling users to pick the version of the docs they want
to look at. The process uses the [mkdocs.yml] configuration file in generating
the site.

[scripts/prepmkdocs.sh]: https://github.com/hyperledger/aries-cloudagent-python/blob/main/scripts/prepmkdocs.sh
[mkdocs.yml]: https://github.com/hyperledger/aries-cloudagent-python/blob/main/mkdocs.yml

## Preparing for a Release

When preparing for a release (or even just a PR to `main`) you can test the
documentation site on your local clone using the following steps. The steps
assume that you have installed `mkdocs` on your system. Guidance for that can be
found in the [MkDocs Material] documentation.

- Note the files changed in your repository that have not been committed. This
process will change and then "unchange" files in your local clone. The
"unchange" may not be perfect, so you want to be sure that no extra changed
files get into your next commit.
- Run the bash script [scripts/prepmkdocs.sh]. It will change a number of files
in your local repository.
- Run `mkdocs`. Watch for warnings of missing documents and broken links in the
startup messages. See the notes below for dealing with those issues.
- Open your browser and browse the site, looking for any issues.
- Update the documents, [mkdocs.yml] and the [scripts/prepmkdocs.sh] as needed,
repeating the generation process as needed.
- When you are happy run [scripts/prepmkdocs.sh] with the parameter `clean`.
This should undo the changes done by the script. You should check that there
no unexpected files changed that you don't want committed into the repo.

If there are missing documents, it may be that they are new Markdown files that
have not yet been added to the [mkdocs.yml] navigation. Update that file to add
the new files, and push the changes to the repository in a pull request. There
are a few files listed below that we don't generate into the documentation site,
and they can be ignored.

- `assets/README.md`
- `design/AnoncredsW3CCompatibility.md`
- `design/UpgradeViaApi.md`
- `features/W3cCredentials.md`

If there are broken links, it is likely because there is a Markdown link that
works using the GitHub UI (e.g. a relative link to a file in the repo) but
doesn't on the generated site. In general there are two ways to fix these:

- Change the link in the Markdown file so that it is a fully qualified URL vs. a
relative link, so that it works in both the GitHub UI and the generated site.
- Extend the [scripts/prepmkdocs.sh] `sed` commands so that the link differs in
the GitHub UI and the generated site -- working in both. A pain, but sometimes
needed...

## Removing RC Releases From the Generated Site

Documentation is added to the site for release candidates (RCs). When those
release candidates are replaced, we want to remove their documentation version
from the documentation site. In the current GitHub Action, the version
documentation is created but never deleted, so the process to remove the
documentation for the RC is manual. It would be nice to create a mechanism in
the GitHub Action to do this automatically, but its not there yet.

To delete the documentation version, do the following:

- In your local fork, checkout the gh-pages: `git checkout -b gh-pages --track
upstream/gh-pages` (or use whatever local branch name you want)
- Check your `git status` and make sure there are no changes in the branch --
e.g., new files that shouldn't be added to the `gh-pages` branch. If there are
any -- delete the files so they are not added.
- Remove the folder for the RC. For example `rm -rf 1.0.0rc4`
- Edit the `versions.json` file and remove the reference to the RC release in
the file.
- Push the changes via a PR to the ACA-Py `gh-pages` branch (don't PR them into
`main`!!).
- Merge the PR and verify (after a few minutes) that the drop down no longer has
the RC in it.

## Adding new 0.11.x Releases

The automatic generation process from ACA-Py started with release 0.12.0.
Unfortunately, we declared release 0.11.x to be an Long Term Support version and
so we still need to add 0.11.x version documentation to the generated site.
Here's the (lousy) process to do this. Typically, [swcurran] will do this and no
one else needs to worry about it. But for completeness, here is the process:

- Follow the instructions in the [aries-acapy-docs] repository to generate and
publish the documentation site for the new 0.11.x version.
- Have a local copy of the [aries-acapy-docs] repository. In that repo, run `git
checkout -b gh-pages --track upstream/gh-pages` to checkout a local copy of
the generated pages from that repo.
- In ACA-Py, run `git checkout -b gh-pages --track upstream/gh-pages` to create
a local branch from which you will push a PR.
- Copy the v0.11.x folder from [aries-acapy-docs] local to a new 0.11.x folder
in the ACA-Py local. Note the "v" that is on the folder in [aries-acapy-docs],
but not in ACA-Py.
- Edit the `versions.json` file to add the 0.11.x reference into the file.
- Push the changes via a PR to the ACA-Py `gh-pages` branch (don't PR them into
`main`!!).
- Merge the PR and verify (after a few minutes) that the drop down includes the
0.11.x version.

Ugly! The LTS for 0.11 ends in January 2025 and this process can be dropped.

[swcurran]: https://github.com/swcurran
[aries-acapy-docs]: https://github.com/hyperledger/aries-acapy-docs
134 changes: 48 additions & 86 deletions PUBLISHING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,86 +15,35 @@ Once ready to do a release, create a local branch that includes the following up
file as necessary. On completion of the testing, run the script
`./scripts/prepmkdocs.sh clean` to undo the temporary changes to the docs. Be
sure to do the last `clean` step -- **DO NOT MERGE THE TEMPORARY DOC
CHANGES.**

3. Update the CHANGELOG.md to add the new release. Only create a new section when working on the first release candidate for a new release. When transitioning from one release candidate to the next, or to an official release, just update the title and date of the change log section.

4. Include details of the merged PRs included in this release. General process to follow:

- Gather the set of PRs since the last release and put them into a list. A good
tool to use for this is the
[github-changelog-generator](https://github.com/github-changelog-generator/github-changelog-generator).
Steps:
- Create a read only GitHub token for your account on this page:
[https://github.com/settings/tokens](https://github.com/settings/tokens/new?description=GitHub%20Changelog%20Generator%20token)
with a scope of `repo` / `public_repo`.
- Use a command like the following, adjusting the tag parameters as
appropriate. `docker run -it --rm -v "$(pwd)":/usr/local/src/your-app
githubchangeloggenerator/github-changelog-generator --user hyperledger
--project aries-cloudagent-python --output 0.11.0rc2.md --since-tag 0.10.4
--future-release 0.11.1rc2 --release-branch main --token <your-token>`
- In the generated file, use only the PR list -- we don't include the list of
closed issues in the Change Log.

In some cases, the approach above fails because of too many API calls. An
alternate approach to getting the list of PRs in the right format is to use [OpenAI ChatGPT].

Prepare the following ChatGPT request. Don't hit enter yet--you have to add the data.

`Generate from this the github pull request number, the github id of the author and the title of the pull request in a tab-delimited list`

Get a list of the merged PRs since the last release by displaying the PR list in
the GitHub UI, highlighting/copying the PRs and pasting them below the ChatGPT
request, one page after another. Hit `<Enter>`, let the AI magic work, and you
should have a list of the PRs in a nice table with a `Copy` link that you should click.

Once you have that, open this [Google Sheet] and highlight the `A1` cell and
paste in the ChatGPT data. A formula in column `E` will have the properly
formatted changelog entries. Double check the list with the GitHub UI to make
sure that ChatGPT isn't messing with you and you have the needed data.
CHANGES.** For more details see the [Managing the ACA-Py Documentation Site] document.

3. Update the CHANGELOG.md to add the new release. Only create a new section
when working on the first release candidate for a new release. When
transitioning from one release candidate to the next, or to an official
release, just update the title and date of the change log section.

4. Collect the details of the merged PRs included in this release -- a list of PR
title, number, link to PR, author's github ID, and a link to the author's
github account. Gathering that data can be painful. Here are is the current
easiest way to do this -- using [OpenAI ChatGPT]:

> Prepare the following ChatGPT request. Don't hit enter yet--you have to add the data.
>
> `Generate from this the github pull request number, the github id of the author and the title of the pull request in a tab-delimited list`
>
> Get a list of the merged PRs since the last release by displaying the PR list in
> the GitHub UI, highlighting/copying the PRs and pasting them below the ChatGPT
> request, one page after another. Hit `<Enter>`, let the AI magic work, and you
> should have a list of the PRs in a nice table with a `Copy` link that you should click.
>
> Once you have that, open this [Google Sheet] and highlight the `A1` cell and
> paste in the ChatGPT data. A formula in column `E` will have the properly
> formatted changelog entries. Double check the list with the GitHub UI to make
> sure that ChatGPT isn't messing with you and you have the needed data.
[OpenAI ChatGPT]: https://chat.openai.com
[Google Sheet]: https://docs.google.com/spreadsheets/d/1gIjPirZ42g5eM-JBtVt8xN5Jm0PQuEv91a8woRAuDEg/edit?usp=sharing

If using ChatGPT doesn't appeal to you, try this scary `sed`/command line approach:

- Put the following commands into a file called `changelog.sed`

``` bash
/Approved/d
/updated /d
/^$/d
/^ [0-9]/d
s/was merged.*//
/^@/d
s# by \(.*\) # [\1](https://github.com/\1)#
s/^ //
s# \#\([0-9]*\)# [\#\1](https://github.com/hyperledger/aries-cloudagent-python/pull/\1) #
s/ / /g
/^Version/d
/tasks done/d
s/^/- /
```

- Navigate in your browser to the paged list of PRs merged since the last
release (using in the GitHub UI a filter such as `is:pr is:merged sort:updated
merged:>2022-04-07`) and for each page, highlight, and copy the text
of only the list of PRs on the page to use in the following step.
- For each page, run the command
`sed -e :a -e '$!N;s/\n#/ #/;ta' -e 'P;D' <<EOF | sed -f changelog.sed`,
paste in the copied text and then type `EOF`.
Redirect the output to a file, appending each page of output to the file.
- The first `sed` command in the pipeline merges the PR title and PR number
plus author lines onto a single line. The commands in the `changelog.sed`
file just clean up the data, removing unwanted lines, etc.
- At the end of that process, you should have a list of all of the PRs in a form you can
use in the CHANGELOG.md file.
- To verify you have right number of PRs, you can do a `wc` of the file and there
should be one line per PR. You should scan the file as well, looking for
anomalies, such as missing `\`s before `#` characters. It's a pretty ugly process.
- Using a `curl` command and the GitHub API is probably a much better and more
robust way to do this, but this was quick and dirty...

Once you have the list of PRs:

- Organize the list into suitable categories, update (if necessary) the PR description and add notes to clarify the changes. See previous release entries to understand the style -- a format that should help developers.
Expand Down Expand Up @@ -124,7 +73,7 @@ Once you have the list of PRs:
to better follow the semver rules.

8. Regenerate openapi.json and swagger.json by running
`../scripts/generate-open-api-spec` from within the `aries_cloudagent` folder.
`scripts/generate-open-api-spec` from within the `aries_cloudagent` folder.

Command: `cd aries_cloudagent;../scripts/generate-open-api-spec;cd ..`

Expand All @@ -142,7 +91,7 @@ Once you have the list of PRs:
PRs in the release, to complement the manually curated Changelog. Verify on
PyPi that the version is published.

11. New images for the release are automatically published by the GitHubAction
11. New images for the release are automatically published by the GitHubAction
Workflows: [publish.yml] and [publish-indy.yml]. The actions are triggered
when a release is tagged, so no manual action is needed. The images are
published in the [Hyperledger Package Repository under
Expand All @@ -153,16 +102,29 @@ Once you have the list of PRs:
Additional information about the container image publication process can be
found in the document [Container Images and Github Actions](docs/deploying/ContainerImagesAndGithubActions.md).

In addition, the published documentation site [https://aca-py.org] should be automatically updated to include the new release via the [publish-docs] GitHub Action.
Additional information about that process and some related maintainance activities that are needed from time to time can be found in the [Updating the ACA-Py Documentation Site] document.

[publish.yml]: https://github.com/hyperledger/aries-cloudagent-python/blob/main/.github/workflows/publish.yml
[publish-indy.yml]: https://github.com/hyperledger/aries-cloudagent-python/blob/main/.github/workflows/publish-indy.yml

12. Update the ACA-Py Read The Docs site by building the new "latest" (main
branch) and activating and building the new release. Appropriate permissions
are required to publish the new documentation version.
1. When a new release is tagged, create a new branch at the same commit with
the branch name in the format `docs-v<version>`, for example, `docs-v1.0.0`.
The creation of the branch triggers the execution of the [publish-docs]
GitHub Action which generates the documentation for the new release,
publishing it at [https://aca-py.org]. The GitHub Action also executes when
the `main` branch is updated via a merge, publishing an update to the `main`
branch documentation. Additional information about that documentation
publishing process and some related maintenance activities that are needed
from time to time can be found in the [Managing the ACA-Py Documentation Site] document.

[publish-docs]: https://github.com/hyperledger/aries-cloudagent-python/blob/main/.github/workflows/publish-docs.yml
[Managing the ACA-Py Documentation Site]: Managing-ACA-Py-Doc-Site.md
[https://aca-py.org]: https://aca-py.org

13. Update the [https://aca-py.org] website with the latest documentation by
creating a PR and tag of the latest documentation from this site. Details
are provided in the [aries-acapy-docs] repository.
13. Update the [ACA-Py Read The Docs site] by logging into Read The Docs
administration site, building a new "latest" (main branch) and activating
and building the new release by version ID. Appropriate permissions are
required to publish the new documentation version.

[https://aca-py.org]: https://aca-py.org
[aries-acapy-docs]: https://github.com/hyperledger/aries-acapy-docs
[ACA-Py Read The Docs site]: https://aries-cloud-agent-python.readthedocs.io/en/latest/
Loading

0 comments on commit e8d89e8

Please sign in to comment.