TriBITS CDash Dashboard
TriBITs is currently being tested using at using the GitHub Actions with test results submitted to the CDash site:
TriBITS is tested for a few different versions of CMake, compilers, Python, and other variations post to the following CDash groups:
-
Nightly: Nightly builds posted every day regardless if there have been any repository activity that day [1]. -
Continuous: Pushes to 'master' [2]. -
Pull Request: Pull requests against 'master' withpr-<id>_prepended to the build names for the PR ID. -
Experimental: Any other submits of TriBITS test results using, for example,make dashboardon local machines.
For more details on the testing that is done, see the active GitHub Actions drivers.
- [1]: Nightly builds primarily test that underlying infrastructure to ensure that everything is working so that when PRs and pushes are done, it will work when needed.
- [2]: Even through each branch on a PR is tested before merging to 'master', it is still important to test when PR branches are merged to 'master' because two independent PRs that are tested at the same time can pass all of the tests independently but fail when merged together (i.e. a violation of the "additive test assumption of branches").
Automated testing of TriBITS nightly builds and Pull Requests is currently performed using GitHub Actions as described above. TriBITS has implemented bidirectional links from the GitHub actions jobs to the results on CDash and from the builds on CDash back to the GitHub Actions. The builds on CDash for a single PR testing iteration look like:
(NOTE: The current set of builds being done may be different that what is shown above.)
As shown above, each build on CDash includes a "Revision" field that gives the shortened Git SHA1 for the commit being tested. In the case of a PR, this is the SHA1 for a merge commit merging the PR topic branch into the target branch (i.e. 'master'). If one clicks on that SHA1 on CDash it takes you to a CDash page like this that gives the full SHA1 and is hyperlinked to the commits on GitHub. Amazingly, GitHub stores these merge commits so they can be examined on GitHub.
As shown above, each build on CDash shows the icon 
that provides a link back to the PR (PR #401 in this case).
In addition, the GitHub Actions Jobs for a PR testing iteration provide hyperlinks to the build result on CDash under the "URL to results on CDash" step of the job as shown below.
Screenshots of the GitHub Actions Job hyperlink to results on CDash
On each passing GitHub Actions testing job, there is a "URL to results on CDash" step that contains a link to the results on CDash as shown below:
The first Link to this build's results on CDash "Link to this build's results on CDash" takes you to a CDash build view as shown below:
However, when there is a failure, the "URL to results on CDash" step is not run but a hyperlink to the results on CDash is also provided at the end of the "Run configure, build, test and submit to CDash" step as shown below:
By printing the URL to the results on CDash at end of this job step, it will be seen at the bottom.
In addition, the URLs to results on CDash is also printed near the top of the output in the "Run configure, build, test and submit to CDash" job step to allow the user to watch those results as they show up on CDash as shown below:
As shown above, the builds for any given PR <pr-id> (401 in this case) is given the build-name prefix pr-<pr-id>_. This makes it easy to create a CDash query locating all of the configure, build, and test results associated with a PR using a query URL like this for PR #401 (just replace 401 in the Build Name field with the <pr-id> for the PR of interest). This shows results like shown below.
Screenshot of all build and test results on CDash for PR #410
One can view all of the GitHub Actions PR testing iterations for a given PR using a GitHub Actions Query like this for PR #401 with topic branch 154-improve-cdash-github-interaction with results as shown below.
Screenshot of query for all GitHub Actions test iterations for a given PR branch
It is not too hard to match up those GitHub Actions PR testing iterations and the builds on CDash to go back in history. But again, going from a given GitHub Action job to results on CDash is a direct hyperlink. It is just that going from an arbitrary CDash build to its driving GitHub Actions Job is not as direct. (Adding direct links for that would be a feature request.)
For details on the implementation of this, see GitHub Actions drivers, PR #389, #401, and other commits related to Issues #363 and #154.