Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: Use codecov for coverage reports #632

Merged
merged 3 commits into from
Oct 31, 2023
Merged

ci: Use codecov for coverage reports #632

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
084554c
ci: Use codecov for coverage reports
aborgna-q Oct 30, 2023
240eb0b
Only run coverage if tests succeed
aborgna-q Oct 31, 2023
f78e789
Bring back daily coverage message
aborgna-q Oct 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,4 +88,29 @@ jobs:
- name: Tests with no features
run: cargo test --verbose --no-default-features
- name: Tests with all features
run: cargo test --verbose --all-features
run: cargo test --verbose --all-features

coverage:
if: github.event_name != 'merge_group'
needs: [tests, check]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: mozilla-actions/sccache-action@v0.0.3
- uses: dtolnay/rust-toolchain@nightly
with:
components: llvm-tools-preview
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov
- name: Run tests with coverage instrumentation
run: |
cargo llvm-cov clean --workspace
cargo llvm-cov --doctests
- name: Generate coverage report
run: cargo llvm-cov report --codecov --output-path coverage.json
- name: Upload coverage to codecov.io
uses: codecov/codecov-action@v3
with:
files: coverage.json
name: ubuntu
token: ${{ secrets.CODECOV_TOKEN }}
32 changes: 14 additions & 18 deletions .github/workflows/coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
schedule:
# 04:00 daily
- cron: '0 4 * * *'
workflow_dispatch: {}

jobs:
check-coverage:
Expand All @@ -12,12 +13,12 @@ jobs:
msg: ${{ steps.make_msg.outputs.msg }}
steps:
- uses: actions/checkout@v4
- name: Install stable toolchain
uses: dtolnay/rust-toolchain@stable
- name: Install llvm
uses: KyleMayes/install-llvm-action@v1
- name: Install rust toolchain
uses: dtolnay/rust-toolchain@nightly
with:
version: "16.0" # should match version used by rustc
components: llvm-tools-preview
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov
- name: Download previous summary
uses: dawidd6/action-download-artifact@v2
with:
Expand All @@ -26,19 +27,13 @@ jobs:
name: coverage-summary
- name: Rename file
run: mv coverage-summary.txt coverage-summary-prev.txt
- name: Run tests with coverage instrumentation
run: |
cargo llvm-cov clean --workspace
cargo llvm-cov --doctests
- name: Generate coverage report and summary
env:
RUSTFLAGS: "-C instrument-coverage"
run: |
TGT=`cargo test --tests 2>&1 | grep Running | awk -F'[()]' '{print $2}'`
llvm-profdata merge -sparse default_*.profraw -o hugr.profdata
llvm-cov show --format=html --ignore-filename-regex='/.cargo/registry' --instr-profile=hugr.profdata --output-dir coverage --object ${TGT}
llvm-cov report --ignore-filename-regex='/.cargo/registry' --instr-profile=hugr.profdata --object ${TGT} | grep TOTAL | awk '{print $10}' | tr -dc '[:digit:].' > coverage-summary.txt
- name: Upload full report
uses: actions/upload-artifact@v3
with:
name: coverage
path: coverage/
cargo llvm-cov report --summary-only | grep TOTAL | awk '{print $10}' | tr -dc '[:digit:].' > coverage-summary.txt
- name: Upload summary
uses: actions/upload-artifact@v3
with:
Expand All @@ -48,11 +43,12 @@ jobs:
id: make_msg
run: |
change="`cat coverage-summary-prev.txt`% --> `cat coverage-summary.txt`%"
codecov="https://codecov.io/gh/${{ github.repository }}?search=&trend=7%20days"
if (( $(echo "`cat coverage-summary-prev.txt` < `cat coverage-summary.txt` + 0.04" | bc -l) ))
then
echo "msg=Coverage check for hugr shows no regression (${change}). ✅ ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" >> "$GITHUB_OUTPUT"
echo "msg=Coverage check for hugr shows no regression (${change}). ✅ ${codecov}" >> "$GITHUB_OUTPUT"
else
echo "msg=Coverage check for hugr shows regression (${change}). ❌ ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" >> "$GITHUB_OUTPUT"
echo "msg=Coverage check for hugr shows regression (${change}). ❌ ${codecov}" >> "$GITHUB_OUTPUT"
fi
notify-slack:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way we can still send a notification to slack with the headline number?

Copy link
Collaborator Author

@aborgna-q aborgna-q Oct 31, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hoped the slack app would have this option, but I didn't find it.

I brought back the cron job (simplified a bit with cargo-llvm-cov). Instead of storing the full coverage result it now just links to the codecov page.
Unfortunately there is no web view for comparing two commits (that's only on the API), so I just link to the landing page with a trend graph.

needs: check-coverage
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ quantinuum-hugr

[![build_status][]](https://github.com/CQCL/hugr/actions)
[![msrv][]](https://github.com/CQCL/hugr)
[![codecov][]](https://codecov.io/gh/CQCL/hugr)

The Hierarchical Unified Graph Representation (HUGR, pronounced _hugger_) is the
common representation of quantum circuits and operations in the Quantinuum
Expand Down Expand Up @@ -38,4 +39,5 @@ This project is licensed under Apache License, Version 2.0 ([LICENSE][] or http:

[build_status]: https://github.com/CQCL/hugr/workflows/Continuous%20integration/badge.svg?branch=main
[msrv]: https://img.shields.io/badge/rust-1.70.0%2B-blue.svg
[codecov]: https://img.shields.io/codecov/c/gh/CQCL/hugr?logo=codecov
[LICENSE]: LICENCE