Skip to content

Commit

Permalink
add issue and PR templates (#909)
Browse files Browse the repository at this point in the history
Fix #885 

These templates crib heavily from other open-source repository tooling.

* provide initial template PRs
* provide various issue reporting options
* add label triggers for some common queue maintenance tasks

Added label automation will require addition of the following labels:
- [x] needs-linting
- [x] needs-unique-branch
- [x] needs-testing-environment
- [x] needs-pull-request-template
- [x] needs-more-information
  • Loading branch information
leondz committed Sep 10, 2024
2 parents 61ce5c4 + 5a17096 commit 7a99261
Show file tree
Hide file tree
Showing 8 changed files with 364 additions and 0 deletions.
50 changes: 50 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
name: Bug Report 🐞
about: Something isn't working as expected? Here is the right place to report.
labels: "bug"
---

<!--
Please fill out each section below, otherwise, your issue will be closed. This info allows garak maintainers to diagnose (and fix!) your issue as quickly as possible.
Useful Links:
- Wiki: https://docs.garak.ai/garak
Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
-->

## Steps to reproduce

How'd you do it?

1. ...
2. ...

This section should also tell us any relevant information about the
environment; for example, if a probe that used to work is failing,
tell us the target application and possibly model versions.

## Were you following a specific guide/tutorial or reading documentation?

If yes link the guide/tutorial or documentation you were following here, otherwise you may omit this section.

## Expected behavior

What should happen?

## Current behavior

What happens instead?

### garak version

Get this with the `--version` flag on the command line (or `git log -1 --pretty=oneline` for a source install).

## Additional Information

1. Operating system
2. Python version
3. Install method (`pypi`, `pip` based repo install, direct repository checkout with `git`)
4. Logs from execution run `report.html` / `report.jsonl` / `hitlog.jsonl` and if possible `garak.log`
5. Details of execution config such as command line flags or config files
6. Any relevant hardware or resource information
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
blank_issues_enabled: false
41 changes: 41 additions & 0 deletions .github/ISSUE_TEMPLATE/documentation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
name: Documentation 📝
about: Suggest better docs coverage for a particular tool or process.
labels: "documentation"
---

<!--
To make it easier for us to help you, please include as much useful information as possible.
Useful Links:
- Wiki: https://docs.garak.ai/garak
- Code reference: https://reference.garak.ai/
Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
-->

## Summary

What problem(s) did you run into that caused you to request additional documentation? What questions do you think we should answer? What, if any, existing documentation relates to this proposal?

Some recommended topics to cover:

- List the topics you think should be here.
- This list does not need to be exhaustive!

### Motivation

Why should we document this and who will benefit from it?

## Steps to resolve this issue

<!-- Your suggestion may require additional steps. Remember to add any relevant labels. Note that you'll need to fill in the link to a similar article as well as the correct section. Don't worry if you're not yet sure about these, especially if this is a brand new topic! -->

### Draft the doc

- [ ] If you can, write the doc, following the format listed in these resources:
- [Example of a similar article]()

### Open a pull request

- [ ] Open a pull request with your work including the words "closes #[this issue's number]" in the pull request description
26 changes: 26 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_suggestion.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
name: Feature Suggestion 💡
about: Suggest a new idea for the project.
labels: "enhancement"
---

<!--
To make it easier for us to help you, please include as much useful information as possible.
Useful Links:
- Wiki: https://docs.garak.ai/garak
Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
-->

## Summary

Brief explanation of the feature.

### Basic example

If the proposal involves a new or changed API, include a basic code example. Omit this section if it's not applicable.

### Motivation

Why are we doing this? What use cases does it support? What is the expected outcome?
26 changes: 26 additions & 0 deletions .github/ISSUE_TEMPLATE/plugin_suggestion.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
name: Plugin Suggestion 📦
about: Suggest a new idea for the project.
labels: "new-plugin"
---

<!--
To make it easier for us to help you, please include as much useful information as possible.
Useful Links:
- Wiki: https://docs.garak.ai/garak
Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
-->

## Summary

Brief explanation of the plugin (e.g. probe, generator, detector, ...).

### Basic example

If you have a Proof-of-Concept, blog post or any other useful references please let us know in this section.

### Motivation

Why are we doing this? What use cases does it support? What is the expected outcome?
21 changes: 21 additions & 0 deletions .github/ISSUE_TEMPLATE/question.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
name: Question 🤔
about: Usage question or discussion about garak.
labels: "question"
---

<!--
To make it easier for us to help you, please include as much useful information as possible.
Useful Links:
- Wiki: https://docs.garak.ai/garak
- Code reference: https://reference.garak.ai/
Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
-->

## Summary

## Relevant information

<!-- Provide as much useful information as you can -->
35 changes: 35 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
Tell us what this change does. If you're fixing a bug, please mention
the github issue number.

Please ensure you are submitting **from a unique branch** in your repository to `main` upstream.

## Verification

List the steps needed to make sure this thing works

- [ ] Supporting configuration such as generator configuration file
``` json
{
"huggingface": {
"torch_type": "float32"
}
}
```
- [ ] `garak -m <model_type> -n <model_name>`
- [ ] Run the tests and ensure they pass `python -m pytest tests/`
- [ ] ...
- [ ] **Verify** the thing does what it should
- [ ] **Verify** the thing does not do what it should not
- [ ] **Document** the thing and how it works ([Example](https://github.com/leondz/garak/blob/61ce5c4ae3caac08e0abd1d069d223d8a66104bd/garak/generators/rest.py#L24-L100))

If you are opening a PR for a new plugin that targets a **specific** piece of hardware or requires a **complex or hard-to-find** testing environment, we recommend that you send us as much detail as possible.

Specific Hardware Examples:
* GPU related
* Specific support required `cuda` / `mps` ( Please not `cuda` via `ROCm` if related )
* Minium GPU Memory

Complex Software Examples:
* Expensive proprietary software
* Software with an extensive installation process
* Software without an English language UI
164 changes: 164 additions & 0 deletions .github/workflows/labels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
name: Labels

# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions
permissions:
actions: none
checks: none
contents: none
deployments: none
id-token: none
# This action can update/close issues
issues: write
discussions: none
packages: none
pages: none
# This action can update/close pull requests
pull-requests: write
repository-projects: none
security-events: none
statuses: none

on:
pull_request_target:
types: [labeled]
issues:
types: [labeled]

jobs:
handle-labels:
runs-on: ubuntu-latest
steps:
- uses: actions/github-script@v6
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
// NOTE: The following section is JavaScript. Note that backticks will need to be escaped within
// the multiline comment strings in the following config. When editing this file, using JavaScript
// syntax highlighting might be easier.
//
// This script has intentionally been inlined instead of using third-party Github actions for both
// security and performance reasons.
const allConfig = {
pullRequests: {
'needs-linting': {
close: false,
comment: `
Thanks for your pull request! Before this pull request can be merged, it must pass the checks of our automated linting tools.
We use `black` to ensure the consistent format of our code. This can be run from the root directory of garak:
\`\`\`
black garak
\`\`\`
Please update your branch after these have been made, and reach out if you have any problems.
`
},
'needs-unique-branch': {
close: true,
comment: `
Thanks for your pull request! We require for all contributed code to come from a **from a unique branch** in your repository before it can be merged.
Please create a new branch in your fork of garak and resubmit this from that branch.
If you are using Git on the command line that may look like:
\`\`\`
# Checkout the main branch
git checkout main
# Create a new branch for your feature
git checkout -b <BRANCH_NAME>
# Add your new files
git add plugins/my-cool-new-plugin
# Commit your changes with a relevant message
git commit
# Push your changes to GitHub
git push origin <BRANCH_NAME>
# Now browse to the following URL and create your pull request!
# - https://github.com/leondz/garak/pulls
\`\`\`
This helps protect the process, ensure users are aware of commits on the branch being considered for merge, allows for a location for more commits to be offered without mingling with other contributor changes and allows contributors to make progress while a PR is still being reviewed.
Please do resubmit from a unique branch, we greatly value your contribution! :tada:
`
},
'needs-testing-environment': {
close: false,
comment: `
Thanks for your pull request! As part of our landing process, we manually verify that all plugins work as expected.
We have been unable to test this plugin successfully. This may be due to software or hardware requirements we cannot replicate.
To help unblock this pull request, please:
- Comment with links to documentation on how to set up an environment, and provide exact software version numbers to use
- Or comment guided steps on how to set up our environment for testing this plugin
Once there's a clear path for testing and evaluating this code, we can progress with this further.
`
},
'needs-pull-request-template': {
close: false,
comment: `
When creating a pull request, please ensure that the default pull request template has been updated with the required details.
`
},
},
issues: {
'needs-more-information': {
close: false,
comment: `
It looks like there's not enough information to replicate this issue. Please provide any relevant output and logs which may be useful in diagnosing the issue.
This includes:
- All of the item points within this [template](https://github.com/leondz/garak/blob/master/.github/ISSUE_TEMPLATE/bug_report.md)
- Screenshots showing the issues you're having
- Exact replication steps
The easier it is for us to replicate and debug an issue means there's a higher chance of this issue being resolved.
`
},
// Used for issues that have zero effort applied, potentially bot related
potato: {
close: true,
comment: `
When creating an issue, please ensure that the default issue template has been updated with the required details:
https://github.com/leondz/garak/issues/new/choose
Closing this issue. If you believe this issue has been closed in error, please provide any relevant output and logs which may be useful in diagnosing the issue.
`
},
}
};
const issueType = context.eventName === 'issues' ? 'issues' : 'pullRequests';
const config = allConfig[issueType][context.payload.label.name];
if (!config) {
return;
}
if (config.comment) {
const precedingWhitespaceLength = config.comment.split("\n")[1].search(/\S/);
const commentWithoutPrecedingWhitespace = config.comment.split("\n").map(line => line.substring(precedingWhitespaceLength)).join("\n").trim();
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: commentWithoutPrecedingWhitespace
});
}
if (config.close) {
await github.rest.issues.update({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
state: 'closed'
});
}

0 comments on commit 7a99261

Please sign in to comment.