Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create an issues report template #5628

Closed
juliamagan opened this issue Aug 1, 2024 · 6 comments
Closed

Create an issues report template #5628

juliamagan opened this issue Aug 1, 2024 · 6 comments

Comments

@juliamagan
Copy link
Member

Description

When reporting issues with failed tests by the QA team to other teams, it has been seen that the issues do not follow a standard and that their quality could be improved, so that it is easier for developers to debug the errors.

With this issue, we intend to establish a standard report, as well as document the process of bug reporting. Everything concluded will be documented in the QA documentation. In addition, we can consider creating a template in the different repositories for reporting issues.

@juliamagan
Copy link
Member Author

First approach:

# QA Report

| Version | Revision | Production/Development | Component | Install type | 
|:--:|:--:|:--:|:--:|:--:|
|    |    |    |    | Packages/Repository/Sources/Assistant |  


## Description

<!--
Insert here a brief description of the reported error.

Example:
During the tests performed at #ISSUE_NUMBER, the test TEST_NAME failed. In this test it could be seen that when ...
-->

### Evironment

<!--
Describe the environment used for the failed test.

Example:
| Node | Vagrant/AWS | Image | CPU | RAM (GB) | Disk memory (GB) | Packages/Repository/Sources branch |
|:--:|:--:|:--:|:--:|:--:|:--:|---|
| All-in-one | Vagrant | ubuntu/jammy64 | 2 | 4 | 20 | pre-release |
| Agent node | Vagrant | ubuntu/jammy64 | 1 | 2 | 10 | https://github.com/wazuh/wazuh/tree/4.9.0 |
| Agent node | Vagrant | roboxes/rhel9 | 1 | 2 | 10 | https://packages-dev.wazuh.com/pre-release/yum/wazuh-agent-4.9.0-1.x86_64.rpm |
-->


| Node | Vagrant/AWS | Image | CPU | RAM (GB) | Disk memory (GB) | Packages/Repository/Sources branch|
|:--:|:--:|:--:|:--:|:--:|:--:|---|
| Master node | | | | |  | |
| Worker node | | | | |  | |
| Indexer node | | | | |  | |
| Dashboard node | | | | |  | |
| Agent node | | | | |  | |
| Agent node | | | | |  | |

### Steps to reproduce

<!--
Describe the steps to reproduce the error. If the issue reports an error in an automated test, the steps can't be `launch the test`. We need to describe the steps that the test follows.

Example:


1. Deploy a Wazuh cluster and two agents
2. Enable vulnerability detector in the managers
3. Register the agents to each manager host
4. Wait until feeds are updated
5. Wait until first scan is performed
6. Check the reported vulnerabilities
7. Check that the agents are scanned and the vulnerable packages are detected
8. Change the manager of the agents
9. Wait until the next scan is performed
10. Check the reported vulnerabilities
11. Install a vulnerable package in both agents
12. Wait until the next scan is performed
13. Check that the vulnerable package is detected in both agents

-->

1. 
2. 
3. 


### Current result
<!--
Describe the current result specifying the errors. This means we can't say `The test fails`.

Example:
The manager is not being restarted.
The following log doesn't appear...
-->

### Expected result
<!--
Describe the expected result. We can't say `The test should pass`.

Example:
The manager should be restarted.
The following log should appear...
-->

### Attachments

<!--
Attach here anything that may be useful when debugging.
-->

- Build link:
- Build parameters:
- Artifacts: .zip
- Logs: .txt, .log or .zip

@juliamagan
Copy link
Member Author

A new section in the documentation has been added.

@rauldpm
Copy link
Member

rauldpm commented Aug 6, 2024

Change proposal

Header

  • Packages/Repository both install packages, maybe we should refer to install from a repository or install from a direct download
| Version | Stage | Production/Development | Component | Install type | 
|:--:|:--:|:--:|:--:|:--:|
|    |    |    |    | Download/Repository/Sources/Assistant | 
  • What is a Component? Is it a subsystem reference? Specific module?
  • Also I think it is better to use the stage instead of revision, or maybe the commit used in development tests

Node table

  • A lot of info here is hidden as it is done in the CI, for example, CPU/RAM/Disk are settings defined by default in the code or it is defined by the Allocator module, or even defined in external deployment, like the EBS storage of AWS AMIs
  • What is Disk memory? (Maybe you wanted to say Disk size?
  • Agent node is duplicated
| Node | Vagrant/AWS | Image | CPU | RAM (GB) | Disk memory (GB) | Packages/Repository/Sources branch|
|:--:|:--:|:--:|:--:|:--:|:--:|---|
| Master node | | | | |  | |
| Worker node | | | | |  | |
| Indexer node | | | | |  | |
| Dashboard node | | | | |  | |
| Agent node | | | | |  | |
| Agent node | | | | |  | |

Steps to reproduce

Describe the steps to reproduce the error. If the issue reports an error in an automated test, the steps can't be `launch the test`. We need to describe the steps that the test follows.
  • Replicating a CI error in manual steps is not always possible, those steps are very concrete, if we need to write each step that the CI does, we are going to be writing constantly those steps, also, some procedures are very long, if we have to write a step-by-step of the DTT procedure we are going to have like 50 steps at least

@jnasselle
Copy link
Member

Review

My two cents, adding a different aspect and enriching @rauldpm review.

In my experience, the problem with issue templates is that it's difficult to make users fill every field, add proper labels, and add to a project. I know that this could be enforced by the team, but it's easier to automate it.

After taking a look at GH documentation, I found that https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-issue-forms could be a way to simplify this. @damarisg already implemented this on this PR and we're going to give it a chance.

@juliamagan
Copy link
Member Author

About @rauldpm changes:

Packages/Repository both install packages, maybe we should refer to install from a repository or install from a direct download

I agree, done.

What is a Component? Is it a subsystem reference? Specific module?

By component, we mean manager, agent, indexer, or dashboard. The affected capability may be clearly specified and described in the problem description.

Also I think it is better to use the stage instead of revision, or maybe the commit used in development tests
In #4820 we will work to make the revision represent the stage or the commit, so it will be the same.

A lot of info here is hidden as it is done in the CI, for example, CPU/RAM/Disk are settings defined by default in the code or it is defined by the Allocator module, or even defined in external deployment, like the EBS storage of AWS AMIs

As part of the analysis, we could get this info. However, when this info is not relevant, we can delete it.

What is Disk memory? (Maybe you wanted to say Disk size?

Yes, sorry, updated.

Agent node is duplicated

Yes, it is duplicated on purpose, so that it can be seen that if there is more than one agent it must be specified, just as if there is no cluster environment, or multinode, there is no need to specify them.

Replicating a CI error in manual steps is not always possible, those steps are very concrete, if we need to write each step that the CI does, we are going to be writing constantly those steps, also, some procedures are very long, if we have to write a step-by-step of the DTT procedure we are going to have like 50 steps at least

When there are steps that cannot be replicated you can specify that they cannot be replicated by hand and only by CI. In addition, it is not necessary to replicate all the steps, only those that reproduce the error, i.e., all tests have many steps behind them, but not all of them are the ones that produce the error. As part of the analysis, most of the time it will be necessary to replicate the case manually, from which the steps to be reproduced can be extracted.


About @jnasselle changes:

Yes, I agree. I'll take a look at the forms, I have doubts if the examples with all the markdown will work properly. However, this is the first iteration and it will be only in our internal documentation. I will work on moving this to other repositories templates in the future.

@juliamagan
Copy link
Member Author

As this is a first approach, after several revisions, we will continue to publish the documentation and close this issue. With the use of these reports, we will see what we need to improve and add it to the repository templates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants