Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wazuh packages redesign tier 2 #2904

Open
4 of 12 tasks
havidarou opened this issue Apr 8, 2024 · 1 comment
Open
4 of 12 tasks

Wazuh packages redesign tier 2 #2904

havidarou opened this issue Apr 8, 2024 · 1 comment
Assignees
@damarisg
Labels
level/objective type/enhancement Enhancement issue

Comments

@havidarou
Copy link
Member

havidarou commented Apr 8, 2024 •
edited by damarisg
Loading

Description

Wazuh-packages (QA, Agent, CppServer)

  • Clean up everything
  • Add support to PPC packages (Agent)

Requirements and Restrictions

  • Wazuh packages redesign tier 1 requirements should be taken into account
  • The new repository must include QA related code and tools
    • New Jenkins should be used for everything QA related after 5.0
    • Current wazuh-packages repository is deprecated after 4.10
    • Current wazuh-jenkins repository is deprecated after 4.10
    • Current wazuh-tools repository is deprecated after 4.10
    • Migrate scripts and pipelines that sync GitHub projects with GitHub.(QA)
    • Identify existing pipelines and migrate those that continue to provide coverage to Wazuh 4.x. (QA)
  • The release procedure should be improved by:
    - Improving repository metadata generation (avoid full metadata sync)
    - Reducing execution time
    - Incorporating backups and storage
    - Mirroring production bucket (release to the mirror, sync main with the mirror, restore main with the mirror)
  • The following installers will be migrated from wazuh-packages repository:
    Agent installers:
    - HP-UX
    - Solaris Intel/SPARC
    - AIX
  • The following will be migrated to the new DevOps repositories (both packages and automation):
    - OVA
    - AMI
    - Wazuh unattended scripts
    - Wazuh Puppet
    - Wazuh Docker
    - Wazuh Kubernetes
    - Wazuh Ansible
    - Training environment
    - Demo environment
  • Add support to PPC packages (Agent)

Auditor

  • Involved: DevOps, QA, Agent, CppServer
  • esponsible: @wazuh/devel-qa-div3
  • DRI name: @davidjiglesias
  • Objective: Wazuh packages redesign tier 2

Planning

Previous tier

Dependencies

Spike

Mandatory tasks

QA

Primary

Secondary

DevOps

Agent

Move to 4.10.1

Conclusion (WIP)

QA

Release procedure

  • S3 sign check: S3 sign checking now works in multi-threaded mode, where each thread runs in a separate VM to handle its errors (destroying the VM and leaving a log of the error). It helps achieve asynchronous independence from the executions in S3_sign_check.
  • Reducing execution time: There's no way to speed up DEB metadata generation without creating bigger problems that could compromise the process.
  • Allow release-tool.py execution on AWS: It allows the launch tool to run on AWS infrastructure (4vcpu and 8GB).
    In addition, it was adapted to local execution and in local containers, providing a single entry point for the tool. It balanced the increase in time in metadata generation.
    Stage v4.9.0-beta2 Current implementation on AWS
    RPM - Download 52m 7m
    RPM - Metadata and upload - 4m20s
    RPM - Metadata 20s -
    RPM - Upload 40s -
    DEB - Download 49m 5m
    DEB - Metadata and upload - 46m
    DEB - Metadata 22m -
    DEB - Upload 40s -
  • Improving repository metadata generation: Added new documentation on how to modify recipe files and regenerate metadata without adding files to the repository, added new use cases such as Stage support, release protocol, and metadata regeneration, improved requirements, and removed redundant information.
  • Incorporating backups and storage: It provides an optional backup mechanism and a precondition feature that allows the execution of ordered tasks and allows some operations before starting with the core release procedure.
  • Mirroring production bucket (release to the mirror, sync main with the mirror, restore main with the mirror): Currently, we sync prod into prod-mirror to guarantee that prod-mirror is a mirror and deploy packages into prod-mirror. After that, the backup prod by into the directory (s3://packages.wazuh.com/Backups/Backup_{timestamp}/{major}.x) and we sync prod-mirror into prod. Finally, we AWS S3-to-S3 copy, without localhost intervention, only deltas, and then we invalidate the cache of the prod.

Restore support to PPC packages (Agent - deb ppc64el, rpm ppc64le)

The changes consisted of:

  • Update Wazuh documentation.
  • Adapt release procedures to support PPC packages (package generation, package signing and verification, and release tool deployment).
  • Update release issues (tests).

The new repository must include QA related code and tools

  • Defines a private repository: “wazuh-qa-automation”.
  • Currently, we have two branches: 4.10.0 and the main.
  • The new repository includes:
    • New templates for reporting bugs, requesting new support, and creating PRs.
    • New README with a use guide for the tools migrated.
    • Apply new structure and good practices.
    • The 4.10.0 and 5.0.0 branches have the same migrated code. It contains:
      • Workflow
        • s3_sign_check: Adds the requirement for a GitHub PAT that must be set using the GITHUB_TOKEN environment variable.
        • dtt1: Adds two steps to workflow files. Now, the token must be manually filled in before execution.
      • Provisioning tool
      • Release procedures
      • The new Jenkins
      • Testing tool
      • SCA footprint test (sca_footprint)
      • System tests
      • VD E2E tests
OUT OF SCOPE: "Packages Redesign Tier 2"

  • We are working on https://github.com/wazuh/wazuh-qa-automation/issues/42 which proposes new processes and practices to work.

  • The 5.0.0 branch should remove everything that Migrate has deprecated.

  • Once the target tasks are complete, we will work on implementing some improvements, which are:

    Release Procedures

    Medium priority improvements:

    • tools/release/src/**/docker/Dockerfile_tests: Each module's docker test image.
      We should re-evaluate if it's necessary to run each module's unit tests on a docker image, noting that it consumes time and may not add much value to the development itself.
    • .github/workflows: Main GHA workflows directory
      • Currently, all the Release Procedures modules' Unit Tests are being executed on GHA and running on Docker, which means that we build a Docker image and run it on each GHA, and it is resources-consuming.
      • All the GHA Workflows can be simplified in just one workflows that runs all the UTs (but first, it would be great if we get rid of the Docker build)
    Testing

    High-priority improvements:

    • tools/testing/src/testing/testing.py: Main testing module file
      • The module's parameters are strongly attached to DTT, it must be more generic to re-use it on different suites.
      • It expects the target and dependencies inventories to be on a specific path with a desired format, so it won't detect the target system correctly if the inventory is in a custom directory.
    • tools/testing/src/testing/playbooks/test.yml Test execution playbook
      The test path is generated "dynamically" but it depends completely on the DTT format
    • tests/test_functional/test_system/test_deployability/workflows: DTT workflows directory
      • The workflows are outdated, and its using the legacy allocator path, it must be updated to use them.
      • Wazuh version on the Workflows is hardcoded using an outdated value.
      • Add parametrization on the Workflows where possible.
    • tools/testing/tests: Testing tool Unit Tests directory
      There are no Unit Tests developed, we must create a set of tests to ensure its correct functionality
      Related issues:

    Medium priority improvements:

    • tests/test_functional/test_system/test_deployability/tests: Main DTT tests directory
      The test helpers have a lot of duplicated code, and there are several wrongly handled conditional flows. We could include unit tests for the helpers to validate its correct behavior
    JobFlow

    Medium priority improvements:

    • tools/jobflow/README.md: Main documentation file of the JobFlow tool
      Currently, the documentation is DTT1-specific, it does not help the user to understand what the tool does, and how it can be used for different cases.

    Low priority improvements:

    • tools/jobflow/tests: Unit Tests suite
      This suite uses Python's default test runner UnitTests utilities and pytest as the runner, it could be improved by only using pytest and its utilities, which is recommended.
      Related issues:
    • tools/jobflow/examples: Workflows examples directory
      We must add more Workflows as examples of different use cases.

    Other improvements could be found here: Jobflow engine - Findings and recommendations wazuh-qa#5044

    Provision

    Medium priority improvements:

    • tools/provision/README.md: Main documentation file of the Provision tool
      This README is strongly related to the JobFlow, it could be improved detailing mainly the standalone execution and having the JobFlow workflow as a secondary use.

    Low-priority improvements:

    • tools/provision/src/provision/playbooks/wazuh: Wazuh-related playbooks
      This directory could be re-structured, probably the non install/uninstall actions (register, services, set repo) could be separated on a different folder as these actions are not specifically related to an installation type.
    • tools/provision/tests: Unit Tests suite
      This suite uses Python's default test runner UnitTests utilities and pytest as the runner, it could be improved by only using pytest and its utilities, which is recommended.
      Related issues:
    Jenkins

    Low-priority improvements:

    • tools/jenkins/tests: tool's Unit Tests directory
      Must implement more Unit Tests for this tool
    • tools/jenkins/src/job-builder: Jenkins pipelines as a code
      Add a README.md detailing this folder, it could be implemented here or in the main Jenkins src directory (tools/jenkins/src), in the second case, it should also includes a summary about bootstrap.
@rauldpm
Copy link
Member

rauldpm commented Apr 8, 2024

Issue content migrated from https://github.com/wazuh/internal-devel-requests/issues/993

@rauldpm rauldpm mentioned this issue May 2, 2024
Closed
6 tasks
@vikman90 vikman90 mentioned this issue May 3, 2024
Closed
@davidjiglesias davidjiglesias closed this as not planned Won't fix, can't repro, duplicate, stale Jun 26, 2024
@QU3B1M QU3B1M mentioned this issue Jun 28, 2024
Closed
8 tasks
@damarisg damarisg self-assigned this Jul 3, 2024
@vikman90 vikman90 mentioned this issue Jul 9, 2024
Closed
This was referenced Jul 10, 2024
Open
Closed
Closed
@pcapellan pcapellan mentioned this issue Jul 15, 2024
Closed
2 tasks
This was referenced Aug 6, 2024
Closed
Closed
Closed
This was referenced Aug 8, 2024
Closed
Closed
This was referenced Aug 8, 2024
Closed
Closed
Open
@damarisg damarisg mentioned this issue Sep 11, 2024
Open
7 tasks
@c-bordon c-bordon mentioned this issue Sep 12, 2024
Open
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@damarisg damarisg

Labels
level/objective type/enhancement Enhancement issue
Projects
Release 4.10.0
Status: In progress
Roadmap
Status: In progress
Milestone
No milestone
Development

No branches or pull requests
5 participants
@damarisg @havidarou @rauldpm @davidjiglesias and others