Add support for Debian 11 and Ubuntu 22.04 platforms #2603

narph · 2022-06-03T14:00:32Z

What does this PR do?

Adds support for Debian 11 and Ubuntu 22.04 platforms

Why is it important?

Adds support for Debian 11 and Ubuntu 22.04 platforms.
Latest Debian and Ubuntu versions are already supported for other stack components and customers are asking if we support it as well.
We need to make sure elastic agent/fleet server tests will pass on these 2 platforms as well.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests (make unit-test), and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Related issues

Relates [Elastic Agent] Support for Debian 11 platform elastic-agent#235

mergify · 2022-06-03T14:01:24Z

This pull request does not have a backport label. Could you fix it @narph? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-06-03T14:05:20Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-06-07T08:02:51.548+0000
Duration: 77 min 7 sec

Test stats 🧪

Test	Results
Failed	2
Passed	384
Skipped	0
Total	386

Test errors

Expand to view the tests failures

`Initializing / End-To-End Tests / fleet_sles15_fleet_mode / Deploying the agent – Fleet Mode`

Expand to view the error details

 Step the "elastic-agent" process is in the "started" state on the host

no stacktrace

`Initializing / End-To-End Tests / fleet_debian_10_arm64_system_integration / Adding diskio system/metrics Integration to a Policy – System Integration`

Expand to view the error details

 Step "system/metrics" with "diskio" metrics are present in the datastreams

no stacktrace

Steps errors

Expand to view the steps failures

`Executing ansible-playbook`

Took 21 min 24 sec . View more details here
Description: /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-2603/.venv/bin/ansible-playbook --private-key="/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-2603/src/github.com/elastic/e2e-testing/e2essh" --extra-vars="workspace=/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-2603/src/github.com/elastic/e2e-testing/ runId=5a71cdc9 sshPublicKey=/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-2603/src/github.com/elastic/e2e-testing/e2essh.pub" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-2603/src/github.com/elastic/e2e-testing/.ci/ansible/playbook.yml -i "18.191.46.99," -t setup-stack --extra-vars="buildURL=https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/PR-2603/12/ gitSha=0a59572acf470885ac6bd2988c82473a61e240f9 build=12 repo=e2e-testing branch=pr-2603 type=ci stackRunner=18.191.46.99 nodeLabel=stack nodeUser=admin nodeImage=ami-0d90bed76900e679a nodeInstanceType=t3.xlarge"

`Sleep`

Took 0 min 5 sec . View more details here
Description: 5

`Shell Script`

Took 9 min 28 sec . View more details here
Description: ssh -tt -o TCPKeepAlive=yes -o ServerAliveInterval=60 -o ServerAliveCountMax=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/jenkins/workspace/PR-2603-12-bbfa62ed-7d79-472a-b060-03d3e0c22f52/e2essh ec2-user@52.15.75.181 -- 'sudo bash /home/ec2-user/e2e-testing/.ci/scripts/functional-test.sh'

`Archive the artifacts`

Took 0 min 0 sec . View more details here
Description: [2022-06-07T09:04:10.911Z] Archiving artifacts script returned exit code 2

`Shell Script`

Took 23 min 18 sec . View more details here
Description: ssh -tt -o TCPKeepAlive=yes -o ServerAliveInterval=60 -o ServerAliveCountMax=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/jenkins/workspace/PR-2603-12-a4fa5bb0-82da-4f2f-bc3f-9a63e6193ea9/e2essh admin@3.16.23.219 -- 'sudo bash /home/admin/e2e-testing/.ci/scripts/functional-test.sh'

`Archive the artifacts`

Took 0 min 0 sec . View more details here
Description: [2022-06-07T09:19:50.477Z] Archiving artifacts script returned exit code 2

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Initializing / End-To-End Tests / fleet_sles15_fleet_mode / Deploying the agent – Fleet Mode
Name: Initializing / End-To-End Tests / fleet_debian_10_arm64_system_integration / Adding diskio system/metrics Integration to a Policy – System Integration

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

.ci/.e2e-platforms.yaml

mdelapenya

This looks good to me, although we need one more thing, as described in https://github.com/elastic/e2e-testing/pull/2589/files#diff-53b78ab29e8f8423188da6c0806aaa9de90d39bd3502d83a655a1a68429fcfe2R43, which is to actually run the desired scenarios in the new platforms

.ci/.e2e-platforms.yaml

mdelapenya · 2022-06-03T15:00:09Z

.ci/.e2e-tests-daily.yaml

@@ -5,51 +5,51 @@ SUITES:
    scenarios:


I think we want to include debian11 and ubuntu in the nightly builds. So I guess they should be added in this file too.

@jlind23 could you confirm?

mdelapenya · 2022-06-03T15:01:28Z

.ci/.e2e-tests.yaml

@@ -5,54 +5,54 @@ SUITES:
    scenarios:


This file is mainly used by the test framework while developing it. The other descriptors are used by downstream jobs: Beats, Agent, Nightly...

.ci/.e2e-platforms.yaml

mergify · 2022-06-03T15:46:23Z

This pull request is now in conflict. Could you fix it @narph? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-debian-support upstream/add-debian-support
git merge upstream/main
git push upstream add-debian-support

mdelapenya · 2022-06-06T16:50:53Z

@narph I'm seeing the very same error while creating all the new instances for Debian 11 and Ubuntu:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: InvalidParameterCombinationEnhanced networking with the Elastic Network Adapter (ENA) is required for the 't3.xlarge' instance type. Ensure that you are using an AMI that is enabled for ENA.c59200f8-6a5a-4e28-b750-160b59f2125c

Will fetch this PR locally and run it from local using the build scripts. Will ping you here with my investigation about that issue

mdelapenya · 2022-06-06T17:24:06Z

@narph I replaced current Ubuntu AMI with the one provided by AWS as a Quickstart AMI and the creation works:

I'm currently running a few scenarios on that Ubuntu to check it. If it works, I will push a commit to this branch with the changes. Same for Debian 11.

Obtained from https://wiki.debian.org/Cloud/AmazonEC2Image/Bullseye

mdelapenya · 2022-06-06T17:35:22Z

For Debian, I'm checking their website for the AMIs: https://wiki.debian.org/Cloud/AmazonEC2Image/Bullseye, where they say that ami-0c7c4e3c6b4941f0f is the AMI in us-east-2.

I do not know why I'm not able to discover that AMI ID in the AWS console 🤷 Anyway. I checked locally and I'm able to create a node with that AMI, successfully running the test scenarios on it 🚀

mdelapenya · 2022-06-06T22:34:48Z

Ok, this is working now. There are two failures that I'd recommend to investigate before merging, as they affect Debian 11:

make -C .ci setup-stack  # it will fail as the fleet-server needs a different healthcheck for Kibana + Fleet
make -C .ci recreate-fleet-server # causes the stack to restart fleet-server
make -C .ci provision-node NODE_IMAGE=ami-0c7c4e3c6b4941f0f NODE_LABEL=debian_11_amd64
make -C .ci setup-node NODE_IMAGE=ami-0c7c4e3c6b4941f0f NODE_LABEL=debian_11_amd64
make -C .ci run-tests TAGS="backend_processes" NODE_LABEL=debian_11_amd64
make -C .ci run-tests TAGS="system_integration" NODE_LABEL=debian_11_amd64

narph · 2022-06-07T10:35:55Z

Ok, this is working now. There are two failures that I'd recommend to investigate before merging, as they affect Debian 11:

make -C .ci setup-stack  # it will fail as the fleet-server needs a different healthcheck for Kibana + Fleet
make -C .ci recreate-fleet-server # causes the stack to restart fleet-server
make -C .ci provision-node NODE_IMAGE=ami-0c7c4e3c6b4941f0f NODE_LABEL=debian_11_amd64
make -C .ci setup-node NODE_IMAGE=ami-0c7c4e3c6b4941f0f NODE_LABEL=debian_11_amd64
make -C .ci run-tests TAGS="backend_processes" NODE_LABEL=debian_11_amd64
make -C .ci run-tests TAGS="system_integration" NODE_LABEL=debian_11_amd64

I see there are the same flaky tests, the Debian 11 tests just passed. Can you confirm?

mdelapenya · 2022-06-07T11:25:13Z

Yes, it's seems the debian/Ubuntu support is at the same level than the rest of platforms.

LGTM

mdelapenya

Ready!! 🚀

…port * upstream/main: Add support for Debian 11 and Ubuntu 22.04 platforms (elastic#2603) fix: retire 8.1 snapshot from upgrades (elastic#2616)

* main: Add support for Debian 11 and Ubuntu 22.04 platforms (elastic#2603) fix: retire 8.1 snapshot from upgrades (elastic#2616) bump stack version 8.4.0-42ce0eef (elastic#2613) docs: document AWS requirements (elastic#2589) chore: move elastic-agent's k8s-autodiscover scenarios from beats to elastic-agent profile (elastic#2601)

* main: (22 commits) chore(jjbb): increase build timeout to 2h (elastic#2661) chore: increase build timeout to 2h (elastic#2660) feat: run tests on windows 2019 (elastic#2468) Remove aws instance reaper Jenkins job (elastic#2647) Update k8s version to v1.24 (elastic#2644) chore: simplify extracting info from the supported platforms (elastic#2637) ci: avoid running on branches that are not available anymore (elastic#2627) Add support for Debian 11 and Ubuntu 22.04 platforms (elastic#2603) fix: retire 8.1 snapshot from upgrades (elastic#2616) bump stack version 8.4.0-42ce0eef (elastic#2613) docs: document AWS requirements (elastic#2589) chore: move elastic-agent's k8s-autodiscover scenarios from beats to elastic-agent profile (elastic#2601) Add test scenario for elastic agent tags (elastic#2552) bump stack version 8.4.0-3525aaf7 (elastic#2597) bump stack version 8.4.0-df27a645 (elastic#2591) fix: agent directories must belong to root (elastic#2590) chore: bump maintenance versions in mergify (elastic#2563) chore: bump kubernetes versions (elastic#2562) fix: use fixed, valid version for protobuffers (elastic#2569) chore: do not run apm-server helm chart scenarios for Beats (elastic#2564) ...

mdelapenya · 2022-06-30T11:36:47Z

.ci/.e2e-tests-beats.yaml

  - suite: "fleet"
    scenarios:
      - name: "Fleet"
        tags: "fleet_mode_agent"
-        platforms: ["centos8_arm64", "centos8_amd64", "debian_arm64", "debian_amd64", "sles15"]
+        platforms: ["centos8_arm64", "centos8_amd64", " debian_10_arm64", "debian_10_amd64", "sles15"]


@v1v we found it! 🤦

add images

424dd65

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Jun 3, 2022

fix space

09a4356

mdelapenya reviewed Jun 3, 2022

View reviewed changes

.ci/.e2e-platforms.yaml Outdated Show resolved Hide resolved

mdelapenya suggested changes Jun 3, 2022

View reviewed changes

mdelapenya reviewed Jun 3, 2022

View reviewed changes

.ci/.e2e-platforms.yaml Outdated Show resolved Hide resolved

mdelapenya reviewed Jun 3, 2022

View reviewed changes

.ci/.e2e-platforms.yaml Outdated Show resolved Hide resolved

narph added 4 commits June 3, 2022 16:30

add tests

3d86076

rename debian

0cc8bc6

rename ubuntu

4e2ecf4

rename on tests

483bee3

narph requested a review from mdelapenya June 3, 2022 14:56

narph self-assigned this Jun 3, 2022

narph added v8.4.0 area:ci Anything related to the CI labels Jun 3, 2022

mdelapenya reviewed Jun 3, 2022

View reviewed changes

add nightly tests

890e103

mdelapenya reviewed Jun 3, 2022

View reviewed changes

.ci/.e2e-platforms.yaml Outdated Show resolved Hide resolved

Merge branch 'main' into add-debian-support

854783b

mdelapenya added 3 commits June 6, 2022 12:15

fix: use a community AMI for Ubuntu 22.04

25dfdf9

fix: rename ubuntu key to be YAML compatible

9cd458e

chore: apply debian11 and ubuntu to the elastic-agent descriptor

31e3d20

fix: use official Debian 11 AMI

f1f2194

Obtained from https://wiki.debian.org/Cloud/AmazonEC2Image/Bullseye

mdelapenya added 2 commits June 6, 2022 12:36

chore: move debian platforms to be all together

96bd734

chore: do not run k8s in multiple platforms

8aaa703

Merge branch 'main' into add-debian-support

0a59572

mdelapenya approved these changes Jun 7, 2022

View reviewed changes

narph added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Jun 7, 2022

narph requested review from a team, aleksmaus and michalpristas and removed request for a team June 7, 2022 11:32

narph merged commit cdb396e into elastic:main Jun 7, 2022

narph deleted the add-debian-support branch June 7, 2022 11:33

mdelapenya reviewed Jun 30, 2022

View reviewed changes

v1v mentioned this pull request Jun 30, 2022

fix space in the configuration file #2749

Merged

ferullo mentioned this pull request Aug 22, 2022

[DOCS] Endpoint 8.4.0 release notes elastic/security-docs#2320

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Debian 11 and Ubuntu 22.04 platforms #2603

Add support for Debian 11 and Ubuntu 22.04 platforms #2603

narph commented Jun 3, 2022

mergify bot commented Jun 3, 2022

elasticmachine commented Jun 3, 2022 •

edited

Loading

Build stats

Test stats 🧪

`Initializing / End-To-End Tests / fleet_sles15_fleet_mode / Deploying the agent – Fleet Mode`

`Initializing / End-To-End Tests / fleet_debian_10_arm64_system_integration / Adding diskio system/metrics Integration to a Policy – System Integration`

`Executing ansible-playbook`

`Sleep`

`Shell Script`

`Archive the artifacts`

`Shell Script`

`Archive the artifacts`

Genuine test errors

mdelapenya left a comment •

edited

Loading

mdelapenya Jun 3, 2022 •

edited

Loading

mdelapenya Jun 3, 2022

mergify bot commented Jun 3, 2022

mdelapenya commented Jun 6, 2022

mdelapenya commented Jun 6, 2022

mdelapenya commented Jun 6, 2022

mdelapenya commented Jun 6, 2022

narph commented Jun 7, 2022

mdelapenya commented Jun 7, 2022

mdelapenya left a comment

mdelapenya Jun 30, 2022

Add support for Debian 11 and Ubuntu 22.04 platforms #2603

Add support for Debian 11 and Ubuntu 22.04 platforms #2603

Conversation

narph commented Jun 3, 2022

What does this PR do?

Why is it important?

Checklist

Related issues

mergify bot commented Jun 3, 2022

elasticmachine commented Jun 3, 2022 • edited Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Initializing / End-To-End Tests / fleet_sles15_fleet_mode / Deploying the agent – Fleet Mode

Initializing / End-To-End Tests / fleet_debian_10_arm64_system_integration / Adding diskio system/metrics Integration to a Policy – System Integration

Steps errors

Executing ansible-playbook

Sleep

Shell Script

Archive the artifacts

Shell Script

Archive the artifacts

🐛 Flaky test report

Genuine test errors

🤖 GitHub comments

mdelapenya left a comment • edited Loading

Choose a reason for hiding this comment

mdelapenya Jun 3, 2022 • edited Loading

Choose a reason for hiding this comment

mdelapenya Jun 3, 2022

Choose a reason for hiding this comment

mergify bot commented Jun 3, 2022

mdelapenya commented Jun 6, 2022

mdelapenya commented Jun 6, 2022

mdelapenya commented Jun 6, 2022

mdelapenya commented Jun 6, 2022

narph commented Jun 7, 2022

mdelapenya commented Jun 7, 2022

mdelapenya left a comment

Choose a reason for hiding this comment

mdelapenya Jun 30, 2022

Choose a reason for hiding this comment

elasticmachine commented Jun 3, 2022 •

edited

Loading

`Initializing / End-To-End Tests / fleet_sles15_fleet_mode / Deploying the agent – Fleet Mode`

`Initializing / End-To-End Tests / fleet_debian_10_arm64_system_integration / Adding diskio system/metrics Integration to a Policy – System Integration`

`Executing ansible-playbook`

`Sleep`

`Shell Script`

`Archive the artifacts`

`Shell Script`

`Archive the artifacts`

mdelapenya left a comment •

edited

Loading

mdelapenya Jun 3, 2022 •

edited

Loading