[CI] Enable build stage for arm architecture #21284

v1v · 2020-09-24T08:29:31Z

What does this PR do?

Use pipeline 2.0 to build in arm

Why is it important?

Enable arm64 builds/tests when merging to branches or tags creation.

When

By default on builds from branches.
By default on builds from tags.
For PRs:
- if label arm is added then the following builds will run it.
- if comment /test <module> for arm" it will trigger a build for arm. matches the beat folder name or x-pack/. For instance /test x-pack/auditbeat for arm or /test auditbeat for arm

Notes

Uses arm label
Uses [CI] Pipeline 2.0 for monorepos #20104 as the base branch. (That's the reason for all those commits)
d733967 are the potential changes to enable arm builds in auditbeat.
Support gvm/go installation in arm architecture.

Furthermore

Directory	OS	Mage Target	Status CI
auditbeat	linux/arm64	build unitTest	✅
filebeat	linux/arm64	build unitTest	✅
heartbeat	linux/arm64	build unitTest	✅
journalbeat	linux/arm64	build unitTest	✅
libbeat	linux/arm64	build unitTest	✅
metricbeat	linux/arm64	build unitTest	🔴
packetbeat	linux/arm64	build unitTest	✅
x-pack/auditbeat	linux/arm64	build unitTest	✅
x-pack/elastic-agent	linux/arm64	build unitTest	✅
x-pack/filebeat	linux/arm64	build unitTest	✅
x-pack/libbeat	linux/arm64	build unitTest	✅
x-pack/metricbeat	linux/arm64	build unitTest	🔴
x-pack/functionbeat	linux/arm64	build unitTest	✅

Issues

Test failures in metricbeats and x-pack/metricbeats:

Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>

…ne-2.0 * upstream/master: (41 commits) adding possibility to override content-type checks, it was breaking certain webhooks that is not able to set content-headers at all. Still defaults to application/json (elastic#20232) fix: use a fixed worker type for tests (elastic#20130) [Ingest Manager] Prepare packaging for endpoint and asc files (elastic#20186) [Packetbeat] HTTP: Improve support for 100-continue elastic#15830 (elastic#19349) Increase index.max_docvalue_fields_search to 200 (elastic#20218) [Ingest Manager] Prevent closing closed reader (elastic#20214) [Metricbeat] Use MySQL Host Parser in Query metricset (elastic#20191) Testing: Ignore timestamp from cylance/protect dataset (elastic#20211) [Filebeat] Ignore cylance.protect timestamps while testing (elastic#20207) [CI] remove codecov step (elastic#20102) [docs] Indicate that SYSTEM user is required on Windows to use Endpoint (elastic#20172) Remove f5/firepass rsa2elk fileset (elastic#20160) [Elastic Agent] Improve GRPC stop to be more relaxed. (elastic#20118) Fix fileset field prefixing (elastic#20170) Fix terminating pod autodiscover issue (elastic#20084) Call host parser only once when building light metricsets (elastic#20149) [CI] fix null string with contains (elastic#20182) [Ingest Manager] Fix failing unit tests on windows (elastic#20127) [Filebeat] Update crowdstrike module (elastic#20138) [docs] Add x-pack role to relevant metricsets (elastic#20167) ...

…eature/ci-pipeline-2.0 * 'feature/ci-pipeline-2.0' of github.com:v1v/beats: Update auditbeat/Jenkinsfile.yml Update auditbeat/Jenkinsfile.yml

This reverts commit 56c45d5.

This reverts commit 02e0303.

It's already defined in the <project>/Jenkinsfile.yml file

mdelapenya

Love the readability of the DSL!

elasticmachine · 2020-09-24T17:26:32Z

Pinging @elastic/ingest-management (Team:Ingest Management)

…ne-2.0-arm * upstream/master: (54 commits) [CI] Change x-pack/auditbeat build events (comments, labels) (elastic#21463) [CI] changeset from elastic#20603 was not added to CI2.0 (elastic#21464) Add new log file reader for filestream input (elastic#21450) [CI] Send slack message with build status (elastic#21428) Remove duplicated sources url in dependencies report (elastic#21462) Add implementation of FSWatcher and FSScanner for filestream (elastic#21444) [Ingest Manager] Split index restrictions into type,dataset, namespace parts (elastic#21406) Update Filebeat module expected logs files (elastic#21454) Edit SQL module docs and fix broken doc structure (elastic#21233) [Ingest Manager] Send snapshot flag together with metadata (elastic#21285) Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447) [JJBB] Use reference repo for fast checkouts (elastic#21410) Add initial skeleton of filestream input (elastic#21427) Initial spec file for apm-server (elastic#21225) [Ingest Manager] Upgrade Action: make source URI optional (elastic#21372) Add field limit check for AWS Cloudtrail flattened fields (elastic#21388) [Winlogbeat] Move winlogbeat javascript processor to libbeat (elastic#21402) ci: pipeline to generate the changelog (elastic#21426) [JJBB] Set shallow cloning to 10 (elastic#21409) docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419) ...

Jenkinsfile

…ne-2.0-arm * upstream/master: (93 commits) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) typofix for dns timeout configuration option (elastic#21069) chore: create CI artifacts for DEV usage (elastic#21645) [Ingest Manager] Atomic installed forgotten changelog elastic#21780 [Ingest Manager] Agent atomic installer (elastic#21745) Add missing configuration annotations (elastic#21736) [Filebeat] Add check for context.DeadlineExceeded error (elastic#21732) Remove kafka partition ISR from Metricbeat docs (elastic#21709) Skip flaky test with oauth2 config in httpjson input (elastic#21749) Fix for azure retrieve resource by ids (elastic#21711) Update obs app links (elastic#21682) fix: update fleet test suite name (elastic#21738) Remove dot from file.extension value in Auditbeat FIM (elastic#21644) Fix leaks with metadata processors (elastic#16349) ...

Co-authored-by: cachedout <mike.place@elastic.co>

v1v · 2020-10-15T09:26:24Z

Environmental issue with Build&Test / libbeat-build

$ mage build test
...
[2020-10-14T15:42:03.951Z] Building elasticsearch_kerberos.elastic
[2020-10-14T15:42:12.178Z] Service 'elasticsearch_kerberos.elastic' failed to build: The command '/bin/sh -c /scripts/installkdc.sh && /scripts/addprincs.sh' returned a non-zero code: 127
[2020-10-14T15:42:12.178Z] >> Building docker images again
[2020-10-14T15:42:12.178Z] elasticsearch uses an image, skipping
[2020-10-14T15:42:12.178Z] elasticsearch_monitoring uses an image, skipping
[2020-10-14T15:42:12.178Z] elasticsearchssl uses an image, skipping
[2020-10-14T15:42:12.178Z] logstash uses an image, skipping
[2020-10-14T15:42:12.178Z] Building redis
[2020-10-14T15:42:13.124Z] Building sredis
[2020-10-14T15:42:14.070Z] Building kafka
[2020-10-14T15:42:14.644Z] Building elasticsearch_kerberos.elastic
[2020-10-14T15:42:16.047Z] Service 'elasticsearch_kerberos.elastic' failed to build: The command '/bin/sh -c /scripts/installkdc.sh && /scripts/addprincs.sh' returned a non-zero code: 127
[2020-10-14T15:42:16.047Z] Error: running "docker-compose -p libbeat_8_0_0_b2fbc1b280-snapshot build --force-rm --pull" failed with exit code 1
script returned exit code 1[2020-10-14T15:42:16.417Z] Client: Docker Engine - Community

# Conflicts: # .ci/scripts/install-go.sh

* upstream/master: (29 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...

* upstream/master: (216 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...

* upstream/master: (156 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...

v1v and others added 30 commits July 21, 2020 17:22

[CI] Pipeline 2.0 for monorepos

ac465eb

Update auditbeat/Jenkinsfile.yml

aa6a07a

Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>

Update auditbeat/Jenkinsfile.yml

1b91c12

Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>

[CI] 2.0 YAML to dynamic stage generations

7183f93

Merge branch 'feature/ci-pipeline-2.0' of github.com:v1v/beats into f…

4290714

…eature/ci-pipeline-2.0 * 'feature/ci-pipeline-2.0' of github.com:v1v/beats: Update auditbeat/Jenkinsfile.yml Update auditbeat/Jenkinsfile.yml

Use shared library

ca9a5a6

Use beatsWhen and beatsStages

56c45d5

Revert "Use beatsWhen and beatsStages"

778da9e

This reverts commit 56c45d5.

Simplify

daf8fb1

Refactor

fcab64b

Add run commands

6a0f1bd

use nodeOS step

a8cc866

Fix GO_VERSION

14e039d

Transform to String

7660a6d

Use method-pointer-operator

35d30be

replace with isInstalled

6be81e2

Fix the YAML data

e874d54

Use node

05baf2e

sort variables

2a668df

Use string rather than a list

b22f34a

[CI] verbose and added some missing functions

8126103

Use the right arguments

c5a55db

Use the directory flag in the make

02e0303

Revert "Use the directory flag in the make"

11653c0

This reverts commit 02e0303.

directoryFlag not required

52ec93e

It's already defined in the <project>/Jenkinsfile.yml file

Archive build reasons

1412871

Support windows builds

bd853af

Tear down environment context in the workers

bc5a6a5

[CI] support x-pack/auditbeat and minor changes

2df49f8

v1v requested a review from a team September 24, 2020 17:22

v1v marked this pull request as ready for review September 24, 2020 17:22

v1v requested review from a team as code owners September 24, 2020 17:22

mdelapenya approved these changes Sep 24, 2020

View reviewed changes

botelastic bot added the Team:Ingest Management label Sep 24, 2020

andrewkroh added the ci label Sep 30, 2020

v1v added 4 commits October 2, 2020 14:43

Timout might happen a bit often if we have only one resource for arm

1083b98

Enable on branches/tags

2c87233

Fix typo

fdb0d70

cachedout approved these changes Oct 5, 2020

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

v1v and others added 2 commits October 14, 2020 16:07

Update Jenkinsfile

b2fbc1b

Co-authored-by: cachedout <mike.place@elastic.co>

v1v added the v7.10.0 label Oct 15, 2020

v1v merged commit f8ff7ce into elastic:master Oct 15, 2020

v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020

[CI] Enable build stage for arm architecture (elastic#21284)

2858a18

v1v mentioned this pull request Oct 15, 2020

Cherry-pick to 7.x: [CI] Enable build stage for arm architecture (#21284) #21839

Merged

v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020

[CI] Enable build stage for arm architecture (elastic#21284)

8e4ff57

# Conflicts: # .ci/scripts/install-go.sh

v1v mentioned this pull request Oct 15, 2020

Cherry-pick to 7.10: [CI] Enable build stage for arm architecture (#21284) #21840

Merged

v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020

[CI] Enable build stage for arm architecture (elastic#21284)

339c742

# Conflicts: # .ci/scripts/install-go.sh

v1v mentioned this pull request Oct 15, 2020

Cherry-pick to 7.9: [CI] Enable build stage for arm architecture (#21284) #21841

Closed

v1v added a commit that referenced this pull request Oct 16, 2020

[CI] Enable build stage for arm architecture (#21284) (#21839)

82f94f4

v1v added a commit that referenced this pull request Oct 16, 2020

[CI] Enable build stage for arm architecture (#21284) (#21840)

6a5e7e6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Enable build stage for arm architecture #21284

[CI] Enable build stage for arm architecture #21284

v1v commented Sep 24, 2020 •

edited

Loading

mdelapenya left a comment

elasticmachine commented Sep 24, 2020

v1v commented Oct 15, 2020

[CI] Enable build stage for arm architecture #21284

[CI] Enable build stage for arm architecture #21284

Conversation

v1v commented Sep 24, 2020 • edited Loading

What does this PR do?

Why is it important?

When

Notes

Furthermore

Issues

mdelapenya left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 24, 2020

v1v commented Oct 15, 2020

v1v commented Sep 24, 2020 •

edited

Loading