Add system tests to elasticsearch package

[crespocarlos]

What does this PR do?

This PR adds system tests to the Elasticsearch package

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

• Run elastic-package stack up -v -d --version 8.5.0-SNAPSHOT
• Thencd packages/elasticsearch && elastic-package build && elastic-package test system

Related issues

• Closes [Stack Monitoring] Add system tests #4008

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-10-14T11:09:36.077+0000

• Duration: 32 min 48 sec

Test stats 🧪

Test	Results
Failed	0
Passed	55
Skipped	0
Total	55

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (5/5)	💚
Files	100.0% (9/9)	💚 2.783
Classes	100.0% (9/9)	💚 2.783
Methods	88.073% (96/109)	👎 -1.738
Lines	91.98% (562/611)	👍 0.513
Conditionals	100.0% (0/0)	💚

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[crespocarlos]

Elasticsearch container must not be started with root user. With this in memory mount, log_generation container, which is logged in with root, is able to get the files generated by the elasticsearch container and copy them to ${SERVICE_LOGS_DIR}

[matschaffer]

Not sure what a good "dev loop" for this stuff is. But these seem to take about 20 minutes to run and eventually I get 2 failures

--- Test results for package: elasticsearch - START ---
FAILURE DETAILS:
elasticsearch/gc default (variant: elasticsearch_8.5.0):
[0] parsing field value failed: field "process.pid"'s Go type, string, does not match the expected field type: long (field value: 172)
[1] parsing field value failed: field "process.pid"'s Go type, string, does not match the expected field type: long (field value: 191)
[2] parsing field value failed: field "process.pid"'s Go type, string, does not match the expected field type: long (field value: 209)
elasticsearch/slowlog default (variant: elasticsearch_8.5.0):
[0] parsing field value failed: field "elasticsearch.slowlog.id"'s Go type, <nil>, does not match the expected field type: keyword (field value: <nil>)


╭───────────────┬────────────────┬───────────┬────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────┬────────────────╮
│ PACKAGE       │ DATA STREAM    │ TEST TYPE │ TEST NAME                              │ RESULT                                                                                          │   TIME ELAPSED │
├───────────────┼────────────────┼───────────┼────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┼────────────────┤
│ elasticsearch │ audit          │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │   50.42037067s │
│ elasticsearch │ cluster_stats  │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │  49.489491231s │
│ elasticsearch │ gc             │ system    │ default (variant: elasticsearch_8.5.0) │ FAIL: one or more errors found in documents stored in logs-elasticsearch.gc-ep data stream      │  53.643696827s │
│ elasticsearch │ index          │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │   54.83371928s │
│ elasticsearch │ index_recovery │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │   54.26436754s │
│ elasticsearch │ index_summary  │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │  57.360332617s │
│ elasticsearch │ ml_job         │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │ 1m16.96065921s │
│ elasticsearch │ node           │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │  58.722088277s │
│ elasticsearch │ node_stats     │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │  58.491270634s │
│ elasticsearch │ server         │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │  58.943940543s │
│ elasticsearch │ shard          │ system    │ default (variant: elasticsearch_8.5.0) │ PASS                                                                                            │  54.375799125s │
│ elasticsearch │ slowlog        │ system    │ default (variant: elasticsearch_8.5.0) │ FAIL: one or more errors found in documents stored in logs-elasticsearch.slowlog-ep data stream │  54.671515062s │
╰───────────────┴────────────────┴───────────┴────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────╯

I'll start poking around to see if I can figure out what's up. Maybe it's just full docker disk again.

[matschaffer]

Wonder if this should be 8.6.0-SNAPSHOT now.

[crespocarlos]

Or both versions? Though it will take twice as much time to complete the test run.

[matschaffer]

yikes

[crespocarlos]

@matschaffer, It unfortunately takes a big chunk of time to complete.

Regarding the error above, did you start a clean elastic-package stack? In the slowlog pipeline, we explicitly tell it to remove the slowlog.id when it comes with a null value. And for the process.pid it's also explicitly converted to long in the gc ingest pipeline.

Which makes me wonder whether your elastic-package stack had the Filebeat version of the pipelines installed when the tests ran. If that's the case, I think fleet for some reason didn't replace them or picked up Filebeat's pipelines instead. If that's true, we may have problems. I'll try to reproduce this.

[klacabane]

I think most time is spent setting and tearing down the _dev docker-compose, as well as enrolling an agent with the test's configuration for each data stream. There's maybe a way to provide a single test configuration so all data stream's are collected in a single run, but I can't see any notes on that in the documentation

[crespocarlos]

I think most time is spent setting and tearing down the _dev docker-compose, as well as enrolling an agent with the test's configuration for each data stream. There's maybe a way to provide a single test configuration so all data stream's are collected in a single run, but I can't see any notes on that in the documentation

That would be nice, because the datastream is always deleted after the tests runs. But by looking at the elastic-package system test code, the runner always creates and destroys the container on each run

[crespocarlos]

@matschaffer , there are 2 possible explanations:
1 - My testing steps didn't mention that you had to run elastic-package build, so maybe you had an old build on your env which was used when you ran the tests.
2 - If you build the package, package registry container could still be looking at an old build, if you either didn't restart the elastic package stack completely or didn't run elastic-package stack up -v -d --services package-registry

[klacabane]

LGTM - we can reach out to the ecosystem team to investigate ways to make the system tests cheaper to run as this will become increasingly problematic if we add variants