[Uptime] Update functional test directory to use a pinned version of package registry via docker

[dominiqueclarke]

Summary

Fixes #109260
Fixes #119265
Fixes #119266
Fixes #103390
Fixes #119783
Fixes #119785
Fixes #119782
Fixes #119781
Fixes #116980

This PR has been run 100 times in flaky test runner. Results: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/83#399dcf17-8cc5-48e6-99d4-8b18032e11a6

Please note, the one failure is from a different test suite.

---

The Elastic Synthetics Integration tests rely on retrieval of the Elastic Synthetics package. If these connections are made to a package registry deployment outside of Kibana CI, these tests can fail at any time for two reasons:

• the deployed registry is temporarily unavailable
• the packages served by the registry do not match the expectation of the code under test

For that reason, we run a dockerized version of the package registry in Kibana CI. For this to work, our tests must run against a custom test configuration and be kept in a custom directory, x-pack/test/functional_synthetics.

How to run the tests locally

Usually, having the test server and the test runner in two different shells is most efficient, as it is possible to keep the server running and only rerun the test runner as often as needed. To do so, in one shell in the main kibana directory, run:

$ export FLEET_PACKAGE_REGISTRY_PORT=12345
$ node scripts/functional_tests_server.js --config test/functional_synthetics/config.js

In another shell in the same directory, run

$ export FLEET_PACKAGE_REGISTRY_PORT=12345
$ node ../scripts/functional_test_runner --config test/functional_synthetics/config.js

Port 12345 is used as an example here, it can be anything, but the environment variable has to be present for the tests to run at all.

DockerServers service setup

We use the DockerServers service provided by kbn-test. The documentation for this functionality can be found here:
https://github.com/elastic/kibana/blob/master/packages/kbn-test/src/functional_test_runner/lib/docker_servers/README.md

The main configuration for the DockerServers service for our tests can be found in x-pack/test/functional_synthetics/config.ts:

Specify the arguments to pass to docker run:

  const dockerArgs: string[] = [
    '-v',
    `${path.join(
      path.dirname(__filename),
      './apis/fixtures/package_registry_config.yml'
    )}:/registry/config.yml`,
    '-v',
    `${path.join(
      path.dirname(__filename),
      './apis/fixtures/test_packages'
    )}:/registry/packages/test-packages`,
  ];

-v mounts local paths into the docker image. The first one puts a custom configuration file into the correct place in the docker container, the second one mounts a directory containing additional packages.

Specify the docker image to use

image: 'docker.elastic.co/package-registry/package-registry:kibana-testing-1'

This image contains the content of docker.elastic.co/package-registry/package-registry:master on June 26 2020. The image used here should be stable, i.e. using master would defeat the purpose of having a stable set of packages to be used in Kibana CI.

Packages available for testing

The containerized package registry contains a set of packages which should be sufficient to run tests. The list of the packages are logged to the console when the docker container is initialized during testing.

Some DockerServers background

For the DockerServers servers to run correctly in CI, the FLEET_PACKAGE_REGISTRY_PORT environment variable needs to be under control of the CI environment. The reason behind this: it is possible that several versions of our tests are run in parallel on the same worker in Jenkins, and if we used a hard-coded port number here, those tests would run into port conflicts. (This is also the case for a few other ports, and the setup happens in vars/kibanaPipeline.groovy).

Also, not every developer has docker installed on their workstation, so it must be possible to run the testsuite as a whole without docker, and preferably this should be the default behaviour. Therefore, our DockerServers service is only enabled when FLEET_PACKAGE_REGISTRY_PORT is set. This needs to be checked in every test like this:

describe('When on the Synthetics Integration Policy Create Page', function () {
    skipIfNoDockerRegistry(providerContext);
    ...
})

[dominiqueclarke]

@elasticmachine merge upstream

[dominiqueclarke]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[dominiqueclarke]

@elasticmachine merge upstream

[dominiqueclarke]

@elasticmachine merge upstream

[dominiqueclarke]

@elasticmachine merge upstream

[dominiqueclarke]

@elasticmachine merge upstream

[shahzad31]

Who is responsible for keeping this up-to-date?

[dominiqueclarke]

Our team, whenever we want to add new tests that depend on a later version.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 598cee2
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #18499 succeeded 65791df
• 💚 Build #17691 succeeded b9382e2
• 💚 Build #17583 succeeded 008eb64
• 💚 Build #17350 succeeded 9f9c8ba
• 💔 Build #17299 failed 2017aef
• 💔 Build #17097 failed a242542

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.0	Backport failed because of merge conflicts

How to fix

Re-run the backport manually:

node scripts/backport --pr 117736

Questions ?

Please refer to the Backport tool documentation