Upgrade test Docker images to latest versions

[ycombinator]

This PR upgrades the Elasticsearch and Kibana Docker images to their latest versions in their respective Metricbeat module tests.

In addition to the actual version upgrade changes, this PR also contains several changes necessary to get the integration and system tests for the elasticsearch/ccr metricset passing. These tests existed before this PR but were skipped because:

• the CCR feature only exists in Elasticsearch 6.5.0 onwards,
• the tests had a check to skip if the version of Elasticsearch being used in the tests was < 6.5.0, and
• we were using Elasticsearch 6.4.3 until this PR came along and upgraded it.

[ycombinator]

I think the CI test failure is legitimate and should be fixed once #9179 is merged.

[ycombinator]

jenkins, test this

[ycombinator]

jenkins, test this

[ycombinator]

@ruflin @jsoriano As you can see from the evolution of this PR, upgrading the ES docker image to 6.5.1 caused the elasticsearch/ccr integration and system tests to start failing. So this PR ended up having a lot of fixes to these tests — mostly around getting ES to generate valid CCR stats before running the tests — in addition to the actual version upgrade changes.

Would you like me to pull out the CCR test-related fixes into their own PR(s) and keep this PR just for the actual version upgrade changes?

[jsoriano]

I am fine with leaving the upgrade of the image in the same PR, this way we know what was needed to support it. I'd add some explanation about why CCR tests existed for previous versions but weren't failing.

Also I don't like so much to have the implementation of CCR setup in both python and Go, but I guess we'd have to live with that while we have system testing in python.

[ycombinator]

Thanks @jsoriano. Added the note about why these tests were not failing before this PR in the PR description.

Agreed about the duplication of code.

[jsoriano]

Could you update the branch after #9179 to see if tests pass in CI?

[ycombinator]

Hmmmm, now the ES module system tests are all failing because the GET http://elasticsearch:9200/_xpack/license API is returning a 404.

However, when I run the exact same Docker image and make the same API call against it locally, it works:

$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
7dc94dc45d22        docker.elastic.co/elasticsearch/elasticsearch:6.5.1   "/usr/local/bin/dock…"   2 seconds ago       Up 1 second         0.0.0.0:9200->9200/tcp, 9300/tcp   quirky_poincare

$ curl 'http://localhost:9200/_xpack/license'
{
  "license" : {
    "status" : "active",
    "uid" : "78324d0b-da61-44ff-98e8-399df1f82063",
    "type" : "basic",
    "issue_date" : "2018-11-22T15:43:54.207Z",
    "issue_date_in_millis" : 1542901434207,
    "max_nodes" : 1000,
    "issued_to" : "docker-cluster",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}

I'm investigating further but if something jumps out to you, @jsoriano or @ruflin, please let me know. Thanks!

[ruflin]

Did the code work for the ml test only with 6.5.1? Or did it also break? The reason I'm asking is because it seems the trial activation breaks the tests. So if the trial activation works with the old code for ML, then the problem is likely in the code, if it didn't work, my assumption is that something changed in 6.5.1 docker containers?

[ycombinator]

Good thinking, @ruflin. Let me put up a "test" PR (so CI can run on it) with just the 6.5.1 version upgrade and no changes to the tests code. Let's see what happens to that one.

[ycombinator]

@jsoriano @ruflin This PR is ready for review again.

I did a bunch of testing and eventually found the fix to be changing this line: https://github.com/elastic/beats/pull/9198/files#diff-5a5de1a556d9b66b8140e73286c014aeR2. Before this change I believe we were hitting a race condition between the new code in this PR calling the GET _xpack/license ES API endpoint before the x-pack code in ES actually registered that API route (or something to that effect).

[jsoriano]

Would it also work with a more generic health check like /_cluster/health?

[ycombinator]

Let me try this and see.

[ycombinator]

I don't think it works reliably with _cluster/health. Jenkins CI passed but Travis CI failed with the same 404 error on GET _xpack/license that we were seeing earlier.

I think it's okay to base the health check on the URL we expect to call, i.e. GET _xpack/license, no?

[jsoriano]

Yeah, it's fine, lets leave it with _xpack/license then.

[jsoriano]

But lets take this into account elastic/elasticsearch#35959 :(

[ycombinator]

I'm going to fix all deprecated _xpack endpoints across the codebase in #9656.