Run kubernetes integration tests inside of a pod and use kind to setup a kubernetes cluster #17656
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
blakerouse
added
enhancement
Team:Platforms
|
Pinging @elastic/integrations-platforms (Team:Platforms) |
|
@exekias @jsoriano @ChrsMark Whats your thoughts on running the intergration tests for Kubernetes using kind? Feels like to me this works rather well and we could add more detailed integration tests this way. Kind also allows mounting of volumes, could possible use this with filebeat to access the log path and have filebeat connect to kubernetes API server to test autodiscovery. At the moment this works as long as |
|
Been looking at the failed test it seems that |
|
Added more tests, got almost all metrcsets tested except for |
|
Thank you for starting this! I understand that this will work in any linux box with Docker installed? If that's the case this sounds like a nice approach to me! any thoughts @jsoriano? In the past we discussed treating this similar to the cloud provider integration tests, but this looks simpler.
This is probably related to some ECS mapping, where we put some info about the container under ECS. |
|
@blakerouse do you know if this will allow us to access container logs? |
jsoriano
reviewed
Apr 14, 2020
There was a problem hiding this comment.
Whats your thoughts on running the intergration tests for Kubernetes using kind?
Using kind sounds good to me.
I would prefer to avoid adding more python code, but our kubernetes tests are already in python and this PR improves them, and we also have unit tests based on real responses, so let's continue like this.
|
Btw, it may be also good to add some environment variable that can be used to run the tests using an existing cluster. For example think on making that if |
|
Thanks for the feedback! I am still working on this because currently the testing is all over the place. Travis tests with minikube (by deploying the manifests, without even checking if they actually deployed), Jenkins tests with kind (by deploying the manifests, without even checking if they actually deployed), and the Kubernetes python integration tests that where disabled and didn't work. My goal is to get it all working with Kind and across multiple versions of Kubernetes. @jsoriano Will update to use @exekias I believe it will allow use to get access to the logs because you can bind mount into the k8s node with kind allowing the host to get the logs from the containers. If we transition to building the images and deploying those into the cluster, then it definitely could always access them. |
|
I would avoid building + pushing the image if we don't really need to, that's something we will test in e2e tests anyway (ie helm chart tests are doing this already). |
|
If we are going to avoid that and save those tests for e2e, then I will not add the I also think we should remove the @exekias wydt? |
I understand the use of KUBECONFIG aims to be able to launch tests against a running you may have (when developing). KUBECONFIG should work also when running from outside the cluster, isn't it?
I would hope we can keep tests separted by concern, for instance, Filebeat & Metricbeat tests are different, or Kubernetes module vs autodiscover. They belong in different places in the code. As we move towards the agent & integrations living somewhere else, I would expect this split to happen anyway at some point, so it's probably better to keep things separated today. That said, I don't see anything wrong with sharing the code to launch kind across the board, for instance |
To expose |
|
@exekias @jsoriano I have actually updated the PR to run the tests with This means we can easily run integration tests against GKE, AKS, EKS, or others without having to adjust the integration tests, as the tests can assume they are running inside of a kubernetes pod that has all rights to the cluster itself. |
blakerouse
changed the title
blakerouse
force-pushed
the
k8s-integration-use-kind
branch
from
a2b0742
to
a442bee
Compare
|
I also forgot to mention that this currently does not test all metricsets in the kubernetes module. The branch has gotten so big that I wanted to stop adding and just get it working and merged before adding more, as the branch is already huge! But after this is merged I will be adding more tests to cover all metricsets in kubernetes, add in field coverage, add tests for filebeat and heartbeat, add tests for autodiscover for metricbeat, filebeat, and heartbeat. |
exekias
reviewed
Apr 17, 2020
exekias
reviewed
Apr 17, 2020
|
Thank you for digging deeper, this is looking really promising! Just a few comments:
|
|
@exekias I am currently working on splitting the mage work out into its own PR. It does not require |
2 tasks
|
#17799 splits out the mage work, once that merges I will rebase and cleanup this PR. |
Thanks! I will take a look. |
|
Does this work with the Kubernetes built into the Docker for Mac install? I think a lot of us develop on Mac with this setup so if that just worked it would be nice. |
|
@andrewkroh Yes as long as you have |
blakerouse
force-pushed
the
k8s-integration-use-kind
branch
from
8510353
to
431afe2
Compare
andrewkroh
reviewed
Apr 29, 2020
dev-tools/mage/gotest.go
Outdated
Comment on lines
157
to
159
| // Set MODULE because only want that modules tests to run inside the testing environment. | ||
| os.Setenv("MODULE", fi.Name()) |
There was a problem hiding this comment.
Is there a way you can pass the module value to wherever it is being used without modifying this process's environment? Haven't looked at the code yet, but like maybe pass it into the runner and then directly add it to what ever process needs it. It's just kind of hard to track what's using it and could cause issues if you ever try to do things in parallel.
💔 Build Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
|
Okay all comments have either been answered or resolved. Anyway I can get a +1 to get this landed. Blocking me adding even more kubernetes tests. |
blakerouse
added a commit
to blakerouse/beats
that referenced
this pull request
Apr 29, 2020
…p a kubernetes cluster (elastic#17656) * Use kind to bring up a local kubernetes cluster and then run the integration tests against the kind cluster. * Run mage update. * Add more tests. * Fix more tests, install kind in prepare-tests. * Switch to running kubernetes integration tests inside of Kubernetes. * Use golang 1.13.9. * Fix for other beats. * Run mage fmt. * Don't run kubernetes integration tests if not inside kubernetes environment. * Fix metricbeat to use mage and the makefile shim for mage. * Improve the error message when kind or kubectl is not available. * Refactor the integration tests into a more module system. * Fix go vet. * Setup travis to use kind. * Run kubernetes integration tests in Jenkins. * Fix filebeat magefile. * Fix travis and Jenkins. * Check requirements of the test runner before actually running the tests. * Add return on parsebool error. * Don't return err on missing requirements for tester. * Run make update. * Move the kubernetes items to its own module, import that module only by metricbeat. (cherry picked from commit db29874)
5 tasks
|
I have seen after updating #17480 with master that with this change integration tests for modules without docker compose or kubernetes files are not executed, not so serious because it only affects some system tests, and cloud tests, that are not merged yet. I will prepare a fix. |
|
@jsoriano Could you file a bug about what is not working? Be happy to fix it. |
@blakerouse I opened a PR with a temptative fix #18125, let me know what you think. |
|
@blakerouse The Metricbeat generator test is failing in master. Do you think it's related to your changes? |
blakerouse
added a commit
that referenced
this pull request
Apr 30, 2020
… a pod and use kind to setup a kubernetes cluster (#18111) * Run kubernetes integration tests inside of a pod and use kind to setup a kubernetes cluster (#17656) * Use kind to bring up a local kubernetes cluster and then run the integration tests against the kind cluster. * Run mage update. * Add more tests. * Fix more tests, install kind in prepare-tests. * Switch to running kubernetes integration tests inside of Kubernetes. * Use golang 1.13.9. * Fix for other beats. * Run mage fmt. * Don't run kubernetes integration tests if not inside kubernetes environment. * Fix metricbeat to use mage and the makefile shim for mage. * Improve the error message when kind or kubectl is not available. * Refactor the integration tests into a more module system. * Fix go vet. * Setup travis to use kind. * Run kubernetes integration tests in Jenkins. * Fix filebeat magefile. * Fix travis and Jenkins. * Check requirements of the test runner before actually running the tests. * Add return on parsebool error. * Don't return err on missing requirements for tester. * Run make update. * Move the kubernetes items to its own module, import that module only by metricbeat. (cherry picked from commit db29874) * Update fields.
|
@andrewkroh I am working on this here - #18086 The latest run I got with that MP showed something different. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Refactors how
mage goIntegTestrun for metricbeat. Before each module was run inside of a docker container usingdocker-compose. Now depending on the module it will either run in docker or kubernetes. At the moment only thekubernetesmodule runs inside of kubernetes.Each module is determined if it should run inside of docker or inside of kubernetes. If the module has a
docker-compose.ymlfile then it will run in docker, if the module has akubernetes.ymlthen it will run inside of kubernetes.The docker runner stuff is the same so nothing really changes there, but the kubernetes runner is what is new. It works by executing the tests inside of a Pod running inside of the kubernetes.
If
KUBECONFIGis defined in the environment then the runner just executes the tests against that defined kubernetes. If its not defined then kind will be used to setup a cluster and run the tests against that cluster.Running the tests inside of kubernetes is handled by
KubeRemotewhich uses the kubernetes client-go to setup all the requirements to get it running. Below breaks down the steps that are taken to get it running:sshd-rsyncand the test executer container that starts once init container exits. The init container exits after the first SSH connection is made and disconnects.rsyncis ran locally to sync the beats repo to the init container sync directory which was mounted as anemptyDirand shared between the init container and exec container.kubernetes.Why is it important?
Currently all of the Kubernetes integration tests are skipped, this fixes them so they can actually be ran inside of a real Kubernetes cluster.
Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist
How to test this PR locally
Related issues