Go modules

[kvch]

This PR introduces the new dependency management system for beats. From now on we are using go mod instead of govendor. The name of the module we provide is named github.com/elastic/beats/v7.

We adopted go modules, but we still keep dependencies under the folder vendor. However, it is maintained by go mod now by running mage vendor. Thus, it is not possible to add local changes to the dependencies there because next time someone runs the command it will be overwritten. If you need to apply patches to dependencies either fork the repository or try to contribute it back to the original repo.

This PR does not address the changes in Beat generators. Those are going to be updated in a follow-up PR.

FAQ

How can I add a new dependency?

TL;DR

go get {{ module_name }}@{{ required_version }}
mage vendor
make notice

Run go get {{ module_name }}@{{ required_version }} in the repository. This adds the module to the requirements list in go.mod. In order to add the dependency to the vendor folder, run mage vendor. If the dependency contains files which are not copied by this command (e.g. C source files), add them to this list: https://github.com/elastic/beats/tree/master/dev-tools/mage/gomod.go

How can I upgrade a dependency?

The process is similar to adding a new dependency. Except for one step. If the dependency is updated to a newer major version, make sure to follow up changes in the root import paths of those libs. I suggest using the tool named mod to upgrade it: https://github.com/marwan-at-work/mod

How can I use a fork of a dependency?

If you need to use a fork of a dependency, add it to go.mod either manually or by running the following command:

go mod edit -replace {{ dependency_name }}={{ fork_name }}@{{ fork_version }}

Why is the major version part of the import path?

In a nutshell, it is needed to tell apart major version changes. Their import paths are different because those are incompatible. See more: https://blog.golang.org/v2-go-modules

Thus, when we are releasing v8.0.0, we need to change the import path again. The tool named mod is going to be useful for us in this case, too.

Why is make check failing?

Are you seeing the following error?

make[1]: Leaving directory `/home/travis/gopath/src/github.com/elastic/beats'
diff --git a/go.mod b/go.mod
index 170f6660..fc5fb6ee 100644
--- a/go.mod
+++ b/go.mod
@@ -53,6 +53,7 @@ require (
 	github.com/dop251/goja_nodejs v0.0.0-20171011081505-adff31b136e6
 	github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4
 	github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2
+	github.com/elastic/beats v7.6.0+incompatible
 	github.com/elastic/ecs v1.4.0
 	github.com/elastic/go-libaudit v0.4.0
 	github.com/elastic/go-licenser v0.2.1

If your go.mod lists github.com/elastic/beats as a requirement, the repo contains an outdated root import path. Change github.com/elastic/beats to github.com/elastic/beats/v7.

What is next?

After this PR is merged, please rebase all of your open PRs. In most cases rebasing will lead to conflicts. Possible sources of conflicts:

1. New import path conflicts with the old. To resolve it switch every github.com/elastic/beats import to github.com/elasitc/beats/v7.
2. A new dependency has been added or existing was updated. Update the dependency using go modules.

Further reading for the interested

• Golang blog: Using Go modules
• Golang wiki

Related issues

#15868

[kvch]

jenkins test this

[kvch]

run beats-ci/package

[kvch]

run beats-ci/package

[urso]

One more tip: run go mod tidy before commiting go.mod. go get puts every dependency that have been seen into your go.mod. Some tools (e.g. vim-go) can add unwanted entries.

btw. do we have a make check target that checks if go.mod is tidy?

[kvch]

No, I haven't added a check, because mage vendor runs tidy for you. So if you have done anything to vendor, e.g. update or add a new one, go mod tidy is called. You think it is necessary to add it on top of this?

[urso]

Its not about packages we want to vendor. It is about people installing tools that are actually no dependency. When I install a custom tool (or vim-go does because I want to update some tools) while being in the Beats repository, then go will automatically add this tool as a dependency to the go.mod file, which will stay there if I don't call go mod vendor by chance. Not sure if this behavior changed in go1.14, but I managed to pollute my go.mod files a few times with this :)

[kvch]

run beats-ci/package

[kvch]

run beats-ci/package

[kvch]

Issue with node_integration_test.go of redis in x-pack/metricbeat is unrelated to this change.

[bschaeffer]

👋 New to go modules here.... why do all the tagged versions remove the go.mod? If I try to require github.com/elastic/beats v7.6.2 it just complains about a missing go.mod, which is present in master but none of the tagged releases.

[jhump]

@bschaeffer, FYI, there was no go.mod file at version v7.6.2. This PR wasn't merged until after that. Looks like the first release with it is v7.7.0. Also, with Go modules, a major version suffix is required when the major version is >1, e.g. /v7.

So you have to do something like this to get it to work:

require github.com/elastic/beats/v7 v7.7.0