Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce apm-server.auth.* config #5457

Merged
merged 5 commits into from
Jun 17, 2021
Merged

Conversation

axw
Copy link
Member

@axw axw commented Jun 15, 2021

Motivation/summary

Introduce the new AgentAuth config structure, which holds API Key and secret token auth. Later we will add "anonymous" auth here too, and deprecate/replace some RUM config (rate limiting and allowed service names).

We also introduce a new YAML naming scheme for the config, apm-server.auth.*. The old config is deprecated and copied across to the new config fields.

Checklist

How to test these changes

  1. Set apm-server.secret_token and apm-server.api_key.*, make sure they are honoured (e.g. query "GET /" with/out auth)
  2. Set apm-server.auth.secret_token and apm-server.auth.api_key.*, same again.
  3. Set both new and old with different values: check that a warning is logged that the old config is ignored, and check that it is possible to auth using the new but not old

Related issues

#5347

@apmmachine
Copy link
Contributor

apmmachine commented Jun 15, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #5457 updated

  • Start Time: 2021-06-17T02:01:44.290+0000

  • Duration: 42 min 30 sec

  • Commit: e0cc895

Test stats 🧪

Test Results
Failed 0
Passed 6114
Skipped 120
Total 6234

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 2

Expand to view the steps failures

Test Sync
  • Took 3 min 21 sec . View more details on here
  • Description: ./.ci/scripts/sync.sh
Build packages
  • Took 29 min 6 sec . View more details on here
  • Description: ./.ci/scripts/package.sh

Log output

Expand to view the last 100 lines of log output

[2021-06-17T02:37:26.368Z] === RUN   TestUnstartedAPMServer
[2021-06-17T02:37:26.368Z] --- PASS: TestUnstartedAPMServer (0.00s)
[2021-06-17T02:37:26.368Z] === RUN   TestAPMServerStartTLS
[2021-06-17T02:37:26.368Z] --- PASS: TestAPMServerStartTLS (0.11s)
[2021-06-17T02:37:26.368Z] === RUN   TestExpvar
[2021-06-17T02:37:26.368Z] --- PASS: TestExpvar (0.11s)
[2021-06-17T02:37:26.368Z] PASS
[2021-06-17T02:37:26.368Z] ok  	github.com/elastic/apm-server/systemtest/apmservertest	2.646s
[2021-06-17T02:37:26.368Z] ?   	github.com/elastic/apm-server/systemtest/benchtest	[no test files]
[2021-06-17T02:37:26.368Z] ?   	github.com/elastic/apm-server/systemtest/cmd/apmbench	[no test files]
[2021-06-17T02:37:26.368Z] ?   	github.com/elastic/apm-server/systemtest/estest	[no test files]
[2021-06-17T02:37:26.368Z] ?   	github.com/elastic/apm-server/systemtest/fleettest	[no test files]
[2021-06-17T02:37:26.368Z] + cleanup
[2021-06-17T02:37:26.368Z] + rm -rf /tmp/tmp.MWXM1RZ1mR
[2021-06-17T02:37:26.368Z] + .ci/scripts/docker-get-logs.sh
[2021-06-17T02:37:27.432Z] Post stage
[2021-06-17T02:37:27.444Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5457/src/github.com/elastic/apm-server/build
[2021-06-17T02:37:27.466Z] Archiving artifacts
[2021-06-17T02:37:27.782Z] Recording test results
[2021-06-17T02:37:28.655Z] [Checks API] No suitable checks publisher found.
[2021-06-17T02:37:28.970Z] + tar --version
[2021-06-17T02:37:29.309Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2021-06-17T02:37:29.587Z] Archiving artifacts
[2021-06-17T02:37:30.094Z] Terminated
[2021-06-17T02:37:30.080Z] Terminated
[2021-06-17T02:40:54.892Z] [INFO] For detailed information see: https://apm-ci.elastic.co/job/apm-integration-tests-selector-mbp/job/master/17744/display/redirect
[2021-06-17T02:40:55.141Z] Copied 18 artifacts from "APM Integration Test MBP Selector » master" build number 17744
[2021-06-17T02:40:56.292Z] Post stage
[2021-06-17T02:40:56.302Z] Recording test results
[2021-06-17T02:40:56.957Z] [Checks API] No suitable checks publisher found.
[2021-06-17T02:41:03.358Z] >> package: Building apm-server type=zip for platform=windows/amd64
[2021-06-17T02:41:03.359Z] >> package: Building apm-server type=tar.gz for platform=linux/386
[2021-06-17T02:41:03.359Z] >> package: Building apm-server type=deb for platform=linux/amd64
[2021-06-17T02:41:03.359Z] >> package: Building apm-server-oss type=deb for platform=linux/amd64
[2021-06-17T02:41:03.359Z] >> package: Building apm-server-oss type=tar.gz for platform=darwin/amd64
[2021-06-17T02:41:05.898Z] >> package: Building apm-server type=tar.gz for platform=darwin/amd64
[2021-06-17T02:41:08.441Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:08.441Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:08.704Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:08.704Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:08.982Z] >> package: Building apm-server-oss type=tar.gz for platform=linux/386
[2021-06-17T02:41:14.286Z] >> package: Building apm-server-oss type=deb for platform=linux/386
[2021-06-17T02:41:15.241Z] >> package: Building apm-server-oss type=rpm for platform=linux/386
[2021-06-17T02:41:19.484Z] >> package: Building apm-server-oss type=docker for platform=linux/amd64
[2021-06-17T02:41:20.869Z] >> package: Building apm-server-oss type=rpm for platform=linux/amd64
[2021-06-17T02:41:21.445Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:21.445Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:24.016Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:24.016Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:30.675Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:30.675Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:30.675Z] >> package: Building apm-server-oss type=rpm for platform=linux/arm64
[2021-06-17T02:41:40.695Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:40.695Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:44.908Z] >> package: Building apm-server type=rpm for platform=linux/amd64
[2021-06-17T02:41:57.180Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:57.180Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:59.721Z] >> package: Building apm-server type=docker for platform=linux/amd64
[2021-06-17T02:42:06.305Z] >> package: Building apm-server type=docker for platform=linux/amd64
[2021-06-17T02:42:32.906Z] >> package: Building apm-server-oss type=tar.gz for platform=linux/arm64
[2021-06-17T02:42:39.517Z] >> package: Building apm-server-oss type=deb for platform=linux/arm64
[2021-06-17T02:42:47.699Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:42:47.699Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:42:58.010Z] >> package: Building apm-server type=tar.gz for platform=linux/amd64
[2021-06-17T02:43:01.394Z] >> package: Building apm-server type=rpm for platform=linux/arm64
[2021-06-17T02:43:09.566Z] >> package: Building apm-server type=tar.gz for platform=linux/arm64
[2021-06-17T02:43:11.526Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:43:11.527Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:43:16.889Z] >> package: Building apm-server type=deb for platform=linux/arm64
[2021-06-17T02:43:26.948Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:43:26.948Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:43:37.024Z] >> package: Building apm-server type=rpm for platform=linux/386
[2021-06-17T02:43:39.588Z] >> package: Building apm-server type=deb for platform=linux/386
[2021-06-17T02:43:42.121Z] >> package: Building apm-server-oss type=tar.gz for platform=linux/amd64
[2021-06-17T02:43:44.651Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:43:44.651Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:43:49.972Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:43:49.972Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:43:49.972Z] >> package: Building apm-server type=zip for platform=windows/386
[2021-06-17T02:43:53.288Z] >> package: Building apm-server-oss type=zip for platform=windows/386
[2021-06-17T02:43:56.618Z] >> package: Building apm-server-oss type=zip for platform=windows/amd64
[2021-06-17T02:44:11.584Z] >> Testing package contents
[2021-06-17T02:44:11.584Z] # command-line-arguments
[2021-06-17T02:44:11.584Z] ../../../../pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20210614232151-2871d29be93a/dev-tools/packaging/package_test.go:39:2: missing go.sum entry for module providing package github.com/blakesmith/ar; to add:
[2021-06-17T02:44:11.584Z] 	go mod download github.com/blakesmith/ar
[2021-06-17T02:44:11.584Z] FAIL	command-line-arguments [setup failed]
[2021-06-17T02:44:11.584Z] FAIL
[2021-06-17T02:44:11.584Z] package ran for 28m2.983237017s
[2021-06-17T02:44:11.584Z] Error: running "go test /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5457/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20210614232151-2871d29be93a/dev-tools/packaging/package_test.go -files /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5457/src/github.com/elastic/apm-server/build/distributions/*" failed with exit code 1
[2021-06-17T02:44:11.584Z] Makefile:323: recipe for target 'release' failed
[2021-06-17T02:44:11.584Z] make: *** [release] Error 1
[2021-06-17T02:44:12.900Z] Stage "Publish" skipped due to earlier failure(s)
[2021-06-17T02:44:13.005Z] Failed in branch Package
[2021-06-17T02:44:13.405Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5457
[2021-06-17T02:44:13.458Z] [INFO] getVaultSecret: Getting secrets
[2021-06-17T02:44:13.519Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-17T02:44:14.467Z] + chmod 755 generate-build-data.sh
[2021-06-17T02:44:14.467Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5457/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5457/runs/13 FAILURE 2549912
[2021-06-17T02:44:14.467Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5457/runs/13/steps/?limit=10000 -o steps-info.json
[2021-06-17T02:44:14.717Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5457/runs/13/tests/?status=FAILED -o tests-errors.json

@axw axw force-pushed the beater-config-auth branch 3 times, most recently from f43315f to 410009c Compare June 15, 2021 13:37
Copy link
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great.

docs/configuration-process.asciidoc Outdated Show resolved Hide resolved
Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.
@axw
Copy link
Member Author

axw commented Jun 16, 2021

@bmorelli25 would you please take a look at the docs changes? I've renamed some config, and added new deprecation sections for the deprecated config names.

I've also moved "api_key.* configuration options" into the "API keys" section, not sure if it was intentional that it was separate? I can move back if you prefer, but then I'm not sure where the deprecation section should go.

@axw axw marked this pull request as ready for review June 16, 2021 04:31
@axw axw requested a review from a team June 16, 2021 04:32
@mergify
Copy link
Contributor

mergify bot commented Jun 16, 2021

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beater-config-auth upstream/beater-config-auth
git merge upstream/master
git push upstream beater-config-auth

Copy link
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also change the apmpackage please. I suggest we remove the deprecated settings and only support the new ones.

@axw
Copy link
Member Author

axw commented Jun 16, 2021

@simitt will do. I was going to wait for your changes (#5444) to land, but I'll just update now for the purposes of review and update again when that lands.

@axw axw requested a review from simitt June 16, 2021 07:28
@mergify
Copy link
Contributor

mergify bot commented Jun 16, 2021

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beater-config-auth upstream/beater-config-auth
git merge upstream/master
git push upstream beater-config-auth

Copy link
Member

@bmorelli25 bmorelli25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs look great–thanks!

It looks like there's one additional change that needs to be made to the command reference file. I'll fix that in the Beats repo and copy it over to apm-server before 7.14.

@axw axw added the v7.14.0 label Jun 17, 2021
@axw
Copy link
Member Author

axw commented Jun 17, 2021

Failure is related to the beats update, which will be resolved by #5471

@axw axw merged commit fc60576 into elastic:master Jun 17, 2021
@axw axw deleted the beater-config-auth branch June 17, 2021 03:17
mergify bot pushed a commit that referenced this pull request Jun 17, 2021
* Introduce `apm-server.auth.*` config

Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.

* docs: update config names

* apmpackage: update auth config keys

(cherry picked from commit fc60576)

# Conflicts:
#	changelogs/head.asciidoc
axw added a commit that referenced this pull request Jun 17, 2021
* Introduce `apm-server.auth.*` config (#5457)

* Introduce `apm-server.auth.*` config

Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.

* docs: update config names

* apmpackage: update auth config keys

(cherry picked from commit fc60576)

Co-authored-by: Andrew Wilkins <axw@elastic.co>
@simitt simitt self-assigned this Jul 9, 2021
mergify bot pushed a commit that referenced this pull request Jul 9, 2021
* Introduce `apm-server.auth.*` config

Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.

* docs: update config names

* apmpackage: update auth config keys

(cherry picked from commit fc60576)

# Conflicts:
#	apmpackage/apm/agent/input/template.yml.hbs
#	beater/config/config.go
#	beater/jaeger/server.go
#	beater/processors.go
#	beater/server.go
#	changelogs/head.asciidoc
@simitt
Copy link
Contributor

simitt commented Jul 9, 2021

Tested with BC2:

  • old settings work as expected
  • new settings work as expected
  • API Key auth disabled when:
apm-server.api_key.enabled: true
apm-server.auth.api_key.enabled: false
  • Secret token 'abcd' expected
apm-server.secret_token: 'xxx'
apm-server.auth.secret_token: 'abcd'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants