Introduce apm-server.auth.anonymous config

[axw]

Motivation/summary

Generalise the ability for agents to send events unauthenticated (anonymous) but rate-limited. Until now this has been a RUM-only feature, but we now find ourselves needing it also for the iOS agent.

Anonymous auth is disabled by default, but is automatically enabled when RUM is enabled as long as apm-server.auth.anonymous hasn't been explicitly configured. The existing RUM config for allowed services and rate limiting are deprecated and replaced with equivalent config under apm-server.auth.anonymous.*.

Instead of restricting anonymous auth to requests going by endpoint (i.e. RUM intake and agent config), we now restrict based on the provided agent and service names. There was previously nothing stopping clients from spoofing RUM agents, e.g. sending events to the RUM intake with a non-RUM agent name, so this is not any less secure.

Checklist

• Update CHANGELOG.asciidoc
  - [ ] Documentation has been updated docs: document apm-server.auth.anonymous config #5698

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

1. Ensure existing config works as before: RUM should be allowed to send data without any auth token when secret_token or api_key auth is enabled; ensure no other agents can send data without an auth token.
2. Define apm-server.auth.anonymous.allow_agent: [iOS/swift], check that the iOS/swift agent can send data without an auth token.
3. Define apm-server.auth.anonymous.allow_service: [opbeans-rum], check that opbeans-rum can send data. Change it to something else and check that opbeans-rum cannot send data.

Related issues

Closes #5347

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-07-14T06:41:20.376+0000

• Duration: 53 min 21 sec

• Commit: 3d2a881

Test stats 🧪

Test	Results
Failed	0
Passed	5961
Skipped	14
Total	5975

Trends 🧪

[simitt]

Generally looks good, left a couple of minor comments for now.

[simitt]

Can you also add the new settings to the apmpackage please and remove the deprecated ones there?

[axw]

Added:

• anonymous_enabled -- on by default, always shown; everything else is advanced config
• anonymous_allow_agent -- defaults to [rum-js, js-base, iOS/swift] (iOS/swift is not in the standalone defaults)
• anonymous_allow_service (was rum_allow_service_names)
• anonymous_rate_limit_event_limit (was rum_event_rate_limit)
• anonymous_rate_limit_ip_limit (was rum_event_rate_lru_size)

[axw]

/test

[simitt]

Wait until elastic/kibana#109237 is resolved and part of the BC and then ensure the new config options are also tested for the apm package (managed by agent)

[simitt]

Tested with BC3:

The following works as expected:

• rum.enabled: true: RUM requests are accepted without auth
• auth.anonymous.enabled: true: RUM requests are denied unless rum.enabled: true; then only requests with rum-js and js-base are allowed without auth; providing secret-tooken or API Key all agents are allowed
• limiting anonymous access to services defined in apm-server.auth.anonymous.allow_service works as expected

Testing with the iOS agent:
Given following apm-server config:

apm-server:
  auth:
    api_key:
      enabled: true
    secret_token: "1234"
    anonymous:
      enabled: true
      allow_agent: [iOS/swift, rum-js, js-base]

• specifying a secret-token in the iOS agent, the ingestion works as expected.
• ingestion fails if the agent does not provide a secret-token, despite adding iOS/swift to apm-server.auth.anonymous.allow_agent

{"log.level":"error","@timestamp":"2021-08-31T14:59:06.897+0200","log.logger":"beater.grpc","log.origin":{"file.name":"interceptors/logging.go","file.line":63},"message":"{error 26 0  rpc error: code = Unauthenticated desc = authentication failed}","service.name":"apm-server","source.address":"127.0.0.1","grpc.request.method":"/opentelemetry.proto.collector.trace.v1.TraceService/Export","event.duration":24098,"grpc.response.status_code":"Unauthenticated","error.message":"authentication failed","ecs.version":"1.6.0"}

Created a bug description in #6095

[simitt]

followed up on the iOS testing issue - this was purely an issue with the test app (sending an empty auth header instead of none). Everything works as expected.