[Elastic Agent] Add the ability to run the Fleet Server #23736

blakerouse · 2021-01-28T14:28:12Z

What does this PR do?

Adds the ability for Fleet Server to be started by Elastic Agent.

This just adds the basics for Fleet Server to be ran by Elastic Agent. This does not handle the enrollment piece of starting Fleet Server to perform the enrollment for the running Elastic Agent parent. That work will be present in a follow up branch.

This provides enough for the Fleet Server to be started by the Elastic Agent and for the release manager build of Elastic Agent to include the Fleet Server.

The Fleet Server requires a custom output configuration for elasticsearch because Fleet Server requires custom permissions and connection information before Elastic Agent has even enrolled into Fleet. This can currently be set manually in the fleet.yml once the Elastic Agent is enrolled in the current Kibana.

id: d9abff40-60bf-11eb-96c5-a7508ca0826b
fleet:
  enabled: true
  access_api_key: RHRuQlJIY0JXd3N4Q3A3TmZRbW46RVpPbHp1czVRY21zVzJXQlYtZzZhZw==
  kibana:
    protocol: http
    host: localhost:5601
    hosts:
    - localhost:5601
    timeout: 5m0s
    ssl:
      verification_mode: none
      renegotiation: never
  reporting:
    threshold: 10000
    check_frequency_sec: 30
  agent:
    id: ""
  server:
    output:
      elasticsearch:
        hosts: '${ELASTICSEARCH_HOSTS:localhost:9200}'
        username: '${ELASTICSEARCH_USERNAME:elastic}'
        password: '${ELASTICSEARCH_PASSWORD:changeme}'

The follow up branch will handle writing the fleet.server.output.elasticsearch into the fleet.yml.

Why is it important?

Needed so Elastic Agent can run the Fleet server.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes [Elastic Agent] Embed Fleet Server as part of the Elastic Agent #22686

Logs

2021-01-28T09:14:23.382-0500    INFO    operation/operator.go:245       operation 'operation-install' skipped for fleet-server.8.0.0
2021-01-28T09:14:23.772-0500    INFO    log/reporter.go:40      2021-01-28T09:14:23-05:00: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0[d9abff40-60bf-11eb-96c5-a7508ca0826b]: State changed to STARTING: Starting
2021-01-28T09:14:23.780-0500    INFO    operation/monitoring.go:39      operator.handleStartSidecar: monitoring is not running and disabled, no action taken
2021-01-28T09:14:24.806-0500    INFO    log/reporter.go:40      2021-01-28T09:14:24-05:00: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0[d9abff40-60bf-11eb-96c5-a7508ca0826b]: State changed to STARTING: Waiting on default policy with Fleet Server integration

elasticmachine · 2021-01-28T17:01:06Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-01-28T18:05:09Z

Pinging @elastic/ingest-management (Team:Ingest Management)

ruflin · 2021-01-29T09:37:12Z

x-pack/elastic-agent/spec/fleet-server.yml

+  - remove_key:
+      key: output
+
+  - select_into:


Having the output directly as part of the input would also simplify the spec file here?

It would still need to be moved because the output is a top-level key to Fleet Server.

I also don't think we want that, because we do not want to send username/password from the input. That also would have a chicken-n-egg problem of getting that information initially.

This goes back to a previous discussion. I think the fleet-server config should be designed and optimised for what is needed for Elastic Agent. So either it directly supports input as the default or it maps it internally. But the translation should not be up to the Elastic Agent. It can be a temporary solution for now.

Not sure I follow the chicken-egg part around username / password? Can you elaborate?

Fleet Server does all of that. It has a top level inputs just like Elastic Agent. The issue is the output, which is also top level output and matches Elastic Agent.

The issue is on start-up there is no output, because Elastic Agent has not enrolled but Fleet Server needs to connect to an output so that the Elastic Agent can enroll.

ruflin · 2021-01-29T09:37:40Z

x-pack/elastic-agent/spec/fleet-server.yml

+        - remove_key:
+            key: server
+
+  - map:


In general I would expect this spec file to only require the input routing to fleet-server. Why is this needed? Is this something we just add for every input automatically and because of this need to remove it?

Yes every input automatically gets these items from Elastic Agent, so they need to be removed.

michalpristas

change looks ok

michalpristas · 2021-01-29T10:47:30Z

x-pack/elastic-agent/CHANGELOG.next.asciidoc

@@ -64,3 +64,4 @@
 - Log level reloadable from fleet {pull}22690[22690]
 - Push log level downstream {pull}22815[22815]
 - Add metrics collection for Agent {pull}22793[22793]
+- Add support for Fleet Server {pull}


please fill in PR number

elasticmachine · 2021-01-29T16:04:27Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: blakerouse commented: /test
- Start Time: 2021-02-01T13:04:22.953+0000
Duration: 90 min 37 sec
Commit: 4c3b6ee

Test stats 🧪

Test	Results
Failed	0
Passed	45151
Skipped	4737
Total	49888

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	45151
Skipped	4737
Total	49888

blakerouse · 2021-02-01T13:03:57Z

/test

* Add the ability to run the Fleet Server. * Add test and changelog. * Fix changelog. (cherry picked from commit d59f780)

… Fleet Server (#23785) * [Elastic Agent] Add the ability to run the Fleet Server (#23736) * Add the ability to run the Fleet Server. * Add test and changelog. * Fix changelog. (cherry picked from commit d59f780) * Fixed monitoring filebeat and metricbeat not connecting to Agent over GRPC (#23843) Fixed monitoring filebeat and metricbeat not connecting to Agent over GRPC (#23843) Co-authored-by: Michal Pristas <michal.pristas@gmail.com>

blakerouse added the Team:Elastic-Agent Label for the Agent team label Jan 28, 2021

blakerouse self-assigned this Jan 28, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 28, 2021

blakerouse marked this pull request as ready for review January 28, 2021 17:00

blakerouse added 2 commits January 28, 2021 13:04

Add the ability to run the Fleet Server.

fcdba78

Add test and changelog.

136fb85

blakerouse force-pushed the run-fleet-server branch from b9822f7 to 136fb85 Compare January 28, 2021 18:05

botelastic bot added the Team:Ingest Management label Jan 28, 2021

ruflin mentioned this pull request Jan 29, 2021

[Meta] Fleet Server Phase 1 elastic/fleet-server#50

Closed

13 tasks

ruflin reviewed Jan 29, 2021

View reviewed changes

michalpristas approved these changes Jan 29, 2021

View reviewed changes

Fix changelog.

4c3b6ee

blakerouse merged commit d59f780 into elastic:master Feb 1, 2021

blakerouse deleted the run-fleet-server branch February 1, 2021 15:11

blakerouse mentioned this pull request Feb 1, 2021

Cherry-pick #23736 to 7.x: [Elastic Agent] Add the ability to run the Fleet Server #23785

Merged

4 tasks

blakerouse added the v7.12.0 label Feb 1, 2021

mdelapenya mentioned this pull request Feb 3, 2021

[Elastic Agent] Monitoring filebeat and metricbeat not connecting to Agent over GRPC #23833

Closed

michalpristas mentioned this pull request Feb 4, 2021

Fixed monitoring filebeat and metricbeat not connecting to Agent over GRPC #23843

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Elastic Agent] Add the ability to run the Fleet Server #23736

[Elastic Agent] Add the ability to run the Fleet Server #23736

blakerouse commented Jan 28, 2021 •

edited

Loading

elasticmachine commented Jan 28, 2021

elasticmachine commented Jan 28, 2021

ruflin Jan 29, 2021

blakerouse Jan 29, 2021

ruflin Feb 1, 2021

blakerouse Feb 1, 2021

ruflin Jan 29, 2021

blakerouse Jan 29, 2021

michalpristas left a comment

michalpristas Jan 29, 2021

elasticmachine commented Jan 29, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

blakerouse commented Feb 1, 2021

[Elastic Agent] Add the ability to run the Fleet Server #23736

[Elastic Agent] Add the ability to run the Fleet Server #23736

Conversation

blakerouse commented Jan 28, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

Logs

elasticmachine commented Jan 28, 2021

elasticmachine commented Jan 28, 2021

ruflin Jan 29, 2021

Choose a reason for hiding this comment

blakerouse Jan 29, 2021

Choose a reason for hiding this comment

ruflin Feb 1, 2021

Choose a reason for hiding this comment

blakerouse Feb 1, 2021

Choose a reason for hiding this comment

ruflin Jan 29, 2021

Choose a reason for hiding this comment

blakerouse Jan 29, 2021

Choose a reason for hiding this comment

michalpristas left a comment

Choose a reason for hiding this comment

michalpristas Jan 29, 2021

Choose a reason for hiding this comment

elasticmachine commented Jan 29, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

blakerouse commented Feb 1, 2021

blakerouse commented Jan 28, 2021 •

edited

Loading

elasticmachine commented Jan 29, 2021 •

edited by jenkins-beats-ci bot

Loading