[Fleet] Make default integration install explicit #121628

juliaElastic · 2021-12-20T10:58:52Z

Summary

Making default integration install explicit
Fixes #108456

Manual test cases:

Initial kibana startup:

Navigate to Fleet, verify that /setup has finished (should happen quickly)
verify that no default policies or integrations exist (except for cloud fleet server policy, see below)

Add integration flow:

go to Integrations / Apache / Add integration
verify that UI offers to create a new policy
when submitting, verify that agent policy is being created with system integration, and elastic agent integration is installed as well (first time takes long, about 2 mins)
try again to add integration, should see now the tabbed view New hosts / Existing hosts
test both paths: create new policy and use existing policy

Fleet Server policy flow on-prem:

Go to Fleet tab
verify that in fleet server instructions there is a create button to add fleet server policy
click create button
verify that fleet server policy is created, UI should show a dropdown instead of create button now
verify that fleet server policy has a fleet server integration
verify policy has system integration if system monitoring checkbox was left checked as default
elastic agent integration should be installed as well if monitoring was enabled on the policy
generated fleet server command should include the newly created policy id

Add agent flow (supposing Fleet Server is installed):

go to Fleet / Agents / Add agent
on both Enroll in Fleet / Run standalone tab, see option to create Agent policy (when no policy exists, except fleet server policy)
click on Create button, verify that default policy is created with system integration, and elastic agent integration is installed
verify that generated enroll command (or policy config yml) includes newly created policy id
verify that now agent policy dropdown shows up with pre-selected agent policy
create one more agent policy, and verify that dropdown has no selection by default

Cloud (depends on change in cloud stackpack)

Verify that Elastic Cloud agent policy is created by default with fleet server integration
non fleet server flows are the same as on prem (no default policy, system, elastic agent integration initially)

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

juliaElastic · 2021-12-20T11:00:16Z

x-pack/plugins/fleet/server/services/package_policy.ts

@@ -1382,7 +1384,9 @@ export async function incrementPackageName(
    ? packagePolicyData.items
        .filter((ds) => Boolean(ds.name.match(pkgPoliciesNamePattern)))
        .map((ds) => parseInt(ds.name.match(pkgPoliciesNamePattern)![1], 10))
-        .sort()
+        .sort((a: number, b: number) => {
+          return a - b;


this fixes a bug, default sorting doesn't work correctly on numbers, e.g. "1, 10, 9" - this was giving an error when reaching system-10 name in integrations, and the logic returned system-10 again as incremented name
EDIT: changed this to find max number, no need to sort.

juliaElastic · 2021-12-20T11:01:14Z

x-pack/plugins/fleet/server/routes/agent_policy/handlers.ts

  try {
+    if (withSysMonitoring) {


potential problem:
installing these packages might take long (about 2 mins) which causes network timeout. this results in an error, and fails next agent policy creation request. Could this be improved by bundled default packages?

EDIT: during latest testing, the install took much less time (25s), will keep an eye on this

Bundled default packages aren't going to make much of a difference here based on my investigations in #110500. We'll likely need elastic/elasticsearch#77505 to fix this.

juliaElastic · 2021-12-20T12:28:36Z

.../public/applications/fleet/sections/agent_policy/components/agent_policy_advanced_fields.tsx

+            defaultMessage="Collecting monitoring logs and metrics will also create an {agent} integration. Monitoring data will be written to the default namespace specified above."
+            values={{
+              agent: (
+                <AgentPolicyPackageBadge pkgName={'elastic_agent'} pkgTitle={'Elastic Agent'} />


currently displaying a generic badge. in order to display the package's real badge, I would need to know the latest version number, which seems overkill

elasticmachine · 2021-12-29T15:47:58Z

Pinging @elastic/fleet (Team:Fleet)

juliaElastic · 2021-12-30T09:42:10Z

x-pack/plugins/fleet/server/services/epm/packages/install.ts

@@ -546,6 +545,7 @@ export async function createInstallation(options: {
    ? true
    : undefined;

+  // TODO cleanup removable flag and isUnremovablePackage function


@mostlyjason what about the endpoint package, can it be removable as well?

I don't see why not. Adding @kevinlog for confirmation. For context, we are trying to remove the concept of unremovable packages since they will no longer be installed by default. This would also allow users to reinstall the package if there is a problem. Can you think of any reason users shouldn't be able to remove endpoint, provided they removed all integration policies first?

@mostlyjason @juliaElastic I think it's OK to make the endpoint package removable especially for reasons you stated such as allowing users to uninstall and reinstall if there is a problem.

If the user removed all integration policies first then all of their Endpoints would be uninstalled. I would just like to test what would happen with their data if they uninstalled the package. This would imply that we remove mappings, etc. If we're going to make the Endpoint package removable, I'd like to open up a separate ticket/PR if possible where we can test a few different scenarios before merging/releasing.

@kevinlog thanks for your response, are you comfortable with testing this soon as we are planning to merge this feature for 8.1? If not, should we keep endpoint package unremovable for now?
cc @joshdover

When we uninstall packages, I don't believe any existing data should be affected, though the next rollover of the data stream would cause the write index to pickup the generic mappings for logs-*-* and metrics-*-* rather than the endpoint-specific ones. This would only affect new data that is ingested which shouldn't be happening once a package is uninstalled anyways.

juliaElastic · 2021-12-30T09:46:59Z

x-pack/plugins/fleet/server/routes/agent_policy/handlers.ts

-      // NOTE: we ignore failures in attempting to create package policy, since agent policy might have been created
-      // successfully
-      withSysMonitoring
+      // TODO case when both isDefaultFleetServer and withSysMonitoring is true


@joshdover is it a valid use case to add system monitoring to default fleet server policy? currently default fleet server policy only includes fleet server integration

I think that is probably a valid use case for preconfiguration / kibana.yml, but isn't possible via the UI today.

Is there a major hurdle in supporting this case?

I thought we are removing the default fleet server policy? I didn't read the code, so am I missing context behind this question?

After reading the comments here and in design doc, it seems that we can completely get rid of the flag to mark a policy default (is_default and is_default_fleet_server flags in code). I wasn't aware of this before, probably because on the designs the policies were still called "Default policy" and "Default Fleet Server policy".

@joshdover I think we can support system integration with fleet server policy, I didn't want to start implementing it until confirmed.

Makes sense to me. If we're going to drop the concept of default completely, we should remove that field from our Saved Object mappings and add a Saved Object migration that deletes this field.

juliaElastic · 2022-01-27T12:19:51Z

@elasticmachine merge upstream

nchaulet

One last comment otherwise 🚀

juliaElastic · 2022-01-31T08:50:07Z

@elasticmachine merge upstream

juliaElastic · 2022-01-31T10:34:58Z

@elasticmachine merge upstream

kibana-ci · 2022-02-01T10:57:24Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	557	565	+8

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1169	1142	-27

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	648.4KB	659.6KB	+11.2KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	112.8KB	109.2KB	-3.6KB

Unknown metric groups

API count

id	before	after	diff
`fleet`	1285	1258	-27

ESLint disabled line counts

id	before	after	diff
`fleet`	40	41	+1

Total ESLint disabled count

id	before	after	diff
`fleet`	48	49	+1

History

💔 Build #20963 failed 511c236
💚 Build #20705 succeeded 247c279
💚 Build #20665 succeeded ac8a0ff
💚 Build #20615 succeeded 9d42aec
💚 Build #20088 succeeded 68bd12a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

first draft

fd1deba

juliaElastic commented Dec 20, 2021

View reviewed changes

Merge branch 'main' into feat-remove-default-pkgs

666cb42

juliaElastic self-assigned this Dec 20, 2021

added new text with badge

4312eac

juliaElastic commented Dec 20, 2021

View reviewed changes

juliaElastic and others added 6 commits December 20, 2021 14:00

updated labels

5f7ee9a

added tabs

d80039a

fix checks

8ccfe99

ignore tests

8249dd4

fix i18n

5c28e7b

Merge branch 'main' into feat-remove-default-pkgs

d187ff9

juliaElastic changed the title ~~[Fleet] Draft before I go on holiday~~ [Fleet] Make default integration install explicit Dec 29, 2021

juliaElastic added 2 commits December 29, 2021 13:52

use max instead of sort

23de22c

Add agent and Fleet server policy

a453c20

juliaElastic marked this pull request as ready for review December 29, 2021 15:47

juliaElastic requested a review from a team as a code owner December 29, 2021 15:47

juliaElastic added release_note:enhancement v8.1.0 Team:Fleet Team label for Observability Data Collection Fleet team labels Dec 29, 2021

juliaElastic added 3 commits December 30, 2021 09:40

fix bug with monitoring checkbox

be86eb5

fix test

0eca9f6

skip test

b0d476f

juliaElastic commented Dec 30, 2021

View reviewed changes

juliaElastic added 3 commits December 30, 2021 15:23

fixed validation

fef2032

fixed fleet server policy install and missing enrollment keys

1264ffc

fixed test

3bafb80

kibanamachine and others added 2 commits January 27, 2022 07:19

Merge branch 'main' into feat-remove-default-pkgs

fd62227

fixed cypress test

68bd12a

nchaulet approved these changes Jan 27, 2022

View reviewed changes

Merge branch 'main' into feat-remove-default-pkgs

9d42aec

kibanamachine and others added 6 commits January 31, 2022 05:35

Merge branch 'main' into feat-remove-default-pkgs

ac8a0ff

fixed package policy error message condition

247c279

Merge branch 'main' into feat-remove-default-pkgs

809d192

fixed import

ce5de0c

fixed authz

511c236

fix cypress

a8ed804

juliaElastic merged commit b4b3285 into elastic:main Feb 1, 2022

kibanamachine added the backport:skip This commit does not require backporting label Feb 1, 2022

juliaElastic deleted the feat-remove-default-pkgs branch February 1, 2022 11:27

juliaElastic mentioned this pull request Feb 1, 2022

[Fleet] support setting id in new agent policy API #124255

Merged

1 task

afharo mentioned this pull request Feb 2, 2022

remove deps on data plugin #121955

Closed

patrykkopycinski mentioned this pull request Feb 2, 2022

[Osquery] Update cypress runner to latest Fleet changes #124398

Merged

kpollich mentioned this pull request Feb 7, 2022

[Fleet] [8.0.1] Fix navigation to blank screen when creating inline Agent Policy after clicking "Add Integration" on integration settings page #124835

Merged

joshdover mentioned this pull request Feb 10, 2022

[Fleet] Add UI tests for Integration Policy upgrades with conflicts #125236

Closed

3 tasks

juliaElastic mentioned this pull request Feb 11, 2022

[Fleet] fixed bug where fleet server instructions were incorrectly shown #125357

Merged

1 task

simitt mentioned this pull request Feb 14, 2022

Fix setup with Elastic Agent and Fleet elastic/apm-integration-testing#1440

Closed

juliaElastic mentioned this pull request Feb 15, 2022

[Fleet] [Enhancement] Confirmation message should be displayed for new policy creation while adding integration. #125644

Open

kpollich mentioned this pull request Feb 23, 2022

[Fleet]: Error page on clicking upgrade button for "Windows" integration with v0.8.1. #110202

Closed

juliaElastic mentioned this pull request Mar 29, 2022

[Fleet] Redesign agent flyout #128381

Merged

9 tasks

nchaulet mentioned this pull request May 24, 2022

[Fleet] Add agent flyout is broken for user with more than 1000 agents #132803

Closed

pebrc mentioned this pull request Jun 22, 2022

Should the operator create enrolment tokens instead of configuring Agents to talk to Kibana elastic/cloud-on-k8s#5779

Closed

nchaulet mentioned this pull request Jan 17, 2024

[Fleet] Ensure policies are not out of sync #175065

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Make default integration install explicit #121628

[Fleet] Make default integration install explicit #121628

juliaElastic commented Dec 20, 2021 •

edited by joshdover

Loading

juliaElastic Dec 20, 2021 •

edited

Loading

juliaElastic Dec 20, 2021 •

edited

Loading

joshdover Jan 4, 2022

juliaElastic Dec 20, 2021

elasticmachine commented Dec 29, 2021

juliaElastic Dec 30, 2021

mostlyjason Jan 4, 2022 •

edited

Loading

kevinlog Jan 26, 2022

juliaElastic Jan 26, 2022

joshdover Jan 26, 2022

juliaElastic Dec 30, 2021

joshdover Jan 4, 2022

mostlyjason Jan 4, 2022

juliaElastic Jan 5, 2022

joshdover Jan 5, 2022

juliaElastic commented Jan 27, 2022

nchaulet left a comment

juliaElastic commented Jan 31, 2022

juliaElastic commented Jan 31, 2022

kibana-ci commented Feb 1, 2022

API count

ESLint disabled line counts

Total ESLint disabled count

[Fleet] Make default integration install explicit #121628

[Fleet] Make default integration install explicit #121628

Conversation

juliaElastic commented Dec 20, 2021 • edited by joshdover Loading

Summary

Checklist

For maintainers

juliaElastic Dec 20, 2021 • edited Loading

Choose a reason for hiding this comment

juliaElastic Dec 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Dec 29, 2021

Choose a reason for hiding this comment

mostlyjason Jan 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliaElastic commented Jan 27, 2022

nchaulet left a comment

Choose a reason for hiding this comment

juliaElastic commented Jan 31, 2022

juliaElastic commented Jan 31, 2022

kibana-ci commented Feb 1, 2022

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

juliaElastic commented Dec 20, 2021 •

edited by joshdover

Loading

juliaElastic Dec 20, 2021 •

edited

Loading

juliaElastic Dec 20, 2021 •

edited

Loading

mostlyjason Jan 4, 2022 •

edited

Loading