Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Manager] Upgrade Agents in Fleet #78810

Merged
merged 24 commits into from
Oct 5, 2020

Conversation

neptunian
Copy link
Contributor

@neptunian neptunian commented Sep 29, 2020

Resolves #78469

Summary

Adds the ability to upgrade agents in the Fleet UI

Changes

  • Added POST /api/ingest_manager/fleet/agents/bulk_upgrade endpoint which expects payload:
{
  agents: [array of agent IDs | kuery string],
  version: string - version to update to,
  source_uri: string - optional download url 
}

Agent should not upgrade agents that are currently unenrolled or unenrolling.

  • Update GET api/ingest_manager/fleet/agents endpoint to accept showUpgradeable option. This will filter for agents that are less than this kibana version with local_metadata.elastic.agent.upgradeable: true

  • Agent Details page now reports release based on local_metadata.elastic.agents.snapshot being true/false. For older Agent versions that do not send this data, the value will be '-'. The snapshot version cannot be tested until we have a snapshot with this functionality.

Agent Details screenshot Screen Shot 2020-10-01 at 11 36 03 AM Screen Shot 2020-10-01 at 11 38 53 AM
  • UI for upgrading a single agent and bulk upgrading agents based on design spec . This design spec is based on new design and functionality for Fleet Status Improvements which is not in 7.10, so these changes do not incorporate that.
Agent Details for upgradeable agent Screen Shot 2020-10-01 at 1 41 18 PM
Agent Detail for non upgradeable agent Screen Shot 2020-10-01 at 1 43 50 PM
Agent List non upgradeable action menu Screen Shot 2020-10-01 at 1 39 21 PM
Agent List upgradeable action menu Screen Shot 2020-10-01 at 1 38 53 PM upgradeable action menu
Agent List bulk upgrade Screen Shot 2020-10-01 at 1 48 13 PM
Agent List filter upgradeable This will filter for agents that are less than this kibana version with `local_metadata.elastic.agent.upgradeable: true` Screen Shot 2020-10-01 at 7 32 53 PM
Agent List when upgrading Screen Shot 2020-10-02 at 1 05 43 PM After a few seconds, Agent should turn green and version should be updated. Upgrade Available indication should be gone.

Testing

To test the upgrade flow through Kibana UI (agent gets the download file and no source_uri is provided to the api), follow these steps:

  • Setup a local server that fetches the download
    • cd into beats/x-pack/elastic-agent
    • build a version of the agent which defaults to 8.0.0. run DEV=true PLATFORMS=darwin mage package
    • mkdir -p ~/server/beats/elastic-agent and copy the built files there cp build/distributions/*tar* ~/server/beats/elastic-agent
    • cd ~/server/beats/elastic-agent and start the server python3 -m http.server
  • Build a new version to run
    • cd back into beats/x-pack/elastic-agent
    • make a small commit such as adding a comment (so commit inside a package changes and it is a valid upgrade)
    • modify version to make it report as 7.9.0. vi ../../libbeat/version/version.go and change version to 7.9.0
    • run DEV=true PLATFORMS=darwin mage package again
    • cd into builds/distribution/elastic-agent and unpack the 7.9 *.tar.gz
    • cd into 7.9 directory and enroll an agent but do not run it (make sure this PR is running in kibana)
    • open elastic-agent.yml and uncomment and update sourceUri to where the server is running and saveScreen Shot 2020-10-02 at 12 49 49 PM
    • enroll the agent. ignore the CONFIG_CHANGE action errors in the log
    • Find your agent in Fleet and use the menu to upgrade it

To simulate testing with the source_uri, see testing steps in this PR:

elastic/beats#21002

@neptunian neptunian added release_note:enhancement v8.0.0 v7.10.0 Team:Fleet Team label for Observability Data Collection Fleet team labels Sep 29, 2020
@neptunian neptunian self-assigned this Sep 29, 2020
@michalpristas
Copy link

works ok with elastic/beats#21372
if source uri is not passed we use official store, otherwise we use what's coming.

what in case of error should agent ack the action? if it's not acked will it be resend indefinitely?

@michalpristas
Copy link

correctly detects snapshot, visualisation of non snapshot commented in the code

@@ -60,13 +60,16 @@ export class IngestManagerPlugin
implements
Plugin<IngestManagerSetup, IngestManagerStart, IngestManagerSetupDeps, IngestManagerStartDeps> {
private config: IngestManagerConfigType;
private kibanaVersion: string;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about changing this to env? It seems similar to what we have in config.

useKibanaVersion could be updated to read the EnvContext and other hooks/functions could get branch, or dev vs prod, etc if/when it comes up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me. I didn't put it in IngestManagerConfigType just because those are from the config file, so I think like you said something like "env" sounds good. I opened a follow up issue #79483

@michalpristas
Copy link

michalpristas commented Oct 2, 2020

x-pack/plugins/ingest_manager/server/types/models/agent.ts needs to be updated as well, for model validation
also PostAgentCheckinRequestBodyJSONSchema in x-pack/plugins/ingest_manager/server/types/rest_spec/agent.ts

@michalpristas
Copy link

michalpristas commented Oct 2, 2020

After suggested changes (i did them manually) i started agent without starting a file server so UPDATE fails i got this (as expected)

image

UPDATE is ACK-ed and Error there is reported, under ACK there is line saying STATE was switched to UPDATING

after i started server and rekicked UPDATE action

image

update was ACKed and state switched to RUNNING after it finished.

Status on list page was ONLINE then UPDATING and then back ONLINE. Even though there were errors, so i guess this will be an object of next iteration.

@neptunian neptunian marked this pull request as ready for review October 2, 2020 18:13
@neptunian neptunian requested a review from a team October 2, 2020 18:13
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

type: AGENT_SAVED_OBJECT_TYPE,
filter: _joinFilters(filters),
sortField,
sortOrder,
page,
perPage,
});
// filtering for a range on the version string will not work,
// nor does filtering on a flattened field (local_metadata), so filter here
if (showUpgradeable) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not ideal, but i decided to filter on the results of the soClient.find(). As we discussed earlier I was unable to filter through the flattened field local_metadata and even if I could I still need to use semver to compare local_metadata.elastic.agent.version which is a string so I could not do a range comparison in KQL. @jen-huang @nchaulet

Copy link
Member

@nchaulet nchaulet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀 tested with API calls (not a real agent but work as expected)

@michalpristas
Copy link

Tested ok today with elastic/beats#21461

@neptunian
Copy link
Contributor Author

@elasticmachine merge upstream

@neptunian neptunian merged commit 53f22dc into elastic:master Oct 5, 2020
neptunian added a commit to neptunian/kibana that referenced this pull request Oct 5, 2020
* add kibanaVersion context and hook, add upgrade available indications

* add agent upgrade modals and action buttons

* fix import

* add bulk actions api and remove source_uri as required

* add upgrading to AgentHealth status

* buildKueryForUpgradingAgents

* bulk actions UI

* remove source_uri

* add release type to agent details

* don't allow upgrade of unenrolled/unenrolling agent

* hide upgradeable button when not upgradeable

* fix test

* add udpating agent status

* remove upgrade available filter button for now

* update isUpgradeAvailable to use local_metadata upgradeable

* add UPDATING to agent event subtype

* use saved object for updating agent status

* add updating badge type label

* add upgrade available button and update agent list endpoint to accept showUpgradeable

* add schema and type for UPDATING

* fix type

* dont try to upgrade local_metadata

* exclude from AAD upgrade_started_at and upgraded_at

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
neptunian added a commit that referenced this pull request Oct 5, 2020
* add kibanaVersion context and hook, add upgrade available indications

* add agent upgrade modals and action buttons

* fix import

* add bulk actions api and remove source_uri as required

* add upgrading to AgentHealth status

* buildKueryForUpgradingAgents

* bulk actions UI

* remove source_uri

* add release type to agent details

* don't allow upgrade of unenrolled/unenrolling agent

* hide upgradeable button when not upgradeable

* fix test

* add udpating agent status

* remove upgrade available filter button for now

* update isUpgradeAvailable to use local_metadata upgradeable

* add UPDATING to agent event subtype

* use saved object for updating agent status

* add updating badge type label

* add upgrade available button and update agent list endpoint to accept showUpgradeable

* add schema and type for UPDATING

* fix type

* dont try to upgrade local_metadata

* exclude from AAD upgrade_started_at and upgraded_at

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@neptunian neptunian deleted the 68310-upgrade-agents branch October 6, 2020 14:22
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

@kbn/optimizer bundle module count

id before after diff
alerts - 26 +26
apm - 1276 +1276
beatsManagement - 302 +302
canvas - 1132 +1132
cloud - 6 +6
crossClusterReplication - 159 +159
dashboardEnhanced - 26 +26
dashboardMode - 4 +4
dataEnhanced - 17 +17
discoverEnhanced - 11 +11
embeddableEnhanced - 11 +11
enterpriseSearch - 250 +250
features - 11 +11
fileUpload - 21 +21
globalSearch - 23 +23
globalSearchBar - 13 +13
globalSearchProviders - 8 +8
graph - 169 +169
grokdebugger - 75 +75
indexLifecycleManagement - 206 +206
indexManagement - 509 +509
infra - 1147 +1147
ingestManager - 548 +548
ingestPipelines - 493 +493
lens - 563 +563
licenseManagement - 145 +145
licensing - 12 +12
lists - 128 +128
logstash - 190 +190
maps - 618 +618
mapsLegacyLicensing - 4 +4
ml - 1222 +1222
monitoring - 635 +635
observability - 96 +96
painlessLab - 40 +40
remoteClusters - 141 +141
reporting - 45 +45
rollup - 179 +179
searchprofiler - 87 +87
security - 458 +458
securitySolution - 1998 +1998
snapshotRestore - 168 +168
spaces - 237 +237
transform - 335 +335
triggers_actions_ui - 274 +274
uiActionsEnhanced - 134 +134
upgradeAssistant - 117 +117
uptime - 655 +655
urlDrilldown - 8 +8
watcher - 237 +237
total +15169

async chunks size

id before after diff
apm - 4.2MB ⚠️ +4.2MB
beatsManagement - 683.5KB ⚠️ +683.5KB
canvas - 1.5MB ⚠️ +1.5MB
crossClusterReplication - 456.5KB ⚠️ +456.5KB
enterpriseSearch - 428.6KB ⚠️ +428.6KB
fileUpload - 574.9KB ⚠️ +574.9KB
graph - 1.3MB ⚠️ +1.3MB
grokdebugger - 568.8KB ⚠️ +568.8KB
indexLifecycleManagement - 231.7KB ⚠️ +231.7KB
indexManagement - 1.6MB ⚠️ +1.6MB
infra - 3.8MB ⚠️ +3.8MB
ingestManager - 1.1MB ⚠️ +1.1MB
ingestPipelines - 812.9KB ⚠️ +812.9KB
lens - 1.0MB ⚠️ +1.0MB
licenseManagement - 167.5KB +167.5KB
logstash - 213.4KB ⚠️ +213.4KB
maps - 3.2MB ⚠️ +3.2MB
ml - 10.6MB ⚠️ +10.6MB
monitoring - 1.2MB ⚠️ +1.2MB
observability - 163.2KB +163.2KB
painlessLab - 38.7KB +38.7KB
remoteClusters - 227.8KB ⚠️ +227.8KB
rollup - 323.3KB ⚠️ +323.3KB
searchprofiler - 705.1KB ⚠️ +705.1KB
security - 1.0MB ⚠️ +1.0MB
securitySolution - 10.3MB ⚠️ +10.3MB
snapshotRestore - 622.6KB ⚠️ +622.6KB
spaces - 45.5KB +45.5KB
transform - 1.2MB ⚠️ +1.2MB
triggers_actions_ui - 1.5MB ⚠️ +1.5MB
upgradeAssistant - 179.8KB +179.8KB
uptime - 1.7MB ⚠️ +1.7MB
watcher - 1.0MB ⚠️ +1.0MB
total ⚠️ +52.5MB

distributable file count

id before after diff
default - 47088 +47088

miscellaneous assets size

id before after diff
apm - 49.0KB +49.0KB
enterpriseSearch - 834.9KB ⚠️ +834.9KB
maps - 978.6KB ⚠️ +978.6KB
securitySolution - 326.0KB ⚠️ +326.0KB
triggers_actions_ui - 27.7KB +27.7KB
total ⚠️ +2.2MB

page load bundle size

id before after diff
alerts - 89.8KB +89.8KB
apm - 44.3KB +44.3KB
beatsManagement - 169.1KB +169.1KB
canvas - 1.0MB ⚠️ +1.0MB
cloud - 5.9KB +5.9KB
crossClusterReplication - 49.2KB +49.2KB
dashboardEnhanced - 43.1KB +43.1KB
dashboardMode - 7.5KB +7.5KB
dataEnhanced - 34.3KB +34.3KB
discoverEnhanced - 29.2KB +29.2KB
embeddableEnhanced - 25.5KB +25.5KB
enterpriseSearch - 20.0KB +20.0KB
features - 15.8KB +15.8KB
fileUpload - 9.5KB +9.5KB
globalSearch - 27.9KB +27.9KB
globalSearchBar - 29.0KB +29.0KB
globalSearchProviders - 10.3KB +10.3KB
graph - 15.8KB +15.8KB
grokdebugger - 11.5KB +11.5KB
indexLifecycleManagement - 89.9KB +89.9KB
indexManagement - 122.8KB +122.8KB
infra - 178.6KB +178.6KB
ingestManager - 540.9KB ⚠️ +540.9KB
ingestPipelines - 42.0KB +42.0KB
lens - 76.0KB +76.0KB
licenseManagement - 26.2KB +26.2KB
licensing - 23.1KB +23.1KB
lists - 164.7KB +164.7KB
logstash - 37.6KB +37.6KB
maps - 164.4KB +164.4KB
mapsLegacyLicensing - 5.1KB +5.1KB
ml - 58.5KB +58.5KB
monitoring - 222.7KB ⚠️ +222.7KB
observability - 52.4KB +52.4KB
painlessLab - 160.9KB +160.9KB
remoteClusters - 35.5KB +35.5KB
reporting - 164.5KB +164.5KB
rollup - 80.3KB +80.3KB
searchprofiler - 50.9KB +50.9KB
security - 163.3KB +163.3KB
securitySolution - 587.1KB ⚠️ +587.1KB
snapshotRestore - 62.5KB +62.5KB
spaces - 364.2KB ⚠️ +364.2KB
transform - 25.4KB +25.4KB
triggers_actions_ui - 148.0KB +148.0KB
uiActionsEnhanced - 320.7KB ⚠️ +320.7KB
upgradeAssistant - 64.7KB +64.7KB
uptime - 25.2KB +25.2KB
urlDrilldown - 18.7KB +18.7KB
watcher - 27.9KB +27.9KB
total ⚠️ +5.6MB

Saved Objects .kibana field count

id before after diff
_data_stream_timestamp - 1 +1
action - 5 +5
action_task_params - 3 +3
alert - 30 +30
apm-indices - 8 +8
application_usage_daily - 2 +2
canvas-element - 8 +8
canvas-workpad - 5 +5
canvas-workpad-template - 8 +8
cases - 32 +32
cases-comments - 17 +17
cases-configure - 14 +14
cases-user-actions - 10 +10
config - 2 +2
dashboard - 17 +17
endpoint:user-artifact - 10 +10
endpoint:user-artifact-manifest - 5 +5
epm-packages - 14 +14
exception-list - 39 +39
exception-list-agnostic - 39 +39
file-upload-telemetry - 2 +2
fleet-agent-actions - 9 +9
fleet-agent-events - 11 +11
fleet-agents - 25 +25
fleet-enrollment-api-keys - 10 +10
graph-workspace - 9 +9
index-pattern - 3 +3
ingest_manager_settings - 6 +6
ingest-agent-policies - 11 +11
ingest-outputs - 10 +10
ingest-package-policies - 35 +35
kql-telemetry - 3 +3
lens - 7 +7
lens-ui-telemetry - 5 +5
map - 7 +7
ml-telemetry - 3 +3
monitoring-telemetry - 2 +2
namespace - 1 +1
namespaces - 1 +1
originId - 1 +1
query - 6 +6
references - 4 +4
sample-data-telemetry - 3 +3
search - 9 +9
siem-detection-engine-rule-actions - 8 +8
siem-detection-engine-rule-status - 12 +12
siem-ui-timeline - 90 +90
siem-ui-timeline-note - 8 +8
siem-ui-timeline-pinned-event - 7 +7
space - 9 +9
telemetry - 9 +9
timelion-sheet - 13 +13
tsvb-validation-telemetry - 2 +2
type - 1 +1
ui-metric - 2 +2
updated_at - 1 +1
upgrade-assistant-reindex-operation - 18 +18
upgrade-assistant-telemetry - 13 +13
url - 6 +6
visualization - 9 +9
total +660

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:enhancement Team:Fleet Team label for Observability Data Collection Fleet team v7.10.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Ingest Manager] Add ability to upgrade the agent in Fleet
6 participants