[8.x] [UII] Advanced agent monitoring options UI for HTTP endpoint an…

…d diagnostics (#193361) (#193658)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[UII] Advanced agent monitoring options UI for HTTP endpoint and
diagnostics (#193361)](#193361)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jen
Huang","email":"its.jenetic@gmail.com"},"sourceCommit":{"committedDate":"2024-09-22T10:49:33Z","message":"[UII]
Advanced agent monitoring options UI for HTTP endpoint and diagnostics
(#193361)\n\n## Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/153950.\r\n\r\nThis PR
implements a UI to configure advanced Elastic Agent
monitoring\r\noptions under agent policy settings. These advanced
options include\r\nenabling HTTP monitoring endpoints and various
options for agent\r\ndiagnostics. They are shown under an a toggle under
the existing agent\r\nmonitoring logs and metrics collection
options:\r\n\r\n<img width=\"1326\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/ac8cbe00-d838-4c9a-8a35-3dbf31222dc9\">\r\n\r\nIf
the base HTTP monitoring endpoint is not enabled, the rest of
the\r\nHTTP options are disabled:\r\n\r\n<img width=\"1328\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/2eac787c-3055-4862-b3eb-2566a39ee86c\">\r\n\r\nThe
following new fields are added to agent policy schema to
support\r\nthis:\r\n```\r\nmonitoring_http\r\nmonitoring_pprof_enabled\r\nmonitoring_diagnostics\r\n```\r\n\r\nThis
work supersedes the previous `HTTP monitoring endpoint` options\r\nunder
`Advanced Settings` at the bottom of the page. Any
previous\r\nconfiguration under an agent
policy's\r\n`advanced_settings.agent_monitoring_http` saved object field
are\r\nmigrated over to the new `monitoring_http` field and the old
field is\r\ndeleted. See the migration fn
`backfillAgentPolicyToV4`.\r\n\r\nThese new options are compiled to
agent yaml like this:\r\n\r\n```yml\r\nagent:\r\n monitoring:\r\n
enabled: true\r\n use_output: default\r\n logs: true\r\n metrics:
true\r\n traces: true\r\n namespace: default\r\n pprof:\r\n enabled:
true\r\n http:\r\n enabled: true\r\n host: localhost\r\n port: 6791\r\n
diagnostics:\r\n limit:\r\n interval: 1m\r\n burst: 1\r\n uploader:\r\n
max_retries: 10\r\n init_dur: 1s\r\n max_dur: 10m\r\n```\r\nSummarize
your PR. If it involves visual changes include a screenshot
or\r\ngif.\r\n\r\n### To-do\r\n- [x] API integration tests\r\n- [x] Full
manual test of SO migration\r\n- [x] Full manual test with agent using
these settings\r\n\r\n### Checklist\r\n\r\nDelete any items that are not
applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n -
elastic/ingest-docs#1333 \r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"87cdc2db728b088a44ff6e1977679f326bfd38d2","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Fleet","v9.0.0","release_note:feature","backport:prev-minor"],"title":"[UII]
Advanced agent monitoring options UI for HTTP endpoint and
diagnostics","number":193361,"url":"https://github.com/elastic/kibana/pull/193361","mergeCommit":{"message":"[UII]
Advanced agent monitoring options UI for HTTP endpoint and diagnostics
(#193361)\n\n## Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/153950.\r\n\r\nThis PR
implements a UI to configure advanced Elastic Agent
monitoring\r\noptions under agent policy settings. These advanced
options include\r\nenabling HTTP monitoring endpoints and various
options for agent\r\ndiagnostics. They are shown under an a toggle under
the existing agent\r\nmonitoring logs and metrics collection
options:\r\n\r\n<img width=\"1326\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/ac8cbe00-d838-4c9a-8a35-3dbf31222dc9\">\r\n\r\nIf
the base HTTP monitoring endpoint is not enabled, the rest of
the\r\nHTTP options are disabled:\r\n\r\n<img width=\"1328\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/2eac787c-3055-4862-b3eb-2566a39ee86c\">\r\n\r\nThe
following new fields are added to agent policy schema to
support\r\nthis:\r\n```\r\nmonitoring_http\r\nmonitoring_pprof_enabled\r\nmonitoring_diagnostics\r\n```\r\n\r\nThis
work supersedes the previous `HTTP monitoring endpoint` options\r\nunder
`Advanced Settings` at the bottom of the page. Any
previous\r\nconfiguration under an agent
policy's\r\n`advanced_settings.agent_monitoring_http` saved object field
are\r\nmigrated over to the new `monitoring_http` field and the old
field is\r\ndeleted. See the migration fn
`backfillAgentPolicyToV4`.\r\n\r\nThese new options are compiled to
agent yaml like this:\r\n\r\n```yml\r\nagent:\r\n monitoring:\r\n
enabled: true\r\n use_output: default\r\n logs: true\r\n metrics:
true\r\n traces: true\r\n namespace: default\r\n pprof:\r\n enabled:
true\r\n http:\r\n enabled: true\r\n host: localhost\r\n port: 6791\r\n
diagnostics:\r\n limit:\r\n interval: 1m\r\n burst: 1\r\n uploader:\r\n
max_retries: 10\r\n init_dur: 1s\r\n max_dur: 10m\r\n```\r\nSummarize
your PR. If it involves visual changes include a screenshot
or\r\ngif.\r\n\r\n### To-do\r\n- [x] API integration tests\r\n- [x] Full
manual test of SO migration\r\n- [x] Full manual test with agent using
these settings\r\n\r\n### Checklist\r\n\r\nDelete any items that are not
applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n -
elastic/ingest-docs#1333 \r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"87cdc2db728b088a44ff6e1977679f326bfd38d2"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193361","number":193361,"mergeCommit":{"message":"[UII]
Advanced agent monitoring options UI for HTTP endpoint and diagnostics
(#193361)\n\n## Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/153950.\r\n\r\nThis PR
implements a UI to configure advanced Elastic Agent
monitoring\r\noptions under agent policy settings. These advanced
options include\r\nenabling HTTP monitoring endpoints and various
options for agent\r\ndiagnostics. They are shown under an a toggle under
the existing agent\r\nmonitoring logs and metrics collection
options:\r\n\r\n<img width=\"1326\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/ac8cbe00-d838-4c9a-8a35-3dbf31222dc9\">\r\n\r\nIf
the base HTTP monitoring endpoint is not enabled, the rest of
the\r\nHTTP options are disabled:\r\n\r\n<img width=\"1328\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/2eac787c-3055-4862-b3eb-2566a39ee86c\">\r\n\r\nThe
following new fields are added to agent policy schema to
support\r\nthis:\r\n```\r\nmonitoring_http\r\nmonitoring_pprof_enabled\r\nmonitoring_diagnostics\r\n```\r\n\r\nThis
work supersedes the previous `HTTP monitoring endpoint` options\r\nunder
`Advanced Settings` at the bottom of the page. Any
previous\r\nconfiguration under an agent
policy's\r\n`advanced_settings.agent_monitoring_http` saved object field
are\r\nmigrated over to the new `monitoring_http` field and the old
field is\r\ndeleted. See the migration fn
`backfillAgentPolicyToV4`.\r\n\r\nThese new options are compiled to
agent yaml like this:\r\n\r\n```yml\r\nagent:\r\n monitoring:\r\n
enabled: true\r\n use_output: default\r\n logs: true\r\n metrics:
true\r\n traces: true\r\n namespace: default\r\n pprof:\r\n enabled:
true\r\n http:\r\n enabled: true\r\n host: localhost\r\n port: 6791\r\n
diagnostics:\r\n limit:\r\n interval: 1m\r\n burst: 1\r\n uploader:\r\n
max_retries: 10\r\n init_dur: 1s\r\n max_dur: 10m\r\n```\r\nSummarize
your PR. If it involves visual changes include a screenshot
or\r\ngif.\r\n\r\n### To-do\r\n- [x] API integration tests\r\n- [x] Full
manual test of SO migration\r\n- [x] Full manual test with agent using
these settings\r\n\r\n### Checklist\r\n\r\nDelete any items that are not
applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n -
elastic/ingest-docs#1333 \r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"87cdc2db728b088a44ff6e1977679f326bfd38d2"}}]}]
BACKPORT-->

---------

Co-authored-by: Jen Huang <its.jenetic@gmail.com>

oas_docs/output/kibana.serverless.staging.yaml

@@ -19761,16 +19761,48 @@ components:
             collection will be disabled
           nullable: true
           type: boolean
+        monitoring_diagnostics:
+          type: object
+          properties:
+            limit:
+              type: object
+              properties:
+                burst:
+                  type: number
+                interval:
+                  type: string
+            uploader:
+              type: object
+              properties:
+                init_dur:
+                  type: string
+                max_dur:
+                  type: string
+                max_retries:
+                  type: number
         monitoring_enabled:
           items:
             enum:
               - metrics
               - logs
             type: string
           type: array
+        monitoring_http:
+          type: object
+          properties:
+            enabled:
+              type: boolean
+            host:
+              type: string
+            port:
+              type: number
+          required:
+            - enabled
         monitoring_output_id:
           nullable: true
           type: string
+        monitoring_pprof_enabled:
+          type: boolean
         name:
           type: string
         namespace:
@@ -20242,6 +20274,63 @@ components:
           type: string
         inputs:
           type: string
+        monitoring:
+          type: object
+          properties:
+            diagnostics:
+              type: object
+              properties:
+                limit:
+                  type: object
+                  properties:
+                    burst:
+                      type: number
+                    interval:
+                      type: string
+                uploader:
+                  type: object
+                  properties:
+                    init_dur:
+                      type: string
+                    max_dur:
+                      type: string
+                    max_retries:
+                      type: number
+            enabled:
+              type: boolean
+            http:
+              type: object
+              properties:
+                enabled:
+                  type: boolean
+                host:
+                  type: string
+                port:
+                  type: number
+              required:
+                - enabled
+            logs:
+              type: boolean
+            metrics:
+              type: boolean
+            namespace:
+              type: string
+            pprof:
+              type: object
+              properties:
+                enabled:
+                  type: boolean
+              required:
+                - enabled
+            traces:
+              type: boolean
+            use_output:
+              type: string
+          required:
+            - enabled
+            - metrics
+            - logs
+            - traces
         output_permissions:
           additionalProperties:
             type: object

oas_docs/output/kibana.serverless.yaml

@@ -12634,16 +12634,48 @@ components:
             collection will be disabled
           nullable: true
           type: boolean
+        monitoring_diagnostics:
+          type: object
+          properties:
+            limit:
+              type: object
+              properties:
+                burst:
+                  type: number
+                interval:
+                  type: string
+            uploader:
+              type: object
+              properties:
+                init_dur:
+                  type: string
+                max_dur:
+                  type: string
+                max_retries:
+                  type: number
         monitoring_enabled:
           items:
             enum:
               - metrics
               - logs
             type: string
           type: array
+        monitoring_http:
+          type: object
+          properties:
+            enabled:
+              type: boolean
+            host:
+              type: string
+            port:
+              type: number
+          required:
+            - enabled
         monitoring_output_id:
           nullable: true
           type: string
+        monitoring_pprof_enabled:
+          type: boolean
         name:
           type: string
         namespace:
@@ -13115,6 +13147,63 @@ components:
           type: string
         inputs:
           type: string
+        monitoring:
+          type: object
+          properties:
+            diagnostics:
+              type: object
+              properties:
+                limit:
+                  type: object
+                  properties:
+                    burst:
+                      type: number
+                    interval:
+                      type: string
+                uploader:
+                  type: object
+                  properties:
+                    init_dur:
+                      type: string
+                    max_dur:
+                      type: string
+                    max_retries:
+                      type: number
+            enabled:
+              type: boolean
+            http:
+              type: object
+              properties:
+                enabled:
+                  type: boolean
+                host:
+                  type: string
+                port:
+                  type: number
+              required:
+                - enabled
+            logs:
+              type: boolean
+            metrics:
+              type: boolean
+            namespace:
+              type: string
+            pprof:
+              type: object
+              properties:
+                enabled:
+                  type: boolean
+              required:
+                - enabled
+            traces:
+              type: boolean
+            use_output:
+              type: string
+          required:
+            - enabled
+            - metrics
+            - logs
+            - traces
         output_permissions:
           additionalProperties:
             type: object

oas_docs/output/kibana.staging.yaml

@@ -27556,16 +27556,48 @@ components:
             collection will be disabled
           nullable: true
           type: boolean
+        monitoring_diagnostics:
+          type: object
+          properties:
+            limit:
+              type: object
+              properties:
+                burst:
+                  type: number
+                interval:
+                  type: string
+            uploader:
+              type: object
+              properties:
+                init_dur:
+                  type: string
+                max_dur:
+                  type: string
+                max_retries:
+                  type: number
         monitoring_enabled:
           items:
             enum:
               - metrics
               - logs
             type: string
           type: array
+        monitoring_http:
+          type: object
+          properties:
+            enabled:
+              type: boolean
+            host:
+              type: string
+            port:
+              type: number
+          required:
+            - enabled
         monitoring_output_id:
           nullable: true
           type: string
+        monitoring_pprof_enabled:
+          type: boolean
         name:
           type: string
         namespace:
@@ -28037,6 +28069,63 @@ components:
           type: string
         inputs:
           type: string
+        monitoring:
+          type: object
+          properties:
+            diagnostics:
+              type: object
+              properties:
+                limit:
+                  type: object
+                  properties:
+                    burst:
+                      type: number
+                    interval:
+                      type: string
+                uploader:
+                  type: object
+                  properties:
+                    init_dur:
+                      type: string
+                    max_dur:
+                      type: string
+                    max_retries:
+                      type: number
+            enabled:
+              type: boolean
+            http:
+              type: object
+              properties:
+                enabled:
+                  type: boolean
+                host:
+                  type: string
+                port:
+                  type: number
+              required:
+                - enabled
+            logs:
+              type: boolean
+            metrics:
+              type: boolean
+            namespace:
+              type: string
+            pprof:
+              type: object
+              properties:
+                enabled:
+                  type: boolean
+              required:
+                - enabled
+            traces:
+              type: boolean
+            use_output:
+              type: string
+          required:
+            - enabled
+            - metrics
+            - logs
+            - traces
         output_permissions:
           additionalProperties:
             type: object