[Ingest Management] Push log level downstream

[michalpristas]

What does this PR do?

This PR injects log level settings into fleet section of config as discussed in linked issue.
resulting config will look like this

fleet:
  agent:
    id: {agent_id}
    logging:
      level: warning
  api:
    access_api_key: {agent_api_key}
    kibana:
      host: {host_uri}:443
      hosts:
      - {host_uri}
      protocol: https
      timeout: 5m0s
  host:
    id: {host_id}
inputs:
- artifact_manifest:
    artifacts:
      endpoint-exceptionlist-macos-v1:
...so on

Why is it important?

#20756

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

cc @ph @ferullo

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #22815 updated

• Start Time: 2020-12-08T11:27:09.059+0000

• Duration: 30 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1396
Skipped	4
Total	1400

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	1396
Skipped	4
Total	1400

[blakerouse]

Looks good.

[ph]

Should we have tests for this?

[ph]

@EricDavisX @rahulgupta-qasource This should be available soon see linked issues for endpoint support.

[dikshachauhan-qasource]

Hi @EricDavisX

We have created 03 testcases for this ticket and linked ticket #20756 . testcases links are as follows:
https://elastic.testrail.io/index.php?/cases/edit/35174
https://elastic.testrail.io/index.php?/cases/view/35175
https://elastic.testrail.io/index.php?/cases/view/35176

However, we have 01 query at information provided on point below at #8105
Move all of the existing Endpoint logging config from policy.<os>.logging to policy.<os>.advanced.logging. Keep in mind updating Binaries/Endpoint/production_config.yaml

Could you please confirm where we could locate this file and verify the same.

[EricDavisX]

@dikshachauhan-qasource I believe... you shouldn't see a difference there, it was an internal refactor to make the logging come out right. If the Endpoint logs show up, and have policy items displayed - then it is working.

[dikshachauhan-qasource]

Hi @EricDavisX

We have validated this ticket on 8.0.0 snapshot cloud environment as per code commit in elastic/kibana@62e79ee and found that code for same is not merged. Below are the build details:

BUILD 38666
COMMIT 62e79eec7011cb8ffa6f1c9626f4f989656ddad1
Artifact link: https://artifacts-api.elastic.co/v1/search/8.0.0-SNAPSHOT/

Steps to reproduce:

• Installed agent using default policy
• Verified agent logs were generated successfully and logging level is 'info' by default on agent details page
• Navigated to 'Default policy'. Go to Actions>View policy

Observation: logging level is not displayed under fleet section of default policy. Screenshot:

[EricDavisX]

@dikshachauhan-qasource Hi. I think the commit yesterday morning missed the Tuesday build, and the Wednesday build isn't 'done' yet by the time you were testing. So, let us re-test tomorrow. I wish we had a better mechanism for tracking check-ins to when they are available in SNAPSHOT builds. I don't know of one better than tracing in git and correlating against the Jenkins jobs for the build for the branch in question, tho we can always do that.

The good news is that for many tests, we can use the Observability generated artifacts. I think @rahulgupta-qasource knows of those, so please follow up with him. I will confirm in our sync tomorrow too.

Final comment... I don't expect the Agent policy that the Agent is assigned to would show the log-level change. I'll ask @michalpristas to help confirm, please. I think we should lay out some AC for the expectation here in terms of precedence:

• what happens when the Agent log level is set manually with this new feature, and then the policy is re-applied... does the level get re-set by the policy application? Or is the level not part of policy and so it is left alone?

[dikshachauhan-qasource]

Hi @EricDavisX

We have updated our testcases as per discussion on the expected behaviour of this pr.

Testcases link is as follows:

• C35176
• C35174

Also, created 02 testcases. Link is as follows:

• C35264
• C35265

Please let us know if anything else is required to be covered.

[dikshachauhan-qasource]

Hi @EricDavisX

We have executed 05 testcases under test plan Push log level downstream Test Run

Build details are as follows:

BUILD 37399
COMMIT f3abc08ac648f8b302733c5c22a39048314a027c
Artifact link: https://staging.elastic.co/7.11.0-710164a0/downloads/beats/elastic-agent/elastic-agent-7.11.0-windows-x86_64.zip

Please let us know if anything is required from our side.

Thanks
QAS