Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ingest Manager] Saved Object client issues after ES restart #75497

Closed
jfsiii opened this issue Aug 19, 2020 · 11 comments
Closed

[Ingest Manager] Saved Object client issues after ES restart #75497

jfsiii opened this issue Aug 19, 2020 · 11 comments
Assignees
@jfsiii
Labels
Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@jfsiii
Copy link
Contributor

jfsiii commented Aug 19, 2020 •
edited
Loading

More info coming. This is a WIP to get started

Kibana version:
8.x

Elasticsearch version:
8.0 snapshot

Steps to reproduce:

  1. start kibana: yarn start --no-base-path
  2. start ES: yarn es snapshot --license=trial -E xpack.security.authc.api_key.enabled=true
  3. no ingest-outputs or epm-packages saved objects at first 
    > curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=ingest-outputs" | jq
{
  "page": 1,
  "per_page": 20,
  "total": 0,
  "saved_objects": []
}

> curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages" | jq
{
  "page": 1,
  "per_page": 20,
  "total": 0,
  "saved_objects": []
}
  4. call setup 
    curl -X POST 'http://localhost:5601/api/ingest_manager/setup' -H 'kbn-xsrf: <string>' --user elastic:changeme
{"isInitialized":true}
  5. see expected saved objects 
    curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages"
// two objects, the default
  6. restart ES. Kibana disconnects, then eventually re-establishes with :9200
  7. run same commands, see no saved objects 
    > curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages"
{"page":1,"per_page":20,"total":0,"saved_objects":[]}
> curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=ingest-outputs" | jq
{
  "page": 1,
  "per_page": 20,
  "total": 0,
  "saved_objects": []
}
> curl -X POST 'http://localhost:5601/api/ingest_manager/setup' -H 'kbn-xsrf: <string>' --user elastic:changeme
{"isInitialized":true}
> curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=ingest-outputs" | jq
{
  "page": 1,
  "per_page": 20,
  "total": 0,
  "saved_objects": []
}
> curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages"
{"page":1,"per_page":20,"total":0,"saved_objects":[]}
  8. restart Kibana, ES stays up
  9. previously created (but not shown) objects are now available 
    > curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=ingest-outputs" | jq
{
  "page": 1,
  "per_page": 20,
  "total": 1,
  "saved_objects": [
    {
      "type": "ingest-outputs",
      "id": "8794b999-cb1a-4e58-b551-831ac164a42d",
      "attributes": {
        "name": "default",
        "is_default": true,
        "type": "elasticsearch",
        "hosts": [
          "http://localhost:9200"
        ]
      },
      "references": [],
      "updated_at": "2020-08-19T20:45:29.309Z",
      "version": "WzE5LDFd",
      "score": 0
    }
  ]
}
> curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages"
{"page":1,"per_page":20,"total":2,"saved_objects":[ ... ]}

Describe the feature:

After ES restarts, our SO client isn't able to read previously created saved objects. After Kibana restarts, all the expected objects are available.

Video showing flow like described above
master-fleet-setup-forceRecreate-with-ES-restart-errors-no-default-output-2048-half.m4v.zip

It affects at least two /api/ingest_manager routes, but didn't seem to affect the SO API (curl --user elastic:changeme -X PUT "localhost:9200/_ingest/pipeline/my-pipeline-id").

I tried adding a refresh: 'wait_for' and refresh: true to the soClient.create calls but it didn't have any effect
@jfsiii jfsiii self-assigned this Aug 19, 2020
@jfsiii jfsiii added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 19, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@jen-huang
Copy link
Contributor

Thanks for filing this issue! I know that we had seen this behavior before but didn't log it. One thing I'm curious about, is if this behavior only occurs for our SO types, or does it apply to other Kibana SOs as well?

This was referenced Aug 24, 2020
Closed
Closed
@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 24, 2020

@elastic/kibana-platform any thoughts about how I can debug / fix?

Here's the pattern we use to get the Saved Object client from a route handler and pass it to a service

kibana/x-pack/plugins/ingest_manager/server/routes/setup/handlers.ts

Lines 79 to 80 in cefb62c

const soClient = context.core.savedObjects.client;
const callCluster = context.core.elasticsearch.legacy.client.callAsCurrentUser;

Are we "holding it wrong"? It feels like a flush issue which is what had me trying the refresh values, but if it was about the way we used the ES client, I'd expect it to have the same "write but not read" issue the first time it's run. This behavior only seems to appear after an ES restart.

Let me know if I can do anything more to help diagnose.

@ruflin ruflin mentioned this issue Aug 25, 2020
Closed
@ph
Copy link
Contributor

ph commented Aug 25, 2020

@nchaulet @skh @neptunian for awareness.

@skh
Copy link
Contributor

skh commented Aug 26, 2020 •
edited
Loading

Is wiping all ES data, including .kibana indices, without restarting KIbana a supported scenario? I could imagine that this affects all saved objects (as @jen-huang already mentions), and maybe even more areas in the platform.

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 26, 2020 •
edited
Loading

@skh the failures / issues we're seeing aren't because .kibana or other data was removed. They're because we cannot persist changes after the restart. And that seems related to whatever is going on with the client.

/setup would happily re-create a default output, install default packages, etc after a restart/wipe. The issue is that when it tries, ES says it successfully created an output, even returning the new object, but then it's not available for another part of the system that expects it. Some of the system is resilient enough to create something if it's missing but, again, those updates are "succeeding" but not updating ES (or not available to be read via the ES client).

Updates not failing, but not ... updating is a challenge.

@ruflin
Copy link
Member

ruflin commented Aug 26, 2020

The queries you run above are based on the SO API. I wonder if you would run the query directly on the .kibana index and search for the documents if they are there or not.

@skh
Copy link
Contributor

skh commented Aug 26, 2020 •
edited
Loading

the failures / issues we're seeing aren't because .kibana or other data was removed.

My guess is that Kibana does some housekeeping on startup, including setting up its own indices and the saved object client, and makes the assumption that its indices stay where they are even if the connection to ES is intermittently lost.

Restarting yarn es snapshot is a clean wipe, so (I'm still guessing) the still-running Kibana makes assumptions about indices that are now gone. When Kibana is restarted, it runs through its own startup code again, and finds (or creates?) the new indices.

If I'm correct, I could well imagine that this is a scenario that is not supported, either intentionally or by oversight, hence my question.

@elastic/kibana-platform is of course better qualified to answer that than I am ;-)

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 26, 2020

@skh TL;DR: "different indexes" didn't occur to me & is looking more likely

server    log   [10:26:49.931] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
server    log   [10:26:49.932] [info][savedobjects-service] Starting saved objects migrations
server    log   [10:26:49.954] [info][savedobjects-service] Creating index .kibana_task_manager_1.
server    log   [10:26:49.959] [info][savedobjects-service] Creating index .kibana_1.
server    log   [10:26:50.523] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
server    log   [10:26:50.573] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.

You reminded me that I thought I saw logs from ES during the failing operations.

Here's what's logged in ES during an initial or successful call to curl --user elastic:changeme -X POST 'http://localhost:5601/api/ingest_manager/setup' --header 'kbn-xsrf: <string>'

ES logs from POST /setup success (before restart) 
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata_current-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.network] for index patterns [logs-endpoint.events.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.registry] for index patterns [logs-endpoint.events.registry-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata_current] for index patterns [metrics-endpoint.metadata_current-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metrics] for index patterns [metrics-endpoint.metrics-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.alerts] for index patterns [logs-endpoint.alerts-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.library] for index patterns [logs-endpoint.events.library-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.file] for index patterns [logs-endpoint.events.file-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.process] for index patterns [logs-endpoint.events.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata] for index patterns [metrics-endpoint.metadata-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.policy] for index patterns [metrics-endpoint.policy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.security] for index patterns [logs-endpoint.events.security-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.core] for index patterns [metrics-system.core-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.entropy] for index patterns [metrics-system.entropy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.filesystem] for index patterns [metrics-system.filesystem-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.auth] for index patterns [logs-system.auth-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.diskio] for index patterns [metrics-system.diskio-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.cpu] for index patterns [metrics-system.cpu-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.load] for index patterns [metrics-system.load-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.fsstat] for index patterns [metrics-system.fsstat-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network] for index patterns [metrics-system.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.memory] for index patterns [metrics-system.memory-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network_summary] for index patterns [metrics-system.network_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process_summary] for index patterns [metrics-system.process_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process] for index patterns [metrics-system.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.raid] for index patterns [metrics-system.raid-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket] for index patterns [metrics-system.socket-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.service] for index patterns [metrics-system.service-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket_summary] for index patterns [metrics-system.socket_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.syslog] for index patterns [logs-system.syslog-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.uptime] for index patterns [metrics-system.uptime-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.users] for index patterns [metrics-system.users-*]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
ES logs from POST /setup after restart 
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata_current-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metrics] for index patterns [metrics-endpoint.metrics-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.file] for index patterns [logs-endpoint.events.file-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.security] for index patterns [logs-endpoint.events.security-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata] for index patterns [metrics-endpoint.metadata-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.alerts] for index patterns [logs-endpoint.alerts-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.library] for index patterns [logs-endpoint.events.library-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.network] for index patterns [logs-endpoint.events.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata_current] for index patterns [metrics-endpoint.metadata_current-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.process] for index patterns [logs-endpoint.events.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.policy] for index patterns [metrics-endpoint.policy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.registry] for index patterns [logs-endpoint.events.registry-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.fsstat] for index patterns [metrics-system.fsstat-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.filesystem] for index patterns [metrics-system.filesystem-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.core] for index patterns [metrics-system.core-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.entropy] for index patterns [metrics-system.entropy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.diskio] for index patterns [metrics-system.diskio-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.cpu] for index patterns [metrics-system.cpu-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.auth] for index patterns [logs-system.auth-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.memory] for index patterns [metrics-system.memory-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.load] for index patterns [metrics-system.load-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network] for index patterns [metrics-system.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network_summary] for index patterns [metrics-system.network_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process] for index patterns [metrics-system.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process_summary] for index patterns [metrics-system.process_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.raid] for index patterns [metrics-system.raid-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.service] for index patterns [metrics-system.service-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket] for index patterns [metrics-system.socket-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket_summary] for index patterns [metrics-system.socket_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.syslog] for index patterns [logs-system.syslog-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.uptime] for index patterns [metrics-system.uptime-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.users] for index patterns [metrics-system.users-*]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]

I noticed there are several lines like this with .kibana_1 in the initial logs

   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]

but .kibana after restart

   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]

The "different indexes" hypothesis gets stronger 😃

I captured the logs from Kibana & ES during inital boot & restart
es_initial.log.txt
es_restart.log.txt
kibana_while_es_restarts.log.txt
kibana_initial.log.txt

Looking more closely at the Kibana boot logs for .kibana, I saw

server    log   [10:26:49.931] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
server    log   [10:26:49.932] [info][savedobjects-service] Starting saved objects migrations
server    log   [10:26:49.954] [info][savedobjects-service] Creating index .kibana_task_manager_1.
server    log   [10:26:49.959] [info][savedobjects-service] Creating index .kibana_1.
server    log   [10:26:50.523] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
server    log   [10:26:50.573] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.

That seems relevant 😃

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 26, 2020 •
edited
Loading

Things appear to work as expected, if I start & restart ES with the local binary (like .es/8.0.0/bin/elasticsearch created by yarn es snapshot)

I think I misunderstood why there was no data after restarting with yarn es snapshot

I think I'll close this but, either way, thanks for helping me work through this @skh

@skh
Copy link
Contributor

skh commented Aug 26, 2020

yarn es snapshot, in case it is relevant, also takes an optional parameter to specify a data directory, which will survive restarts if it's outside the snapshot folder:

yarn es snapshot -E path.data=../data

(from https://www.elastic.co/guide/en/kibana/master/running-elasticsearch.html )

@jfsiii jfsiii closed this as completed Sep 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jfsiii jfsiii

Labels
Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ph @jfsiii @skh @ruflin @jen-huang @elasticmachine