ElasticSearch index can be created with incorrect mapping

[mabn]

When collector was running I've changed the ES cluster. As a result an index was created with incorrect mapping -

{
    "jaeger-span-2017-09-03": {
        "mappings": {
            "span": {
                "properties": {

                    (removed for brevity)

                    "traceID": {
                        "fields": {
                            "keyword": {
                                "ignore_above": 256,
                                "type": "keyword"
                            }
                        },
                        "type": "text"
                    }
                }
            }
        }
    }
}

I suspect that setting dynamic=false on the top-level tocument or action.auto_create_index: false would solve the problem.

[tanner-bruce]

This would be due to elasticsearch trying to figure out the mappings on its own, since we aren't checking if the index is their post startup. Since we can't aggregate on text fields without enabling fielddata this would break the query searches.

@yurishkuro In my opinion the fix here is to check and set an index template on collector startup for the jaeger-span and jaeger-service indices. Then we don't have to worry about this happening since the template handles all the mappings automatically.

As an aside there likely should also be support for setting jaeger to write to a predefined index name to allow users to write to an index alias, allowing users to use something like Curator to do automated roll-overs in case they log too many spans and kill their cluster due to overly large indices.

[dmitrygusev]

This happened to us yesterday, although we're just testing: I've removed current-date indices (for cleanup purposes) while collectors were still running.

It looks like Jager didn't notice this and just continued posting the spans as before, Elasticsearch didn't fail the requests and created new indices for incoming requests on-the-fly on its own with guessed mappings. The mappings were obviously incorrect for the jaeger-query.

I couldn't find a way how to fix this as I couldn't afford stopping all the collectors running individually on a set of microservices on different machines. It has fixed itself on date change when it was time to create new indexes with new date in their name, new indexes were fine and jaeger-query resumed working.

Obviously all data collected on previous date were not consumable and I had to remove the index. Although I now understand that I could reindex them into new index somehow if I really needed it, the problem was is I didn't have a schema for mappings anywhere. Having the above mentioned templates in place would help here.

It'd also help if Jaeger used index aliases instead of real index names on write, i.e. jaeger-span-DATE could become an alias for a real index, i.e. jaeger-span-DATE-real. In that case I could reindex current bad index into new index with correct mapping and added the same alias to it to be picked by the jaeger-query with zero downtime. After that old index could be dropped. Just an idea though.

[mikelduke]

I've encountered the same problem. In my case, it looked like it was due to a network timeout when Jaeger collector was creating the new indexes for the day.

To fix this, I created my own index templates by exporting a working set of index mappings and then reindexing to it.

It would be great if Jaeger came with the current index mappings in a json file for use as a template instead of being created in code. By extracting and maintaining the template and index mappings separately, there is a risk that a change will be missed after a version upgrade.

[pavolloffay]

Jaeger collectors handles correctly if the index is already created.

I agree about the benefits of keeping the mapping files separately, on the other hand, it will complicate deployment.

[pavolloffay]

Now the index templates are versioned in a separate files https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/mappings.

@mabn @mikelduke @dmitrygusev @tanner-bruce could you please provide instructions on how I could reproduce the issue?

What would be an ideal solution to this? At the moment jaeger creates an index (if it's not in cache). Maybe we could upload an index template on jaeger start.

[mabn]

This was happening when:

• I've manually deleted the index while jaeger was working
• I've switched to a new ES cluster without restarting jaeger (by changing load balancer config in front of ES)
  I think that the templates cover both of these cases.

[tanner-bruce]

Simply deleting the index while jaeger is running would reproduce the issue back when I commented. I am not sure if this is still the case as I am not currently using jaeger.

…

On Mon, Jun 24, 2019, 4:15 AM Marcin Biegan, ***@***.***> wrote: This was happening when: - I've manually deleted the index while jaeger was working - I've switched to a new ES cluster without restarting jaeger (by changing load balancer config in front of ES) I think that the templates cover both of these cases. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#374?email_source=notifications&email_token=AFC3JFQHW27X6FQGQUU4M3TP4CGDZA5CNFSM4DZNJVQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYMJRDY#issuecomment-504928399>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFC3JFVLPUJQLN4S4XGCW33P4CGDZANCNFSM4DZNJVQQ> .