trying to merge the field stats of field but the field type is incompatible #18135

nellicus · 2016-05-04T14:01:13Z

Elasticsearch version:
5.0.0-alpha2

JVM version:
java version "1.8.0_45"
OS version:
Linux
Description of the problem including expected versus actual behavior:
Error when calling _field_stats API on Logstash 5.0.0-alpha2 generated index

abonuccelli@w530 /opt/elk/PROD/scripts $ curl -XGET "https://192.168.1.105:9200/logstash-syslog-2016.05.04/_field_stats?fields=@timestamp&level=indices&pretty" -k --cacert /opt/elk/PROD/FS/secure/cacert.pem  -u elastic:xxxxxx
{
  "error" : {
    "root_cause" : [ {
      "type" : "illegal_state_exception",
      "reason" : "trying to merge the field stats of field [@timestamp] from index [logstash-syslog-2016.05.04] but the field type is incompatible, try to set the 'level' option to 'indices'"
    } ],
    "type" : "illegal_state_exception",
    "reason" : "trying to merge the field stats of field [@timestamp] from index [logstash-syslog-2016.05.04] but the field type is incompatible, try to set the 'level' option to 'indices'"
  },
  "status" : 500
}

calling the same on .monitoring-es-* index timestamp field works ok

abonuccelli@w530 /opt/elk/PROD/scripts $  curl -XGET "https://192.168.1.105:9200/.monitoring-es-2-2016.05.04/_field_stats?fields=timestamp&level=indices&pretty" -k --cacert /opt/elk/PROD/FS/secure/cacert.pem  -u elastic:xxxxxx
{
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "indices" : {
    ".monitoring-es-2-2016.05.04" : {
      "fields" : {
        "timestamp" : {
          "max_doc" : 21360,
          "doc_count" : 21360,
          "density" : 100,
          "sum_doc_freq" : -1,
          "sum_total_term_freq" : 21360,
          "min_value" : 1462353531772,
          "min_value_as_string" : "2016-05-04T09:18:51.772Z",
          "max_value" : 1462370153801,
          "max_value_as_string" : "2016-05-04T13:55:53.801Z"
        }
      }
    }
  }

}

Provide logs (if relevant):

[2016-05-04 15:50:22,052][WARN ][rest.suppressed          ] /logstash-syslog-2016.05.04/_field_stats Params: {pretty=, level=indices, index=logstash-syslog-2016.05.04, fields=@timestamp}
java.lang.IllegalStateException: trying to merge the field stats of field [@timestamp] from index [logstash-syslog-2016.05.04] but the field type is incompatible, try to set the 'level' option to 'indices'
    at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.newResponse(TransportFieldStatsTransportAction.java:108)
    at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.newResponse(TransportFieldStatsTransportAction.java:58)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction.finishHim(TransportBroadcastAction.java:248)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction.onOperation(TransportBroadcastAction.java:213)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction$1.handleResponse(TransportBroadcastAction.java:193)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction$1.handleResponse(TransportBroadcastAction.java:180)
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:789)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:178)
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:143)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

**other info

@timestamp mapping for problematic logstash-syslog-* index

logstash-syslog-* 

 "timestamp" : {
            "type" : "date",
           }

timestamp mapping for .monitoring-es-* index

 "timestamp" : {
            "type" : "date",
            "format" : "date_time"
          }

The text was updated successfully, but these errors were encountered:

clintongormley · 2016-05-04T16:15:36Z

Trying to work up a recreation

clintongormley · 2016-05-04T17:05:41Z

OK - simple recreation. Start a cluster with two nodes, then run the following:

DELETE t

POST t/t/_bulk
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}
{"index":{}}
{"@timestamp":"2000-01-01T00:00:00.0+00"}

GET t/_field_stats?level=indices&fields=@timestamp

The above returns:

{
   "error": {
      "root_cause": [
         {
            "type": "illegal_state_exception",
            "reason": "trying to merge the field stats of field [@timestamp] from index [t] but the field type is incompatible, try to set the 'level' option to 'indices'"
         }
      ],
      "type": "illegal_state_exception",
      "reason": "trying to merge the field stats of field [@timestamp] from index [t] but the field type is incompatible, try to set the 'level' option to 'indices'"
   },
   "status": 500
}

With the following stack trace:

[2016-05-04 19:04:43,705][WARN ][rest.suppressed          ] /t/_field_stats Params: {level=indices, index=t, fields=@timestamp}
java.lang.IllegalStateException: trying to merge the field stats of field [@timestamp] from index [t] but the field type is incompatible, try to set the 'level' option to 'indices'
  at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.newResponse(TransportFieldStatsTransportAction.java:108)
  at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.newResponse(TransportFieldStatsTransportAction.java:58)
  at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction.finishHim(TransportBroadcastAction.java:248)
  at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction.onOperation(TransportBroadcastAction.java:213)
  at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction$1.handleResponse(TransportBroadcastAction.java:193)
  at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction$1.handleResponse(TransportBroadcastAction.java:180)
  at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:789)
  at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:178)
  at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:143)
  at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

jimczi · 2016-05-04T21:28:35Z

This is a serialization bug in the field stats transport that was introduced when the new point api was added. It occurs only with 5.0.0-alpha2 and has been fixed in master with this commit f600c4a

nellicus · 2016-05-11T15:36:32Z

@jimferenczi this effectively prevents, at least in my environment, any use in kibana of the data coming from logstash. just raising a concern that this might impede proper testing of alpha2 through our user base

clintongormley · 2016-05-12T10:01:59Z

@nellicus yeah - not much we can do about it until the next release

anhlqn · 2016-05-13T14:33:55Z

This also occurred when I tried to use the winlogbeat indexes on a fresh installation without Logstash

gluckspilz · 2016-05-27T00:33:37Z

Will adding "format" : "date_time" to the mapping for the index created by logstash stop this error?

michalterbert · 2016-05-30T19:30:57Z

@gluckspilz: for me doesn't work 👎

clintongormley · 2016-06-01T16:28:37Z

This is fixed in 5.0.0-alpha3, which is out already

nellicus added the discuss label May 4, 2016

clintongormley closed this as completed May 4, 2016

clintongormley reopened this May 4, 2016

clintongormley added the :Data Management/Stats Statistics tracking and retrieval APIs label May 4, 2016

clintongormley assigned jimczi May 4, 2016

clintongormley added >bug and removed discuss labels May 4, 2016

nellicus mentioned this issue May 4, 2016

IllegalStateException: trying to merge the field stats of field but the field type is incompatible elastic/kibana#7127

Closed

jimczi closed this as completed May 4, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

trying to merge the field stats of field but the field type is incompatible #18135

trying to merge the field stats of field but the field type is incompatible #18135

nellicus commented May 4, 2016 •

edited

Loading

clintongormley commented May 4, 2016

clintongormley commented May 4, 2016

jimczi commented May 4, 2016

nellicus commented May 11, 2016

clintongormley commented May 12, 2016

anhlqn commented May 13, 2016

gluckspilz commented May 27, 2016 •

edited

Loading

michalterbert commented May 30, 2016

clintongormley commented Jun 1, 2016

trying to merge the field stats of field but the field type is incompatible #18135

trying to merge the field stats of field but the field type is incompatible #18135

Comments

nellicus commented May 4, 2016 • edited Loading

clintongormley commented May 4, 2016

clintongormley commented May 4, 2016

jimczi commented May 4, 2016

nellicus commented May 11, 2016

clintongormley commented May 12, 2016

anhlqn commented May 13, 2016

gluckspilz commented May 27, 2016 • edited Loading

michalterbert commented May 30, 2016

clintongormley commented Jun 1, 2016

nellicus commented May 4, 2016 •

edited

Loading

gluckspilz commented May 27, 2016 •

edited

Loading