JVM is crashing with SIGSEGV

[rvller]

I'm running an Spring application on the Tomcat which is connecting to ES 1.2.1. Tomcat JVM is crashing from time to time with the following error.

[rvller]

SEE THE FULL ERROR FILE - http://pastebin.com/i7hQ27P2

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fe5d29c0fa6, pid=1472, tid=140624657848064

JRE version: Java(TM) SE Runtime Environment (7.0_72-b14) (build 1.7.0_72-b14)
Java VM: Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode linux-amd64 compressed oops)
Problematic frame:
V [libjvm.so+0x3bdfa6] CardTableModRefBS::dirty_MemRegion(MemRegion)+0xb6

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

If you would like to submit a bug report, please visit:
http://bugreport.sun.com/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x00007fe54c8c9800): JavaThread "elasticsearch[Alexander Goodwin Pierce][transport_client_worker][T#7]{New I/O worker #7}" daemon [_thread_in_Java, id=2497, stack(0x00007fe5bab71000,0x00007fe5bac72000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x00007fe5cd898000

Registers:
RAX=0x0000000000000000, RBX=0x00007fe54c8c9800, RCX=0x00007fe5cd898000, RDX=0x0000000000000001
RSP=0x00007fe5bac6fbf0, RBP=0x00007fe5bac6fbf0, RSI=0x00007fe5cd898000, RDI=0x0008000000000000
R8 =0x0080000000000000, R9 =0x0080000000000000, R10=0x00007fe5cd898000, R11=0x00807fe5cd898000
R12=0x0000000763c30180, R13=0x00007fe54c8c9800, R14=0x00007fe5bac6fcc8, R15=0x00007fe54c8c9800
RIP=0x00007fe5d29c0fa6, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fe5bac6fbf0)
0x00007fe5bac6fbf0: 00007fe5bac6fc30 00007fe5d2a28e91
0x00007fe5bac6fc00: 00007fe5bac6fc40 00007fe5d2bc7066
0x00007fe5bac6fc10: 0000000000000000 4000000000000000
0x00007fe5bac6fc20: 0000000763c30180 00007fe5cc016e20
0x00007fe5bac6fc30: 00007fe5bac6fc80 00007fe5d2a28866
0x00007fe5bac6fc40: 00007fe5bac6fc80 00007fe5d2bc1c0b
0x00007fe5bac6fc50: 000000060bc23d68 00007fe54c8c9800
0x00007fe5bac6fc60: 000000060bc17820 00007fe54c8c9800
0x00007fe5bac6fc70: 00007fe5d3463ef8 0000000000000000
0x00007fe5bac6fc80: 00007fe5bac6fcb0 00007fe5d2eaed6a
0x00007fe5bac6fc90: 0000000761ed39b0 0000000000000000
0x00007fe5bac6fca0: 00000006b49364f0 00007fe5bac6fcc8
0x00007fe5bac6fcb0: 0000000000000001 00007fe5c90624c5
0x00007fe5bac6fcc0: 0000000000000001 00007fe5c94d5428
0x00007fe5bac6fcd0: 0000000761ed39b0 00000006b49364f0
0x00007fe5bac6fce0: 00000006b49364f0 0000000761ed38a8
0x00007fe5bac6fcf0: 00000006b49627b0 0000000761ed39d8
0x00007fe5bac6fd00: d692c4e700000002 0000000761ed3808
0x00007fe5bac6fd10: 00000007d692c4cd 00000001ec785f25
0x00007fe5bac6fd20: 000000060bc18eb8 0000000000000000
0x00007fe5bac6fd30: 000000060bc18bb0 0000000000000000
0x00007fe5bac6fd40: 00007fe5bac6fdb8 00007fe5c9006058
0x00007fe5bac6fd50: 00007fe5c9006233 00007fe5c9006058
0x00007fe5bac6fd60: 00000006b49364f0 00000006b49627b0
0x00007fe5bac6fd70: 0000000761ed38a8 00007fe5bac6fd78
0x00007fe5bac6fd80: 000000060bc0fa9b 00007fe5bac6fdd0
0x00007fe5bac6fd90: 000000060bc10c70 0000000000000000
0x00007fe5bac6fda0: 000000060bc0fac8 00007fe5bac6fd60
0x00007fe5bac6fdb0: 00007fe5bac6fdc8 00007fe5bac6fe18
0x00007fe5bac6fdc0: 00007fe5c9006058 00000006b49627b0
0x00007fe5bac6fdd0: 0000000761ed38a8 00007fe5bac6fdd8
0x00007fe5bac6fde0: 000000060bc0f9a2 00007fe5bac6fe30

Instructions: (pc=0x00007fe5d29c0fa6)
0x00007fe5d29c0f86: 85 c0 74 2e 66 0f ef c0 49 8d 0c 02 31 d2 0f 1f
0x00007fe5d29c0f96: 40 00 48 89 d0 48 83 c2 01 48 c1 e0 04 48 39 fa
0x00007fe5d29c0fa6: 66 0f 7f 04 01 72 eb 4c 01 c6 4d 39 c1 74 0f 0f
0x00007fe5d29c0fb6: 1f 00 c6 06 00 48 83 c6 01 49 39 f3 77 f4 c9 c3

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x00007fe54c8c9800 is a thread
RCX=0x00007fe5cd898000 is an unknown value
RDX=0x0000000000000001 is an unknown value
RSP=0x00007fe5bac6fbf0 is pointing into the stack for thread: 0x00007fe54c8c9800
RBP=0x00007fe5bac6fbf0 is pointing into the stack for thread: 0x00007fe54c8c9800
RSI=0x00007fe5cd898000 is an unknown value
RDI=0x0008000000000000 is an unknown value
R8 =0x0080000000000000 is an unknown value
R9 =0x0080000000000000 is an unknown value
R10=0x00007fe5cd898000 is an unknown value
R11=0x00807fe5cd898000 is an unknown value
R12=0x0000000763c30180 is an oop
org.elasticsearch.search.internal.InternalSearchHit

• klass: 'org/elasticsearch/search/internal/InternalSearchHit'
  R13=0x00007fe54c8c9800 is a thread
  R14=0x00007fe5bac6fcc8 is pointing into the stack for thread: 0x00007fe54c8c9800
  R15=0x00007fe54c8c9800 is a thread

Stack: [0x00007fe5bab71000,0x00007fe5bac72000], sp=0x00007fe5bac6fbf0, free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x3bdfa6] CardTableModRefBS::dirty_MemRegion(MemRegion)+0xb6
V [libjvm.so+0x425e91] CollectedHeap::flush_deferred_store_barrier(JavaThread_)+0x81
V [libjvm.so+0x425866] CollectedHeap::new_store_pre_barrier(JavaThread_, oopDesc_)+0x26
V [libjvm.so+0x8abd6a] OptoRuntime::new_instance_C(klassOopDesc_, JavaThread*)+0x24a

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v ~RuntimeStub::_new_instance_Java
J 1931 C2 org.elasticsearch.search.internal.InternalSearchHits.readFrom(Lorg/elasticsearch/common/io/stream/StreamInput;Lorg/elasticsearch/search/internal/InternalSearchHits$StreamContext;)V (123 bytes) @ 0x00007fe5c94d5428 [0x00007fe5c94d4a40+0x9e8]
j org.elasticsearch.search.internal.InternalSearchHits.readFrom(Lorg/elasticsearch/common/io/stream/StreamInput;)V+11
j org.elasticsearch.search.internal.InternalSearchHits.readSearchHits(Lorg/elasticsearch/common/io/stream/StreamInput;)Lorg/elasticsearch/search/internal/InternalSearchHits;+10
j org.elasticsearch.search.internal.InternalSearchResponse.readFrom(Lorg/elasticsearch/common/io/stream/StreamInput;)V+2
j org.elasticsearch.search.internal.InternalSearchResponse.readInternalSearchResponse(Lorg/elasticsearch/common/io/stream/StreamInput;)Lorg/elasticsearch/search/internal/InternalSearchResponse;+10
j org.elasticsearch.action.search.SearchResponse.readFrom(Lorg/elasticsearch/common/io/stream/StreamInput;)V+7
j org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(Lorg/elasticsearch/common/io/stream/StreamInput;Lorg/elasticsearch/transport/TransportResponseHandler;)V+9
j org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(Lorg/elasticsearch/common/netty/channel/ChannelHandlerContext;Lorg/elasticsearch/common/netty/channel/MessageEvent;)V+490
j org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(Lorg/elasticsearch/common/netty/channel/ChannelHandlerContext;Lorg/elasticsearch/common/netty/channel/ChannelEvent;)V+13
j org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(Lorg/elasticsearch/common/netty/channel/DefaultChannelPipeline$DefaultChannelHandlerContext;Lorg/elasticsearch/common/netty/channel/ChannelEvent;)V+9
j org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Lorg/elasticsearch/common/netty/channel/ChannelEvent;)V+22
J 1883 C2 org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(Lorg/elasticsearch/common/netty/channel/ChannelHandlerContext;Lorg/elasticsearch/common/netty/channel/Channel;Lorg/elasticsearch/common/netty/buffer/ChannelBuffer;Ljava/net/SocketAddress;)V (104 bytes) @ 0x00007fe5c9618df0 [0x00007fe5c96188c0+0x530]
j org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(Lorg/elasticsearch/common/netty/channel/ChannelHandlerContext;Lorg/elasticsearch/common/netty/channel/MessageEvent;)V+62
j org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(Lorg/elasticsearch/common/netty/channel/ChannelHandlerContext;Lorg/elasticsearch/common/netty/channel/ChannelEvent;)V+13
j org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(Lorg/elasticsearch/common/netty/channel/DefaultChannelPipeline$DefaultChannelHandlerContext;Lorg/elasticsearch/common/netty/channel/ChannelEvent;)V+9
j org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(Lorg/elasticsearch/common/netty/channel/ChannelEvent;)V+56
J 1880 C2 org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(Ljava/nio/channels/SelectionKey;)Z (209 bytes) @ 0x00007fe5c90c105c [0x00007fe5c90c0cc0+0x39c]
J 1413% C2 org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run()V (409 bytes) @ 0x00007fe5c946c7cc [0x00007fe5c946c460+0x36c]
j org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run()V+1
j org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run()V+1
j org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run()V+55
j org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run()V+14
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub

[clintongormley]

@rmuir any ideas here?

[janmejay]

Can you install debug symbols for JVM and libc? With that the traces will atleast be meaningful. Eg. installing [package]-dbg on debian env will give you the debug symbols.

Also, depending on what environment you are running in, it may be possible to configure it to take a core-dump on SIGSEGV.

On GNU/Linux, try running 'ulimit -c unlimited' before launching tomcat. Check for presence of file /core after crash.

[clintongormley]

@rvller @janmejay either way, this is a JVM bug not an Elasticsearch bug, so I'm going to close this issue. (@rvller feel free to provide the info requested by @janmejay here even though the ticket is closed)