Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YCSB Multiple Thread Issue #770

Closed
AnkitaD opened this issue Jun 19, 2016 · 5 comments
Closed

YCSB Multiple Thread Issue #770

AnkitaD opened this issue Jun 19, 2016 · 5 comments

Comments

@AnkitaD
Copy link

AnkitaD commented Jun 19, 2016

Hi,

I am using HBase 1.2.0 for my testing when I am using more that one thread,by using following command,its throwing issue:

./bin/ycsb load hbase10 -p columnfamily=cf -P workloads/workloadc -p table=usertable -p recordcount=5000 -p operationcount=5000 -threads 10 -s > Hbase_nfs_worka

jstack results shows following threads waiting to complete:

"hconnection-0x7ad4696c-shared--pool1-t20" #58 daemon prio=5 os_prio=0 tid=0x00007f9f38003000 nid=0xb38 waiting on condition [0x00007fa02befd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t19" #59 daemon prio=5 os_prio=0 tid=0x00007f9f4400b800 nid=0xb37 waiting on condition [0x00007fa02bffe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t18" #60 daemon prio=5 os_prio=0 tid=0x00007f9f54003000 nid=0xb36 waiting on condition [0x00007fa030167000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t14" #54 daemon prio=5 os_prio=0 tid=0x00007f9f4c00c000 nid=0xb35 waiting on condition [0x00007fa030266000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1149)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t13" #56 daemon prio=5 os_prio=0 tid=0x00007f9f4838c800 nid=0xb34 waiting on condition [0x00007fa030369000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t12" #53 daemon prio=5 os_prio=0 tid=0x00007f9f40004000 nid=0xb33 waiting on condition [0x00007fa03046a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t15" #55 daemon prio=5 os_prio=0 tid=0x00007f9f34006800 nid=0xb32 waiting on condition [0x00007fa03056b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t16" #57 daemon prio=5 os_prio=0 tid=0x00007f9f3c00a000 nid=0xb31 waiting on condition [0x00007fa03066c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t11" #52 daemon prio=5 os_prio=0 tid=0x00007f9f30006000 nid=0xb30 waiting on condition [0x00007fa03076d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t7" #43 daemon prio=5 os_prio=0 tid=0x00007f9f44007800 nid=0xb2e waiting on condition [0x00007fa030b6f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t1" #47 daemon prio=5 os_prio=0 tid=0x00007f9f3c002800 nid=0xb2d waiting on condition [0x00007fa030c70000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t6" #48 daemon prio=5 os_prio=0 tid=0x00007f9f30001800 nid=0xb2c waiting on condition [0x00007fa030d71000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t8" #42 daemon prio=5 os_prio=0 tid=0x00007f9f38001800 nid=0xb2b waiting on condition [0x00007fa030e72000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t10" #41 daemon prio=5 os_prio=0 tid=0x00007f9f34001800 nid=0xb2a waiting on condition [0x00007fa030f73000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t9" #44 daemon prio=5 os_prio=0 tid=0x00007f9f4c00a800 nid=0xb29 waiting on condition [0x00007fa031075000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t3" #45 daemon prio=5 os_prio=0 tid=0x00007f9f4838a800 nid=0xb28 waiting on condition [0x00007fa031176000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t2" #49 daemon prio=5 os_prio=0 tid=0x00007f9f40001000 nid=0xb27 waiting on condition [0x00007fa031277000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t5" #50 daemon prio=5 os_prio=0 tid=0x00007f9f28007000 nid=0xb26 waiting on condition [0x00007fa031378000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"hconnection-0x7ad4696c-shared--pool1-t4" #46 daemon prio=5 os_prio=0 tid=0x00007f9f54001000 nid=0xb25 waiting on condition [0x00007fa031479000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a01d8188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"IPC Client (578667173) connection to stlrx2540m1-69.rtpppe.netapp.com/10.63.150.163:60000 from root" #40 daemon prio=5 os_prio=0 tid=0x00007f9f4837b000 nid=0xb24 in Object.wait() [0x00007fa03157a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:559)
- locked <0x00000005a0228000> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:581)

"Thread-3-EventThread" #38 daemon prio=5 os_prio=0 tid=0x00007f9f48207000 nid=0xb22 waiting on condition [0x00007fa031a7b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a0230188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

"Thread-3-SendThread(localhost:2181)" #37 daemon prio=5 os_prio=0 tid=0x00007f9f48206800 nid=0xb21 runnable [0x00007fa0321ae000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000005a01d0408> (a sun.nio.ch.Util$2)
- locked <0x00000005a01d0418> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005a01d03c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

"process reaper" #34 daemon prio=10 os_prio=0 tid=0x00007f9f480c6800 nid=0xb1b waiting on condition [0x00007fa0321e7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a0198490> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

"Thread-11" #32 prio=5 os_prio=0 tid=0x00007fa0b452b000 nid=0xb19 in Object.wait() [0x00007fa0326f0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a0228380> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a03e6f00> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-10" #31 prio=5 os_prio=0 tid=0x00007fa0b4529000 nid=0xb18 in Object.wait() [0x00007fa0327f1000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a01a8470> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0568b50> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-9" #30 prio=5 os_prio=0 tid=0x00007fa0b4527800 nid=0xb17 in Object.wait() [0x00007fa0328f3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02381b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0238230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-8" #29 prio=5 os_prio=0 tid=0x00007fa0b4526000 nid=0xb16 in Object.wait() [0x00007fa0329f4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02401b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0240230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-7" #28 prio=5 os_prio=0 tid=0x00007fa0b4524000 nid=0xb15 in Object.wait() [0x00007fa032af5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02481b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a02481d0> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-6" #27 prio=5 os_prio=0 tid=0x00007fa0b4522800 nid=0xb14 in Object.wait() [0x00007fa032bf6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02501b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0415d18> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-5" #26 prio=5 os_prio=0 tid=0x00007fa0b4521000 nid=0xb13 in Object.wait() [0x00007fa032cf7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02581b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0258230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-4" #25 prio=5 os_prio=0 tid=0x00007fa0b451f000 nid=0xb12 in Object.wait() [0x00007fa032df8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02601b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a025d7b0> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-3" #24 prio=5 os_prio=0 tid=0x00007fa0b451d800 nid=0xb11 in Object.wait() [0x00007fa032ef8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02681b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0268230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-2" #23 prio=5 os_prio=0 tid=0x00007fa0b451c800 nid=0xb10 in Object.wait() [0x00007fa032ff9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02701b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0270230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-12" #33 prio=5 os_prio=0 tid=0x00007fa0b451c000 nid=0xb0f waiting on condition [0x00007fa0331f0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005a0181670> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source)
at java.util.concurrent.CountDownLatch.await(Unknown Source)
at com.yahoo.ycsb.StatusThread.waitForClientsUntil(Client.java:176)
at com.yahoo.ycsb.StatusThread.run(Client.java:98)

"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007fa0b4115800 nid=0xb0c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007fa0b4110800 nid=0xb0b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007fa0b410e000 nid=0xb0a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007fa0b410c800 nid=0xb09 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007fa0b410a000 nid=0xb08 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007fa0b4108000 nid=0xb07 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007fa0b4106000 nid=0xb06 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007fa0b4103800 nid=0xb05 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007fa0b4101800 nid=0xb04 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007fa0b40ff800 nid=0xb03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007fa0b40fd800 nid=0xb02 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007fa0b40f3800 nid=0xb01 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007fa0b40f1000 nid=0xb00 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fa0b40ef800 nid=0xaff waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fa0b40eb000 nid=0xafe waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fa0b40e8000 nid=0xafd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fa0b40e6000 nid=0xafc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fa0b40ba800 nid=0xafb in Object.wait() [0x00007fa038431000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x00000005a01b88a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fa0b40b8800 nid=0xafa in Object.wait() [0x00007fa038532000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
- locked <0x00000005a0190468> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007fa0b400a800 nid=0xad7 in Object.wait() [0x00007fa0bbdd3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000005a0270000> (a com.yahoo.ycsb.ClientThread)
at java.lang.Thread.join(Unknown Source)
- locked <0x00000005a0270000> (a com.yahoo.ycsb.ClientThread)
at java.lang.Thread.join(Unknown Source)
at com.yahoo.ycsb.Client.main(Client.java:931)

"VM Thread" os_prio=0 tid=0x00007fa0b40b3800 nid=0xaf9 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fa0b4020000 nid=0xad8 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fa0b4021800 nid=0xad9 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fa0b4023800 nid=0xada runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fa0b4025000 nid=0xadb runnable

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007fa0b4027000 nid=0xadc runnable

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007fa0b4028800 nid=0xadd runnable

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007fa0b402a800 nid=0xade runnable

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007fa0b402c000 nid=0xadf runnable

"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007fa0b402e000 nid=0xae0 runnable

"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007fa0b402f800 nid=0xae1 runnable

"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007fa0b4031800 nid=0xae2 runnable

"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007fa0b4033000 nid=0xae3 runnable

"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007fa0b4035000 nid=0xae4 runnable

"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007fa0b4036800 nid=0xae5 runnable

"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007fa0b4038800 nid=0xae6 runnable

"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007fa0b403a000 nid=0xae7 runnable

"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007fa0b403c000 nid=0xae8 runnable

"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007fa0b403d800 nid=0xae9 runnable

"GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007fa0b403f800 nid=0xaea runnable

"GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007fa0b4041000 nid=0xaeb runnable

"GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007fa0b4043000 nid=0xaec runnable

"GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007fa0b4044800 nid=0xaed runnable

"GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007fa0b4046800 nid=0xaee runnable

"GC task thread#23 (ParallelGC)" os_prio=0 tid=0x00007fa0b4048000 nid=0xaef runnable

"GC task thread#24 (ParallelGC)" os_prio=0 tid=0x00007fa0b404a000 nid=0xaf0 runnable

"GC task thread#25 (ParallelGC)" os_prio=0 tid=0x00007fa0b404b800 nid=0xaf1 runnable

"GC task thread#26 (ParallelGC)" os_prio=0 tid=0x00007fa0b404d800 nid=0xaf2 runnable

"GC task thread#27 (ParallelGC)" os_prio=0 tid=0x00007fa0b404f000 nid=0xaf3 runnable

"GC task thread#28 (ParallelGC)" os_prio=0 tid=0x00007fa0b4051000 nid=0xaf4 runnable

"GC task thread#29 (ParallelGC)" os_prio=0 tid=0x00007fa0b4052800 nid=0xaf5 runnable

"GC task thread#30 (ParallelGC)" os_prio=0 tid=0x00007fa0b4054800 nid=0xaf6 runnable

"GC task thread#31 (ParallelGC)" os_prio=0 tid=0x00007fa0b4056000 nid=0xaf7 runnable

"GC task thread#32 (ParallelGC)" os_prio=0 tid=0x00007fa0b4058000 nid=0xaf8 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fa0b4118000 nid=0xb0d waiting on condition

JNI global references: 274

Can anyone please suggest what would be the issue and how to resolve this ??

@busbey
Copy link
Collaborator

busbey commented Jun 22, 2016

what's the issue thrown? did the client produce a jstack dump, or did you run jstack when it didn't appear to do much?

@nitsanw
Copy link
Contributor

nitsanw commented Jun 23, 2016

This looks like a jstack from out of process, it would have been kind of the submitter to filter out the many irrelevant threads, and not dump the stack twice... here's what's left after that exercise:

"hconnection-0x7ad4696c-shared--pool1-t14" #54 daemon prio=5 os_prio=0 tid=0x00007f9f4c00c000 nid=0xb35 waiting on condition [0x00007fa030266000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1149)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:705)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)


"IPC Client (578667173) connection to stlrx2540m1-69.rtpppe.netapp.com/10.63.150.163:60000 from root" #40 daemon prio=5 os_prio=0 tid=0x00007f9f4837b000 nid=0xb24 in Object.wait() [0x00007fa03157a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:559)
- locked <0x00000005a0228000> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:581)

"Thread-11" #32 prio=5 os_prio=0 tid=0x00007fa0b452b000 nid=0xb19 in Object.wait() [0x00007fa0326f0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a0228380> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a03e6f00> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-10" #31 prio=5 os_prio=0 tid=0x00007fa0b4529000 nid=0xb18 in Object.wait() [0x00007fa0327f1000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a01a8470> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0568b50> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-9" #30 prio=5 os_prio=0 tid=0x00007fa0b4527800 nid=0xb17 in Object.wait() [0x00007fa0328f3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02381b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0238230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-8" #29 prio=5 os_prio=0 tid=0x00007fa0b4526000 nid=0xb16 in Object.wait() [0x00007fa0329f4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02401b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0240230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-7" #28 prio=5 os_prio=0 tid=0x00007fa0b4524000 nid=0xb15 in Object.wait() [0x00007fa032af5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02481b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a02481d0> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-6" #27 prio=5 os_prio=0 tid=0x00007fa0b4522800 nid=0xb14 in Object.wait() [0x00007fa032bf6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02501b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0415d18> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-5" #26 prio=5 os_prio=0 tid=0x00007fa0b4521000 nid=0xb13 in Object.wait() [0x00007fa032cf7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02581b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0258230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-4" #25 prio=5 os_prio=0 tid=0x00007fa0b451f000 nid=0xb12 in Object.wait() [0x00007fa032df8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02601b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a025d7b0> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-3" #24 prio=5 os_prio=0 tid=0x00007fa0b451d800 nid=0xb11 in Object.wait() [0x00007fa032ef8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02681b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0268230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

"Thread-2" #23 prio=5 os_prio=0 tid=0x00007fa0b451c800 nid=0xb10 in Object.wait() [0x00007fa032ff9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1622)
- locked <0x00000005a02701b8> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1651)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0x00000005a0270230> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1470)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1064)
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:450)
at com.yahoo.ycsb.db.HBaseClient10.insert(HBaseClient10.java:481)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:579)
at com.yahoo.ycsb.ClientThread.run(Client.java:346)

There's a bunch of client threads waiting to finish flushing, and the top thread is sleeping waiting to resubmit something...

@busbey
Copy link
Collaborator

busbey commented Jun 23, 2016

If the issue was the client appearing to hang, could you take a look at your region servers? Were they healthy? Anything in their logs?

The default hbase client retry/timeout is very very generous so it will happily sit around for ages if your cluster falls over during the test.

@busbey
Copy link
Collaborator

busbey commented Aug 2, 2016

@AnkitaD did you get a chance to check on RegionServer health while running into this issue?

@busbey
Copy link
Collaborator

busbey commented Aug 9, 2016

closing as stale. @AnkitaD if you're still having trouble and you get a chance to check the suggestions above, please reopen.

@busbey busbey closed this as completed Aug 9, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants