IllegalStateException: Future Not Started

[benjchristensen]

Originally was issue #80 but that got hijacked by a HystrixCollapser issue that at first was thought to be related or the same but wasn't.

---

We are seeing this exception in logs. It suggests a concurrency bug.

It's a very small volume of errors - 1 or 2 per hour out of 10s of thousands of executions per instance - but it's concerning that it is happening at all.

---

Logging was added in pull request #81 to help identify the issue.

[benjchristensen]

Example log output:

Caused by: java.lang.IllegalStateException: Future was not started.  Key: PlaylistGet  ActualFuture: null  Started: true  actualFutureExecuted: true  futureStarted: 0  isInterrupted: false  actualResponseReceived: 1  isCommandTimedOut: false  Events: [THREAD_POOL_REJECTED, FALLBACK_FAILURE]

Caused by: java.lang.IllegalStateException: Future was not started.  Key: QTGetQTVGenres  ActualFuture: null  Started: true  actualFutureExecuted: true  futureStarted: 0  isInterrupted: false  actualResponseReceived: 1  isCommandTimedOut: false  Events: [THREAD_POOL_REJECTED]

In the machine being monitored there was a small burst of each of these.

[benjchristensen]

More examples from a machine in production. They are very rare but always seem to occur on THREAD_POOL_REJECTED.

Caused by: java.lang.IllegalStateException: Future was not started.  Key: ABTestGetAllocationMap  ActualFuture: null  Started: true  actualFutureExecuted: true  futureStarted: 0  isInterrupted: false  actualResponseReceived: 1  isCommandTimedOut: false  Events: [THREAD_POOL_REJECTED]

Caused by: java.lang.IllegalStateException: Future was not started.  Key: GPSGetGroups  ActualFuture: null  Started: true  actualFutureExecuted: true  futureStarted: 0  isInterrupted: false  actualResponseReceived: 1  isCommandTimedOut: false  Events: [THREAD_POOL_REJECTED, FALLBACK_FAILURE]

Caused by: java.lang.IllegalStateException: Future was not started.  Key: ABTestGetNonExplicitAllocationForTest  ActualFuture: null  Started: true  actualFutureExecuted: true  futureStarted: 0  isInterrupted: false  actualResponseReceived: 1  isCommandTimedOut: false  Events: [THREAD_POOL_REJECTED]

[benjchristensen]

Fixed in previous pull request.