[Reporting] Unlogged error: the reports:monitor task found an expired processing job

[tsullivan]

We sometimes get reports from users involving a report job that takes a very long time to fail with a timeout, with no errors logged and no explanation why.

If Kibana restarts or crashes during report job execution, that job will remain labeled as processing. To clean up that state, the reports:monitor task queries for processing reports that started at a time further back than the timeout limit allows. The report is rescheduled for reattempt.

If Kibana keeps restarting / crashing / going unresponsive while a report job is running, it will keep getting rescheduled until the number of attempts is exhausted. When a delayed processing job is found and is marked with no remaining attempts, Reporting marks the job as failed.

Problem: The rescheduling events may not be very apparent even to someone reading the logs, since these are info or debug level debugs. One would normally be able to see this kind of activity if they are searching for errors in the reporting logs, since they are not harmless events.

The reports:monitor task should log an error when it finds a task that needs to be retried. The error message should explain what happened for things to get into that state: the instance that was executing the report job stopped responding for too long.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServicesUx)