Troubleshooting a running environment

These are scenarios seen during active development with breaking code changes. While every effort has been made to deal with errors in an informative manner in a deployment, these hints will be helpful for supporting the system in the case of a failure.

• github hook fails to hit lambda

  • serverless lambda uninstalled and reinstalled? check endpoint address is correct in github hook config.

• event does not appear on pending sqs queue

  • check hook lambda cloudwatch logs
  • check github hook secret matches lambda env config
  • check .githubTaskManager.json exists in repo and is valid json

• agent not receiving events

  • check cloudwatch pending queue logs
  • check cloudwatch agent logs to see if another agent picked up the events
  • check agent event group matches (although agent logs should show the event skip)

• github pull request checks do not appear as pending

  • check cloudwatch logs to confirm event passed through hook lambda and queue to an agent
  • check that .githubTaskManager.json is well formed and has the expected task executor and context

• github pull requests not updated from 'pending' to 'success' or 'error'

  • check for unhandled errors in cloudwatch agent logs
  • check pending dead letter queue in aws
  • if queue agents are not active, purge the queues from aws console