Elsa v3 None of my Delays or Timers are resuming

[kyle-alm]

I have a development and a production environment. I have used delays successfully in both environments, but now none of my delays or timers are resuming. I put up a simple http endpoint into a 1 minute delay into a console.writeline 40 minutes ago. The suspended event says it should resume 39 minutes ago, and it just hasn't. I've restarted the Elsa studio project and my local environment several times, still nothing.

Perhaps related, I also cannot cancel any workflows. I used to have to click it a couple of times, but now all of my workflows stay uncancelled no matter how many times I hit it. I'm not sure where to even start with debugging because the documentation is pretty thin. I see in previous versions there were settings you could configure with the scheduler, but I haven't seem any indication of that for version 3, but I also doubt my scheduler just suddenly started waiting longer to look for activities to resume. I guess my gut feeling is that it's hitting an error in it's scheduling process and exiting out.

In production, I have been getting a lot of errors telling my an id isn't present in the dictionary, and after looking through the database, I tracked at least one down to a parentactivityinstanceid for a Elsa.Workflow. I can see in my bookmarks a lot of Elsa.Delays where the resumeAt in the serialized payload was yesterday or earlier.

I have many workflows in flight and it will be a monumental task to replace Elsa if I can't straighten this out. Thanks.

[kyle-alm]

After more testing, it may be a database issue. I run Elsa Studio locally with the server also running locally, pointing to my production or development database. I also have a production server running and a development server. I did just discover that I cannot cancel locally, but if I post the cancel endpoint on my production server, that will cancel. I had also opened a connection in azure data studio to the workflow database to query instances against, and closing that connection and my local server seemed to move some things along. I think it's a database locking or concurrency issue.