Retrieving the status for multiple orchestrations (quickly)

[f2bo]

We have an app that needs to query the status for all orchestations that are executed. It seems to take a very long time (almost 3 mins.) just to retrieve information from the last 14 days. Given that the number of orchestrations in that period was not large, less than 500, I'm wondering whether there's anything that can be done to speed this up.

I notice that GetStatusAsync always includes both the input and output of the orchestrations. These can be large, 150Kb is not uncommon for some of our outputs, and there's no option to reduce the level of detail provided. I may be mistaken, but I would think that when you query multiple instances, it would be far more common not to require this information. In our particular case, simply being able to enumerate the instance ID, runtime status, created and last updated dates, as well as the custom status would be sufficient. We can always obtain the remaining information by querying any orchestrations that we want to analyze in more detail using their instance IDs.

Moreover, I don't recall it being this slow probably because, in previous releases, any large outputs were GZip-compressed and uploaded to blob storage and the response from GetStatusAsync only contained a URL (#665). Yes, it was a bit of a nuisance, but we had code to deal with this and more importantly, we had more control. With the current behavior, you retrieve all input and outputs whether you want them or not.

I'm not suggesting reverting to the original behavior but rather to provide and overload to GetStatusAsync where you can specify the level of information you want returned. For example, something like:

GetStatusAsync(
    DateTime createdTimeFrom,
    DateTime? createdTimeTo,
    IEnumerable<OrchestrationRuntimeStatus> runtimeStatus,
    bool includeInputs,
    bool includeOutputs,
    CancellationToken cancellationToken);

[cgillum]

@f2bo indeed, I can see how this could be problematic from a performance perspective.

@ConnorMcMahon looking at this commit, it seems we could resolve this issue by exposing a FetchLargeMessageDataEnabled property to DurableTaskOptions. That would allow someone to opt-out of auto-fetching large messages.

[ConnorMcMahon]

@f2bo and @cgillum, we actually already have the piping to turn off this behavior. You should just need to set the property "fetchLargeMessageDataEnabled" to false in your host.json file.

This is on a per-application level, but I am hesitant to add controls for it on a per-API level, as that could start to add too many parameters for developers to reasonably keep track of. Is this sufficient for you @f2bo?

[f2bo]

Hi @ConnorMcMahon

I cannot find documentation for this setting. Is this something that will change the behavior of DurableOrchetrationClient.GetStatusAsync() only? To be clear, it's not that I don't ever want to access the inputs and outputs, but I don't want to pay the performance cost of retrieving them when querying multiple orchestrations, but rather download them individually using their instance IDs, if I need them.

In any case, I'm not sure if I misunderstood but I quickly tested it running locally and I still see that both inputs and outputs are populated when I execute the following:

var currentStatus = await client.GetStatusAsync(
    createdTimeFrom,
    createdTimeTo,
    Enumerable.Empty<OrchestrationRuntimeStatus>())

This was in a Function app with v1.83 and with the following settings in host.json.

"durableTask": {
     "hubName": "StatsEngine",
     "traceInputsAndOutputs": false,
     "extendedSessionsEnabled": true,
     "extendedSessionIdleTimeoutInSeconds": 600,
     "logReplayEvents": false,
     "maxConcurrentActivityFunctions": 300,
     "fetchLargeMessageDataEnabled": false
}

[ConnorMcMahon]

@f2bo,

I apologize, I actually gave you the wrong setting name. It should actually be "fetchLargeMessagesAutomatically". If you set this to false, it should restore functionality to the way it was before 1.8.3, i.e. it turns off the new automatic fetching for the entire application.

Unfortunately, doing the work to configure it on a per-request basis would involve going back into DurableTask.AzureStorage and adding some substantial changes there. It is definitely possible, but it is not a quick feature.

If you already have the code that worked with the old behavior, I would recommend using this setting and only grabbing the blobs yourself when necessary.

I will add a documentation flag to this issue to address the fact we need to update our documentation with this behavior.

[f2bo]

@ConnorMcMahon

Yes. That did the trick. I'm still getting the inputs, which I don't need, but it's much faster now and yes, we still have the code to retrieve large outputs from the compressed blobs.

Thanks!

[ConnorMcMahon]

Glad to hear @f2bo. I will keep this issue open until we make the relevant documentation improvements.

[ConnorMcMahon]

I have consolidated all of the missing host.json fields into one issue for documentation.

[ConnorMcMahon]

See #1212.