Major CPU performance decrease with ImageSharp.Web 3

[marklagendijk]

Prerequisites

• I have written a descriptive issue title
• I have verified that I am running the latest version of ImageSharp.Web
• I have verified if the problem exist in both DEBUG and RELEASE mode
• I have searched open and closed issues to ensure it has not already been reported

ImageSharp.Web version

3.0.1

Other Six Labors packages and versions

None

Environment (Operating system, version and so on)

Azure App Service Windows

.NET version

7

Description

After upgrading to ImageSharp.Web 3 we noticed a major increase of CPU usage.
We then proceeded with steps to isolate the issue so we would be 100% sure that the version increase was indeed the cause:

1. We isolated the usage of ImageSharp.Web to its own application + App Service.
2. After a number of days we stepped down the version of ImageSharp.Web to version 2.
3. After around half a day I took a screenshot of the CPU performance (see below). I could wait longer, but the difference is so clear that I don't think that will help.

Technical background

• We run a multi-tenant web application that servers around 15 news websites.
• ImageSharp.Web is used for image downsizing.
• Azure FrontDoor is used as a pass-through CDN.
• Azure FrontDoor is configured to cache all the image responses for 1 year.
• Because Azure FrontDoor caches on the POP (point-of-presence) level, there are still numerous requests per image.
• Azure App Service (Windows, .NET 7) is used for serving the web applications. At this point we separated the usage of ImageSharp.Web to its own App Service, although we still run it on the same VM (Azure App Service Plan).
• We currently don't use caching on the ImageSharp.Web level, we configure a NullImageCache.
• Before we switched to the NullImageCache we used the PhysicalFileSystemCache. It ended up breaking our App Service because we eventually ran out of disk space, because the cache does not remove old images (note: I now know that this is a non-trivial issue, I would propose making this more clear in the documentation).
• The images are loaded with a custom RemoteImageProvider which uses HttpClient to load images from an url.
• We use the PresetOnlyQueryCollectionRequestParser.

Background

A few weeks back I found out that version 3 of ImageSharp.Web was released. I also found out that the license had been changed.
After arranging the purchase of the license I upgraded.

Based on the release notes I expected to see a performance improvement, but unfortunately it quickly became clear that we instead had a performance decrease.

I explicitly waited with reporting the issue until I was 100% sure that it is caused by the ImageSharp.Web upgrade.
I checked the changelogs of ImageSharp.Web and ImageSharp, but I could not find any change that which might cause our issue.

I am willing to help with identifying the cause of the issue, but I would like some pointers with regards to how to proceed.

Steps to Reproduce

At this point the "Steps to Reproduce" are not clear beyond "Upgrade from ImageSharp.Web 2.0.2 to 3.0.1".
However, I can share all the relevant code that we use.

This is the library that contains all our ImageSharp.Web code.
PMP.ImageProcessing.zip

The PMP.ImageProcessing library is used in a minimal web application.
PMP.ImageProcessor.zip

Images

The red arrow indicates the point where the version 2 of ImageSharp.Web was deployed.
The graph shows that with ImageSharp.Web 3 both average and maximum CPU usage are at least 2 times higher.

[JimBobSquarePants]

Thanks for the detail. Have you been able to profile this?

I’m wondering if this is related? It might be worth trying one of the nightly builds. Hard to determine the issue without profiling.

#327

[marklagendijk]

I have not been able to profile the issue.

I will try updating to the nightly Feedz.io build to see if that fixes the issue.

We don't use the HMAC-protection. Is it enabled by default? If so, is there a way to disable it?

[JimBobSquarePants]

I've been doing some profiling comparing the two versions and I cannot see anything significant. Are you experiencing a drop in throughput?

On its own, higher CPU usage isn't always bad. It needs to be evaluated in the context of the application's overall performance, including throughput, response time, and resource utilization.

[marklagendijk]

I did try the nightly build.
It did not seem to make any difference.

Based on your comment I compared the Response Time metric for the 2.x and 3.x.
It is significantly higher (worse performance) for 3.x vs 2.x:

• 50th percentile: 47ms vs 36ms
• 95th percentile: 280ms vs 220ms
• 99th percentile: 840ms vs 410ms
• Average: 100ms vs 70ms

Now I do have a license for all Jetbrains .NET tools, including the profiler build into Rider and the separate dotTrace Profiler application, however I don't have much experience with it.
I could quite easily get to the point where I can run a fixed set of image processor requests. I think this might be useful because it could be that the issue only applies to certain types of requests, so doing a set instead of a single one could help pin down the issue.
Is it easy to use the profiling tool to compare such a set of requests?

[JimBobSquarePants]

The Jetbrains Tools don't really provide the detailed information you need running under stress so instead I've benchmarked the solution using Crank

It's fairly straightforward to get started.

1. First install the global tool as per the instructions.
2. Run crank-agent
3. Navigate to each folder in the zip file in turn and run the following, letting each run complete before starting the other.
   crank --config local.benchmarks.yml --scenario imagesharp --profile local

crank.zip

Now... That all said I'm not seeing the issues you are,

Application build and startup churns through the CPU a bit more for some reason but the load tests are equivalent. If anything v3 has a slightly higher max throughput.

[adamsitnik]

My 2 cents:

1. How does the CPU spike relates to the number of requests being handled by the web service? Is it possible that a single instance of a web service is now handling more requests?
2. What is the CPU specification? Is @JimBobSquarePants running the benchmarks on the same architecture (x64 vs arm64), with the same vector size and available instruction set?

[marklagendijk]

I just (yesterday) upgraded to the newly released 3.1 version, and the problematic CPU usage seems to be fixed.
I'll monitor the situation for a few days to be absolutely sure.

@adamsitnik good points.

1. The CPU spikes really stand out in an otherwise stable graph. In other words, such spikes don't occur in the historical data and relate 100% between the old and new version.
2. We run on Azure App Service, P2V3 (4 virtual CPUs, x64). We do run the application in 64 Bit.

[marklagendijk]

I can happily report that our application has been running 3.1 for several days now and has had no CPU issues.
With this I conclude that the original issue (whatever it may have been) has been fixed since 3.0.

[JimBobSquarePants]

Huzzah!!

[marklagendijk]

@JimBobSquarePants I seem to be running into the issue described in SixLabors/ImageSharp#2450.
I see that you fixed that issue and deployed it in ImageSharp 3.1 (which is also part of ImageSharp.Web 3.1).

Do I need to configure the GIF Encoder with specific settings to use this, or may it be that I'm running into a different GIF issue?

[JimBobSquarePants]

No special configuration required.

Can you share the image, there’s an edge case I fixed 3 days ago but it should be fairly rare.

SixLabors/ImageSharp#2605

[marklagendijk]

Sure. We have the issue with GIF advertisement banners.
This is one example:

It seems to happen with a significant percentage of our banners.

[JimBobSquarePants]

It's the deduper. Your images use a global palette with 256 entries and no transparency. I'm introducing transparency but not replacing the palette. I just fixed it locally and will push out a v3.1.1 release with this and other fixes later once I PR it.

Here's my version with the fix. As a bonus it's only 480KB vs 654KB original.

[marklagendijk]

Awesome! That is a size reduction of 26%!
Given that most of our hosting costs are bandwidth (Azure FrontDoor bandwidth is not cheap) this is very significant for us.

When the new version is out I will verify it locally against all our current banners.

[JimBobSquarePants]

You can grab ImageSharp v3.1.1 now from NuGet. It's binary compatible with ImageSharp.Web v3.1.0

If your company hasn't purchased a license already and meet the criterea, please do so. It helps keep the lights on here.

[marklagendijk]

Wow, that was quick! 🚀
Btw, yes, we did buy a license.

I can confirm that this fixes the distorted GIF issue.

When comparing file sizes I did notice a case where the processed image was bigger than the original:
Processed:

Original:

I will create a script to automatically compare the file sizes of all our images to see the distribution of size difference over the entire dataset.
Is it reasonable to expect that the output size of ImageSharp.Web will always be smaller or the same as the input (assuming no size-increasing operations, of course)? Or is this not feasible due to complicated technical details?

[JimBobSquarePants]

There’s no guarantees. When we load a gif we have to coalesce the frame deltas losing all previous optimisations. We do a ton of work to reintroduce optimisations but sometimes we can’t match original meticulous individual encodings. There are changes we could theoretically make to the deduping process to enable better compression but they’d make encoding times too slow.

[marklagendijk]

I get that.

For the web use case I think it would be nice to have the option to use more aggressive compression.
For our case it could be totally acceptable to have processing times be up to 10 times higher if that would help with reducing image size, the result is cached after all.

It's not even only about saving bandwidth costs, but also about faster load times for the end user, and saving them bandwidth.

[JimBobSquarePants]

It’s very intense for gif since there’s no line filtering. You’d have to have multiple deduplication strategies that ignore diffs when there are not multiple in a row. Then you’d have to compress each whole image frame multiple times and compare.

Ideally our current strategy saves overall so individual differences become less relevant.

[marklagendijk]

Aha. Excuse my ignorance about stuff like this. With awesome libraries like ImageSharp most developers rarely need to get into these kind of details.

When looking into ways to save bandwidth I found out that both browsers and ImageSharp support WebP now. I will definitely investigate whether we can save bandwidth by using ImageSharp to convert our GIF images to WebP.