Add beeheads stress benchmarks to the repo

[antonfirsov]

Prerequisites

• I have written a descriptive pull-request title
• I have verified that there are no overlapping pull-requests open
• I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
• I have provided test coverage for my change (where applicable)

Description

Contributes to #1596.

To simplify benchmarking and memory profiling, I'd like to add @saucecontrol 's modified LoadResizeSave benchmarks to our repo:

1. The BDN benchmarks go to ImageSharp.Benchmarks project
2. The MemoryStress stuff goes to our ProfilingSandbox, so we can easily profile WIP source code, without the Xunit and BDN bloat, attaching whatever (memory) profiler needs to be attached

I did a significant refactor on the original code:

• So 1. and 2. shares the same code, implemented in LoadResizeSaveStressRunner.cs
• Parallel/non parallel stuff is unified, by making parallelism a parameter
• Applied [ShortRunJob] and removed FreeImage to reduce the number of total runs

MagicScaler issue

When running the MagicScaler benchmark from within BDN, it fails for me with the following stack trace on my machine:

EDIT: There is a general issue with NuGet references in ImageSharp.Benchmarks, had to workaround by repeating the version numbers.

Performance report [updated with MagicScaler runs]

Posting the report from my run of LoadResizeSaveStressBenchmarks.

I used a workstation with 64GB of RAM and a 10-core i9-10900X @ 3.70GHz, # of threads aka. Environment.ProcessorCount is 20 on this machine.

I benchmarked with MaxDegreeOfParallelism = 1, 5, 10, 20, and did a little analysis about scalability.

                        Parallelism * Mean time [for the current row] 
Scalability  =   ---------------------------------------------------------------------
                        Mean time  [for Parallelism == 1] 

Method	Parallellism	Mean time (ms)	Ratio to ImageSharp with same parallelism	Parallelism * Mean	Scalability (ideal: all lines are 1-s)
ImageSharp	1	5256	1.00	5256	1.00
ImageSharp	5	1537	1.00	7687	1.46
ImageSharp	10	908	1.00	9075	1.73
ImageSharp	20	782	1.00	15644	2.98
Magick	1	11479	2.18	11479	1.00
Magick	5	3306	2.15	16530	1.44
Magick	10	2148	2.37	21482	1.87
Magick	20	1627	2.08	32542	2.83
SystemDrawing	1	9505	1.81	9505	1.00
SystemDrawing	5	3413	2.22	17063	1.80
SystemDrawing	10	2992	3.30	29917	3.15
SystemDrawing	20	2995	3.83	59902	6.30
SkiaBitmap	1	6995	1.33	6995	1.00
SkiaBitmap	5	1995	1.30	9973	1.43
SkiaBitmap	10	1147	1.26	11473	1.64
SkiaBitmap	20	887	1.13	17748	2.54
NetVips	1	3070	0.58	3070	1.00
NetVips	5	594	0.39	2969	0.97
NetVips	10	391	0.43	3908	1.27
NetVips	20	388	0.50	7766	2.53
MagicScaler	1	2089	0.40	2089	1.00
MagicScaler	5	670	0.44	3350	1.60
MagicScaler	10	482	0.53	4823	2.31
MagicScaler	20	401	0.51	8010	3.84

Conclusions

• We outperform every other lib except NetVips and MagicScaler
• SystemDrawing scales very poorly -- due to the global locks I guess
• NetVips scales the best, SkiaSharp scales slightly better than us
• For some reason MagicScaler doesn't seem to champion on my CPU as much as in @saucecontrol 's runs. With more CPU-s NetVips seems to outperform it. Any idea why?

[codecov]

Codecov Report

Merging #1687 (20040bd) into master (3f96b1e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1687   +/-   ##
=======================================
  Coverage   84.33%   84.33%           
=======================================
  Files         816      816           
  Lines       35908    35908           
  Branches     4173     4173           
=======================================
  Hits        30282    30282           
  Misses       4804     4804           
  Partials      822      822           

Flag	Coverage Δ
unittests	84.33% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f96b1e...20040bd. Read the comment docs.

[antonfirsov]

I see what's going wrong with MagicScaler within ImageSharp.Benchmarks, however, no idea what's causing the restore issue:

warning NU1701: Package 'PhotoSauce.MagicScaler 0.5.0' was restored using '.NETFramework,Version=v4.6.1, .NETFramework,Version=v4.6.2, .NETFramework,Version=v4.7, .NETFramework,Version=v4.7.1, .NETFramework,Version=v4.7.2, .NETFramework,Version=v4.8' instead of the project target framework '.NETCoreApp,Version=v3.1'. This package may not be fully compatible with your project.

A few more:

warning NU1604: Project dependency NetVips does not contain an inclusive lower bound. Include a lower bound in the dependency version to ensure consistent restore results.
warning NU1604: Project dependency NetVips.Native does not contain an inclusive lower bound. Include a lower bound in the dependency version to ensure consistent restore results.
warning NU1604: Project dependency PhotoSauce.MagicScaler does not contain an inclusive lower bound. Include a lower bound in the dependency version to ensure consistent restore results.

[antonfirsov]

Repeating the package versions in the .csproj seems to fix the issue: 2566879
Will post Posted an update with the MagicScaler numbers included.

@JimBobSquarePants any idea what's going wrong with the package references?

[JimBobSquarePants]

@antonfirsov Great stuff!!

I've fixed up the includes for you. Just waiting on a stable build (no OOME) for merging.

[br3aker]

Question. Isn't it better to save images to the memory stream per 'core' so it won't stress and degrade hdd/ssd? It should also eliminate disk write overhead.

[antonfirsov]

@br3aker I think the point of these benchmarks is to keep them end-to-end, and as close to the user scenario as possible, so including the physical stream parsing is part of the story. We have other benchmarks where do pre-load stuff.

[br3aker]

Got it, thanks for clarification!

[antonfirsov]

@JimBobSquarePants thanks for the help! If you have an explanation why did 17b0e3e fix things, I'm curious.

Looks like there is major difference in the big picture when it comes to progressive Jpegs. MagicScaler suffers very much from the WIC codec (we actually outperform it!), and ImageSharp's performance gap compared to Vips (= libjpeg-turbo) is also smaller:

This is the result of LoadResizeSaveStressBenchmarks filtered to progressive jpegs only:

Method	Parallellism	Mean time (ms)	Ratio to ImageSharp	Parallelism * Mean	Scalability (ideal: all lines are 1-s)
ImageSharp	1	4490	1.00	4490	1.00
ImageSharp	5	1200	1.00	6002	1.34
ImageSharp	10	835	1.00	8347	1.86
ImageSharp	20	652	1.00	13030	2.90
Magick	1	10639	2.37	10639	1.00
Magick	5	3359	2.80	16796	1.58
Magick	10	2219	2.66	22194	2.09
Magick	20	1901	2.92	38018	3.57
SystemDrawing	1	10177	2.27	10177	1.00
SystemDrawing	5	3233	2.69	16167	1.59
SystemDrawing	10	2827	3.39	28274	2.78
SystemDrawing	20	2687	4.12	53740	5.28
SkiaBitmap	1	5998	1.34	5998	1.00
SkiaBitmap	5	1605	1.34	8027	1.34
SkiaBitmap	10	1097	1.31	10965	1.83
SkiaBitmap	20	798	1.22	15956	2.66
NetVips	1	3072	0.68	3072	1.00
NetVips	5	835	0.70	4175	1.36
NetVips	10	575	0.69	5753	1.87
NetVips	20	576	0.88	11522	3.75
MagicScaler	1	4623	1.03	4623	1.00
MagicScaler	5	1307	1.09	6536	1.41
MagicScaler	10	903	1.08	9026	1.95
MagicScaler	20	678	1.04	13562	2.93

[JimBobSquarePants]

@antonfirsov i did notice you were referencing System.Drawing twice so I remove the older reference and normalised the entries to only include required packages in the individual libraries. Seemed to just work after then. (I wish the UI made these things easier)

I think our progressive decoder compares better because we go through the same mcu allocation process as other libraries there and our deficit compared to NetVips will likely be due to color conversion differences since we do more work. I think we’ll get a speed up when we optimised the baseline decoding allocation process.