Create a memory benchmark for watcher

[clux]

What problem are you trying to solve?

Would like to know if a PR improves/degrades the memory profile of a watcher significantly.
We can't rely on people producing such excellent manual benchmarks such as in #1494.

Describe the solution you'd like

Not sure if this would work so take this with a grain of salt. Any practical suggestions would be considered.

• cargo bench with some custom timings (maybe counts can help)
• actual test spawning 10k generated ConfigMaps and sending through watcher synthetically
• checking memory usage with dhat-rs
• some kind of benchmark comparison action to feedback it into PRs if more than 5% different e.g. https://github.com/benchmark-action/github-action-benchmark

Describe alternatives you've considered

Have only really skimmed https://nnethercote.github.io/perf-book/profiling.html
I'm sure there are better solutions.

Target crate for feature

kube-runtime

[clux]

Case-in-point: the buffer follow-up in #1504 making individual pages in watcher held as a slowly draining VecDeque may actually have undone a portion of the memory improvements in the original PR, but I only e2e tested one controller case at the time.

You can still get the gains by lowering the page_size (as noted in the results section of the 0.92 blog post), or using streaming lists, so am still happy with the state of things (got both ergonomic improvements and big memory gains). But it was just one of those unwelcome surprises that made me doubt myself quite a bit yesterday when going over the benchmarks.