Long Term Testing: Memory Leaks

[puckpuck]

After running a 4-week test on a stable EKS cluster using m6i.large nodes I have found the following services to be experiencing memory leaks:

• emailservice (severe)
• loadgenerator (moderate)
• redis (moderate)

This chart shows the 3 services' memory consumption and pod restarts over 3 weeks

This PR seems to be related to the emailservice though I'm unclear on it.
The redis leak is likey attributed to a lack of TTL when the cartservice is adding items to the cart
The loadgenerator may need further investigation to understand how and why the leak is being caused.

I have also found the frauddetectionservice to be experiencing a mild leak, though the service has not restarted after 4 weeks of running. I don't think we need to take any action on this at this point, but wanted to report it just in case.

[cartersocha]

Would basic garbage collection potentially help the loadgenerator?

[cartersocha]

@ahayworth I took a look at the threads Pierre linked and you responded to some of the comments about the ruby / otel memory leak. Was there every any resolution or work around?

[austinlparker]

Closing this as part of issue cleanup. If there's anything that needs to be done, please create a new issue.