Memory leak in ScriptEvaluate(string)

[Funbit]

Hello,
First of all, I want to say that I've been using StackExchange.Redis package for a couple of years without any problems. However, recently, I started experiencing strange memory leaks. Usually, I always used prepared LUA scripts (via LuaScript.Prepare), but after some point I had to introduce some manually generated LUA scripts. The new code I'm using looks like this:

var builder = new StringBuilder("redis.call('HMSET',");
builder.Append(...); 
// build some script here and then run it as string:
_connectionMultiplexer.GetDatabase().ScriptEvaluate(builder.ToString());

After several days of running, the server memory goes above 1GB (usually it stayed around 200MB). I dumped the process a couple of times and result is the same: there are many strings (500000+) with reference count = 1 in a Hashtable bucket. I'm not sure why it's there and why it cannot be freed (the script is not prepared, so it shouldn't be cached, etc).

I have attached a couple of screenshots (from MemoScope 0.9.999).

Is there any way to fix/avoid such behavior?

PS. If you need more information please let me know.
PPS. I'm using the latest version of the NuGet package (1.2.1).

Total string count:

This is how strings look like (builder.ToString() output):

Here is the reference map for each string:

[mgravell]

The expectation of the code is that you are going to be issuing the *same* parameterized commands over and over - it uses SCRIPT LOAD and EVALSHA to optimize things, and maintains a lookup of string to SHA mappings. I would be open to adding a "NoScriptCache" option to CommandFlags - with the intent of just issuing EVAL directly, but : before doing so, I'd like to understand the scenario more. What type of script is it that is so dynamic? Can it not be parameterized?

…

On 25 April 2017 at 11:19, Funbit ***@***.***> wrote: Hello, First of all, I want to say that I've been using StackExchange.Redis package for a couple of years without any problems. However, recently, I started experiencing strange memory leaks. Usually, I always used prepared LUA scripts (via LuaScript.Prepare), but after some point I had to introduce some manually generated LUA scripts. The new code I'm using looks like this: var builder = new StringBuilder("redis.call('HMSET',"); builder.Append(...); // build some script here and then run it as string: _connectionMultiplexer.GetDatabase().ScriptEvaluate(builder.ToString()); After several days of running, the server memory goes above 1GB (usually it stayed around 200MB). I dumped the process a couple of times and result is the same: there are many strings (500000+) with reference count = 1 in a Hashtable bucket. I'm not sure why it's there and why it cannot be freed (the script is not prepared, so it shouldn't be cached, etc). I have attached a couple of screenshots (from MemoScope 0.9.999). Is there any way to fix/avoid such behavior? PS. If you need more information please let me know. Total string count: [image: capture1] <https://cloud.githubusercontent.com/assets/5512433/25380544/8fed0f48-29eb-11e7-84aa-566d276a3c12.PNG> This is how strings look like (builder.ToString() output): [image: capture] <https://cloud.githubusercontent.com/assets/5512433/25380433/2967ea0e-29eb-11e7-8af4-6ee505cccdbc.PNG> Here is the reference map for each string: [image: capture2] <https://cloud.githubusercontent.com/assets/5512433/25380434/2968cfaa-29eb-11e7-936a-a2921f65932a.PNG> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#617>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABDsC5eJ2ZUx2_ZJSsVE47gi6pWJt4Yks5rzciUgaJpZM4NHRyB> .

-- Regards, Marc

[mgravell]

From the screenshot, it actually looks like an ideal case for parameterization - you can see https://redis.io/commands/eval for what this means in terms of the Lua - you just use KEYS[n] and ARGS[n] in the script to lookup values. The SE.Redis API exposes separate RedisKey[] keys and RedisValue[] values parameters on ScriptEvaluate that are mapped into KEYS and ARGS respectively.

As a minor note: HMSET is exposed directly via HashSet and HashSetAsync

[Funbit]

Thanks for the quick response!
Generally speaking, it's not that simple. The script I'm trying to evaluate looks like this:

redis.call('HMSET','key1','hashkey1','NEWVALUE','hashkey2','NEWVALUE', ...); 
redis.call('DEL','keys:hashkey2', ...);

HMSET sets "NEWVALUE" for some hash keys AND at the same time (i.e. as a single atomic transaction) DEL deletes some other keys. I'm not sure whether it's doable using script parameterization.

Probably, I can achieve the same effect via _connectionMultiplexer.GetDatabase().CreateTransaction() (can I?) instead of LUA, but it will help only this time. There's a big chance that at some point I would need to perform a condition ("if"/"for" etc) and use some local variables, which can be done only via LUA scripting.
So I think it would be great to have a new NoScriptCache option anyway.

[mgravell]

Targeting 1.2.3, later today

[mgravell]

https://www.nuget.org/packages/StackExchange.Redis/1.2.3

[Funbit]

Thank you very much for the update!