SystemService and backend improvements

[a1ext]

1. gc collect is safer, avoid loading all the redis tasks into memory
2. optimization of metrics adjustment

The only place which stays unsafe is @ backend.py get_all_tasks:

tasks = self.redis.keys(f"{KARTON_TASK_NAMESPACE}:*")

At this place all the redis tasks keys are loaded into memory, which could lead to MemoryError/redis connection drop in case there are not enough RAM on the server (we faced such issue when there were 9Mil tasks)

[ITAYC0HEN]

Sweet!

[a1ext]

It seems like I didn't fix karton.core.inspect.KartonState, it's actual code makes few variables which will hold huge objects in RAM:

• self.tasks
• self.pending_tasks
• self.analyses

How this class is used?

[nazywam]

Hey, thanks for the changes. I like it!

AFAIK KartonState is used primarily in karton-dashboard.

I'll see how we could make it compatible with the changes you've proposed 👍

[a1ext]

Hey, thanks for the changes. I like it!

AFAIK KartonState is used primarily in karton-dashboard.

I'll see how we could make it compatible with the changes you've proposed 👍

I think it is better to not keep all the objects as fields, but add some funcs to get needed data, probably with pagination

[nazywam]

Yep, was thinking about the same thing. Even iterators would be okay as they are right now but we'd also need to add information about the number of tasks (of course this should be pretty light in comparison)

[nazywam]

Okay, did some digging and I think I've spotted all problematic/unoptimal code fragments.

• KartonQueue->pending_tasks should use the redis queue instead of loading all tasks and dividing them by the queue name
• KartonQueue->crashed_tasks should use a newly-created redis set of all crashed tasks for given queue
  • This of course needs to be implemented. We should also remember to clear the sets when recycling outdated crashed tasks
• KartonState->analyses needs to be implemented in a more efficient way. rakovskij-stanislav proposed an interesting solution here: Get task tree status #178 (comment). This would definitely speed up the process of handling analyses but we still have to determine how backwards-(non)-compatible that would be.

I like the direction where this is going though! 👍

[psrok1]

Continued in #207, where I reimplemented some of your changes. Thanks for your contribution!