misc: reduce memory usage of performance recalculation script

[TrueRou]

Cache every map during recalculation may cause memory lack on low-end server if there are too many maps (may eat many gigs of ram). It's better to make cache optional which can easily enable or disable by providing args (--use-cache or other something)

[tsunyoku]

maybe too high effort solution but we could use a cache which keeps the most frequently accessed maps and evicts the others to keep the memory down

[minisbett]

maybe instead of recalculating pp on a score id going upwards basis calculate it per map? so it goes through all maps in the maps table and recalcs all scores for that particular map?

[TrueRou]

maybe instead of recalculating pp on a score id going upwards basis calculate it per map? so it goes through all maps in the maps table and recalcs all scores for that particular map?

I agree with that, it just like the old way we calculate by command.

[cmyui]

is this query the problem?

bancho.py/tools/recalc.py

Lines 183 to 190 in 701c462

	scores = [
	dict(row)
	for row in await ctx.database.fetch_all(
	"SELECT scores.id, scores.mode, scores.mods, scores.acc, nmiss, scores.max_combo, scores.map_md5, scores.pp, maps.id as map_id FROM scores INNER JOIN maps ON scores.map_md5 = maps.md5 "
	"WHERE scores.status = 2 AND scores.mode = :mode ORDER BY scores.pp DESC",
	{"mode": mode},
	)
	]

looks like it's only chunked for processing once in memory which is not particularly useful, it should be pulled from sql in chunks to keep memory usage down

[minisbett]

The problem is that everytime a map gets recalced the whole map needs to be read by rosu-pp and therefore loaded into ram I think

[TrueRou]

is this query the problem?

bancho.py/tools/recalc.py

Lines 183 to 190 in 701c462

	scores = [
	dict(row)
	for row in await ctx.database.fetch_all(
	"SELECT scores.id, scores.mode, scores.mods, scores.acc, nmiss, scores.max_combo, scores.map_md5, scores.pp, maps.id as map_id FROM scores INNER JOIN maps ON scores.map_md5 = maps.md5 "
	"WHERE scores.status = 2 AND scores.mode = :mode ORDER BY scores.pp DESC",
	{"mode": mode},
	)
	]

looks like it's only chunked for processing once in memory which is not particularly useful, it should be pulled from sql in chunks to keep memory usage down

This won't take too much because millions of scores will only cost about 100 mb. It's worthwhile to cost memory in enchange of some io performance