Listing huge number of users floods the heap

[espen42]

It seems that listing (or other kinds of aggregation?) huge numbers of users causes the heap to fill up, probably via an only semi-handled issue in nashorn.

When researching the multiple instances at support ticket 5445 and looking into the heap dump from a full heap (causing the garbage collector to choke the CPU), we found several calls to <domain>/admin/tool/com.enonic.xp.app.users/main/_/service/com.enonic.xp.app.users/graphql with queries like:

{"query":"query ($query: String, $itemIds: [String], $start: Int, $count: Int) {\n                    types (query: $query, itemIds: $itemIds, start: $start, count: $count) {\n                        totalCount\n                        aggregations {\n                            name,\n                            buckets {\n                                key,\n                                docCount\n                            }\n                        }\n                    }\n                }"}

This in turn made calls to java classes concerned with the nashorn issue above, for example WeakReference and SoftReference - in essence, nashorn handles JS objects with a large number (10000+) of fields badly and causes an unreasonably large strain on the heap. Multiple simultaneous calls/threads make this even worse.

The customer has over 75 000 users in the repo. When I log in and enter their user app, and try to expand the "Members" item, the same graphql endpoint is called...

{"query":"query($idprovider: String, $types: [PrincipalType], $start: Int, $count: Int, $sort: String) {\n                  principalsConnection (idprovider: $idprovider, types: $types, start: $start, count: $count, sort: $sort) {\n                        totalCount\n                    }\n                }","variables":{"types":["USER"],"idprovider":"members"}}

...and fails ("canceled" in the network tab of dev tools, probably a timeout?). Note how start and count aren't filled in, so default (or unbounded?) values are most likely used.

It should be possible to reproduce this by creating ~100 000 random users and repeating the call.

• Will it help to paginate these calls? @rymsha doubts it.
• Should it be the app-users handling this at all? The graphql endpoint is still exposed after admin-console login (should it be?), so it would still be "internally vulnerable" to custom calls.

[espen42]

Consequences of this:
#779
#822

[alansemenov]

@reisfmb we need a new query that uses exists to check whether idProvider has users and groups. this query should return true/false and be used instead of totalCount here

[sigdestad]

AFAIK: Using totalCount should not have any big impact on server as long as "count = 0" - i.e. not fetching any items, just the totalCount. Do we still have to rewrite this?

[rymsha]

... but not when count (size) is -1 which is the case after #779

Elasticsearch does have terminate_after search parameter, but we don't expose it, so my hopes to use "exists" are ruined.

Solution is to show "expand" arrow based on elements prefetched into the folder.

[sigdestad]

Why not change this to regular pagination in UI? List 100 at a time, with load more option?
Can you record a deviation that we somehow set this to -1