Better defaults for DocumentSubsetBitsetCache

[jpountz]

Initially the DLS cache was unbounded. This was not due to us being careless, but rather because Lucene makes the assumption that accessing live docs is cheap, and so we didn't it to run in linear time on cache miss. Unfortunately the ability to use templated role queries means you can have an unbounded number of bit sets in the cache, so we had to rollback this decision and move to a size-bound cache in order to avoid out-of-memory errors. We got some feedback that this has considerably slowed down some clusters, and the default of 50MB that this cache may use is way too low.

We should probably move to a number that is more generous, and also depends on the available memory on the node, e.g. 10%?

I wonder whether we should also decrease the default TTL. I think the TTL is important on this cache because it helps avoid keeping that cache full all the time in case some role queries are rarely used, which is probably typical when templated role queries are used. But the current default of 1 week means that someone running only one query every week would always use memory for this cache that might be better spent elsewhere. What about decreasing this default to maybe a couple hours?

[elasticmachine]

Pinging @elastic/es-security (:Security/Authorization)

[tvernum]

Broadly there's 2 options for cache sizing

1. make them relatively small, but with a long TTL so that it caters for objects that are used occasionally
2. make them large, but with a shorter TTL so that it caters for objects that are used multiple times over a short period, but then quickly release that memory.

(Small with a short TTL is also an option, but is not particularly helpful, and Large with a long ttl just ends up holding too much memory for too long).

When adding this cache, I opted for approach (1) based on the premise that some users/roles would only be used a few times a week.
I think in retrospect that was the wrong choice. This cache is most helpful when indices are large and calculating live-docs is relatively slow. The current defaults are so small that it sometimes cannot hold a single bitset for a large index and is effectively useless.

I think it makes sense to switch the defaults to large-weight+short-ttl because that matches the use cases where the cache is most needed.

[jpountz]

+1