[APM] Number of instances are limited to ~50-100 instances

[sorenlouv]

Related: https://github.com/elastic/sdh-apm/issues/1090 (internal)

Problem

The instance table on the Service Overview page is limited to 50 instances (it may show up to max 100 instances due to how items are fetched):

kibana/x-pack/plugins/apm/server/routes/services/get_service_instances/main_statistics.ts

Line 44 in 343c04c

size: 50,

This means that if the user has more than 50 instances they are not guarenteed to show up. If they have more than 100 instances they can be sure that not every instance shows up in the list.

Client side sorting issue
Having a limit is fine but in that case it is important that sorting happens on the server, in order to surface the "top instances" based on the selected sort dimension. Currently when the user clicks a table heading, sorting happens on the client which means that we don't necesarily surface the true "top instances" but only the instances already loaded.

Solution

• Rename header from "Instances" to "Top 100 instances"
• Ensure that sorting happens on the server instead of the client

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[crespocarlos]

@sqren , to sort on the server, did you have in mind using the order parameter from the terms aggs?

So, I briefly looked at the code and the queries run avg aggregations. Sorting by sub-aggregations is not safe and not recommended either, due to the possibility of ES producing wrong results (see here).I think this is the similar blocker we've had in the hosts view.

memory_usage_cgroup and memory_usage_system both are bucket script aggregations. I might be wrong, but I don't think it's possible by to sort Pipeline aggregations.

I'll keep investigating, but these are my first thoughts after looking into it.