Remove some more joins on room_memberships

[erikjohnston]

This involves changing the way we filter out forgotten rooms, which needed a new index.

These commits are independently reviewable.

[codecov]

Codecov Report

Merging #5752 into develop will increase coverage by 0.03%.
The diff coverage is 84.21%.

@@            Coverage Diff             @@
##           develop   #5752      +/-   ##
==========================================
+ Coverage    63.36%   63.4%   +0.03%     
==========================================
  Files          331     331              
  Lines        36399   36424      +25     
  Branches      6007    6009       +2     
==========================================
+ Hits         23063   23093      +30     
+ Misses       11697   11692       -5     
  Partials      1639    1639

[richvdh]

so is this in aid of trying to support unforgetting of rooms? is that a thing that's possible?

[erikjohnston]

Yup, you can just rejoin the room (I believe)

[richvdh]

actually could you update the comment to make this a bit clearer?

[richvdh]

won't this include membership events that happened before the "forget"?

[erikjohnston]

Ah, the non-obvious way forgetting works is to mark all room memberships with forgotten = 1, so to check if a room is forgotten or not is a case of checking if any memberships have forgotten = 0

[richvdh]

this comment looks a bit lost?

[erikjohnston]

Err, yes :/

[richvdh]

lgtm, once the CI is sorted out

[richvdh]

this is going to be very painful until that background reindex completes, isn't it.

[erikjohnston]

It takes about ~500ms to 900ms for large accounts, small accounts take a few ms.

The query plan does a bitmap scan across room_id and user_id indices:

matrix=> explain analyze SELECT room_id, (
                    SELECT count(*) FROM room_memberships
                    WHERE room_id = m.room_id AND user_id = m.user_id AND forgotten = 0
                ) AS count
                FROM room_memberships AS m
                WHERE user_id = '@e:matrix.org' AND forgotten = 1
                GROUP BY room_id, user_id;
                                                                         QUERY PLAN                                                                          
-------------------------------------------------------------------------------------------------------------------------------------------------------------
 Group  (cost=2518.90..4262.71 rows=4 width=68) (actual time=0.412..0.412 rows=0 loops=1)
   Group Key: m.room_id, m.user_id
   ->  Sort  (cost=2518.90..2518.91 rows=4 width=60) (actual time=0.411..0.411 rows=0 loops=1)
         Sort Key: m.room_id
         Sort Method: quicksort  Memory: 25kB
         ->  Index Scan using room_memberships_user_id on room_memberships m  (cost=0.57..2518.86 rows=4 width=60) (actual time=0.407..0.407 rows=0 loops=1)
               Index Cond: (user_id = '@e:matrix.org'::text)
               Filter: (forgotten = 1)
               Rows Removed by Filter: 46
   SubPlan 1
     ->  Aggregate  (cost=435.94..435.95 rows=1 width=8) (never executed)
           ->  Bitmap Heap Scan on room_memberships  (cost=433.92..435.93 rows=1 width=0) (never executed)
                 Recheck Cond: ((user_id = m.user_id) AND (room_id = m.room_id))
                 Filter: (forgotten = 0)
                 ->  BitmapAnd  (cost=433.92..433.92 rows=1 width=0) (never executed)
                       ->  Bitmap Index Scan on room_memberships_user_id  (cost=0.00..32.24 rows=1289 width=0) (never executed)
                             Index Cond: (user_id = m.user_id)
                       ->  Bitmap Index Scan on room_memberships_room_id  (cost=0.00..401.43 rows=17981 width=0) (never executed)
                             Index Cond: (room_id = m.room_id)
 Planning time: 0.144 ms
 Execution time: 0.468 ms
(21 rows)