feat: dataType.on('updated-docs') when docs update

[gmaclennan]

Emits an array of updated docs for the data type whenever the docs
update (rather than emitting one doc at a time, it waits for indexing to
complete, then emits all the docs that have been updated. If there are
several updates to a particular docId, then only the resolved head
version is returned).

Attaching this listener to a data type that has many values / is
frequently updated will have a performance impact, because large amounts
of data could be read from the database and there is no limit, e.g. if
this was attached to the observation dataType and a new device synced
with a project with 10,000 observations, an attached listener would be
called with 10,000 observations.

The main reason for this feature is for internal use for capabilities,
listening for changes and modifying permissions accordingly, initially in order to implement #268, which requires adding cores as device capabilities are updated.

This is implemented in a particular way with performance considerations in mind. My initial idea was to return all decoded indexed docs from the index writer, but this would have a high performance cost because:

1. the plan is to move the index writer to a separate worker thread, and sending all docs back over the worker message channel would be expensive.
2. we are not interested in subscribing to all doc updates (it wouldn't make sense to listen for all observations being updated because it would be too many, particularly after a large sync),
3. The docs that are indexed could contain multiple versions of a doc id (e.g. all the edits to that doc) and for an 'update' event we're only actually interested in the resolved "head" doc, so we have to re-request the doc from the index anyway (e.g. getByDocId(), which returns the "head").

So, the indexer now returns the docIds of the documents it indexed, then the DataStore de-dupes those and emits them for each schemaName (but only if a listener is attached) when indexing finishes, and the DataType class listens for updated doc ids (but only if a listener is attached) and looks up the docs by those docIds, and emits the updated-docs event.

[achou11]

lgtm. seems like you ran into some potentially gnarly TS issues - think you made the right choice to ignore them 😄

[gmaclennan]

lgtm. seems like you ran into some potentially gnarly TS issues - think you made the right choice to ignore them 😄

Yeah, I wasted quite a lot of time trying to make TS happy, then just had a realization "why am I bothering". In the internals of functions like these dealing with complex data types, I don't think jumping through hoops to keep TS happy really gains us anything, especially we're hitting limitations of TS. The issue I keep getting frustrated by is trying to narrow a type with arrayOfValidTypes.includes(maybeValidType), but dependent types is often an issue (TS does not support them).