fix: fix file storage race condition #388
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For now this simplifies the API (because we are only supporting local
sync, not remote sync over the internet) to:
- `project.$sync.getState()`
- `project.$sync.start()`
- `project.$sync.stop()`
- Events
- `sync-state`
It's currently not possible to stop local discovery, nor is it possible
to stop sync of the metadata namespaces (auth, config, blobIndex). The
start and stop methods stop the sync of the data and blob namespaces.
Fixes #134. Stacked on #360, #358 and #356.
Fixes Add project method to download auth + config cores #233 Rather than call this inside the `client.addProject()` method, instead I think it is better for the API consumer to call `project.$waitForInitialSync()` after adding a project, since this allows the implementer to give user feedback about what is happening.
This was a big lift, but necessary to be able to debug sync issues since temporarily adding console.log statements was too much work, and debugging requires knowing the deviceId associated with each message.
This caused a day of work: a bug from months back
gmaclennan
added a commit
that referenced
this pull request
Nov 28, 2023
gmaclennan
added a commit
that referenced
this pull request
Nov 29, 2023
gmaclennan
added a commit
that referenced
this pull request
Nov 29, 2023
This was referenced Apr 11, 2024
This was referenced Jun 20, 2024
This was referenced Jul 25, 2024
This was referenced Aug 21, 2024
This was referenced Aug 29, 2024
This was referenced Sep 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stacked on #386 -> #362
Fixes #387
Switching to file storage exposes a race condition when writing documents (create or update). The indexer relies on the hypercore
appendevent, which can fire beforecore.append()resolves, which results in the indexer batch function resolving beforecore.append()resolves. Because we wait for the indexer to index the document before resolving the create or update method, this situation leads to a hang. The clunky solution is to await any pending appends first, so that we know that the deferred promise is set, and then the indexer batch can resolve it.Switching to file storage unfortunately causes the e2e sync tests to fail because of other reasons - sometimes the indexing does not complete before we read the synced docs, which results in failed tests. Because of this, in this PR I have switched off file storage for tests. With file storage on (
FAST_TESTSset tofalse), the project-crud.js test passes (it does not before this PR), but the sync.js is failing. This is fixed by #389 (stacked on this PR), which turns on file storage for tests.