Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fix file storage race condition #388

Merged
merged 84 commits into from
Nov 28, 2023
Merged

Commits on Oct 26, 2023

  1. WIP initial work

    gmaclennan committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    d687f1d View commit details
    Browse the repository at this point in the history
  2. rename Rpc to LocalPeers

    gmaclennan committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    74a77e2 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e301c85 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ae371fd View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    ccdf39f View commit details
    Browse the repository at this point in the history
  6. return replication stream

    gmaclennan committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    a52254b View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    50698e6 View commit details
    Browse the repository at this point in the history
  8. rename and re-organize

    gmaclennan committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    ae35e9c View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    be64a3d View commit details
    Browse the repository at this point in the history
  10. WIP initial work

    gmaclennan committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    ee2020e View commit details
    Browse the repository at this point in the history
  11. Tie everything together

    gmaclennan committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    0cd25da View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. rename getProjectInstance

    gmaclennan committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    395345d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7963a3e View commit details
    Browse the repository at this point in the history
  3. feat: add $sync API methods

    For now this simplifies the API (because we are only supporting local
    sync, not remote sync over the internet) to:
    
    - `project.$sync.getState()`
    - `project.$sync.start()`
    - `project.$sync.stop()`
    - Events
        - `sync-state`
    
    It's currently not possible to stop local discovery, nor is it possible
    to stop sync of the metadata namespaces (auth, config, blobIndex). The
    start and stop methods stop the sync of the data and blob namespaces.
    
    Fixes #134. Stacked on #360, #358 and #356.
    gmaclennan committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    9bff9ac View commit details
    Browse the repository at this point in the history
  4. feat: Add project.$waitForInitialSync() method

    Fixes Add project method to download auth + config cores #233
    
    Rather than call this inside the `client.addProject()` method, instead I
    think it is better for the API consumer to call
    `project.$waitForInitialSync()` after adding a project, since this
    allows the implementer to give user feedback about what is happening.
    gmaclennan committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    1c0dc6b View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    1f4fad2 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2023

  1. Configuration menu
    Copy the full SHA
    d00bf10 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    989b429 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bbda3f0 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0a32989 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. Configuration menu
    Copy the full SHA
    dc7a7c3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1895679 View commit details
    Browse the repository at this point in the history
  3. Add debug logging

    This was a big lift, but necessary to be able to debug sync issues since
    temporarily adding console.log statements was too much work, and
    debugging requires knowing the deviceId associated with each message.
    gmaclennan committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    763b57b View commit details
    Browse the repository at this point in the history
  4. fix timeout

    gmaclennan committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    2fb1072 View commit details
    Browse the repository at this point in the history
  5. fix: Add new cores to the indexer (!!!)

    This caused a day of work: a bug from months back
    gmaclennan committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    b5f1a00 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d3fb8ad View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    32c4d32 View commit details
    Browse the repository at this point in the history
  8. fix invite test

    gmaclennan committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    fe01caa View commit details
    Browse the repository at this point in the history
  9. keep blob cores sparse

    gmaclennan committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    5a47b56 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. Configuration menu
    Copy the full SHA
    9921f66 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. Merge branch 'main' into feat/initial-download

    * main:
      feat: add `$sync` API methods (#361)
      feat: `listLocalPeers()` & `local-peers` event (#360)
      feat: integrate LocalDiscovery & LocalPeers (#358)
      implement IconApi (#335)
      feat: MapeoRPC -> LocalPeers (#356)
      chore: update @mapeo/schema and @mapeo/sqlite-indexer (#357)
    gmaclennan committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    6dfa0d6 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. re-org sync and replication

    Removes old replication code attached to CoreManager
    Still needs tests to be updated
    gmaclennan committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    68455be View commit details
    Browse the repository at this point in the history
  2. update package-lock

    gmaclennan committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    7513788 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'main' into feat/initial-download

    * main:
      implement media server (#365)
      feat: implement icon server plugin (#315)
      fix: fix core storage initialization in MapeoManager (#367)
    gmaclennan committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    f58bbaf View commit details
    Browse the repository at this point in the history
  4. chore: Add debug logging

    gmaclennan committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    8c6a081 View commit details
    Browse the repository at this point in the history
  5. Merge branch 'chore/add-logging' into feat/initial-download

    * chore/add-logging:
      chore: Add debug logging
    gmaclennan committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    3bad62a View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8a63746 View commit details
    Browse the repository at this point in the history
  7. Get invite test working

    gmaclennan committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    ed69a78 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2023

  1. fix manager logger

    gmaclennan committed Nov 11, 2023
    Configuration menu
    Copy the full SHA
    7300da3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b2eda92 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. Configuration menu
    Copy the full SHA
    2224f2b View commit details
    Browse the repository at this point in the history
  2. Merge branch 'main' into fix/local-peers

    * main:
      fix: fix writing to blob store when also calculating content hash (#370)
      integrate icon plugin into MediaServer (#369)
    gmaclennan committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    97df1c0 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bc5e8d6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c1b473c View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    07cac79 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3407e0c View commit details
    Browse the repository at this point in the history
  7. Merge branch 'fix/local-peers' into feat/initial-download

    * fix/local-peers:
      new tests for duplicate connections
      fixed fake timers implementation for tests
      send invite to non-existent peer
      fix stream close before channel open
      fix: handle duplicate connections to LocalPeers
      fix: fix writing to blob store when also calculating content hash (#370)
      integrate icon plugin into MediaServer (#369)
      chore: Add debug logging (#373)
    gmaclennan committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    6174ce0 View commit details
    Browse the repository at this point in the history
  8. cleanup and small fix

    gmaclennan committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    8387f84 View commit details
    Browse the repository at this point in the history
  9. Better state debug logging

    gmaclennan committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    6a886b8 View commit details
    Browse the repository at this point in the history
  10. chain of invites test

    gmaclennan committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    d50fcea View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    5bf1941 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. fix: only request a core key from one peer

    Reduces the number of duplicate requests for the same keys.
    gmaclennan committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    34d2d34 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    158ac7c View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    687b9bc View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e6225fe View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. add e2e sync tests

    gmaclennan committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    a912ede View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. Configuration menu
    Copy the full SHA
    bbfe301 View commit details
    Browse the repository at this point in the history
  2. fix: don't open cores in sparse mode

    Turns out this changes how core.length etc. work, which confuses things
    gmaclennan committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    ea442a3 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    72f0dff View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a07c278 View commit details
    Browse the repository at this point in the history
  5. fix coreManager unit tests

    gmaclennan committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    800d136 View commit details
    Browse the repository at this point in the history
  6. fix blob store tests

    gmaclennan committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    3afafbb View commit details
    Browse the repository at this point in the history
  7. Merge branch 'main' into feat/initial-download

    * main:
      chore: pin more hyper deps (#381)
      chore: extract and test construction of icon url (#366)
      chore: fix release workflow configuration (#379)
      chore: remove unused/unneeded deps (#376)
      chore: remove authstore (#377)
      fix: handle duplicate connections to LocalPeers (#378)
    gmaclennan committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    ca60188 View commit details
    Browse the repository at this point in the history
  8. fix discovery-key event

    gmaclennan committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    420e21a View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    5cd349d View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    04520f6 View commit details
    Browse the repository at this point in the history
  11. fix datatype unit tests

    gmaclennan committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    53e388e View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    41895a7 View commit details
    Browse the repository at this point in the history
  13. remote peer-sync-controller unit test

    This is now tested in e2e tests
    gmaclennan committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    7df2c73 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    1133e75 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. ignore debug type errors

    gmaclennan committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    5d7c251 View commit details
    Browse the repository at this point in the history
  2. fixes for review comments

    gmaclennan committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    a2f8a1b View commit details
    Browse the repository at this point in the history
  3. move utils-new into utils

    gmaclennan committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    cd89ec4 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2023

  1. Merge branch 'main' into feat/initial-download

    * main:
      chore: fix Brittle type definitions (#383)
      feat: dbMigrationPathOpt - add db migration path to `MapeoProject` and `MapeoManager` (#384)
      [OPTIC-RELEASE-AUTOMATION] release/v9.0.0-alpha.2 (#382)
    gmaclennan committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    1f67ba8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    83ffe92 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4b0c71b View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2023

  1. Configuration menu
    Copy the full SHA
    d560d0a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6896021 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    aaa46f1 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8050b2d View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a4da0fb View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    a7e32b2 View commit details
    Browse the repository at this point in the history
  7. fix flakey test

    This test relied on `peer.connectedAt` changing in order to distinguish
    connections, but sometimes `connectedAt` was the same for both peers.
    This adds a 1ms delay before making the second connection, to attempt to
    stop the flakiness.
    gmaclennan committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    3931887 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. Merge branch 'main' into fix/file-storage-race-condition

    * main:
      feat: remove project.ready() (#392)
      feat: integrate sync and project invites (#362)
    gmaclennan committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    fdb28ab View commit details
    Browse the repository at this point in the history