Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TxPool v2 General architecture #2162

Merged
merged 83 commits into from
Oct 3, 2024
Merged

TxPool v2 General architecture #2162

merged 83 commits into from
Oct 3, 2024

Commits on Sep 4, 2024

  1. Add skeleton txpool v2

    AurelienFT committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    2e07d37 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Configuration menu
    Copy the full SHA
    4cbb11c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bef4aff View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Configuration menu
    Copy the full SHA
    552af83 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Configuration menu
    Copy the full SHA
    8d39499 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3d1a1bb View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3d65076 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    19830cb View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Configuration menu
    Copy the full SHA
    795567e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    67afb9d View commit details
    Browse the repository at this point in the history
  3. Add all tests

    AurelienFT committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    b956e2c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    9ed577e View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Configuration menu
    Copy the full SHA
    29ec06d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9a22983 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ca38a25 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    fe9e995 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    34c7e07 View commit details
    Browse the repository at this point in the history
  6. fmt fix

    AurelienFT committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    4576797 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    c8e86ec View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Configuration menu
    Copy the full SHA
    16a7e8f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    60ada65 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0cf8835 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b328bc8 View commit details
    Browse the repository at this point in the history
  5. fmt

    AurelienFT committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0d69dc1 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    75a9180 View commit details
    Browse the repository at this point in the history
  7. Fix comments in tests

    AurelienFT committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    87e5027 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. Configuration menu
    Copy the full SHA
    4e3e3ce View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    826f13e View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. Configuration menu
    Copy the full SHA
    bafbf75 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ed2302a View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Configuration menu
    Copy the full SHA
    d4ebf3c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    45d5296 View commit details
    Browse the repository at this point in the history
  3. Update cargo lock

    AurelienFT committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    1c1e5db View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Configuration menu
    Copy the full SHA
    6e5c7f8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2a5aabd View commit details
    Browse the repository at this point in the history
  3. Fix changelog

    AurelienFT committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    e78544f View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Configuration menu
    Copy the full SHA
    dd7c095 View commit details
    Browse the repository at this point in the history
  2. Update crates/services/txpool_v2/src/lib.rs

    Co-authored-by: Green Baneling <XgreenX9999@gmail.com>
    AurelienFT and xgreenx authored Sep 20, 2024
    Configuration menu
    Copy the full SHA
    14b9b53 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    68b1157 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    1074e5a View commit details
    Browse the repository at this point in the history
  5. fmt

    AurelienFT committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f89174f View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3ddea7e View commit details
    Browse the repository at this point in the history
  7. fmt

    AurelienFT committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    a5ff2ab View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    dd9ef6e View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    f0476e2 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    2f5a199 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. Configuration menu
    Copy the full SHA
    ade8569 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6beb83d View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. elllude lifetime

    AurelienFT committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    3dde832 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. update review comments

    AurelienFT committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    f5ef031 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2bd0382 View commit details
    Browse the repository at this point in the history
  3. remove wrong debug assert

    AurelienFT committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    e9c78fc View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f955ee5 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Configuration menu
    Copy the full SHA
    3be0fc7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    24cc034 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    aa8cb24 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0bbdaf5 View commit details
    Browse the repository at this point in the history
  5. Txpool v2 insertion (#2193)

    REVIEWER NOTE : Review #2162
    first for context
    ## Linked Issues/PRs
    Closes #2186 
    
    ## Description
    
    This PR allows you to insert `Transaction` in the `TxPool v2` service
    and so perform all the necessary verifications and the insertion.
    
    ### Changes in code logic from `TxPool v1`
    
    - The verifications are performed in a new order specified in #2186. The
    goal is to avoid making the computation heavy work if the simple checks
    aren't valid. In this new version we also ensure that verifications are
    done in order by having wrapper type around each step to allow only one
    verification path.
    
    - The insertion is performed in a separate thread pool, the goal is to
    not block the pool on any verifications/insertions and to manage the
    ressources we allocate to these works
    
    - The insertion rules and conditions has change to the following : 
    - A transaction with dependencies can collide only with one other
    transaction
    - A transaction without dependencies can collide with multiple
    transaction
      - Rules to free up space for new transaction
    - If a transaction is colliding with another verify if deleting the
    colliding transaction and dependents subtree is enough otherwise refuses
    the tx
    - If a transaction is dependent and not enough space, don't accept
    transaction
    - If a transaction is executable, try to free has much space used by
    less profitable transactions as possible in the pool to include it
    
    - New limits on the size of the pool : `max_pool_bytes_size and
    max_pool_gas`
    
    ### Changes from the base branch `txpool-v2`
    
    All the tests has been refactored to be way more clearer to understand
    and easier to read. They also now all follow the naming and the GWT
    convention.
    
    ## Checklist
    - [x] Breaking changes are clearly marked as such in the PR description
    and changelog
    - [x] New behavior is reflected in tests
    - [x] [The specification](https://github.com/FuelLabs/fuel-specs/)
    matches the implemented behavior (link update PR if changes are needed)
    
    ### Before requesting review
    - [x] I have reviewed the code myself
    - [x] I have created follow-up issues caused by this PR and linked them
    here
    
    ---------
    
    Co-authored-by: Green Baneling <XgreenX9999@gmail.com>
    AurelienFT and xgreenx authored Sep 27, 2024
    Configuration menu
    Copy the full SHA
    c128c75 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. Minimized the number of Result in the code in places where we can't…

    … get an error.
    
    Handled more gracefully cases when transactions is not found the storage.
    Improved performance of the collision check in the parents by collecting all parents once.
    Removed dependency collections from the public API of the storage. Now we have a separate `has_dependencies` method.
    Extracted the logic about collision verification to the Pool level.
    Fixed the issues with `find_free_space` where we could subtract the same subtree several times.
    Fixed the issues with non updating the `number_dependents_in_chain` and `dependents_cumulative_bytes_size`.
    Changed the return type for `get` function to be `Option` instead of `Result`. Added handling of `None`.
    Reworked `gather_best_txs` function to not abandon the old iteration and continue it instead since it is harder to manipulate the algorithm. But still it is still has the same worse complexity as before.
    Minimized the number of places where we pass the storage and storage id. Instead, pass the `&StorageData`. It gurantees that the caller already got storage values from the storage(and handled cases when it is `None`). It is more safe from API perspective.
    xgreenx committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    aeb2935 View commit details
    Browse the repository at this point in the history
  2. Added fuzzer for TxPoolV1 and TxPoolV2 to create dependent graphs. It…

    … creates a random graph and internal asserts catch the error or unexpected state. The tests verify at the end that all state is empty after full cycle of TxPoolV2 usage.
    
    Fixed several issues found by the fuzzer.
    
    Now `insert` method uses the output of the `can_insert_transaction` method. Now the logic is the same for insert and can insert(before it was a little bit different).
    xgreenx committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    8a746bf View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2024

  1. Configuration menu
    Copy the full SHA
    0fcb185 View commit details
    Browse the repository at this point in the history
  2. Make clippy happy

    xgreenx committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    88cec23 View commit details
    Browse the repository at this point in the history
  3. Moved the logic of collision and depdednencies verificaiton to the po…

    …ol instead of the storage. It allows to not bye aware of the collisions in the storage level.
    xgreenx committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    47ebb4c View commit details
    Browse the repository at this point in the history
  4. Revert rocksdb

    xgreenx committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    eed0d13 View commit details
    Browse the repository at this point in the history
  5. Fix tests

    xgreenx committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    d88a016 View commit details
    Browse the repository at this point in the history
  6. Reuse direct dedpednencies.

    Return the error earlier if transaciton has too many deodendencies
    xgreenx committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    2fd7c15 View commit details
    Browse the repository at this point in the history
  7. Merge branch 'refs/heads/master' into txpool-v2

    # Conflicts:
    #	crates/services/txpool/src/transaction_selector.rs
    xgreenx committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    99da36b View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Reject transactions that creates diamond dependencies. It allows simp…

    …lify the algorithm and to nto waste a lot of resources on cyclies handling.
    
    Removed subtracting of the collisions from the `can_fit_into_pool` method. Having this logic requires handling cases when collided transaction is sub transaction of the collision graph, or in the case of candidate graph in the `find_free_space` method.
    
    Added graph itegrity check to be sure that its state it valid after all manipualtions.
    
    Handle the case with zero inputs in stability test.
    
    Re-wrote the `remove_node_and_dependent_sub_graph_recursion` to be a bfs algorithm instead of dfs.
    xgreenx committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a540eda View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b25b832 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Configuration menu
    Copy the full SHA
    acf0ab5 View commit details
    Browse the repository at this point in the history
  2. Applied comments from the PR

    xgreenx committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    6df300a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d16424c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6fe6d98 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5def1d3 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3b7fc49 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. fix tests

    AurelienFT committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    48090ea View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1cd89e4 View commit details
    Browse the repository at this point in the history
  3. update insert return type

    AurelienFT committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    1a96d77 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. address tests reviews

    AurelienFT committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    de93d33 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    55e770f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e86ddad View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    414a8c7 View commit details
    Browse the repository at this point in the history
  5. remove copy bound

    AurelienFT committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    b7e93c0 View commit details
    Browse the repository at this point in the history