Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an off-chain worker #1671

xgreenx · 2024-02-15T15:52:17Z

The change moves the FuelBlockSecondaryKeyBlockHeights table and related logic to the off-chain worker. Along with this, the change adds a new Merklelized blueprint that maintains the binary Merkle tree over the storage data. It supports only the insertion of the objects without removing them. This blueprint replaces the logic that previously was defined in the Database.

Some side changes caused by the main change:

Now, each blueprint provides its own set of tests related to the logic of this blueprint.
The TransactionStatus uses block_height inside instead of the block_id.

The key for the dense merkle tree looks like:

pub enum DenseMetadataKey<PrimaryKey> {
    /// The primary key of the `DenseMerkleMetadata`.
    Primary(PrimaryKey),
    #[default]
    /// The latest `DenseMerkleMetadata` of the table.
    Latest,
}

…-chain worker

crates/fuel-core/Cargo.toml

# Conflicts: # crates/fuel-core/src/database/storage.rs # crates/fuel-core/src/graphql_api/database.rs # crates/fuel-core/src/graphql_api/ports.rs # crates/fuel-core/src/graphql_api/storage.rs # crates/fuel-core/src/graphql_api/worker_service.rs # crates/storage/src/column.rs

MitchTurner · 2024-02-17T20:28:20Z

crates/client/src/client.rs

@@ -745,9 +745,12 @@ impl FuelClient {
        Ok(block)
    }

-    pub async fn block_by_height(&self, height: u32) -> io::Result<Option<types::Block>> {
+    pub async fn block_by_height(


The types on latest_gas_price include a u32 to represent block height. Should we use BlockHeight there?

MitchTurner · 2024-02-17T21:04:05Z

crates/storage/src/blueprint/merklelized.rs

+/// The macro that generates basic storage tests for the table with the merklelized structure.
+/// It uses the [`InMemoryStorage`](crate::structured_storage::test::InMemoryStorage).
+#[macro_export]
+macro_rules! basic_merklelized_storage_tests {


Should these macros be behind a test-helpers flag?

MitchTurner · 2024-02-17T21:23:01Z

crates/types/src/blockchain/header.rs

@@ -112,26 +112,31 @@ impl BlockHeader {
    /// Set the block height for the header
    pub fn set_block_height(&mut self, height: BlockHeight) {
        self.consensus_mut().height = height;
+        self.recalculate_metadata();


It looks like we have this method in a separate impl block for some reason. In fact, we have multiple blocks for BlockHeader have the same (no) constraints. Can we combine the impl blocks?

They are segregated by their business logic. I'm okay with joining them into one/two impl blocks, but since they are in different places in the file, I prefer to not shuffle them in this PR to minimize number of changes=)

Okay. I just got confused where a method was just reading the PR diffs, but with an IDE it doesn't really affect anything.

MitchTurner · 2024-02-17T21:29:55Z

crates/storage/src/tables.rs

+    pub enum DenseMetadataKey<PrimaryKey> {
+        /// The primary key of the `DenseMerkleMetadata`.
+        Primary(PrimaryKey),
+        #[default]
+        /// The latest `DenseMerkleMetadata` of the table.
+        Latest,
+    }


I want to understand this better. If Latest gives you the entire latest Merkle tree, the PrimaryKey gives you a previous tree? Or a subsection?

Maybe this would be obvious if I understood the MTs terminology better.

Yes. The Merkle tree that we are using here is append-only. Because of that, we can only add new stuff at the end, and the Latest key allows us to do that. At the same time, appending a new leaf doesn't invalidate all previous Merkle trees, and PrimaryKey allows us to get the valid MerkleTree at some Key that we added before. For example, in the case of the FuelBlocks, we can get a valid Merkle tree at height X, X-1, .., X - Y and use it in proofs that some block is part of the Merkle tree at a specific height.

Voxelot · 2024-02-18T04:00:10Z

crates/types/src/blockchain/header.rs

    }

    /// Set the previous root for the header
    pub fn set_previous_root(&mut self, root: Bytes32) {
        self.consensus_mut().prev_root = root;
+        self.recalculate_metadata();


Should we use a builder in the hot paths instead of calling these setters one by one, now that they all have to recalculate the application header hash? We could save several hashing calls in the executor by calculating this after all the fields are set.

We use these methods only in the tests, so it doesn't hurt us=) I will put them under test-helper feature

Voxelot · 2024-02-18T04:01:46Z

crates/types/src/services/txpool.rs

@@ -180,7 +180,7 @@ pub enum TransactionStatus {
    /// Transaction was successfully included in a block
    Success {
        /// Included in this block
-        block_id: BlockId,
+        block_height: BlockHeight,


If there's a possibility of reorgs in the future, would only using block height on the api be problematic?

With a shared sequencer, I don't see how we can occur fork with several valid blocks for the same height. It is possible to have invalid blocks and valid blocks, but we will skip invalid due to invalidity and commit only a valid one.

so if there are several proposed block ids for the same block height, and a user tries to see the block containing their tx by height, the tx might not exist in the block that actually gets accepted right?

Yes, the transaction has Committed status only when the corresponding block is finalized by the shared sequencer. Until finalisation point it is TxPool level and we can only provide soft confirmations without block information

Voxelot · 2024-02-18T05:25:28Z

crates/storage/src/blueprint/merklelized.rs

+        $crate::paste::item! {
+        #[cfg(test)]
+        #[allow(unused_imports)]
+        mod [< $table:snake _basic_tests >] {


These look identical to the basic storage tests, should we just reuse those here?

Also, perhaps we should have some merkle tree related tests as part of this macro? I.e similar to the root tests used on the sparse structures?

The tests are not the same since the removal of elements is not allowed=)

I moved old tests for the FuelBlocks table in the blocks.rs instead of adding new tests here. I did it for the simplicity of the PR, but if you think it is better to make it part of this macro now, then later, I will do it=)

bvrooman

Reviewed 22/47 files - will continue the review at a later point. A few questions/comments so far.

CHANGELOG.md

crates/client/src/client/types.rs

bvrooman · 2024-02-19T15:58:08Z

crates/client/src/client/schema/tx.rs

@@ -178,7 +178,7 @@ pub struct SubmittedStatus {
 #[cynic(schema_path = "./assets/schema.sdl")]
 pub struct SuccessStatus {
    pub transaction_id: TransactionId,
-    pub block: BlockIdFragment,
+    pub block: BlockHeightFragment,


nit: Maybe this should be named block_height now?

It is named block since from the GraphQL perspective you receive the block=) The block_height is ID, but it could be BlockId as well(both works)

Agreed - since this could return a full block, we must use a fragment query here to only return a partial view of the object.

bvrooman · 2024-02-19T16:20:01Z

crates/fuel-core/src/graphql_api/storage/blocks.rs

+/// It links the `BlockId` to corresponding `BlockHeight`.
+pub struct FuelBlockSecondaryKeyBlockHeights;
+
+impl Mappable for FuelBlockSecondaryKeyBlockHeights {


If I understand correctly, this simply maps BlockId to BlockHeight. The naming suggests that this is a secondary storage for BlockHeight, i.e. a second way to query BlockHeight. The metadata table is the first way, and metadata is on-chain.

Since BlockHeight is now off-chain, I don't think it is "secondary" - we have separate on-chain and off-chain domains. I think now this could just be called BlockHeights, which is more clear to the reader.

maybe FuelBlockIdsToHeights? I also find it confusing to parse out from this table name what it's actually doing without having to inspect the blueprint.

bvrooman · 2024-02-19T16:23:45Z

crates/storage/src/column.rs

    /// See [`SealedBlockConsensus`](crate::tables::SealedBlockConsensus)
-    FuelBlockConsensus = 19,
+    FuelBlockConsensus = 18,


Will this be a breaking change for existing databases and require a migration? My understanding is that we have the manual numbering to make sure that there isn't any breakage when adding/removing columns. Otherwise, we could just use the default implicit numbering.

Since we don't have mainnet it's okay to break the numbering and invalidate the database. But yeah, later we will try to avoid breaking stuff like this

bvrooman · 2024-02-19T16:44:29Z

crates/fuel-core/src/query/block.rs

 }

-impl<D: OnChainDatabase + ?Sized> SimpleBlockData for D {
+impl<D: OnChainDatabase + OffChainDatabase + ?Sized> SimpleBlockData for D {


It seems a bit odd that we are giving access to on-chain data to off-chain views. Shouldn't on-chain and off-chain databases be fully self-sufficient?

It looks like this constraint isn't even needed. I removed it from here and line 53 impl<D: OnChainDatabase + ?Sized> BlockQueryData for D { and the tests passed.

# Conflicts: # CHANGELOG.md # crates/types/src/services/txpool.rs

@Dentosal

## Version v0.23.0 ### Added - [#1713](#1713): Added automatic `impl` of traits `StorageWrite` and `StorageRead` for `StructuredStorage`. Tables that use a `Blueprint` can be read and written using these interfaces provided by structured storage types. - [#1671](#1671): Added a new `Merklized` blueprint that maintains the binary Merkle tree over the storage data. It supports only the insertion of the objects without removing them. - [#1657](#1657): Moved `ContractsInfo` table from `fuel-vm` to on-chain tables, and created version-able `ContractsInfoType` to act as the table's data type. ### Changed - [#1723](#1723): Notify about imported blocks from the off-chain worker. - [#1717](#1717): The fix for the [#1657](#1657) to include the contract into `ContractsInfo` table. - [#1657](#1657): Upgrade to `fuel-vm` 0.46.0. - [#1671](#1671): The logic related to the `FuelBlockIdsToHeights` is moved to the off-chain worker. - [#1663](#1663): Reduce the punishment criteria for mempool gossipping. - [#1658](#1658): Removed `Receipts` table. Instead, receipts are part of the `TransactionStatuses` table. - [#1640](#1640): Upgrade to fuel-vm 0.45.0. - [#1635](#1635): Move updating of the owned messages and coins to off-chain worker. - [#1650](#1650): Add api endpoint for getting estimates for future gas prices - [#1649](#1649): Add api endpoint for getting latest gas price - [#1600](#1640): Upgrade to fuel-vm 0.45.0 - [#1633](#1633): Notify services about importing of the genesis block. - [#1625](#1625): Making relayer independent from the executor and preparation for the force transaction inclusion. - [#1613](#1613): Add api endpoint to retrieve a message by its nonce. - [#1612](#1612): Use `AtomicView` in all services for consistent results. - [#1597](#1597): Unify namespacing for `libp2p` modules - [#1591](#1591): Simplify libp2p dependencies and not depend on all sub modules directly. - [#1590](#1590): Use `AtomicView` in the `TxPool` to read the state of the database during insertion of the transactions. - [#1587](#1587): Use `BlockHeight` as a primary key for the `FuelsBlock` table. - [#1585](#1585): Let `NetworkBehaviour` macro generate `FuelBehaviorEvent` in p2p - [#1579](#1579): The change extracts the off-chain-related logic from the executor and moves it to the GraphQL off-chain worker. It creates two new concepts - Off-chain and On-chain databases where the GraphQL worker has exclusive ownership of the database and may modify it without intersecting with the On-chain database. - [#1577](#1577): Moved insertion of sealed blocks into the `BlockImporter` instead of the executor. - [#1574](#1574): Penalizes peers for sending invalid responses or for not replying at all. - [#1601](#1601): Fix formatting in docs and check that `cargo doc` passes in the CI. - [#1636](#1636): Add more docs to GraphQL DAP API. #### Breaking - [#1725](#1725): All API endpoints now are prefixed with `/v1` version. New usage looks like: `/v1/playground`, `/v1/graphql`, `/v1/graphql-sub`, `/v1/metrics`, `/v1/health`. - [#1722](#1722): Bugfix: Zero `predicate_gas_used` field during validation of the produced block. - [#1714](#1714): The change bumps the `fuel-vm` to `0.47.1`. It breaks several breaking changes into the protocol: - All malleable fields are zero during the execution and unavailable through the GTF getters. Accessing them via the memory directly is still possible, but they are zero. - The `Transaction` doesn't define the gas price anymore. The gas price is defined by the block producer and recorded in the `Mint` transaction at the end of the block. A price of future blocks can be fetched through a [new API nedopoint](#1641) and the price of the last block can be fetch or via the block or another [API endpoint](#1647). - The `GasPrice` policy is replaced with the `Tip` policy. The user may specify in the native tokens how much he wants to pay the block producer to include his transaction in the block. It is the prioritization mechanism to incentivize the block producer to include users transactions earlier. - The `MaxFee` policy is mandatory to set. Without it, the transaction pool will reject the transaction. Since the block producer defines the gas price, the only way to control how much user agreed to pay can be done only through this policy. - The `maturity` field is removed from the `Input::Coin`. The same affect can be achieve with the `Maturity` policy on the transaction and predicate. This changes breaks how input coin is created and removes the passing of this argument. - The metadata of the `Checked<Tx>` doesn't contain `max_fee` and `min_fee` anymore. Only `max_gas` and `min_gas`. The `max_fee` is controlled by the user via the `MaxFee` policy. - Added automatic `impl` of traits `StorageWrite` and `StorageRead` for `StructuredStorage`. Tables that use a `Blueprint` can be read and written using these interfaces provided by structured storage types. - [#1712](#1712): Make `ContractUtxoInfo` type a version-able enum for use in the `ContractsLatestUtxo`table. - [#1657](#1657): Changed `CROO` gas price type from `Word` to `DependentGasPrice`. The dependent gas price values are dummy values while awaiting updated benchmarks. - [#1671](#1671): The GraphQL API uses block height instead of the block id where it is possible. The transaction status contains `block_height` instead of the `block_id`. - [#1675](#1675): Simplify GQL schema by disabling contract resolvers in most cases, and just return a ContractId scalar instead. - [#1658](#1658): Receipts are part of the transaction status. Removed `reason` from the `TransactionExecutionResult::Failed`. It can be calculated based on the program state and receipts. Also, it is not possible to fetch `receipts` from the `Transaction` directly anymore. Instead, you need to fetch `status` and its receipts. - [#1646](#1646): Remove redundant receipts from queries. - [#1639](#1639): Make Merkle metadata, i.e. `SparseMerkleMetadata` and `DenseMerkleMetadata` type version-able enums - [#1632](#1632): Make `Message` type a version-able enum - [#1631](#1631): Modify api endpoint to dry run multiple transactions. - [#1629](#1629): Use a separate database for each data domain. Each database has its own folder where data is stored. - [#1628](#1628): Make `CompressedCoin` type a version-able enum - [#1616](#1616): Make `BlockHeader` type a version-able enum - [#1614](#1614): Use the default consensus key regardless of trigger mode. The change is breaking because it removes the `--dev-keys` argument. If the `debug` flag is set, the default consensus key will be used, regardless of the trigger mode. - [#1596](#1596): Make `Consensus` type a version-able enum - [#1593](#1593): Make `Block` type a version-able enum - [#1576](#1576): The change moves the implementation of the storage traits for required tables from `fuel-core` to `fuel-core-storage` crate. The change also adds a more flexible configuration of the encoding/decoding per the table and allows the implementation of specific behaviors for the table in a much easier way. It unifies the encoding between database, SMTs, and iteration, preventing mismatching bytes representation on the Rust type system level. Plus, it increases the re-usage of the code by applying the same blueprint to other tables. It is a breaking PR because it changes database encoding/decoding for some tables. ### StructuredStorage The change adds a new type `StructuredStorage`. It is a wrapper around the key-value storage that implements the storage traits(`StorageInspect`, `StorageMutate`, `StorageRead`, etc) for the tables with blueprint. This blueprint works in tandem with the `TableWithBlueprint` trait. The table may implement `TableWithBlueprint` specifying the blueprint, as an example: ```rust impl TableWithBlueprint for ContractsRawCode { type Blueprint = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } ``` It is a definition of the blueprint for the `ContractsRawCode` table. It has a plain blueprint meaning it simply encodes/decodes bytes and stores/loads them into/from the storage. As a key codec and value codec, it uses a `Raw` encoding/decoding that simplifies writing bytes and loads them back into the memory without applying any serialization or deserialization algorithm. If the table implements `TableWithBlueprint` and the selected codec satisfies all blueprint requirements, the corresponding storage traits for that table are implemented on the `StructuredStorage` type. ### Codecs Each blueprint allows customizing the key and value codecs. It allows the use of different codecs for different tables, taking into account the complexity and weight of the data and providing a way of more optimal implementation. That property may be very useful to perform migration in a more easier way. Plus, it also can be a `no_std` migration potentially allowing its fraud proving. An example of migration: ```rust /// Define the table for V1 value encoding/decoding. impl TableWithBlueprint for ContractsRawCodeV1 { type Blueprint = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } /// Define the table for V2 value encoding/decoding. /// It uses `Postcard` codec for the value instead of `Raw` codec. /// /// # Dev-note: The columns is the same. impl TableWithBlueprint for ContractsRawCodeV2 { type Blueprint = Plain<Raw, Postcard>; fn column() -> Column { Column::ContractsRawCode } } fn migration(storage: &mut Database) { let mut iter = storage.iter_all::<ContractsRawCodeV1>(None); while let Ok((key, value)) = iter.next() { // Insert into the same table but with another codec. storage.storage::<ContractsRawCodeV2>().insert(key, value); } } ``` ### Structures The blueprint of the table defines its behavior. As an example, a `Plain` blueprint simply encodes/decodes bytes and stores/loads them into/from the storage. The `SMT` blueprint builds a sparse merkle tree on top of the key-value pairs. Implementing a blueprint one time, we can apply it to any table satisfying the requirements of this blueprint. It increases the re-usage of the code and minimizes duplication. It can be useful if we decide to create global roots for all required tables that are used in fraud proving. ```rust impl TableWithBlueprint for SpentMessages { type Blueprint = Plain<Raw, Postcard>; fn column() -> Column { Column::SpentMessages } } | | \|/ impl TableWithBlueprint for SpentMessages { type Blueprint = Sparse<Raw, Postcard, SpentMessagesMerkleMetadata, SpentMessagesMerkleNodes>; fn column() -> Column { Column::SpentMessages } } ``` ### Side changes #### `iter_all` The `iter_all` functionality now accepts the table instead of `K` and `V` generics. It is done to use the correct codec during deserialization. Also, the table definition provides the column. #### Duplicated unit tests The `fuel-core-storage` crate provides macros that generate unit tests. Almost all tables had the same test like `get`, `insert`, `remove`, `exist`. All duplicated tests were moved to macros. The unique one still stays at the same place where it was before. #### `StorageBatchMutate` Added a new `StorageBatchMutate` trait that we can move to `fuel-storage` crate later. It allows batch operations on the storage. It may be more performant in some cases. - [#1573](#1573): Remove nested p2p request/response encoding. Only breaks p2p networking compatibility with older fuel-core versions, but is otherwise fully internal. ## What's Changed * Fix the example tx in README.md by @Dentosal in #1543 * Weekly `cargo update` by @github-actions in #1557 * Update LICENSE by @Voxelot in #1558 * Remove duplicating logic in the `KeyValueStore` trait by @xgreenx in #1559 * Move `KeyValueStore` to the `fuel-core-storage` crate by @xgreenx in #1566 * Update `libp2p` from `0.50.0` to `0.53.1` by @MitchTurner in #1379 * Weekly `cargo update` by @github-actions in #1575 * Weekly `cargo update` by @github-actions in #1578 * Moved insertion of the blocks into the `BlockImporter` instead of the executor by @xgreenx in #1577 * Let `NetworkBehaviour` macro generate `FuelBehaviorEvent` in p2p by @MitchTurner in #1585 * Remove RustSec Ignore for RUSTSEC-2022-0093 by @MitchTurner in #1586 * Simplify p2p request/response message serialization by @Dentosal in #1573 * Weekly `cargo update` by @github-actions in #1588 * Depend on libp2p directly instead of submodules by @MitchTurner in #1591 * Make `Block` versionable by @MitchTurner in #1593 * Switch katyo/publish-crates@v2 to xgreenx/publish-crates@v1 for the CI by @Dentosal in #1603 * Weekly `cargo update` by @github-actions in #1599 * Fix cargo doc, check for cargo doc in the CI by @Dentosal in #1601 * Move storage traits implementation to the `fuel-core-storage` crate by @xgreenx in #1576 * Extract off chain logic from the executor by @xgreenx in #1579 * Applying comments from the PR for the storage crate by @xgreenx in #1610 * Use `BlockHeight` as a primary key for the `FuelsBlock` table by @xgreenx in #1587 * Use `AtomicView` in the `TxPool` by @xgreenx in #1590 * Unify namespacing in P2P codebase by @MitchTurner in #1597 * Weekly `cargo update` by @github-actions in #1611 * Make `Consensus` version-able by @MitchTurner in #1596 * feat: add api endpoint to retrieve messages by nonce by @matt-user in #1613 * Update Docker tags for Flux integration by @liviudm in #1617 * Remove dev consensus key by @MujkicA in #1614 * Make `BlockHeader` a versionable enum by @MitchTurner in #1616 * feat: Versionable CompressedCoin by @bvrooman in #1628 * Use `AtomicView` in all services by @xgreenx in #1612 * Notify services about importing of the genesis block by @xgreenx in #1633 * Weekly `cargo update` by @github-actions in #1634 * feat: Versionable `Message` by @bvrooman in #1632 * Making relayer independent from the executor by @xgreenx in #1625 * Upgrade to the fuel-vm 0.44.0 by @Dentosal in #1600 * Upgrade to fuel-vm 0.45.0 by @MitchTurner in #1640 * feat: Versionable Merkle metadata by @bvrooman in #1639 * Add docs for GraphQL DAP endpoints by @Dentosal in #1636 * Use a separate database for each data domain by @xgreenx in #1629 * feat: dry run multiple transactions by @matt-user in #1631 * Weekly `cargo update` by @github-actions in #1644 * Decrease peer reputation on request timeouts and decode errors by @Dentosal in #1574 * refactor: remove redudant receipts queries by @matt-user in #1646 * Latest gas price endpoint by @MitchTurner in #1649 * Estimate gas price API Endpoint by @MitchTurner in #1650 * Fix for the race condition with tx status and receipts by @xgreenx in #1658 * Bump up rust to 1.75 by @MitchTurner in #1656 * Weekly `cargo update` by @github-actions in #1655 * Duplication of the `0.22.1` release by @xgreenx in #1665 * Move updating of the owned messages and coins to off-chain worker by @xgreenx in #1635 * Weekly `cargo update` by @github-actions in #1682 * Remove nested contract resolvers in GQL schema by @Voxelot in #1675 * Soften txpool p2p reputation requirements by @Voxelot in #1663 * Remove failing deployment job by @Voxelot in #1692 * chore: format cargo tomls and add ci check by @segfault-magnet in #1691 * Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an off-chain worker by @xgreenx in #1671 * Weekly `cargo update` by @github-actions in #1707 * Increase liveness probe timeout to 5 minutes by @rfuelsh in #1709 * feat: Add `ContractInfo` table and versionable `ContractInfoType` to on-chain storage by @bvrooman in #1657 * feat: Versionable `ContractUtxoInfo` for the `ContractsLatestUtxo` table by @bvrooman in #1712 * Weekly `cargo update` by @github-actions in #1719 * Bump `fuel-vm` dep to 0.47.1 by @MitchTurner in #1714 * Bugfix: Calling of the contract with enabled `utxo_validation` fails by @bvrooman in #1717 * Bugfix: Zero `predicate_gas_used` field during validation of the produced block by @xgreenx in #1722 * Notify about imported blocks from the off-chain worker by @xgreenx in #1723 * Added versioning suffix to the GraphQL endpoint by @xgreenx in #1725 ## New Contributors * @liviudm made their first contribution in #1617 **Full Changelog**: v0.22.0...v0.23.0

@Dentosal

## Version v0.23.0 ### Added - [#1713](FuelLabs/fuel-core#1713): Added automatic `impl` of traits `StorageWrite` and `StorageRead` for `StructuredStorage`. Tables that use a `Blueprint` can be read and written using these interfaces provided by structured storage types. - [#1671](FuelLabs/fuel-core#1671): Added a new `Merklized` blueprint that maintains the binary Merkle tree over the storage data. It supports only the insertion of the objects without removing them. - [#1657](FuelLabs/fuel-core#1657): Moved `ContractsInfo` table from `fuel-vm` to on-chain tables, and created version-able `ContractsInfoType` to act as the table's data type. ### Changed - [#1723](FuelLabs/fuel-core#1723): Notify about imported blocks from the off-chain worker. - [#1717](FuelLabs/fuel-core#1717): The fix for the [#1657](FuelLabs/fuel-core#1657) to include the contract into `ContractsInfo` table. - [#1657](FuelLabs/fuel-core#1657): Upgrade to `fuel-vm` 0.46.0. - [#1671](FuelLabs/fuel-core#1671): The logic related to the `FuelBlockIdsToHeights` is moved to the off-chain worker. - [#1663](FuelLabs/fuel-core#1663): Reduce the punishment criteria for mempool gossipping. - [#1658](FuelLabs/fuel-core#1658): Removed `Receipts` table. Instead, receipts are part of the `TransactionStatuses` table. - [#1640](FuelLabs/fuel-core#1640): Upgrade to fuel-vm 0.45.0. - [#1635](FuelLabs/fuel-core#1635): Move updating of the owned messages and coins to off-chain worker. - [#1650](FuelLabs/fuel-core#1650): Add api endpoint for getting estimates for future gas prices - [#1649](FuelLabs/fuel-core#1649): Add api endpoint for getting latest gas price - [#1600](FuelLabs/fuel-core#1640): Upgrade to fuel-vm 0.45.0 - [#1633](FuelLabs/fuel-core#1633): Notify services about importing of the genesis block. - [#1625](FuelLabs/fuel-core#1625): Making relayer independent from the executor and preparation for the force transaction inclusion. - [#1613](FuelLabs/fuel-core#1613): Add api endpoint to retrieve a message by its nonce. - [#1612](FuelLabs/fuel-core#1612): Use `AtomicView` in all services for consistent results. - [#1597](FuelLabs/fuel-core#1597): Unify namespacing for `libp2p` modules - [#1591](FuelLabs/fuel-core#1591): Simplify libp2p dependencies and not depend on all sub modules directly. - [#1590](FuelLabs/fuel-core#1590): Use `AtomicView` in the `TxPool` to read the state of the database during insertion of the transactions. - [#1587](FuelLabs/fuel-core#1587): Use `BlockHeight` as a primary key for the `FuelsBlock` table. - [#1585](FuelLabs/fuel-core#1585): Let `NetworkBehaviour` macro generate `FuelBehaviorEvent` in p2p - [#1579](FuelLabs/fuel-core#1579): The change extracts the off-chain-related logic from the executor and moves it to the GraphQL off-chain worker. It creates two new concepts - Off-chain and On-chain databases where the GraphQL worker has exclusive ownership of the database and may modify it without intersecting with the On-chain database. - [#1577](FuelLabs/fuel-core#1577): Moved insertion of sealed blocks into the `BlockImporter` instead of the executor. - [#1574](FuelLabs/fuel-core#1574): Penalizes peers for sending invalid responses or for not replying at all. - [#1601](FuelLabs/fuel-core#1601): Fix formatting in docs and check that `cargo doc` passes in the CI. - [#1636](FuelLabs/fuel-core#1636): Add more docs to GraphQL DAP API. #### Breaking - [#1725](FuelLabs/fuel-core#1725): All API endpoints now are prefixed with `/v1` version. New usage looks like: `/v1/playground`, `/v1/graphql`, `/v1/graphql-sub`, `/v1/metrics`, `/v1/health`. - [#1722](FuelLabs/fuel-core#1722): Bugfix: Zero `predicate_gas_used` field during validation of the produced block. - [#1714](FuelLabs/fuel-core#1714): The change bumps the `fuel-vm` to `0.47.1`. It breaks several breaking changes into the protocol: - All malleable fields are zero during the execution and unavailable through the GTF getters. Accessing them via the memory directly is still possible, but they are zero. - The `Transaction` doesn't define the gas price anymore. The gas price is defined by the block producer and recorded in the `Mint` transaction at the end of the block. A price of future blocks can be fetched through a [new API nedopoint](FuelLabs/fuel-core#1641) and the price of the last block can be fetch or via the block or another [API endpoint](FuelLabs/fuel-core#1647). - The `GasPrice` policy is replaced with the `Tip` policy. The user may specify in the native tokens how much he wants to pay the block producer to include his transaction in the block. It is the prioritization mechanism to incentivize the block producer to include users transactions earlier. - The `MaxFee` policy is mandatory to set. Without it, the transaction pool will reject the transaction. Since the block producer defines the gas price, the only way to control how much user agreed to pay can be done only through this policy. - The `maturity` field is removed from the `Input::Coin`. The same affect can be achieve with the `Maturity` policy on the transaction and predicate. This changes breaks how input coin is created and removes the passing of this argument. - The metadata of the `Checked<Tx>` doesn't contain `max_fee` and `min_fee` anymore. Only `max_gas` and `min_gas`. The `max_fee` is controlled by the user via the `MaxFee` policy. - Added automatic `impl` of traits `StorageWrite` and `StorageRead` for `StructuredStorage`. Tables that use a `Blueprint` can be read and written using these interfaces provided by structured storage types. - [#1712](FuelLabs/fuel-core#1712): Make `ContractUtxoInfo` type a version-able enum for use in the `ContractsLatestUtxo`table. - [#1657](FuelLabs/fuel-core#1657): Changed `CROO` gas price type from `Word` to `DependentGasPrice`. The dependent gas price values are dummy values while awaiting updated benchmarks. - [#1671](FuelLabs/fuel-core#1671): The GraphQL API uses block height instead of the block id where it is possible. The transaction status contains `block_height` instead of the `block_id`. - [#1675](FuelLabs/fuel-core#1675): Simplify GQL schema by disabling contract resolvers in most cases, and just return a ContractId scalar instead. - [#1658](FuelLabs/fuel-core#1658): Receipts are part of the transaction status. Removed `reason` from the `TransactionExecutionResult::Failed`. It can be calculated based on the program state and receipts. Also, it is not possible to fetch `receipts` from the `Transaction` directly anymore. Instead, you need to fetch `status` and its receipts. - [#1646](FuelLabs/fuel-core#1646): Remove redundant receipts from queries. - [#1639](FuelLabs/fuel-core#1639): Make Merkle metadata, i.e. `SparseMerkleMetadata` and `DenseMerkleMetadata` type version-able enums - [#1632](FuelLabs/fuel-core#1632): Make `Message` type a version-able enum - [#1631](FuelLabs/fuel-core#1631): Modify api endpoint to dry run multiple transactions. - [#1629](FuelLabs/fuel-core#1629): Use a separate database for each data domain. Each database has its own folder where data is stored. - [#1628](FuelLabs/fuel-core#1628): Make `CompressedCoin` type a version-able enum - [#1616](FuelLabs/fuel-core#1616): Make `BlockHeader` type a version-able enum - [#1614](FuelLabs/fuel-core#1614): Use the default consensus key regardless of trigger mode. The change is breaking because it removes the `--dev-keys` argument. If the `debug` flag is set, the default consensus key will be used, regardless of the trigger mode. - [#1596](FuelLabs/fuel-core#1596): Make `Consensus` type a version-able enum - [#1593](FuelLabs/fuel-core#1593): Make `Block` type a version-able enum - [#1576](FuelLabs/fuel-core#1576): The change moves the implementation of the storage traits for required tables from `fuel-core` to `fuel-core-storage` crate. The change also adds a more flexible configuration of the encoding/decoding per the table and allows the implementation of specific behaviors for the table in a much easier way. It unifies the encoding between database, SMTs, and iteration, preventing mismatching bytes representation on the Rust type system level. Plus, it increases the re-usage of the code by applying the same blueprint to other tables. It is a breaking PR because it changes database encoding/decoding for some tables. ### StructuredStorage The change adds a new type `StructuredStorage`. It is a wrapper around the key-value storage that implements the storage traits(`StorageInspect`, `StorageMutate`, `StorageRead`, etc) for the tables with blueprint. This blueprint works in tandem with the `TableWithBlueprint` trait. The table may implement `TableWithBlueprint` specifying the blueprint, as an example: ```rust impl TableWithBlueprint for ContractsRawCode { type Blueprint = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } ``` It is a definition of the blueprint for the `ContractsRawCode` table. It has a plain blueprint meaning it simply encodes/decodes bytes and stores/loads them into/from the storage. As a key codec and value codec, it uses a `Raw` encoding/decoding that simplifies writing bytes and loads them back into the memory without applying any serialization or deserialization algorithm. If the table implements `TableWithBlueprint` and the selected codec satisfies all blueprint requirements, the corresponding storage traits for that table are implemented on the `StructuredStorage` type. ### Codecs Each blueprint allows customizing the key and value codecs. It allows the use of different codecs for different tables, taking into account the complexity and weight of the data and providing a way of more optimal implementation. That property may be very useful to perform migration in a more easier way. Plus, it also can be a `no_std` migration potentially allowing its fraud proving. An example of migration: ```rust /// Define the table for V1 value encoding/decoding. impl TableWithBlueprint for ContractsRawCodeV1 { type Blueprint = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } /// Define the table for V2 value encoding/decoding. /// It uses `Postcard` codec for the value instead of `Raw` codec. /// /// # Dev-note: The columns is the same. impl TableWithBlueprint for ContractsRawCodeV2 { type Blueprint = Plain<Raw, Postcard>; fn column() -> Column { Column::ContractsRawCode } } fn migration(storage: &mut Database) { let mut iter = storage.iter_all::<ContractsRawCodeV1>(None); while let Ok((key, value)) = iter.next() { // Insert into the same table but with another codec. storage.storage::<ContractsRawCodeV2>().insert(key, value); } } ``` ### Structures The blueprint of the table defines its behavior. As an example, a `Plain` blueprint simply encodes/decodes bytes and stores/loads them into/from the storage. The `SMT` blueprint builds a sparse merkle tree on top of the key-value pairs. Implementing a blueprint one time, we can apply it to any table satisfying the requirements of this blueprint. It increases the re-usage of the code and minimizes duplication. It can be useful if we decide to create global roots for all required tables that are used in fraud proving. ```rust impl TableWithBlueprint for SpentMessages { type Blueprint = Plain<Raw, Postcard>; fn column() -> Column { Column::SpentMessages } } | | \|/ impl TableWithBlueprint for SpentMessages { type Blueprint = Sparse<Raw, Postcard, SpentMessagesMerkleMetadata, SpentMessagesMerkleNodes>; fn column() -> Column { Column::SpentMessages } } ``` ### Side changes #### `iter_all` The `iter_all` functionality now accepts the table instead of `K` and `V` generics. It is done to use the correct codec during deserialization. Also, the table definition provides the column. #### Duplicated unit tests The `fuel-core-storage` crate provides macros that generate unit tests. Almost all tables had the same test like `get`, `insert`, `remove`, `exist`. All duplicated tests were moved to macros. The unique one still stays at the same place where it was before. #### `StorageBatchMutate` Added a new `StorageBatchMutate` trait that we can move to `fuel-storage` crate later. It allows batch operations on the storage. It may be more performant in some cases. - [#1573](FuelLabs/fuel-core#1573): Remove nested p2p request/response encoding. Only breaks p2p networking compatibility with older fuel-core versions, but is otherwise fully internal. ## What's Changed * Fix the example tx in README.md by @Dentosal in FuelLabs/fuel-core#1543 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1557 * Update LICENSE by @Voxelot in FuelLabs/fuel-core#1558 * Remove duplicating logic in the `KeyValueStore` trait by @xgreenx in FuelLabs/fuel-core#1559 * Move `KeyValueStore` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1566 * Update `libp2p` from `0.50.0` to `0.53.1` by @MitchTurner in FuelLabs/fuel-core#1379 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1575 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1578 * Moved insertion of the blocks into the `BlockImporter` instead of the executor by @xgreenx in FuelLabs/fuel-core#1577 * Let `NetworkBehaviour` macro generate `FuelBehaviorEvent` in p2p by @MitchTurner in FuelLabs/fuel-core#1585 * Remove RustSec Ignore for RUSTSEC-2022-0093 by @MitchTurner in FuelLabs/fuel-core#1586 * Simplify p2p request/response message serialization by @Dentosal in FuelLabs/fuel-core#1573 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1588 * Depend on libp2p directly instead of submodules by @MitchTurner in FuelLabs/fuel-core#1591 * Make `Block` versionable by @MitchTurner in FuelLabs/fuel-core#1593 * Switch katyo/publish-crates@v2 to xgreenx/publish-crates@v1 for the CI by @Dentosal in FuelLabs/fuel-core#1603 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1599 * Fix cargo doc, check for cargo doc in the CI by @Dentosal in FuelLabs/fuel-core#1601 * Move storage traits implementation to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1576 * Extract off chain logic from the executor by @xgreenx in FuelLabs/fuel-core#1579 * Applying comments from the PR for the storage crate by @xgreenx in FuelLabs/fuel-core#1610 * Use `BlockHeight` as a primary key for the `FuelsBlock` table by @xgreenx in FuelLabs/fuel-core#1587 * Use `AtomicView` in the `TxPool` by @xgreenx in FuelLabs/fuel-core#1590 * Unify namespacing in P2P codebase by @MitchTurner in FuelLabs/fuel-core#1597 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1611 * Make `Consensus` version-able by @MitchTurner in FuelLabs/fuel-core#1596 * feat: add api endpoint to retrieve messages by nonce by @matt-user in FuelLabs/fuel-core#1613 * Update Docker tags for Flux integration by @liviudm in FuelLabs/fuel-core#1617 * Remove dev consensus key by @MujkicA in FuelLabs/fuel-core#1614 * Make `BlockHeader` a versionable enum by @MitchTurner in FuelLabs/fuel-core#1616 * feat: Versionable CompressedCoin by @bvrooman in FuelLabs/fuel-core#1628 * Use `AtomicView` in all services by @xgreenx in FuelLabs/fuel-core#1612 * Notify services about importing of the genesis block by @xgreenx in FuelLabs/fuel-core#1633 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1634 * feat: Versionable `Message` by @bvrooman in FuelLabs/fuel-core#1632 * Making relayer independent from the executor by @xgreenx in FuelLabs/fuel-core#1625 * Upgrade to the fuel-vm 0.44.0 by @Dentosal in FuelLabs/fuel-core#1600 * Upgrade to fuel-vm 0.45.0 by @MitchTurner in FuelLabs/fuel-core#1640 * feat: Versionable Merkle metadata by @bvrooman in FuelLabs/fuel-core#1639 * Add docs for GraphQL DAP endpoints by @Dentosal in FuelLabs/fuel-core#1636 * Use a separate database for each data domain by @xgreenx in FuelLabs/fuel-core#1629 * feat: dry run multiple transactions by @matt-user in FuelLabs/fuel-core#1631 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1644 * Decrease peer reputation on request timeouts and decode errors by @Dentosal in FuelLabs/fuel-core#1574 * refactor: remove redudant receipts queries by @matt-user in FuelLabs/fuel-core#1646 * Latest gas price endpoint by @MitchTurner in FuelLabs/fuel-core#1649 * Estimate gas price API Endpoint by @MitchTurner in FuelLabs/fuel-core#1650 * Fix for the race condition with tx status and receipts by @xgreenx in FuelLabs/fuel-core#1658 * Bump up rust to 1.75 by @MitchTurner in FuelLabs/fuel-core#1656 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1655 * Duplication of the `0.22.1` release by @xgreenx in FuelLabs/fuel-core#1665 * Move updating of the owned messages and coins to off-chain worker by @xgreenx in FuelLabs/fuel-core#1635 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1682 * Remove nested contract resolvers in GQL schema by @Voxelot in FuelLabs/fuel-core#1675 * Soften txpool p2p reputation requirements by @Voxelot in FuelLabs/fuel-core#1663 * Remove failing deployment job by @Voxelot in FuelLabs/fuel-core#1692 * chore: format cargo tomls and add ci check by @segfault-magnet in FuelLabs/fuel-core#1691 * Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an off-chain worker by @xgreenx in FuelLabs/fuel-core#1671 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1707 * Increase liveness probe timeout to 5 minutes by @rfuelsh in FuelLabs/fuel-core#1709 * feat: Add `ContractInfo` table and versionable `ContractInfoType` to on-chain storage by @bvrooman in FuelLabs/fuel-core#1657 * feat: Versionable `ContractUtxoInfo` for the `ContractsLatestUtxo` table by @bvrooman in FuelLabs/fuel-core#1712 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1719 * Bump `fuel-vm` dep to 0.47.1 by @MitchTurner in FuelLabs/fuel-core#1714 * Bugfix: Calling of the contract with enabled `utxo_validation` fails by @bvrooman in FuelLabs/fuel-core#1717 * Bugfix: Zero `predicate_gas_used` field during validation of the produced block by @xgreenx in FuelLabs/fuel-core#1722 * Notify about imported blocks from the off-chain worker by @xgreenx in FuelLabs/fuel-core#1723 * Added versioning suffix to the GraphQL endpoint by @xgreenx in FuelLabs/fuel-core#1725 ## New Contributors * @liviudm made their first contribution in FuelLabs/fuel-core#1617 **Full Changelog**: FuelLabs/fuel-core@v0.22.0...v0.23.0

Extraction of the FuelBlockSecondaryKeyBlockHeights table to an off…

3783579

…-chain worker

xgreenx requested a review from a team February 15, 2024 15:52

xgreenx self-assigned this Feb 15, 2024

Updated CHANGELOG.md

3c2d424

xgreenx commented Feb 15, 2024

View reviewed changes

crates/fuel-core/Cargo.toml Outdated Show resolved Hide resolved

xgreenx added 3 commits February 17, 2024 10:16

Merge with master

1d95e63

Fix CI

95dfa58

MitchTurner reviewed Feb 17, 2024

View reviewed changes

Voxelot reviewed Feb 18, 2024

View reviewed changes

xgreenx added 2 commits February 18, 2024 20:16

Apply comments from the PR

111efbd

Merge branch 'master' into feature/blocks-second-key-extraction

cabca80

xgreenx requested a review from a team February 18, 2024 19:23

Merge branch 'master' into feature/blocks-second-key-extraction

0e20633

bvrooman reviewed Feb 19, 2024

View reviewed changes

Voxelot and others added 3 commits February 20, 2024 10:52

Merge branch 'master' into feature/blocks-second-key-extraction

4e8ebc6

# Conflicts: # CHANGELOG.md # crates/types/src/services/txpool.rs

Merge branch 'master' into feature/blocks-second-key-extraction

71dbc41

Apply comments from the review

593ad52

xgreenx requested review from Voxelot, MitchTurner and bvrooman February 21, 2024 15:54

xgreenx added 2 commits February 21, 2024 16:56

Renamed FuelBlockSecondaryKeyBlockHeights

97a96d5

Remove contstrain

3ce8c17

xgreenx requested a review from a team February 21, 2024 18:58

Merge branch 'master' into feature/blocks-second-key-extraction

802578e

Voxelot approved these changes Feb 21, 2024

View reviewed changes

xgreenx merged commit ed92738 into master Feb 22, 2024
31 checks passed

xgreenx deleted the feature/blocks-second-key-extraction branch February 22, 2024 00:34

xgreenx mentioned this pull request Mar 4, 2024

Release v0.23.0 #1726

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an off-chain worker #1671

Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an off-chain worker #1671

xgreenx commented Feb 15, 2024 •

edited

Loading

MitchTurner Feb 17, 2024

MitchTurner Feb 17, 2024 •

edited

Loading

MitchTurner Feb 17, 2024 •

edited

Loading

xgreenx Feb 18, 2024

MitchTurner Feb 20, 2024

MitchTurner Feb 17, 2024

xgreenx Feb 18, 2024

Voxelot Feb 18, 2024

xgreenx Feb 18, 2024

Voxelot Feb 18, 2024

xgreenx Feb 18, 2024

Voxelot Feb 18, 2024 •

edited

Loading

xgreenx Feb 18, 2024 •

edited

Loading

Voxelot Feb 18, 2024 •

edited

Loading

xgreenx Feb 18, 2024

bvrooman left a comment

bvrooman Feb 19, 2024

xgreenx Feb 19, 2024

Voxelot Feb 20, 2024

bvrooman Feb 19, 2024

Voxelot Feb 20, 2024 •

edited

Loading

bvrooman Feb 19, 2024 •

edited

Loading

xgreenx Feb 19, 2024

bvrooman Feb 19, 2024

Voxelot Feb 20, 2024

Extraction of the FuelBlockSecondaryKeyBlockHeights table to an off-chain worker #1671

Extraction of the FuelBlockSecondaryKeyBlockHeights table to an off-chain worker #1671

Conversation

xgreenx commented Feb 15, 2024 • edited Loading

Choose a reason for hiding this comment

MitchTurner Feb 17, 2024 • edited Loading

Choose a reason for hiding this comment

MitchTurner Feb 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Voxelot Feb 18, 2024 • edited Loading

Choose a reason for hiding this comment

xgreenx Feb 18, 2024 • edited Loading

Choose a reason for hiding this comment

Voxelot Feb 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bvrooman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Voxelot Feb 20, 2024 • edited Loading

Choose a reason for hiding this comment

bvrooman Feb 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an off-chain worker #1671

Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an off-chain worker #1671

xgreenx commented Feb 15, 2024 •

edited

Loading

MitchTurner Feb 17, 2024 •

edited

Loading

MitchTurner Feb 17, 2024 •

edited

Loading

Voxelot Feb 18, 2024 •

edited

Loading

xgreenx Feb 18, 2024 •

edited

Loading

Voxelot Feb 18, 2024 •

edited

Loading

Voxelot Feb 20, 2024 •

edited

Loading

bvrooman Feb 19, 2024 •

edited

Loading