trace_block no longer works after Hertz upgrade

[0xStrobe]

System information

Erigon version: 1.1.9-dev-4b352f69

OS & Version: Linux

Commit hash: 4b352f69

Erigon Command (with flags/config):

./erigon --chain=bsc --port=xxx --sentry.drop-useless-peers --torrent.port=xxx --torrent.upload.rate=15mb --torrent.download.rate=1024mb --txpool.disable --metrics.addr=0.0.0.0 --metrics.port=xxx --db.pagesize=16k --db.read.concurrency 192 --datadir=/xxx/bsc-erigon/data --private.api.addr=localhost:xxx

Chain/Network: BSC

Expected behaviour

When sending an RPC request with method call trace_block, the erigon node should return trace entries of the block.

Example request:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "trace_block",
  "params": [
    "0x1dda1a0" // this is block 31302048
  ]
}

Actual behaviour

Erigon responds with an error. It seems that the Hertz upgrade has not been correctly implemented.

{
    "jsonrpc": "2.0",
    "id": 1,
    "error": {
        "code": -32000,
        "message": "convert tx into msg: eip-1559 transactions require London"
    }
}

[Mister-EA]

@0xStrobe Did you make sure Berlin, London and Hertz blocks are set at the same height?

[0xStrobe]

@0xStrobe Did you make sure Berlin, London and Hertz blocks are set at the same height?

@Mister-EA what do you mean specifically? i didn't manually configure that

[blxdyx]

It's hardcoded in the source code, don't need manually configure

[0xStrobe]

anything i could do from the user's side to help u debug and identify this issue? :)

[blxdyx]

Could you please help to check trace_block when blockNum between 28500000 to 29500000？

[blxdyx]

As a temporary fix, can use --snap.stop, It will stop move historical data from DB to new immutable snapshots.

[0xStrobe]

Could you please help to check trace_block when blockNum between 28500000 to 29500000？

@blxdyx Sure :)

I tested with this block 28700000 and the response is correct. (This is before 31302048 though, which is the Hertz upgrade block)

The request is this:

curl --request POST \
  --url http://1.2.3.4:5678/ \
  --header 'Content-Type: application/json' \
  --data '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "trace_block",
    "params": [
      "0x1b5ed60"
    ]
}'

The response is a huge payload which is what we expect.

[0xStrobe]

As a temporary fix, can use --snap.stop, It will stop move historical data from DB to new immutable snapshots.

Will this fix the past blocks or only will be fixing the new ones from now?

[0xStrobe]

Meanwhile I have started a new node based on the snapshots the Ankr team published, now I'm getting a different error when requesting block 31302048 (the json req payload is same as the OP):

{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"first run for txIndex 0 error: nonce too high: address 0xD98eA98A4aCC0eAcF180c75600e365867D13b51c, tx: 1 state: 0"}}

This is also true for the official Ankr RPC, if you send this request:

curl --request POST \
  --url https://rpc.ankr.com/bsc \
  --header 'Content-Type: application/json' \
  --data '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "trace_block",
    "params": [
      "0x1dda1a0"
    ]
}'

So in summary to make clear of the context, since this issue has been open for a while -

1. I started a node using the MDBX snapshot published here - https://github.com/bnb-chain/bsc-snapshots. I documented the workflow of exactly the steps I took here - How to use Erigon snapshot correctly ? bnb-chain/bsc-snapshots#253 (comment)
2. The node was running for a few months, erigon updates were always successfully applied during this time
3. Node was upgraded to v1.1.6, in preparation for the Hertz upgrade
4. I observed that all trace block calls have failed after the Hertz upgrade at block 31302048. I upgraded it to the latest v1.1.9, but kept receiving error as in OP: convert tx into msg: eip-1559 transactions require London (I don't remember what error v1.1.6 gave at that time)
5. The tx logs API etc seem to be okay, although the trace API errors out
6. After doing research, I thought there might be an issue with the way I bootstrapped my node, so I downloaded Ankr's snapshot from Chainsnap. Today I started a new server, and attempted to request the same trace_block, and I got the error first run for txIndex 0 error: nonce too high: address 0xD98eA98A4aCC0eAcF180c75600e365867D13b51c, tx: 1 state: 0 instead.
7. I checked with the official Ankr RPC endpoint, they return the identical error, ruling out that I didn't bootstrap it correctly

[blxdyx]

As a temporary fix, can use --snap.stop, It will stop move historical data from DB to new immutable snapshots.

Will this fix the past blocks or only will be fixing the new ones from now?

the new ones

[0xStrobe]

As a temporary fix, can use --snap.stop, It will stop move historical data from DB to new immutable snapshots.

Will this fix the past blocks or only will be fixing the new ones from now?

the new ones

Thank you, is there a way to fix and backfill the faulty blocks since the Hertz hard fork?

[blxdyx]

As a temporary fix, can use --snap.stop, It will stop move historical data from DB to new immutable snapshots.

Will this fix the past blocks or only will be fixing the new ones from now?

the new ones

Thank you, is there a way to fix and backfill the faulty blocks since the Hertz hard fork?

You can try to sync from snapshot before Hertz hard fork with this flag

[0xStrobe]

As a temporary fix, can use --snap.stop, It will stop move historical data from DB to new immutable snapshots.

Will this fix the past blocks or only will be fixing the new ones from now?

the new ones

Thank you, is there a way to fix and backfill the faulty blocks since the Hertz hard fork?

You can try to sync from snapshot before Hertz hard fork with this flag

Is there a way to reset an updated node to a block before Hertz then sync from there? Unfortunately I can't find a snapshot before Hertz anymore, there is this one but it's very up to date - https://github.com/bnb-chain/bsc-snapshots#endpointmainnet-update-bi-weekly

(Also would appreciate it if this bug is fixed in the codebase and there is a short guide for users to fix/resync the faulty blocks or snapshots :))

[blxdyx]

One way is delete all the snapshots more than 29500000 blocks in ./data/snapshots and delete db in this dir. And then use /integration stage_headers --reset --chain=bsc --datadir=./data. after that resync with --snap.stop,

[0xStrobe]

One way is delete all the snapshots more than 29500000 blocks in ./data/snapshots and delete db in this dir. And then use /integration stage_headers --reset --chain=bsc --datadir=./data. after that resync with --snap.stop,

Thank you, I will give it a shot! Hopefully the bug will soon be identified and fixed though, so no more hacks are needed. :)

Let me know if you need any other digging from user side

[blxdyx]

Seems fix in latest commit in devel. You could try this.

[0xStrobe]

Syncing with the latest commit in devel, will report back

[blxdyx]

Follow this: #234

[0xStrobe]

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "trace_block",
  "params": [
    "0x1dda1a0" // this is block 31302048
  ]
}

Thank you! I forgot to delete /snapshots/db but ran all the other stuff and synced... The block I used in the OP is working with traces now (block 0x1dda1a0).

However to be completely sure, I will delete /snapshots/db now and redo the steps and report back to confirm.

[blxdyx]

Reopen if you still have problem