nonce too high error calling debug_traceTransaction

[quick-pawiromitchel]

System information

Erigon version: erigon/1.1.6/linux-amd64/go1.20.6

OS & Version: Linux

Chain/Network: BSC Mainnet

Expected behaviour

A correct response

Actual behaviour

{"code":-32000,"message":"transaction 8550de69437bf36fbd2e030c84d96cd763fa76d21860689c477e8fcb84aa1fca failed: nonce too high: address 0xeeD3EF2BA1797AeEF04be6bbEbd5BeD0Cb7f8528, tx: 3567779 state: 22634"}

Steps to reproduce the behaviour

Run with the following payload

{"method": "debug_traceTransaction", "params": ["0xd3395b9d41988c0c4b27c984843fdff96d60f1908ba15abc02e9571093d61a32", {"tracer": "callTracer", "timeout": "32s"}], "id": 1, "jsonrpc": "2.0"}

Backtrace

Does not generate any errors in the logs

[quick-pawiromitchel]

Tracing the following transactions also errors out

0x3a4ff95a0a89b74ba989fa1ba90edab6c18b16709205702f2778dc35eff5a0e5
0xfb678bfdcd581f6ac2064df80d8d0e629ba0775ed1a87d0edb430f6d170a5113
0xabd260b521ed066460fd22f4d144b8ee7aa6fc7d3958cc7efc705dd6685b0128
0x064f4adfd670597da6540f4d0690223a3b7dcc527b12d1c4719dfe94c7b74b70
0x6867068db34c3888bd33adba275a2e73005f2f529b83953cf7fc1ee89e9fa34e
0xd3395b9d41988c0c4b27c984843fdff96d60f1908ba15abc02e9571093d61a32
0xb8b0688f9c35d2924613b417ef6fe233184407976a13e18b750ffa71d74e6810
0x74bf277f491d31610bcfa6fa116d59b24c7bdb64a271eb147654fa8c1dd70f90
0xf25f34426faf1cfe844671d3c00c3a3e0b92721f8cc1992a4c5556001e7fb526
0xe982631e2f15a4524491fc685889f6c39d00f973a33283d04fca4bd295d4a8ac
0xc3890d0fed602170dbca0a282f9f6455f1da2968e31d8f7088b3eedc0a3f86e6
0x0d84d7cfd41dd7c77c0241d648d3f0dba538c215e997819ffac78b975d20d36b
0x8de25226036ff4bab1adf3ff30f8c04211abec8467d0562ced7f135e741ed154
0xd6bac0062ee10908e277a42451c39a40d965f7e837049420f9c35afe0f8431dd
0xe6d9392de18dee5aaf07769cc9ea47b8fc355d006ecc2b5c5a5874e71920d05c
0x73efff9f80602b7a165f85ebff90ccd824da406b34e530ec0a5fcf4fe594b254

[qk-santi]

Adding another tx:
0x245e1521e1418a26438ca37502a0b7aaa3a8ffd92a40ed99eef9edf106386a33

[0x1un]

I have the same problem here

[qk-santi]

We found the following:

Affects >90% of transactions within these blocks:

• From 0x1D3CDD0 - 30.658.000 Aug-07-2023 11:41:54 PM +UTC
• To 0x1D6A078 - 30.843.000 Aug-14-2023 10:18:09 AM +UTC

Affects both debug_traceTransaction and debug_traceByBlockNumber and probably hash too, when using callTracer.

Affects Erigon nodes that are doing snapshots.

[0x1un]

This is the block number where I recently encountered a nonce too hight error using the debug_traceByBlockNumber interface, and this is only a portion of what was affected.

https://gist.github.com/00db8568422edc786a4261a4a3fe8853.git

[quickchase]

As far as I can tell, there is something wrong with snapshots. The block range that is affected is increasing, which is to be expected as the node removes data from the database and moves it to snapshots.

[tpalaz]

Can also confirm that older blocks (~29,500,000) are affected. If someone could test the following transactions:

• 0xfc1c891f898e03bac4c3d86209e2b79853ab44f5a76790639af08589f377322b
• 0x35fb12129333715cb7e58f1b8a2e7ae5c1667adc03a4f21bdf90ec1c9ce24aec

RPC replies with the error:
first run for txIndex 0 error: nonce too high: address 0x9512628e773cb8eF25eB7ea0291d066cB12c3e09, tx: 480981 state: 25323

[0x1un]

@tpalaz
These two trading addresses are normal for me here, so it seems that everyone is affected by a different address and block number.

[tpalaz]

@0x1un interesting. Was your node synced from scratch or did you use public snapshots, such as the ones from BNB 48 Club?

[quickchase]

Originally our nodes were sync'd from genesis, we then tested the 48Club snapshots:

When loading the 48Club snapshots, we ran into missing data and an error that said:

[WARN] [08-23|20:56:36.627] [snapshots] retire blocks                err="DumpBlocks: DumpHeaders: header missed in db: block_num=30000000,  hash=0fe1b87f3d62477a866bbd2327139b884c0dc057f11dc273f8bed34fe699efbd" fromBlock=30000000 toBlock=30500000

and after doing a header reset:

integration stage_headers --chain bsc --datadir /home/bsc/erigon --reset

It seemed to fix the retire blocks error and the node seemed happy, however, we're right back where we started. Same error as original post, going full circle.

Also: We did the reset on devel and are running current devel now.

[0x1un]

@0x1un interesting. Was your node synced from scratch or did you use public snapshots, such as the ones from BNB 48 Club?

@tpalaz synced from public snapshots: https://github.com/bnb-chain/bsc-snapshots

[du5]

I think this is caused by a bug in a certain version of erigon, please update upstream as soon as possible

erigontech#7892

erigontech#7934

https://github.com/search?q=repo%3Aledgerwatch%2Ferigon%20BadHeaderNumber&type=code

[quickchase]

So, we sync'd from genesis (snapshots enabled) and this error happens basically all the time.

We sync'd another node from genesis with snapshots disabled and it seems fine...

So snapshots are definitely busted

[blxdyx]

So, we sync'd from genesis (snapshots enabled) and this error happens basically all the time.

We sync'd another node from genesis with snapshots disabled and it seems fine...

So snapshots are definitely busted

You means nearly all history block in snapshots are busted? Or just block after 29000000 ?

[quickchase]

I THINK it's just blocks after 29000000 I will double check

[quickchase]

Actually, I take it back. The node never actually finished syncing from Genesis because of an error in execution at 30M:

[7/15 Execution] Execution failed        block=30000000 hash=0x0fe1b87f3d62477a866bbd2327139b884c0dc057f11dc273f8bed34fe699efbd err="could not apply tx 0 from block 30000000 [0x27e6d19a16f6081d794bd882ff8ab0d6065d023d9d59809cefc59af08e5ea588]: nonce too low: address 0xeb6DB8aC2E53E7dc543b0036bAB272e421521914, tx: 32 state: 68703"

[INFO] [09-20|19:56:39.368] UnwindTo                                 block=29999999 bad_block_hash=0x0fe1b87f3d62477a866bbd2327139b884c0dc057f11dc273f8bed34fe699efbd

[INFO] [09-20|19:56:39.368] [7/15 Execution] Completed on            block=29999999

[INFO] [09-20|19:56:42.845] Timings (slower than 50ms)               Headers=28m23.562s CumulativeIndex=226ms BlockHashes=4.635s Bodies=5h1m45.365s Senders=8m5.821s Execution=50ms Unwind Headers=2.975s

[INFO] [09-20|19:56:42.845] [2/15 Headers] Waiting for headers...    from=29999999

💀

[blxdyx]

Stuck at block 29999999, seems like this #226 (comment).
Use the latest release?

[pyinto]

@blxdyx same issue, erigon version 1.1.9-dev

Sep 29 14:27:24 bscnode erigon[490492]: [INFO] [09-29|14:27:24.354] [2/15 Headers] No block headers to write in this log period block number=29999999
Sep 29 14:27:24 bscnode erigon[490492]: [INFO] [09-29|14:27:24.354] Req/resp stats                           req=8693 reqMin=29926428 reqMax=30464883 skel=20 skelMin=29999998 skelMax=30000190 resp=57603 respMin=29911525 respMax=32165665 dups=1230946
Sep 29 14:27:44 bscnode erigon[490492]: [INFO] [09-29|14:27:44.355] [2/15 Headers] No block headers to write in this log period block number=29999999
Sep 29 14:27:44 bscnode erigon[490492]: [INFO] [09-29|14:27:44.355] Req/resp stats                           req=7070 reqMin=29919659 reqMax=30464499 skel=20 skelMin=29999998 skelMax=30000190 resp=51806 respMin=29910598 respMax=32165671 dups=1161816

This result I get with and without using snapshots btw

[blxdyx]

Same issue, will update in this: #211