Feature/rebuild trie from flat

[garyschulte]

PR description

Implement a storage subcommand to rebuild the bonsai state trie from flat db. This subcommand can be used to fix a broken state trie by re-inserting all of the elements from the flat database into the state trie.

usage:

besu --config-file=<config_toml> storage rebuild-bonsai-state-trie

This is a destructive action, but has appropriate safeguards to ensure the state trie reconstruction will be successful.

Fixed Issue(s)

fixes #7934

Thanks for sending a pull request! Have you done the following?

• Checked out our contribution guidelines?
• Considered documentation and added the doc-change-required label to this PR if updates are required.
• Considered the changelog and included an update if required.
• For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

• unit tests: ./gradlew build
• acceptance tests: ./gradlew acceptanceTest
• integration tests: ./gradlew integrationTest
• reference tests: ./gradlew ethereum:referenceTests:referenceTests

[github-actions]

This pr is stale because it has been open for 30 days with no activity.

[github-actions]

This pr was closed because it has been inactive for 14 days since being marked as stale.

[garyschulte]

reopening

[pinges]

Under what circumstances do we have a "good" flat DB that can be used to rebuild a broken trie? And how do we know that the flat DB is "good"?

[garyschulte]

Under what circumstances do we have a "good" flat DB that can be used to rebuild a broken trie? And how do we know that the flat DB is "good"?

In its current incarnation, this is a 'repair' type command. If for example besu ends up with a corrupted worldstate, this will rebuild the state trie from the flat db.

I had this happen on a long running node yesterday where trielog, blockchain and worldstate all ended up at different places on restart due to a kill. When Besu started it attempted to "heal" the worldstate it wrote garbage to the trie. Using this PR I was able to salvage an archive node that had been syncing for quite some time.

Longer term, this subcommand would be useful to rebuild the trie from a state-trie-less full sync strategy, where we can trust the chain head either via static checkpoints we have in the code or from finalized blocks from the CL. That sync method is still a WIP though.