cmd/evm: add enginetest command for direct engine fixture execution by spencer-tb · Pull Request #34650 · ethereum/go-ethereum

spencer-tb · 2026-04-03T14:10:50Z

Summary

Add evm enginetest command that runs blockchain_test_engine fixtures directly against a lightweight Engine API handler, without requiring Hive or full client startup. Also add --workers flag to all three test runners (enginetest, blocktest, statetest) for parallel fixture file processing.

`evm enginetest`

A new direct runner for Engine API test fixtures. Implements a lightweight engine handler that mirrors the core logic of eth/catalyst.ConsensusAPI:

Version-specific NewPayloadV1-V5 parameter validation
ExecutableDataToBlock payload conversion and block insertion via InsertBlockWithoutSetHead
ForkchoiceUpdated with SetCanonical head management (including initial FCU to genesis)
Invalid block ancestor tracking with proper PayloadStatusV1 responses
EngineAPIError code validation against fixture expectations (errorCode, validationError)

This exercises the actual engine code path (two-phase insert-then-canonicalize via forkchoice), not just InsertChain like blocktest.

Benchmarks

Tested against EEST v5.3.0 stable fixtures on Apple M-series.

For reference, Hive runs of the same test suite on the same geth version:

consume engine: 2h 48m
consume rlp: 4h 29m

evm enginetest — exercises the same engine code paths as consume engine (40,523 tests):

Workers	Time	Speedup vs serial	vs Hive consume engine
1	1m02s	1x	~162x
8	12.0s	5.2x	~840x

evm blocktest — exercises the same execution paths as consume rlp (43,924 tests):

Workers	Time	Speedup vs serial
1	1m06s	1x
8	12.7s	5.2x

evm statetest (40,553 tests):

Workers	Time	Speedup
1	21.8s	1x
8	4.4s	4.9x

Hive parity

Tested against v5.3.0 stable release — exact same 4 failures as Hive consume engine on geth master:

eip7002/test_system_contract_deployment[CancunToPragueAtTime15k-deploy_after_fork-nonzero_balance]
eip7002/test_system_contract_deployment[CancunToPragueAtTime15k-deploy_after_fork-zero_balance]
eip7251/test_system_contract_deployment[CancunToPragueAtTime15k-deploy_after_fork-nonzero_balance]
eip7251/test_system_contract_deployment[CancunToPragueAtTime15k-deploy_after_fork-zero_balance]

Usage

# Run engine fixtures
evm enginetest /path/to/blockchain_tests_engine/

# With parallel workers
evm enginetest --workers 8 /path/to/blockchain_tests_engine/

# Filter by regex
evm enginetest --run "eip4844" /path/to/fixtures/

# Human-readable output
evm enginetest --human /path/to/fixtures/

# Same --workers flag on blocktest and statetest
evm blocktest --workers 8 /path/to/blockchain_tests/
evm statetest --workers 8 /path/to/state_tests/

kevaundray · 2026-04-03T18:50:03Z

+			return engine.PayloadStatusV1{Status: engine.INVALID}, engineParamsErr("nil versionedHashes post-cancun")
+		case p.BeaconRoot == nil:
+			return engine.PayloadStatusV1{Status: engine.INVALID}, engineParamsErr("nil beaconRoot post-cancun")
+		case !h.checkFork(params.Timestamp, forks.Cancun, forks.Prague, forks.Osaka, forks.BPO1, forks.BPO2, forks.BPO3, forks.BPO4, forks.BPO5):


Will need to double check this for my own sanity -- for some reason I was expecting it to just say forks.Cancun

This matches the real ConsensusAPI.NewPayloadV3 at api.go:204 which allows V3 for Cancun through BPO5. So maybe can be changed there too but not 100% certain :)

Ah I see, seems I was remembering this:

go-ethereum/eth/catalyst/api.go

Line 708 in d8cb8a9

case !api.checkFork(params.Timestamp, forks.Cancun):

kevaundray · 2026-04-05T12:22:48Z

+	if postCheck != nil {
+		defer postCheck(result, chain)
+	}


Minor nit: I think we may want to use a closure here becauseresult here would be evaluated at the callsite and not when the defer is triggered.

Small self-contained example to elaborate:

package main import "fmt" func runBuggy(postCheck func(error, string)) (result error) { chain := "some-chain" // `result` is evaluated at the callsite, so its nil and not when the defer fires. if postCheck != nil { defer postCheck(result, chain) } result = fmt.Errorf("payload 3: expected VALID, got INVALID") return result } func runFixed(postCheck func(error, string)) (result error) { chain := "some-chain" // closure captures `result` by reference, so it reads the final value when the defer fires. if postCheck != nil { defer func() { postCheck(result, chain) }() } result = fmt.Errorf("payload 3: expected VALID, got INVALID") return result } func main() { check := func(res error, chain string) { fmt.Println(" postCheck got error:", res) } fmt.Println("buggy:") runBuggy(check) fmt.Println("fixed:") runFixed(check) }

kevaundray · 2026-04-05T12:35:26Z

 	var tests map[string]*tests.BlockTest
 	if err = json.Unmarshal(src, &tests); err != nil {
-		return nil, err
+		return nil, nil // Skip non-fixture JSON files


Just want to confirm that this also skips errors from malformed fixture files?

- Add lastBlockHash to blocktest/enginetest, lastPayloadStatus to enginetest - Remove stateRoot from blocktest/enginetest (only statetest has it) - Report validation/rejection error in `error` even when test passes, for negative tests (expected exceptions) - Enables EELS consume direct to map errors through ExceptionMapper and verify correct exception for every invalid test

MariusVanDerWijden · 2026-04-07T11:37:39Z

I like the idea. I don't know if we need --worker though, we could just default to runtime.NumCPU()

New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319

jwasinger · 2026-04-09T17:53:13Z

There's a lot of log output when I just run this against a test dir:

iB state-history="last 90000 blocks"
INFO [04-09|13:52:08.614] Resuming snapshot generation             root=56e81f..63b421 accounts=0 slots=0 storage=0.00B dangling=0 elapsed="33.5µs"
INFO [04-09|13:52:08.614] Generated snapshot                       accounts=0 slots=0 storage=0.00B dangling=0 elapsed="60.542µs"
INFO [04-09|13:52:08.615] Load database journal from disk
INFO [04-09|13:52:08.615] State snapshot generator is not found
INFO [04-09|13:52:08.615] Starting snapshot generation             root=56e81f..63b421 accounts=0 slots=0 storage=0.00B dangling=0 elapsed=459ns
INFO [04-09|13:52:08.615] Initialized path database                triecache=16.00MiB statecache=16.00MiB buffer=64.00MiB state-history="last 90000 blocks"
INFO [04-09|13:52:08.615] Resuming snapshot generation             root=56e81f..63b421 accounts=0 slots=0 storage=0.00B dangling=0 elapsed="30.334µs"
INFO [04-09|13:52:08.615] Generated snapshot                       accounts=0 slots=0 storage=0.00B dangling=0 elapsed="54.25µs"
...

I'm not sure if the best solution is to suppress this entirely, or only dump log output for failing cases. The latter may be impossible.

jwasinger · 2026-04-09T17:58:28Z

 }
+
+// reportNDJSON prints one JSON object per result as it completes.
+func reportNDJSON(r testResult) {


can remove this it seems

yeah lets remove it, i was trialing whether we could stream the json output using ndjson to pytest in ethereum/execution-specs#2622

jwasinger · 2026-04-09T19:56:10Z

+		Alloc:         t.json.Pre,
+		BaseFee:       t.json.Genesis.BaseFeePerGas,
+		BlobGasUsed:   t.json.Genesis.BlobGasUsed,
+		ExcessBlobGas: t.json.Genesis.ExcessBlobGas,


We need to set the SlotNumber here. And later when BALs are merged, the BAL hash

jwasinger · 2026-04-09T20:03:49Z

+
+// validateEnginePostState verifies the post-state accounts match the expected values.
+// Mirrors BlockTest.validatePostState.
+func validateEnginePostState(post types.GenesisAlloc, statedb *state.StateDB) error {


It's almost exactly the same as BlockTest.validatePostState. We should deduplicate them

jwasinger · 2026-04-09T20:06:49Z

+type engineHandler struct {
+	chain             *core.BlockChain
+	invalidBlocksHits map[common.Hash]int
+	invalidTipsets    map[common.Hash]*types.Header


The naming is kind of weird. invalidHeaders seems more appropriate.

jwasinger · 2026-04-09T20:09:19Z

+// newPayloadVersioned dispatches to the appropriate version-specific validation
+// before calling the core newPayload logic. Mirrors NewPayloadV1-V5 in
+// eth/catalyst/api.go.
+func (h *engineHandler) newPayloadVersioned(p etNewPayload) (engine.PayloadStatusV1, error) {


Is there a reason to do validation here specific to the payload version? It feels like that's behavior we're trying to test here.

Also (same line) newPayloadVersioned -> newPayloadVersion. typo fix

Oh sorry, it's not calling the engine API handler. It's calling the engine API chasse that's added in this PR

Would be nice if we could just call into existing engine API path instead of introducing a mirror of the logic here. I'm not sure what hurdles would need to be overcome to make that a possibility.

jwasinger · 2026-04-09T21:22:27Z

Assigned myself because I'm going to be pushing some changes to this to try improve where I can and pair down the size of the diff (e.g. #34650 (comment))

spencer-tb · 2026-04-12T11:35:54Z

Just checking this now! Sorry for the referenced commit PR spam! Feel free to start a new PR based off of this one or push to my branch etc

I mostly wanted to gauge whether this was actually feasible from all clients, so I'm glad it seems like a reasonable addition.

We'd like to use: ethereum/execution-specs#2622 as the new default for running tests on clients. It will give us a much faster feedback loop across the board if we go this route :)

jwasinger · 2026-05-01T11:17:06Z

Sorry for the late reply @spencer-tb ! Yes, I intend to finish this one. Will push updates to this branch when I get it to a state I am satisfied with.

spencer-tb force-pushed the feat/evm-enginetest branch 5 times, most recently from 29e3f4c to 00957ea Compare April 3, 2026 16:47

spencer-tb added 3 commits April 3, 2026 17:52

cmd/evm: add enginetest command for direct engine fixture execution

2ef6227

cmd/evm: add --workers flag to blocktest for parallel file processing

58fe592

cmd/evm: add --workers flag to statetest for parallel file processing

d46370a

spencer-tb force-pushed the feat/evm-enginetest branch from 00957ea to d46370a Compare April 3, 2026 16:53

spencer-tb marked this pull request as ready for review April 3, 2026 17:01

spencer-tb requested review from MariusVanDerWijden and lightclient as code owners April 3, 2026 17:01

This was referenced Apr 3, 2026

cmd/evm: add enginetest command and parallel workers for test runners erigontech/erigon#20315

Closed

feat: consume direct support for all EL clients with engine fixture runners ethereum/execution-specs#2650

Open

kevaundray reviewed Apr 3, 2026

View reviewed changes

This was referenced Apr 4, 2026

nethtest: add --engineTest, --stateTest, --jsonout, --workers flags NethermindEth/nethermind#11035

Draft

evmtool: add engine-test subcommand and parallel workers for test runners besu-eth/besu#10184

Draft

cmd/evm: add initial forkchoice update to genesis in enginetest

1063416

kevaundray reviewed Apr 5, 2026

View reviewed changes

spencer-tb added 4 commits April 5, 2026 19:45

cmd/evm: always include error field in JSON output

ffc5899

cmd/evm: add --ndjson flag for streaming JSON output

6f0ae11

tests: move block insertion debug output from stdout to stderr

2ca9720

cmd/evm: remove --ndjson flag (not needed for consume direct)

c8fdb1d

spencer-tb mentioned this pull request Apr 5, 2026

feat(validate): new validate command for direct client binary testing ethereum/execution-specs#2622

Draft

4 tasks

spencer-tb marked this pull request as draft April 5, 2026 23:04

jwasinger reviewed Apr 9, 2026

View reviewed changes

jwasinger self-assigned this Apr 9, 2026

Conversation

spencer-tb commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

evm enginetest

Benchmarks

Hive parity

Usage

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spencer-tb Apr 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MariusVanDerWijden commented Apr 7, 2026

Uh oh!

jwasinger commented Apr 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jwasinger commented Apr 9, 2026

Uh oh!

spencer-tb commented Apr 12, 2026

Uh oh!

jwasinger commented May 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

spencer-tb commented Apr 3, 2026 •

edited

Loading

`evm enginetest`

spencer-tb Apr 4, 2026 •

edited

Loading