Skip to content

fix(block): fix replay logic#3053

Merged
julienrbrt merged 4 commits intomainfrom
julien/claude-replay-fix
Feb 5, 2026
Merged

fix(block): fix replay logic#3053
julienrbrt merged 4 commits intomainfrom
julien/claude-replay-fix

Conversation

@julienrbrt
Copy link
Member

The replay logic was wrong and leading to app hashes.
This PR fixes it. The correct solution was one shot by Claude 👏🏾

@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2026

The latest Buf updates on your PR. Results from workflow CI / buf-check (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed⏩ skipped✅ passed✅ passedFeb 5, 2026, 7:37 PM

@claude
Copy link
Contributor

claude bot commented Feb 5, 2026

Claude finished @julienrbrt's task in 1m 58s —— View job


Code Review Complete

I've reviewed the replay logic fix. This PR addresses a critical bug in the app hash verification during block replay. The fix is fundamentally correct and well-tested.

Tasks:

  • Read repository guidelines and changed files
  • Analyze the replay logic fix for correctness
  • Review test coverage and quality
  • Check for Go best practices and potential issues
  • Provide detailed feedback

Overall Assessment

Verdict: Approved with minor suggestions

This is a solid fix for a subtle but critical bug in the replay logic. The core issue was that the code was comparing the execution result against header.AppHash, which is actually the input hash (previous state), not the expected output hash. The fix correctly retrieves the expected state at the current height for verification.


Detailed Findings

1. Code Quality & Architecture ✅

Strengths:

  • Clear separation of concerns in the replayer component
  • Good use of interfaces (HeightProvider, Rollbackable)
  • Proper error wrapping with context
  • Well-structured test suite with comprehensive coverage

Observations:

  • The fix correctly distinguishes between "input app hash" (from previous state) and "output app hash" (result of execution)
  • Comments have been significantly improved to clarify the app hash flow (block/internal/common/replay.go:159, 163, 189-190)

2. Correctness Analysis ✅

The Bug:
The original code at line ~206 compared:

if !bytes.Equal(newAppHash, header.AppHash) // WRONG!

This was incorrect because:

  • header.AppHash = the app hash from the previous block (input to execution)
  • newAppHash = the result of executing the current block (output)
  • Comparing input to output would always fail unless app hash doesn't change

The Fix:

expectedState, err := s.store.GetStateAtHeight(ctx, height)
if err == nil {
    if !bytes.Equal(newAppHash, expectedState.AppHash) // CORRECT!

This correctly compares the execution result against the stored state at the current height, which contains the expected app hash.

Critical Insight (replay.go:163-164):
The comment clarifies that GetStateAtHeight(height-1) returns the state AFTER block height-1 was executed, which contains the correct input hash. This is key to understanding the fix.

Graceful Degradation (replay.go:214-220):
The code gracefully handles the case where state doesn't exist yet at the current height during fresh replay, which is smart defensive programming.

3. Error Handling & Resilience ✅

Strengths:

  • Proper error wrapping with fmt.Errorf and %w
  • Clear error messages with height and hash values for debugging (replay.go:197-200)
  • Graceful handling of missing state during replay
  • Good logging at appropriate levels (Debug for normal flow, Error for failures)

Minor Suggestion:
Consider logging the specific error when GetStateAtHeight(ctx, height) fails at line 193, for easier debugging:

expectedState, err := s.store.GetStateAtHeight(ctx, height)
if err == nil {
    // verification logic...
} else {
    s.logger.Debug().
        Uint64("height", height).
        Err(err).  // Add this
        Str("app_hash", hex.EncodeToString(newAppHash)).
        Msg("replayBlock: ExecuteTxs completed (no stored state to verify against)")
}

4. Security ✅

Observations:

  • The fix maintains security by still validating app hash when state is available
  • Graceful degradation for missing state is acceptable since blocks are "known-good" (already validated)
  • No security regressions identified
  • Proper handling of cryptographic hashes (no truncation, proper comparison)

Note on Trust Model:
The comment at replay.go:215-216 states "we trust the execution result since we're replaying validated blocks." This is a reasonable trust assumption for replay scenarios, but ensure the blocks in the store are indeed validated before being stored initially.

5. Testing & Reliability ✅

Test Coverage - Excellent:
The test suite is comprehensive and has been updated to match the corrected logic:

  1. TestReplayer_SyncToHeight_ExecutorBehind (L18-98): Tests basic replay with proper input/output hash distinction
  2. TestReplayer_SyncToHeight_InSync (L100-126): Tests no-op case
  3. TestReplayer_SyncToHeight_ExecutorAhead (L128-155): Tests error case
  4. TestReplayer_SyncToHeight_NoHeightProvider (L157-178): Tests interface check
  5. TestReplayer_SyncToHeight_AtGenesis (L180-204): Tests genesis edge case
  6. TestReplayer_SyncToHeight_MultipleBlocks (L206-311): Tests batch replay
  7. TestReplayer_ReplayBlock_FirstBlock (L313-371): Tests genesis block handling with missing state
  8. TestReplayer_AppHashMismatch (L373-443): Tests verification failure
  9. TestReplayer_AppHashMismatch_NoStoredState (L445-516): NEW TEST - Validates graceful handling when state doesn't exist

Test Quality:

  • Proper use of mocks with testify
  • Clear test names following Go conventions
  • Good coverage of edge cases
  • Mock expectations properly distinguish input vs output hashes (e.g., L50, L65, L72-73)

Key Test Update (replay_test.go:50, 72-73):

AppHash: []byte("app-hash-99"), // This is the input app hash (from state 99)
// ...
mockExec.On("ExecuteTxs", ..., []byte("app-hash-99")).
    Return([]byte("app-hash-100"), nil)

This correctly shows that block 100 takes app-hash-99 as input and produces app-hash-100 as output.

6. Performance & Resource Efficiency ✅

Observations:

  • The fix adds one additional GetStateAtHeight call per replay (line 193)
  • This is acceptable overhead for correctness and is only executed during replay scenarios
  • No allocations or blocking operations in hot paths
  • Proper cleanup with batch operations

Minor Optimization Opportunity:
The code could potentially cache the expected state if multiple operations need it, but this would be premature optimization for replay scenarios.

7. Documentation & Developer Experience ✅

Strengths:

  • Excellent inline comments explaining the app hash flow
  • Clear distinction between "input" and "output" app hash in comments
  • Good logging for debugging (hex-encoded hashes, height information)
  • Proper CHANGELOG.md entry (L23)

CHANGELOG Entry:

- Fix replay logic leading to app hashes by verifying against the wrong block [#3053]

Minor suggestion: Could be more specific: "Fix replay logic app hash mismatch by verifying execution result against correct stored state instead of input header hash"


Specific Code Comments

block/internal/common/replay.go

Line 159 ✅ Good comment clarifying genesis app hash is input

Lines 163-165 ✅ Excellent comment explaining GetStateAtHeight(height-1) semantics

Lines 166-168 ✅ Good simplification - removed redundant GetBlockData call and manual state updates

Lines 188-190 ✅ Critical comment explaining header.AppHash vs expected output

Lines 193-220 ✅ Excellent verification logic with graceful degradation

block/internal/common/replay_test.go

Line 5 ✅ Added errors import for new test case

Lines 42-73 ✅ Properly updated mocks to reflect input/output distinction

Lines 76-84 ✅ New verification mock with correct expected output hash

Lines 329 ✅ Added missing GetLatestHeight mock for first block test

Lines 445-516 ✅ New test case for missing state - excellent edge case coverage


Potential Issues & Suggestions

Minor Issues:

  1. Line 193 logging: Consider logging the error when state fetch fails (mentioned above)

  2. Comment at line 216: "which would be unusual since we fetched the block" - This comment might be misleading. During fresh replay or recovery scenarios, it's actually expected that the state might not exist yet at the current height. Consider rewording:

    // If this state doesn't exist (which can occur during fresh replay), we skip verification.
  3. Type assertion check: At line 48, the code checks if executor implements HeightProvider. This is good defensive programming, but consider documenting what executors should implement this interface and when it's expected to be missing.

No Breaking Changes:

The fix maintains the same external interface and behavior, only correcting the internal verification logic.


Recommendations

  1. Merge this PR - The fix is correct and well-tested
  2. Consider the minor logging and comment improvements suggested above
  3. Ensure monitoring/alerting exists for app hash mismatches in production
  4. Document the app hash flow in architecture docs if not already present

Summary

This PR fixes a critical bug where replay logic was comparing execution output against the wrong hash (input instead of expected output). The fix:

  • ✅ Correctly retrieves and compares against stored state at current height
  • ✅ Gracefully handles missing state during fresh replay
  • ✅ Adds comprehensive test coverage including new edge cases
  • ✅ Improves documentation with clear comments
  • ✅ Maintains backward compatibility

The fact that Claude suggested this solution in "one shot" speaks to the clarity of the problem once the input/output hash distinction was understood. Excellent work! 👏🏾

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @julienrbrt, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical bug in the block replay mechanism where the application hash (AppHash) logic was flawed, potentially causing incorrect state synchronization. The changes ensure that the correct AppHash is used as input for block execution and that the verification process accurately compares the execution output against the expected state, thereby stabilizing the replay process and preventing discrepancies in the application state.

Highlights

  • Corrected Previous State Retrieval for Replay: The logic for obtaining the previous block's application hash (AppHash) during replay has been refined. Previously, the AppHash was redundantly fetched from the previous block's header, potentially leading to inconsistencies. Now, the GetStateAtHeight(height-1) call is solely relied upon, as it correctly provides the state after the previous block's execution, including its output AppHash.
  • Improved App Hash Verification Logic: The mechanism for verifying the application hash after executing transactions has been overhauled. The previous implementation incorrectly compared the newAppHash (result of current block execution) against the header.AppHash (which is the input hash from the previous block). The new logic correctly fetches the expectedState.AppHash for the current block's height from the store for verification. If the state for the current height is not found (e.g., during a fresh replay), verification is gracefully skipped, trusting the execution result.
  • Enhanced Test Coverage for Replay Logic: Unit tests for the replayer have been updated and expanded to reflect the corrected logic. This includes adjusting mock expectations for AppHash values to distinguish between input and output hashes, removing redundant GetBlockData calls for previous states, and adding a new test case to ensure correct behavior when state for the current height is not yet available for verification.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • block/internal/common/replay.go
    • Removed redundant fetching of previous block's AppHash, relying solely on GetStateAtHeight(height-1) for the correct input AppHash.
    • Updated comment for genesis block's AppHash to clarify its role as input.
    • Refactored AppHash verification logic to compare newAppHash against the expectedState.AppHash for the current height.
    • Added a fallback to skip AppHash verification if the state for the current height is not found, which is appropriate during initial replay.
  • block/internal/common/replay_test.go
    • Added errors package import.
    • Updated mock expectations in TestReplayer_SyncToHeight_ExecutorBehind to align with the refined previous state and verification logic.
    • Adjusted AppHash values in test mocks to correctly represent input vs. output hashes.
    • Removed GetBlockData mock expectations for previous blocks, reflecting changes in replay.go.
    • Added GetStateAtHeight mock expectations for the current height to support the new verification logic.
    • Introduced mockExec.On("GetLatestHeight") for TestReplayer_ReplayBlock_FirstBlock to simulate starting from genesis.
    • Added a new test case TestReplayer_AppHashMismatch_NoStoredState to validate behavior when current state is not found for verification.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@julienrbrt julienrbrt requested a review from tac0turtle February 5, 2026 19:36
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request provides a critical fix to the block replay logic. The previous implementation incorrectly determined the input AppHash for transaction execution and performed an invalid verification of the resulting AppHash. The changes correct this by using the state from height-1 to get the input AppHash and verifying the new AppHash against the stored state at the current height. The test suite has been updated accordingly to reflect these logic changes and includes new test cases for app hash mismatches and scenarios where state for verification is not yet present. The fix appears correct and robust. I have one suggestion to improve the error handling during the state verification step to make it more resilient.

Comment on lines +194 to 220
if err == nil {
// State exists, verify the app hash matches
if !bytes.Equal(newAppHash, expectedState.AppHash) {
err := fmt.Errorf("app hash mismatch at height %d: expected %s got %s",
height,
hex.EncodeToString(expectedState.AppHash),
hex.EncodeToString(newAppHash),
)
s.logger.Error().
Str("expected", hex.EncodeToString(expectedState.AppHash)).
Str("got", hex.EncodeToString(newAppHash)).
Uint64("height", height).
Err(err).
Msg("app hash mismatch during replay")
return err
}
s.logger.Debug().
Uint64("height", height).
Str("app_hash", hex.EncodeToString(newAppHash)).
Msg("replayBlock: app hash verified against stored state")
} else {
// State doesn't exist yet, we trust the execution result since we're replaying validated blocks.
s.logger.Debug().
Uint64("height", height).
Err(err).
Msg("app hash mismatch during replay")
return err
Str("app_hash", hex.EncodeToString(newAppHash)).
Msg("replayBlock: ExecuteTxs completed (no stored state to verify against)")
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This logic handles errors from s.store.GetStateAtHeight by assuming any error means the state was not found, and then proceeds without verification. This is risky because it could mask other critical errors from the store, such as data corruption, disk I/O issues, or unmarshalling problems. In such cases, the replayer would silently proceed and potentially persist an incorrect state.

It would be safer to explicitly check for a 'not found' error and treat all other errors as fatal for the replay of this block.

A more robust implementation would be to check for the specific 'not found' error. For example:

expectedState, err := s.store.GetStateAtHeight(ctx, height)
if err != nil {
    // It's important to distinguish 'not found' from other errors.
    // This might require exporting an error from the store package or using errors.Is with the underlying datastore error.
    if !isNotFoundError(err) { // isNotFoundError is a placeholder for the actual check
        return fmt.Errorf("failed to get expected state for verification at height %d: %w", height, err)
    }

    // State not found, proceed without verification (as intended)
    s.logger.Debug().
        Uint64("height", height).
        Str("app_hash", hex.EncodeToString(newAppHash)).
        Msg("replayBlock: ExecuteTxs completed (no stored state to verify against)")
} else {
    // State exists, perform verification
    if !bytes.Equal(newAppHash, expectedState.AppHash) {
        // ... error handling for mismatch
    }
}

Please consider refining this error handling to make the replay process more resilient to unexpected storage issues.

@julienrbrt julienrbrt enabled auto-merge February 5, 2026 19:51
@codecov
Copy link

codecov bot commented Feb 5, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.62%. Comparing base (0425059) to head (da6a0d3).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3053      +/-   ##
==========================================
+ Coverage   56.56%   56.62%   +0.06%     
==========================================
  Files         118      118              
  Lines       12151    12152       +1     
==========================================
+ Hits         6873     6881       +8     
+ Misses       4536     4530       -6     
+ Partials      742      741       -1     
Flag Coverage Δ
combined 56.62% <100.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@julienrbrt julienrbrt added this pull request to the merge queue Feb 5, 2026
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Feb 5, 2026
@julienrbrt julienrbrt merged commit aedd573 into main Feb 5, 2026
30 of 31 checks passed
@julienrbrt julienrbrt deleted the julien/claude-replay-fix branch February 5, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants