Skip to content

Update block validation script to use reflink#6999

Merged
wileyj merged 15 commits intostacks-network:developfrom
wileyj:feat/reflink_validation
Apr 6, 2026
Merged

Update block validation script to use reflink#6999
wileyj merged 15 commits intostacks-network:developfrom
wileyj:feat/reflink_validation

Conversation

@wileyj
Copy link
Copy Markdown
Collaborator

@wileyj wileyj commented Mar 17, 2026

Update block validation script to use less disk

  • if slice dir or supplied local chainstate is mounted on an xfs partition with reflink enabled, use it to reduce storage required (cp_arg)
  • removed html output, only create the txt file recording the results of all validation processes
  • cleanup and comments

The diff is larger than the change suggests from the comment/formatting cleanup, as well as removing the html output which isn't particularly useful for this script in this repo (better to handle that, if desired, in a wrapper script or separate process).

The main changes here are the addition of checking for a reflink enabled disk, and setting a specific cp arg if it's available for the filesystem where the chainstate, or (in the case of downloading an archive) where the slice data is.

this allows for multiple copies of the chainstate with no addtional required storage (ex: 12 copies of the chainstate uses ~1.1TB, whereas in the past it would be close to 6TB).

Previous functionality persists, so if you run this on an ext4 etc partition, it will execute as it has in the past.

cc @cylewitruk

- if slice  dir or supplied local chainstate is mounted on an xfs partition with reflink enabled, use it to reduce storage required (cp_arg)
- removed html output, only create the txt file recording the results of all validation processes
- cleanup and comments
@wileyj
Copy link
Copy Markdown
Collaborator Author

wileyj commented Mar 17, 2026

One thing to note, i have not run shellcheck on this - if you do, there are likely several (many) things to fix but i felt it was better to get this change merged and update later since it's only a helper/testing script.

Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh
Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh
Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh Outdated
Copy link
Copy Markdown
Contributor

@federico-stacks federico-stacks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an issue with check_reflink that need to be addressed.
I also added a bunch of nits while looking at the script. Feel free to address them or keep them for later as you mentioned here: #6999 (comment)

Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh Outdated
Copy link
Copy Markdown
Contributor

@federico-stacks federico-stacks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check_reflink logic should work properly now. Added a couple of nits coming from the fixes

Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh Outdated
wileyj and others added 3 commits March 18, 2026 07:29
Co-authored-by: Cyle Witruk <236413682+cylewitruk-stacks@users.noreply.github.com>
Co-authored-by: Cyle Witruk <236413682+cylewitruk-stacks@users.noreply.github.com>
@wileyj
Copy link
Copy Markdown
Collaborator Author

wileyj commented Mar 18, 2026

@cylewitruk-stacks mentioned an alternate way to check/use reflink which would apply to other filesytem types vs the xfs only option in this PR.

will try and refactor slightly to account for non-xfs but reflink enabled filesystems (currently i'm checking specifically for xfs via the xfs_info binary which obviously will not work for btrfs/apfs etc)

@wileyj
Copy link
Copy Markdown
Collaborator Author

wileyj commented Mar 19, 2026

@cylewitruk-stacks mentioned an alternate way to check/use reflink which would apply to other filesytem types vs the xfs only option in this PR.

will try and refactor slightly to account for non-xfs but reflink enabled filesystems (currently i'm checking specifically for xfs via the xfs_info binary which obviously will not work for btrfs/apfs etc)

Refactored - seems to work well!

Comment thread contrib/tools/block-validation.sh Outdated
Comment thread contrib/tools/block-validation.sh Outdated
@coveralls
Copy link
Copy Markdown

coveralls commented Mar 25, 2026

Pull Request Test Coverage Report for Build 23864564498

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 1325 unchanged lines in 51 files lost coverage.
  • Overall coverage remained the same at 85.712%

Files with Coverage Reduction New Missed Lines %
libsigner/src/session.rs 1 65.41%
stackslib/src/burnchains/burnchain.rs 1 71.93%
stackslib/src/chainstate/stacks/index/node.rs 1 86.61%
stackslib/src/chainstate/stacks/miner.rs 1 83.37%
stackslib/src/clarity_vm/clarity.rs 1 93.7%
stackslib/src/net/download/nakamoto/download_state_machine.rs 1 90.57%
stackslib/src/net/mod.rs 1 77.99%
stackslib/src/net/stackerdb/mod.rs 1 87.13%
stackslib/src/util_lib/bloom.rs 1 92.48%
stacks-signer/src/client/stacks_client.rs 1 86.19%
Totals Coverage Status
Change from base Build 23400582460: 0.0%
Covered Lines: 186632
Relevant Lines: 217744

💛 - Coveralls

jcnelson
jcnelson previously approved these changes Apr 1, 2026
Copy link
Copy Markdown
Member

@jcnelson jcnelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it works, then it works. I don't think this is in CI yet, but have you run shellcheck on the new script? I don't think I saw any footguns, but shellcheck can find them better than I can ;)

@wileyj
Copy link
Copy Markdown
Collaborator Author

wileyj commented Apr 1, 2026

If it works, then it works. I don't think this is in CI yet, but have you run shellcheck on the new script? I don't think I saw any footguns, but shellcheck can find them better than I can ;)

it wasn't done on this update, only the original script.
will do them now -i'm 100% certain there will be changes required (unquoted variables and such).

@wileyj wileyj dismissed stale reviews from jcnelson and federico-stacks via 3db0140 April 1, 2026 18:33
@jcnelson jcnelson self-requested a review April 6, 2026 02:34
@wileyj wileyj added this pull request to the merge queue Apr 6, 2026
Merged via the queue into stacks-network:develop with commit dbf3086 Apr 6, 2026
9 checks passed
@wileyj wileyj deleted the feat/reflink_validation branch April 6, 2026 03:35
@github-actions
Copy link
Copy Markdown

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Apr 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants