Skip to content

implement validator custody with column refill mechanism #7127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: column-syncer
Choose a base branch
from

Conversation

agnxsh
Copy link
Contributor

@agnxsh agnxsh commented May 5, 2025

No description provided.

for i in startIndex..<SLOTS_PER_EPOCH:
let blck = vcus.dag.getForkedBlock(blocks[int(i)]).valueOr: continue
withBlck(blck):
when typeof(forkyBlck).kind < ConsensusFork.Fulu: continue
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this ever happen? it's much cheaper to detect this before the block is ever fetched from the database, based on the slot number + fork schedule.

Copy link
Contributor Author

@agnxsh agnxsh May 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

detect this before the block is ever fetched from the database

the only way to make this cheaper is to see if the earliest data column in the database is against a Fulu block only then it makes sense to skip this check, imo, otherwise everytime we pull a finalized block from the DB, it's safe to check the fork, also say it can be a situation just a couple epochs after Fulu transition that on wanting block range i get EEEEEEFFFFFFF, i want to refill only against the last Fulu blocks, and wanna continue for the first few Es, note that Es and Fs should ideally way more in number, this is just for an example

Copy link

github-actions bot commented May 5, 2025

Unit Test Results

       15 files    2 645 suites   1h 21m 25s ⏱️
  6 546 tests   6 022 ✔️ 524 💤 0
45 406 runs  44 658 ✔️ 748 💤 0

Results for commit 6d364fb.

♻️ This comment has been updated with latest results.

@tersec
Copy link
Contributor

tersec commented May 5, 2025

https://github.com/status-im/nimbus-eth2/actions/runs/14842398481/job/41668253800?pr=7127

2025-05-05T18:21:02.9784725Z Hint: mm: refc; threads: on; opt: speed; options: -d:release
2025-05-05T18:21:02.9787538Z 378176 lines; 215.223s; 10.377GiB peakmem; proj: /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/tests/all_tests.nim; out: /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/nimcache/release/all_tests/all_tests.json [SuccessX]
2025-05-05T18:24:28.9640792Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/iterators.nim: In function ‘_ZN9confutils8loadImplE8typeDescIN4conf14BeaconNodeConfEE3seqI6stringE6string6string4bool4bool3refIN9anonymous16SecondarySourcesEE4procIN4conf14BeaconNodeConfE3refIN9anonymous16SecondarySourcesEEE6string.constprop’:
2025-05-05T18:24:28.9649194Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-confutils/confutils.nim:910:15: note: variable tracking size limit exceeded with ‘-fvar-tracking-assignments’, retrying without
2025-05-05T18:24:28.9786592Z   910 | proc loadImpl[C, SecondarySources](
2025-05-05T18:24:28.9787035Z       |               ^
2025-05-05T18:25:06.9981057Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/iterators.nim: In function ‘_ZN9confutils8loadImplE8typeDescIN4conf14BeaconNodeConfEE3seqI6stringE6string6string4bool4bool3refIN14initBeaconNode16SecondarySourcesEE4procIN4conf14BeaconNodeConfE3refIN14initBeaconNode16SecondarySourcesEEE6string.constprop’:
2025-05-05T18:25:06.9985700Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-confutils/confutils.nim:910:15: note: variable tracking size limit exceeded with ‘-fvar-tracking-assignments’, retrying without
2025-05-05T18:25:06.9986981Z   910 | proc loadImpl[C, SecondarySources](
2025-05-05T18:25:06.9987472Z       |               ^
2025-05-05T18:25:26.2738160Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/iterators_1.nim: In function ‘_ZN17validator_custody25detectNewValidatorCustodyE3refIN17validator_custody16ValidatorCustodyEE’:
2025-05-05T18:25:26.2740847Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/beacon_chain/sync/validator_custody.nim:67:15: error: stack usage is 11312192 bytes [-Werror=stack-usage=]
2025-05-05T18:25:26.2742260Z    67 | proc detectNewValidatorCustody(vcus: ValidatorCustodyRef): seq[ColumnIndex] =
2025-05-05T18:25:26.2742997Z       |               ^
2025-05-05T18:26:16.7631562Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-unittest2/unittest2.nim: In function ‘_ZN22test_gossip_transition18runTestX60gensym3_E6string6string’:
2025-05-05T18:26:16.7736235Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-unittest2/unittest2.nim:1101:15: note: variable tracking size limit exceeded with ‘-fvar-tracking-assignments’, retrying without
2025-05-05T18:26:16.7737632Z  1101 |   proc runTest(suiteName, testName: string): TestStatus {.raises: [], gensym.} =
2025-05-05T18:26:16.7738172Z       |               ^
2025-05-05T18:26:18.2169452Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-unittest2/unittest2.nim:1101:15: note: variable tracking size limit exceeded
2025-05-05T18:26:19.0940375Z lto1: some warnings being treated as errors
2025-05-05T18:26:19.1458084Z make[2]: *** [/tmp/ccScWUVj.mk:305: /tmp/ccDZizXd.ltrans101.ltrans.o] Error 1
2025-05-05T18:26:19.1459044Z make[2]: *** Waiting for unfinished jobs....
2025-05-05T18:26:45.1057573Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-unittest2/unittest2.nim: In function ‘_ZN19test_toblindedblock21runTestX60gensym1069_E6string6string’:
2025-05-05T18:26:45.1134244Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-unittest2/unittest2.nim:1101:15: note: variable tracking size limit exceeded with ‘-fvar-tracking-assignments’, retrying without
2025-05-05T18:26:45.1135690Z  1101 |   proc runTest(suiteName, testName: string): TestStatus {.raises: [], gensym.} =
2025-05-05T18:26:45.1136273Z       |               ^
2025-05-05T18:27:20.9297450Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-unittest2/unittest2.nim: In function ‘_ZN19test_toblindedblock21runTestX60gensym2038_E6string6string’:
2025-05-05T18:27:20.9300358Z /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nim-unittest2/unittest2.nim:1101:15: note: variable tracking size limit exceeded with ‘-fvar-tracking-assignments’, retrying without
2025-05-05T18:27:22.1837786Z lto-wrapper: fatal error: make returned 2 exit status
2025-05-05T18:27:22.1838783Z compilation terminated.
2025-05-05T18:27:22.3302505Z /usr/bin/ld: error: lto-wrapper failed
2025-05-05T18:27:22.3436028Z collect2: error: ld returned 1 exit status
2025-05-05T18:27:22.3468012Z make[1]: *** [nimcache/release/all_tests/all_tests.makefile:3760: build] Error 1

dataColumnRefillEpoch.start_slot, blocks.toOpenArray(0, SLOTS_PER_EPOCH - 1))
for i in startIndex..<SLOTS_PER_EPOCH:
dataColumnRefillEpoch.start_slot, blocks.toOpenArray(0, slot.epoch().int - 1))
for i in startIndex..<slot.epoch().int:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

practically save but not that great to do this because (a) in theory 32/64-bit differences and (b) introduces potential Defect due to int conversion

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see i also has to be int(i) a couple lines later, worth looking at the types here holistically.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's the same technique used in pruneSidecars

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not really great there either, but sure, it's consistent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants