boba-community docker images for geth mainnet do not sync as-is

[letsbangout]

I've been running RPCs for this chain for about a year now. I have experienced persistent synchronization problems which I've documented in another Github issue. I see that a fresh pull of this project, along with a fresh pull of the snapshot URL mentioned in docker-compose-boba-mainnet-geth.yml does not apparently yield a working RPC. It seems stuck on a particular block according to the logs below from op-node. The problematic portion seems to be "Derivation process temporary error". I have pointed by node at a local ETH full node, as well as a local ETH archive node. Both L1's experience this problem. Also I notice that by default, op-node v1.6.3 is used, isn't this an error? Should it not be 1.6.10 or 1.6.11 to account for all hardforks?

Logs:

t=2024-10-28T23:00:10+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmH38oE9z8iD6Z9KFdZE5jM6URsdseY9T5yw7neH9VpYir
t=2024-10-28T23:00:10+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm2wktiKuKNdFhTYM6uZGEJqLvWWjgHEDgSSKz9q17udsm
t=2024-10-28T23:00:10+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkyS5AoREMmntTSR52Yd4LZt1WGXqboVA6AVKMG8Q7fJhi
t=2024-10-28T23:00:13+0000 lvl=info msg="Received signed execution payload from p2p" id=0x51690c6f75cd68f982fdfcf2edbacb498c73cda51ecb3697c64dfdfb32f78cf3:9575785 peer=16Uiu2HAm9GxuiUzG7JcxxMJureGStSVH9DRmfDfsMLaetXR9LpJc
t=2024-10-28T23:00:13+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x51690c6f75cd68f982fdfcf2edbacb498c73cda51ecb3697c64dfdfb32f78cf3:9575785
t=2024-10-28T23:00:13+0000 lvl=warn msg="Derivation process temporary error" attempts=38 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x2c1e6d76bb36f1379ad3dd6fe5f0e842d9aea184d5d2b25dc38bc1971e28b71e:19671339: failed to fetch blob sidecars for slot 8873866 block 0x2c1e6d76bb36f1379ad3dd6fe5f0e842d9aea184d5d2b25dc38bc1971e28b71e:19671339: failed request with status 404: {\"code\":404,\"message\":\"NOT_FOUND: beacon block at slot 8873866\",\"stacktraces\":[]}"
t=2024-10-28T23:00:15+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm2wktiKuKNdFhTYM6uZGEJqLvWWjgHEDgSSKz9q17udsm
t=2024-10-28T23:00:15+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmH38oE9z8iD6Z9KFdZE5jM6URsdseY9T5yw7neH9VpYir
t=2024-10-28T23:00:15+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkyS5AoREMmntTSR52Yd4LZt1WGXqboVA6AVKMG8Q7fJhi
t=2024-10-28T23:00:16+0000 lvl=info msg="Received signed execution payload from p2p" id=0x6214b113463f3800ab8f22dd85ad1a69ada21838c18f6e95f1b98663fdfc67a0:9575787 peer=16Uiu2HAm9GxuiUzG7JcxxMJureGStSVH9DRmfDfsMLaetXR9LpJc
t=2024-10-28T23:00:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x6214b113463f3800ab8f22dd85ad1a69ada21838c18f6e95f1b98663fdfc67a0:9575787
t=2024-10-28T23:00:19+0000 lvl=info msg="Received signed execution payload from p2p" id=0x47876d7ba8c2de06b7126509f31381459d2f2e1d0055af6f62b0eccce06cd02c:9575788 peer=16Uiu2HAm9GxuiUzG7JcxxMJureGStSVH9DRmfDfsMLaetXR9LpJc
t=2024-10-28T23:00:19+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x47876d7ba8c2de06b7126509f31381459d2f2e1d0055af6f62b0eccce06cd02c:9575788
t=2024-10-28T23:00:20+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmH38oE9z8iD6Z9KFdZE5jM6URsdseY9T5yw7neH9VpYir
t=2024-10-28T23:00:20+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkyS5AoREMmntTSR52Yd4LZt1WGXqboVA6AVKMG8Q7fJhi
t=2024-10-28T23:00:20+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm2wktiKuKNdFhTYM6uZGEJqLvWWjgHEDgSSKz9q17udsm
t=2024-10-28T23:00:24+0000 lvl=warn msg="Derivation process temporary error" attempts=39 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x2c1e6d76bb36f1379ad3dd6fe5f0e842d9aea184d5d2b25dc38bc1971e28b71e:19671339: failed to fetch blob sidecars for slot 8873866 block 0x2c1e6d76bb36f1379ad3dd6fe5f0e842d9aea184d5d2b25dc38bc1971e28b71e:19671339: failed request with status 404: {\"code\":404,\"message\":\"NOT_FOUND: beacon block at slot 8873866\",\"stacktraces\":[]}"
t=2024-10-28T23:00:25+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm2wktiKuKNdFhTYM6uZGEJqLvWWjgHEDgSSKz9q17udsm
t=2024-10-28T23:00:25+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkyS5AoREMmntTSR52Yd4LZt1WGXqboVA6AVKMG8Q7fJhi
t=2024-10-28T23:00:25+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmH38oE9z8iD6Z9KFdZE5jM6URsdseY9T5yw7neH9VpYir
t=2024-10-28T23:00:25+0000 lvl=info msg="Received signed execution payload from p2p" id=0x95a1c84a958647a35e7029d4e477c8f75f5d57eab164037f3afaf112d3dbd8b7:9575791 peer=16Uiu2HAm9GxuiUzG7JcxxMJureGStSVH9DRmfDfsMLaetXR9LpJc
t=2024-10-28T23:00:25+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x95a1c84a958647a35e7029d4e477c8f75f5d57eab164037f3afaf112d3dbd8b7:9575791
t=2024-10-28T23:00:29+0000 lvl=info msg="Received signed execution payload from p2p" id=0xc4cc65c7faade1a719f02f66d8c52649b8bf0b7b8e90dfdae25751b7329ec5df:9575793 peer=16Uiu2HAm9GxuiUzG7JcxxMJureGStSVH9DRmfDfsMLaetXR9LpJc
t=2024-10-28T23:00:29+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xc4cc65c7faade1a719f02f66d8c52649b8bf0b7b8e90dfdae25751b7329ec5df:9575793
t=2024-10-28T23:00:30+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm2wktiKuKNdFhTYM6uZGEJqLvWWjgHEDgSSKz9q17udsm
t=2024-10-28T23:00:30+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkyS5AoREMmntTSR52Yd4LZt1WGXqboVA6AVKMG8Q7fJhi
t=2024-10-28T23:00:30+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmH38oE9z8iD6Z9KFdZE5jM6URsdseY9T5yw7neH9VpYir
t=2024-10-28T23:00:30+0000 lvl=info msg="Received signed execution payload from p2p" id=0xb7b13be50c2e2bc6c3627085809412520995e26908bc540ecc5c2041528dccc9:9575794 peer=16Uiu2HAm9GxuiUzG7JcxxMJureGStSVH9DRmfDfsMLaetXR9LpJc
t=2024-10-28T23:00:30+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xb7b13be50c2e2bc6c3627085809412520995e26908bc540ecc5c2041528dccc9:9575794
t=2024-10-28T23:00:34+0000 lvl=warn msg="Derivation process temporary error" attempts=40 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x2c1e6d76bb36f1379ad3dd6fe5f0e842d9aea184d5d2b25dc38bc1971e28b71e:19671339: failed to fetch blob sidecars for slot 8873866 block 0x2c1e6d76bb36f1379ad3dd6fe5f0e842d9aea184d5d2b25dc38bc1971e28b71e:19671339: failed request with status 404: {\"code\":404,\"message\":\"NOT_FOUND: beacon block at slot 8873866\",\"stacktraces\":[]}"
t=2024-10-28T23:00:35+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkyS5AoREMmntTSR52Yd4LZt1WGXqboVA6AVKMG8Q7fJhi
t=2024-10-28T23:00:35+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm2wktiKuKNdFhTYM6uZGEJqLvWWjgHEDgSSKz9q17udsm
t=2024-10-28T23:00:35+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmH38oE9z8iD6Z9KFdZE5jM6URsdseY9T5yw7neH9VpYir
t=2024-10-28T23:00:40+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm2wktiKuKNdFhTYM6uZGEJqLvWWjgHEDgSSKz9q17udsm
t=2024-10-28T23:00:40+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkyS5AoREMmntTSR52Yd4LZt1WGXqboVA6AVKMG8Q7fJhi
t=2024-10-28T23:00:40+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmH38oE9z8iD6Z9KFdZE5jM6URsdseY9T5yw7neH9VpYir
t=2024-10-28T23:00:40+0000 lvl=info msg="Received signed execution payload from p2p" id=0x7939a07b090850bbc282c9f21a6b397c3eea77b8564012f9ac19b53ca508dc3b:9575799 peer=16Uiu2HAm9GxuiUzG7JcxxMJureGStSVH9DRmfDfsMLaetXR9LpJc
t=2024-10-28T23:00:40+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x7939a07b090850bbc282c9f21a6b397c3eea77b8564012f9ac19b53ca508dc3b:9575799
t=2024-10-28T23:00:42+0000 lvl=info msg="Received signed execution payload from p2p" id=0xcb88aa3a8c7ac39513b903f9c4634769df7c58c7e245ad383f80f46960d93aca:9575800 peer=16Uiu2HAm9GxuiUzG7JcxxMJureGStSVH9DRmfDfsMLaetXR9LpJc
t=2024-10-28T23:00:42+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xcb88aa3a8c7ac39513b903f9c4634769df7c58c7e245ad383f80f46960d93aca:9575800
t=2024-10-28T23:00:44+0000 lvl=warn msg="Derivation process temporary error" attempts=41 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x2c1e6d76bb36f1379ad3dd6fe5f0e842d9aea184d5d2b25dc38bc1971e28b71e:19671339: failed to fetch blob sidecars for slot 8873866 block 0x2c1e6d76bb36f1379ad3dd6fe5f0e842d9aea184d5d2b25dc38bc1971e28b71e:19671339: failed request with status 404: {\"code\":404,\"message\":\"NOT_FOUND: beacon block at slot 8873866\",\"stacktraces\":[]}"

[letsbangout]

Your website at https://docs.boba.network/developer/node-operators/snapshot-downloads lists two geth snapshots, one is https://boba-db.s3.us-east-2.amazonaws.com/mainnet/boba-mainnet-geth-db-114909.tgz from April, the other is https://boba-db.s3.us-east-2.amazonaws.com/mainnet/boba-mainnet-geth-snapshot-5683043.tgz from July - the former is the one I mentioned above which is also linked in the docker-compose..yml

I just wiped my data directory and tried the latter snapshot. I also forced docker to use op-node 1.6.11 and geth v1.101408.0. I obtained these versions from your website at https://docs.boba.network/developer/node-operators/software-release which mentions they are the minimum acceptable software versions. Now I see a similar error, but it errors on a different block. I have again tried a different RPC with no change. The common variable is obviously the snapshot data.

t=2024-10-29T02:23:22+0000 lvl=warn msg="Engine temporary error" err="derivation failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x8842812c4dc4eb11b122f188f82506b6e286d3ac755a4374c7a5f7880591500d:20421299: failed to fetch blob sidecars for slot 9628482 block 0x8842812c4dc4eb11b122f188f82506b6e286d3ac755a4374c7a5f7880591500d:20421299: #returned blobs(0) != #requested blobs(1)"

[letsbangout]

So this may be obvious to some of you, but it seems the importance of a good ETH beacon archive is crucial. The basis of these errors seems to be a lack of ETH beacon blob data. Many beacon clients are set to prune this data. For example, the pruning policy of Lighthouse by default is 4096 epochs, at 6.4 minutes each, that only gives about 18 days worth of blob data retention from the tip.

Therefore, perhaps the following changes should be made:

1. A documentation update in the docker-compose-boba-mainnet-geth.yml to point to the later snapshot from July, along with mention that simply having a 'full Ethereum node' is not sufficient, it must be at least a "full ETH1" plus "many weeks/months of ETH2" or however you choose to word it. In any event, it should be emphasized that it is necessary to point ETH2_HTTP at a Beacon endpoint which retains at least all of the blobs since the snapshot date.

2. Static version updates to the version of op-node and geth used to run new nodes, so that newbies are not confused.