feat: Add reproducible builds release workflows and push images to DockerHub #7614
Conversation
|
|
|
Ubuntu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
MoeMahhouk
force-pushed
the
unstable
branch
2 times, most recently
from
82dcfef to
238fbaa
Compare
|
Doing some testing on this, will post the comment when ready |
|
Some required checks have failed. Could you please take a look @MoeMahhouk? 🙏 |
mergify
bot
added
waiting-on-author
| - name: Install build dependencies and cross-compilation tools | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y libclang-dev cmake gcc-aarch64-linux-gnu |
There was a problem hiding this comment.
This wouldn't compile for now: https://github.com/sigp/lighthouse/actions/runs/15705132140/job/44642272292?pr=7614
I have tested it in my local branch and added some dependencies, you can refer to: chong-he#8
There was a problem hiding this comment.
yeah it is probably missing the g++-aarch64-linux-gnu as it might be needed for cross compilation for ARM architecture. I will experiment more on seeing what makes it build sucessfully
| if cmp lighthouse-build-1-arm64 lighthouse-build-2-arm64; then | ||
| echo "✅ ARM64 binaries are identical - reproducible build PASSED" | ||
| else | ||
| echo "❌ ARM64 binaries differ - reproducible build FAILED" |
There was a problem hiding this comment.
In my testing, the reproducible for ARM64 is failing as the SHA256 for both builds are different (x86_64 passes): https://github.com/chong-he/lighthouse/actions/runs/15815928865/job/44574695858
There was a problem hiding this comment.
Yes, something I couldn't grasp on why the github runner is having some non-deterministic factors affecting reproducibility while running this locally does not cause different hashes.
I will by-pass this issue but building those binaries within the reproducible docker that we have now and extract the binaries from those builds to compare. So far it seems more successful and having clean build environment without potential caching residues affecting reproducibility.
There was a problem hiding this comment.
Yes, I can see that it passes the CI now, for the reproducible-build
| @@ -0,0 +1,142 @@ | |||
| name: release-reproducible | |||
There was a problem hiding this comment.
Some comments:
-
From my understanding this is about pushing the docker image, without checking if it is reproducible? Would it be better if we can add a condition such as
"if reproducible-build is successful, then only push this docker image"? -
In my testing, the ARM64 docker image is taking ~5 hours to build: https://github.com/chong-he/lighthouse/actions/workflows/release-reproducible.yml
I am not sure if it is because of my local repo, or due to the runner being slow, or due to configs in the file -
Since this is also pushing docker images, would it be better to put under: https://github.com/sigp/lighthouse/blob/unstable/.github/workflows/docker.yml
We can also modify some parameters to be consistent with other docker builds, for example, the OS that's running on (pinnedubuntu-22.04orubuntu-latest),docker/build-push-action@v5orv6
There was a problem hiding this comment.
- From my understanding this is about pushing the docker image, without checking if it is reproducible? Would it be better if we can add a condition such as
"if reproducible-build is successful, then only push this docker image"?
You are correct, I left that out in case you don't want to block docker container releases based on reproducibility but we can for sure added a script that tests the reproducibility similar to reproducible build workflow and only then push the resulting artifacts once successful to dockerhub
- In my testing, the ARM64 docker image is taking ~5 hours to build: https://github.com/chong-he/lighthouse/actions/workflows/release-reproducible.yml
I am not sure if it is because of my local repo, or due to the runner being slow, or due to configs in the file
Yes, the github runner seems to be x86 architecture, so the docker build for ARM64 is being emulated by qemu and therefore very slow compared to building it on a native ARM architecture machine which takes something like 20 mins similar to the x86 build on a x86 architecture hardware. Do you have suggestions or options to customize/select the runner machine architecture for ARM builds?
Edit: I found out today that github is offering a preview version of arm runners. I am trying them now and seems to be working. It build the binary within docker build in less than 20mins similar to the x86 based runner
- Since this is also pushing docker images, would it be better to put under: https://github.com/sigp/lighthouse/blob/unstable/.github/workflows/docker.yml
We can also modify some parameters to be consistent with other docker builds, for example, the OS that's running on (pinned ubuntu-22.04 or ubuntu-latest), docker/build-push-action@v5 or v6
I'd say that is your call to make, I separated them here to be targeting reproducible docker container images instead than a normal docker image. Hence, the version pinning of all dependencies and the specific docker image hash we are building with to avoid any factor that might impact the reproducibility of the build process.
There was a problem hiding this comment.
Reply to:
- Makes sense, not sure what is the best way, we will hear from the devs about this
- Good to know!
- will hear from the devs too
| cargo cache -a | ||
| rm -rf target/ | ||
|
|
||
| - name: Build Lighthouse (second build) |
There was a problem hiding this comment.
The two builds in CI will take about 40 mins to complete, which could be time consuming for every PR to run. One option is to run theese CI checks when there is a new release. We could add a condition like the syncing test:
to only trigger this reproducible build checks by having a label that's tagged with a release, for example, a label that starts with
v
There was a problem hiding this comment.
I can remove the on pull requests and only leave the cronjob which runs it individually every 2 days and giving you insights in-case something broke reproducibility, wdyt?
We could also add the condition like syncing in case you still would like to have this be triggered for tagged pull requests.
Thanks for reviewing it. |
Issue Addressed
This pull request introduces workflows and updates to ensure reproducible builds for the Lighthouse project. It adds two GitHub Actions workflows for building and testing reproducible Docker images and binaries, updates the
Makefileto streamline reproducible build configurations, and modifies theDockerfile.reproducibleto align with the new build process. Additionally, it removes thereproducibleprofile fromCargo.toml.Proposed Changes
New GitHub Actions Workflows:
.github/workflows/release-reproducible.yml: Adds a workflow to build and push reproducible multi-architecture Docker images for releases, including support for dry runs..github/workflows/reproducible-build.yml: Adds a workflow to test reproducibility by comparing binaries built twice forx86_64andaarch64architectures, with failure reporting and artifact uploads.Build Configuration Updates:
Makefile: Refactors reproducible build targets, centralizes environment variables for reproducibility, and updates Docker build arguments forx86_64andaarch64architectures.Dockerfile.reproducible: Updates the base Rust image to version 1.86, removes hardcoded reproducibility settings, and delegates build logic to theMakefile.Profile Removal:
Cargo.toml: Removes thereproducibleprofile, simplifying build configurations and relying on external tooling for reproducibility.Additional Info
This is mainly a follow up to this work #6799 where I refine the reproducible build configuration to simplify the CI workflow to generate the reproducible images and pushes them to DockerHub. I also added a cron job workflow (inspired from the Reth repo) that checks every two days or pull requests that touches files that might affect reproducibility to catch potential regressions.
In case, this is too much, let me know and I can create a separate PR for this to be merged later when necessary
close #7486
close #7485