Rust build stops be reproducible, after adding/removing one assert

[qarmin]

One user of my app reported that builds of my app are not reproducible.
I started to minimize the project, but found that randomly after removing code or dependencies, it either produces reproducible or non-reproducible builds.
Finally, I managed to get reproducible builds just by removing a single assert in the code.
This fails in exactly the same way on my OS, with both Mold and the default linker.

The strangest part for me is that the bug seems to be related to the size of the project.
After removing other unrelated parts, I was also able to make the project reproducible.

Steps to reproduce

• download and unpack project
• execute this command inside folder, to compile app multiple times and compare md5 hashes

rm czkawka_cli_*;N=3; for i in $(seq 1 $N); do rm -rf target || true && cargo build && mv target/debug/czkawka_cli czkawka_cli_$i; done; md5sum czkawka_cli_*

• you should get 3 different hashes
  
• go to czkawka_core/src/progress_data.rs
• remove inside file, comment and assert below 56 line
• run the oneliner again
• all hashes should be equal

Project - czkawka.zip

Meta

rustc --version --verbose:

rustc 1.86.0 (05f9846f8 2025-03-31)
binary: rustc
commit-hash: 05f9846f893b09a1be1fc8560e33fc3c815cfecb
commit-date: 2025-03-31
host: x86_64-unknown-linux-gnu
release: 1.86.0
LLVM version: 19.1.7

and also

rustc 1.88.0-nightly (077cedc2a 2025-04-19)
binary: rustc
commit-hash: 077cedc2afa8ac0b727b7a6cbe012940ba228deb
commit-date: 2025-04-19
host: x86_64-unknown-linux-gnu
release: 1.88.0-nightly
LLVM version: 20.1.2

but according to qarmin/czkawka#1507, this happens also with rust 1.85.0 from debian repo - https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/rust-czkawka-cli.html

[bjorn3]

It seems like something is causing mono items to get reordered within a codegen unit. Can you run rustc -Zls=root /path/to/libczkawka_core.rlib on two runs that should be identical yet are not? I'm curious if the crate hash is already different or if the non-determinism happens later.

[qarmin]

Only hash is different

1.txt
2.txt

[saethlin]

I have debugged quite a few of those reports by Debian packagers (not users). In every case, they were caused by a build script somewhere in your dependency tree, usually one iterating over a HashMap (which with the default hasher, is not reproducible).

[saethlin]

The responsible crate seems to be czkawka_core. To recreate the problem, run:

cargo clean -p czkawka_core && cargo build -p czkawka_core && md5sum target/debug/deps/libczkawka_core-f7031ec472ee5a6f.rlib 

A few times. Each time produces a different hash. Crates which depend on this one also have an unstable hash, but nothing else does.

I've also confirmed that the hash becomes stable if I remove everything from the crate.

Unfortunately the module graph in this crate has a LOT of cycles so it's going to be very hard to reduce this.

[saethlin]

I found a loop over a HashMap in a proc-macro crate that you are using: https://github.com/kellpossible/cargo-i18n/blob/ceb3da0ee3acf91b17a7a52e02642267ddb47a3d/i18n-embed-fl/src/lib.rs#L550

Which exactly lines up with the instability in the output of cargo expand I'm seeing:

cargo expand -p czkawka_core > old
cargo clean
cargo expand -p czkawka_core > new
git diff --no-index old new

diff --git a/old b/new
index f2be1cc..ef8766a 100644
--- a/old
+++ b/new
@@ -4182,11 +4182,11 @@ pub mod common_dir_traversal {
                                 "core_cannot_read_entry_dir",
                                 {
                                     let mut args = std::collections::HashMap::new();
+                                    args.insert("reason", e.to_string().into());
                                     args.insert(
                                         "dir",
                                         current_folder.to_string_lossy().to_string().into(),
                                     );
-                                    args.insert("reason", e.to_string().into());
                                     args
                                 },
                             )
@@ -4211,11 +4211,11 @@ pub mod common_dir_traversal {
                                 "core_cannot_read_metadata_dir",
                                 {
                                     let mut args = std::collections::HashMap::new();
-                                    args.insert("reason", e.to_string().into());
                                     args.insert(
                                         "dir",
                                         current_folder.to_string_lossy().to_string().into(),
                                     );
+                                    args.insert("reason", e.to_string().into());
                                     args
                                 },
                             )
@@ -4291,11 +4291,11 @@ pub mod common_dir_traversal {
                                     "core_folder_no_modification_date",
                                     {
                                         let mut args = std::collections::HashMap::new();
-                                        args.insert("reason", e.to_string().into());
                                         args.insert(
                                             "name",
                                             current_file_name.to_string_lossy().to_string().into(),
                                         );
+                                        args.insert("reason", e.to_string().into());
                                         args
                                     },
                                 )
@@ -4308,11 +4308,11 @@ pub mod common_dir_traversal {
                                     "core_file_no_modification_date",
                                     {
                                         let mut args = std::collections::HashMap::new();
+                                        args.insert("reason", e.to_string().into());
                                         args.insert(
                                             "name",
                                             current_file_name.to_string_lossy().to_string().into(),
                                         );
-                                        args.insert("reason", e.to_string().into());
                                         args
                                     },
                                 )

[saethlin]

Filed kellpossible/cargo-i18n#150