Implement a faster checksum method in "/usr/sbin/ocs-sr"

[keks24]

Currently it is only possible to generate SHA1 checksums of all created backup files. This is very slow and since 2005 SHA1 is considered as insecure.

BLAKE2 (b2sum) or the even faster implementation BLAKE3 (b3sum) on the other hand, are way faster; even faster than MD5. They also claim to be at least as secure as SHA384.

The b2sum binary is already available in the coreutils package, which should be already installed on all Debian distributions:

$ dpkg --search b2sum
coreutils: /usr/share/man/man1/b2sum.1.gz
coreutils: /usr/bin/b2sum

It would be awesome to implement one of these checksum methods.

[stevenshiau]

Thanks for this nice suggestion. We have implement that in Clonezilla live >= 2.7.2-9 or 20200309-*:

https://clonezilla.org/downloads.php

In the expert mode, you can choose "-gb" to create the checksum by BLAKE2 when saving. The corresponding option "-cb" is available for you to check the BLAKE2 checksum when restoring.

Please give it a try and let us know the results.
Thanks.

[keks24]

Nice! I will give it a try in the next days!

[keks24]

I tested it yesterday and it works nicely!

Unfortunately, the execution time of b2sum does not appear in any log file, but I had the impression, that it was significantly faster. For some reason, the checksum of the file B2SUMS is saved in the file itself, so this checksum will always be wrong, since new content gets appended during the backup:

$ b2sum --check --quiet B2SUMS
B2SUMS: FAILED

I also checked the file SHA1SUMS of my older backups and there was no checksum of the file itself, which is correct.

The backup hard drive is encrypted via LUKS version 2 aes-xts-plain64 (512 bytes) and the data was transferred via USB 3.

---

blkdev.list.txt
clonezilla-img.txt
Info-lshw_cpu.txt

[stevenshiau]

Thanks for reporting this. The issue was fixed and now in the repository. It will be used in the next testing Clonezilla live.

[keks24]

Awesome!

I will give it a try next month, when another backup is due.

In the meantime, maybe you want to compare BLAKE3 with other checksum methods and implement to Clonezilla as well. There is also a binary available at their repository, which could be copied to /usr/local/bin/b3sum, if the variable $PATH is correctly set.

There are also ideas to make BLAKE3 even faster via GPU implementations:

• BLAKE3 in OpenCL BLAKE3-team/BLAKE3#136
• Vulkan implementation of b3sum BLAKE3-team/BLAKE3#80

[stevenshiau]

Thanks.
As for BLAKE3, when it's available in Debian repository:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978472
we will include that.

Steven

[keks24]

@stevenshiau

I downloaded the newest testing image version 2.7.2-15 and I successfully created another backup using b2sum. I have not tested the restore process (-cb), since I have no testing environment, but I successfully checked the checksums manually.

$ cd "/<path_to_image_backup>/"
$ du --summarize --human-readable
56G     total
$ time b2sum --check --quiet B2SUMS
b2sum --check --quiet B2SUMS  156.10s user 20.69s system 35% cpu 8:17.98 total

-Ramon

[stevenshiau]

Great. Thanks for your feedback.

[keks24]

I just stumbled across this issue again.

It looks like, that b3sum has been implemented two years ago:

$ apt-cache show b3sum
Package: b3sum
Source: rust-b3sum (1.2.0-1)
Version: 1.2.0-1+b3
Installed-Size: 1192
Maintainer: Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net>
Architecture: amd64
Depends: libc6 (>= 2.34), libgcc-s1 (>= 4.2)
Description-en: Command line tool for the BLAKE3 hash
 BLAKE3 is a cryptographic hash function that is much faster than MD5, SHA-1,
  SHA-2, SHA-3, and BLAKE2 and
  Secure, unlike MD5 and SHA-1. And secure against length extension.
  Note, This package includes binary "b3sum" which is built from official
  multithreaded  implementation of BLAKE3 in rust.
Description-md5: 20e51b2306ef32c102665b1e0fca9724
Multi-Arch: allowed
Homepage: https://github.com/BLAKE3-team/BLAKE3

Message #15 from the bug report.

[stevenshiau]

Thanks for your feedback. Will include b3sum in the near future.

[keks24]

Awesome! Looking forward to it!