Update firmware packages from bookworm-backports on Debian 12 #184
Conversation
marmarek
force-pushed
the
bookworm-firmware
branch
from
682fbb2 to
73e9bc0
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #184 +/- ##
=======================================
Coverage 73.03% 73.03%
=======================================
Files 10 10
Lines 1157 1157
=======================================
Hits 845 845
Misses 312 312 ☔ View full report in Codecov by Sentry. |
|
@marmarek, firmware-iwlwifi from testing is a far better fit, imo. firmware-iwlwifi from bookworm-backports is five months older and anecdotally I couldn't get it to work with my BE200 card (which NovaCustom also sells). Judging by the forum others have had problems with it too. https://forum.qubes-os.org/t/intel-wi-fi-7-be200-working/24017 |
|
AFAIK the "proper" Debian way of bringing packages from testing to stable is via backports, otherwise it isn't really safe to mix packages from testing and stable. While in this particular case it may work today, there isn't really guarantee it wouldn't break some other day (for example package in testing pulling some other dependencies from testing, that would break stable). |
OpenQA test summaryComplete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025030800-4.3&flavor=pull-requests Test run included the following:
New failures, excluding unstableCompared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025021804-4.3&flavor=update
Failed tests25 failures
Fixed failuresCompared to: https://openqa.qubes-os.org/tests/129058#dependencies 9 fixed
Unstable testsPerformance TestsPerformance degradation:32 performance degradations
Remaining performance tests:40 tests
|
marmarek
force-pushed
the
bookworm-firmware
branch
3 times, most recently
from
033949d to
078c9db
Compare
| # first, add bookworm-backports if not already there: | ||
| with open(sources_list) as sources: | ||
| current_sources = sources.read() | ||
| if "bookworm-backports" not in current_sources: |
There was a problem hiding this comment.
Whonix enables backports in /etc/apt/sources.list.d/debian.list.
It is best to check if it is enabled in any sources list, be it:
- sources.list
- sources.list.d/*.list
- sources.list.d/*.sources
Does not lead to an error, but a huge warning wall.
Quite difficult to validate this considering there are mirrors, protocols can vary (https, http, tor+https, tor+http, http://HTTPS) and URLs can also very, considering debian.org, .onion variant and mirrors.
A stricter validation would allow to not fail on false positives of external repositories having the suite bookworm-backports.
There was a problem hiding this comment.
Look for other deb.debian.org line and copy it? If not found, do nothing. The main target for this change is a template that may be used for sys-net. Without updating firmware from backports, QubesOS/qubes-issues#9794 breaks wifi in most cases (for those using Debian for sys-net). It doesn't need to work in Whonix or other cases like that
There was a problem hiding this comment.
I understand Whonix is not the focus, Kicksecure for example is becoming a template and there, as the base of Whonix, which could also become a sys-net template, has https://github.com/Kicksecure/anon-apt-sources-list/blob/master/etc/apt/sources.list.d/debian.list
Therefore I suggest testing against both of these URLs
- deb.debian.org/debian
- 2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian
There was a problem hiding this comment.
Is that an official onion repo? I don't see it listed at https://www.debian.org/mirror/list
There was a problem hiding this comment.
Source: https://onion.debian.org/
Bottom of the page.
| with open(sources_list) as sources: | ||
| current_sources = sources.read() | ||
| if "bookworm-backports" not in current_sources: | ||
| with open(sources_list, "a") as sources: |
There was a problem hiding this comment.
I believe it is better to use a file in sources.list.d so it is easier to remove that configuration if necessary one day.
|
|
||
| APT_CONF = "/etc/apt/apt.conf.d/01qubes-update" | ||
| sources_list = "/etc/apt/sources.list" | ||
| backports_line = "deb https://deb.debian.org/debian bookworm-backports main contrib non-free-firmware" |
There was a problem hiding this comment.
Using https as protocol for default is sane but may break user setups. Every apt-cacher-ng user, Whonix Workstation app qubes that have configured their gateway to disable transparent proxy.
Can also differ from what the user has configured especially on Whonix, such as https, tor+https, tor+http, tor with onion.
There should be a qube a hook to allow package maintainers to set the correct URL.
There was a problem hiding this comment.
Using new file in sources.list.d is a good idea. I can simply make it to not do anything if the file already exists (lets say, backports.list). apt-cacher-ng users can either adjust this file post-change, or proactively create it earlier with desired content.
| backports_line = "deb https://deb.debian.org/debian bookworm-backports main contrib non-free-firmware" | ||
| prefs_path = "/etc/apt/preferences.d/firmware_backports" | ||
| prefs_data = """\ | ||
| Package: src:firmware-nonfree |
There was a problem hiding this comment.
marmarek
force-pushed
the
bookworm-firmware
branch
4 times, most recently
from
306db22 to
5291a73
Compare
marmarek
force-pushed
the
bookworm-firmware
branch
from
5291a73 to
edffda1
Compare
| ) | ||
| if b"bpo" in output: | ||
| # version from backports already installed | ||
| return |
There was a problem hiding this comment.
Now that pipewire was added, the check should not return as soon as firmware from backports is found.
There was a problem hiding this comment.
Now I see why pipewire should be in a separate plugin, to avoid returning early, the plugin name is firmware_backports and the function has the same name.
Maybe switch the file name to backports, place backports related code in one function and separate the check for pipewire and firmware-linux-non-free?
There was a problem hiding this comment.
I can also change this line to not return early.
| ) | ||
| if b"bpo" in output: | ||
| # version from backports already installed | ||
| return |
There was a problem hiding this comment.
Now I see why pipewire should be in a separate plugin, to avoid returning early, the plugin name is firmware_backports and the function has the same name.
Maybe switch the file name to backports, place backports related code in one function and separate the check for pipewire and firmware-linux-non-free?
|
I must caution against use of Quote https://www.kicksecure.com/wiki/Dev/APT_Pinning
Question: Can a package pull a another package from another repository to be added? Answers:
I rarely got such as direct request |
|
So, back to the original question: how to mark a package to be installed from backports, similar to |
|
The issues discussed in the threads @adrelanos linked to are mainly with dynamically adding new sources.list and pin entries during package installation or upgrade. Using pinning and the like is not inherently bad, it's just something that you should not be doing on already installed, running systems. People can and will customize their templates in all sorts of different ways, and adding a sources.list entry to their setup could cause major headaches if they have a non-standard setup. The ideal way to do this is the way you're already doing - make sure the repos are right from the get-go on freshly built systems. But you need older systems to be updated as well so that users get fixes and don't suffer problems during a kernel upgrade. I don't think there's any good solution here sadly - Debian dislikes the solution being proposed here, but I think it might be the best that can be done. Some solutions that I've thought of looking at this:
|
Firmware packages in standard Debian repo are too old for recent kernels, and recent hardware in general. This affects all kernel-latest versions, but also soon-to-be-default kernel-6.12. This is at least necessary to support wifi network in NovaCustom laptops, but other modern systems need that too. Fresh template builds have firmware from bookworm-backports already selected, but do that for older versions too. Build the URL based on `apt-get --print-uris update` output to match protocol used for the main Debian repo (tor, apt-cacher-ng etc). Thanks @aronowski for the hint on apt properties. Fixes QubesOS/qubes-issues#9815
marmarek
force-pushed
the
bookworm-firmware
branch
from
edffda1 to
9c3ba48
Compare
|
I would discourage any user interactive solution. The solution that I recommend would be:
That way, Qubes developers stay in full control, such updates remain fully tested by Qubes automated tests, and the package manager won't break due to the risks of using APT pinning. Building the package from source code might be overkill. This might have an impact on reprodubile builds, rebuilders. The package matching Debian's package is something that rebuilders should verify.
I highly doubt Debian has (or even wants) special permissions. The Debian developer only made a strong request but nobody made a legal argument. |
I don't like this, because that requires us to do that with all the future updates to the package too. Rules for backports are specifically crafted to not conflict with stable. This is the difference compared to plain "testing", which might cause problems, and that is likely also why you got such negative response about depending on package from "testing" in the posts you linked. |
Firmware packages in standard Debian repo are too old for recent
kernels, and recent hardware in general. This affects all kernel-latest
versions, but also soon-to-be-default kernel-6.12. This is at least
necessary to support wifi network in NovaCustom laptops, but other
modern systems need that too.
Fresh template builds have firmware from bookworm-backports already
selected, but do that for older versions too.
Thanks @aronowski for the hint on apt properties.
Fixes QubesOS/qubes-issues#9815