Include Ubuntu as a variant for faster patch releases

[nicktimko]

When using security scanners, there are often complaints about unfixed CVEs in the Debian image. Debian, for better or worse, seems to value increased stability, leading to more, older (found) bugs vs. Ubuntu's quicker releases which probably cause more new bugs. Which is better? ¿Porque no los dos?

Adding focal to the list of variants alongside buster and bullseye would seem to add Ubuntu to the list of Python images without much trouble. Both focal (20.04) and jammy (22.04, prerelease until ~Apr 21) tags are available as buildpack-deps tags (https://hub.docker.com/_/buildpack-deps).

I can get a PR together if the maintainers are amenable to the idea.

[yosifkit]

My stance remains unchanged: #61 (comment).

[nicktimko]

I think the "underlying distribution is irrelevant" doesn't always hold. Frequently, in order to build packages that don't ship manylinux* binaries (which don't work on Alpine, so we're past that immediately), some system libraries need to be installed, so you can't really be distro agnostic.

You also can't be agnostic when dealing with security issues. Unless your application is pretty narrow, it's going to be calling out to many other libraries to access a database, format an image, compress a file, send some network traffic...

Because these CVEs come and get squashed, it's hard to "replicate" this issue, but for some example as of now, here's Apache, which has a known vuln in Debian Bullseye, but it's patched in Ubuntu 20.04 Focal. I don't use Apache as much as nginx, but I've had vulnerable libpng's and such show up. In general the trend seems to be that Ubuntu security fixes can roll out faster.

• CVE-2022-22720 - Apache OOB vuln
  • Debian tracker - 2.4.52-1~deb11u2 is vulnerable
  • Ubuntu tracker patched as of March 14

Some commands to check package versions and their output...

$ docker run --pull always --rm -it ubuntu:latest bash -c \
    "apt-get update > /dev/null;apt-cache madison apache2"
latest: Pulling from library/ubuntu
Digest: sha256:bea6d19168bbfd6af8d77c2cc3c572114eb5d113e6f422573c93cb605a0e2ffb
Status: Image is up to date for ubuntu:latest
   apache2 | 2.4.41-4ubuntu3.10 | http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
   apache2 | 2.4.41-4ubuntu3.10 | http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
   apache2 | 2.4.41-4ubuntu3 | http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

$ docker run --pull always --rm -it python:3 bash -c \
    "apt-get update > /dev/null;apt-cache madison apache2"
3: Pulling from library/python
Digest: sha256:6441e2f0bd2e566de0df6445cb8e7e395ea1a376dd702de908d70401d3700961
Status: Image is up to date for python:3
   apache2 | 2.4.52-1~deb11u2 | http://security.debian.org/debian-security bullseye-security/main amd64 Packages
   apache2 | 2.4.51-1~deb11u1 | http://deb.debian.org/debian bullseye/main amd64 Packages

[tianon]

Sure, in a perfect world we'd support every combination of every popular distribution across every actively supported/maintained version of both Python and the distributions, but both from a support/maintenance perspective and a build time/complexity perspective, that just isn't reasonable. For example, each version of just Python 3.11 on each version of Alpine we support takes upwards of 30 minutes to build (so that's a full hour of straight compilation just for two versions of Alpine on one single version of Python). 😬 🙈

I imagine our Dockerfile for the Debian-based variants is probably generic enough that swapping the FROM over to Ubuntu will probably "just work" (especially given folks doing "out there" things like what's described in #706), but unfortunately that's the extent to which we're willing to take this given the other constraints involved.

[marc-pensa]

I imagine our Dockerfile for the Debian-based variants is probably generic enough that swapping the FROM over to Ubuntu will probably "just work"

Can confirm, replacing FROM debian:bullseye-slim with FROM ubuntu:20.04 in the https://github.com/docker-library/python/blob/master/3.10/slim-bullseye/Dockerfile works like a charm.

We were unable to use Debian for our base images due to missing library versions for what we are attempting to install, but Ubuntu works for our purposes.

[tianon]

Closing, as this is unfortunately not something we plan to support. Sorry! ❤️ (see #708 (comment))

[consideRatio]

I understand the decision to not take on additional complexity by providing support for ubuntu based images as its essential to ensure that a project is sustainable to maintain. This comment is solely made to convey an example of the value of ubuntu over debian for reference.

I distribute Docker images where we didn't mind either debian or ubuntu, so we switched to using the python images based to not need to install python ourselves via an ubuntu image. We are now considering reverting back because of the amounts of known vulnerabilities reported by container scanners.

I tracked it down to debian:bullseye vs ubuntu:22.04 and the vast difference in vulnerabilities in the default packages installed and for additional optional packages. Container security scanner trivy reported the following differences between the distributions after we installed some optional apt packages.

debian:bullseye, without vim as an apt package installed - Total: 95 (... HIGH: 40, CRITICAL: 4)
debian:bullseye, with vim as an apt package installed - Total: 625 (... HIGH: 454, CRITICAL: 20)
ubuntu:22.04, with vim as an apt package installed - Total: 0 (... HIGH: 0, CRITICAL: 0)

In practice I don't know if this is a security problem, but the noise of reported known vulnerabilities is a problem. For example, one of the critical vulnerabilities as reported by trivy is disputed to be a problem at all. At the same time, another critical vulnerability known since 2019 is fixed in debian:bookworm/sid but not part of bullseye. This CVE was patched 2019 by ubuntu it seems, and part of the ubuntu:20.04 and ubuntu:22.04 when they were released.

Due to this, the open source project I'm working in is moving back towards ubuntu for now - jupyterhub/zero-to-jupyterhub-k8s#2921.

Thank you for your work in this project, the python images are very helpful!!! ❤️ 🎉

[tianon]

Thank you for your kind words ❤️

For some added insight, the primary reason I've found over the years that CVE scanners are less kind to Debian than they are to Ubuntu is that Ubuntu clearly marks CVEs as "WONTFIX" (which the scanners than interpret as "hide this CVE"), where the Debian Security Team never says outright that something will not be fixed because any Debian Developer / member of the project (or someone with enough time to do the work and find a sponsor for it) can fix any issue they like.

For example, the maintainer of a given package could choose to fix every CVE filed against their packages in stable, if they so chose, even if the Security Team at large has decided that most of them aren't worth fixing in stable. The way the Security Team normally telegraphs their evaluation of the CVE's severity within the context of the Debian package and how it is used is with a "no-dsa" tag in the notes, which IMO all the scanning vendors should pick up and interpret the same as they do "WONTFIX" in Ubuntu (because it really is the exact same meaning), but I have not seen any of them doing so, even in spite of the many long meetings I've had with several of them describing this. 😞

See docker-library/mysql#188 (comment) for an example where I went through a very long list of CVEs on the mysql image and highlighted the notes from the Debian Security Team to illustrate this better.

[consideRatio]

@tianon wow amazing work in detailing this situation!!!

I pinged you in jupyterhub/zero-to-jupyterhub-k8s#2918 (comment), asking if you know a place where this is being tracked by trivy (or debian).