Update external docs

Author: garrett

external/bots/README.md

@@ -32,8 +32,8 @@ For managing these images:
 
  - `image-download`: Download selected or all test images
  - `image-create`: Create test machine images from scratch (usually through
-   virt-install or downloading a cloud image), with common build and test
-   dependencies for Cockpit projects preinstalled
+   downloading a cloud image), with common build and test dependencies for
+   Cockpit projects preinstalled
  - `image-upload`: Upload a locally built test image to the official image servers
 
 For running and debugging the images:
@@ -210,6 +210,39 @@ progress and at the end there will be links to commits with the new
 images.  You can then include these commits into the pull request in
 any way you like.
 
+### Creating a new image
+
+Creating a new image from scratch requires some
+[images/scripts/](./images/scripts) files. For a new image called `tux` we need:
+
+- `tux.bootstrap`: Download or create an initial qcow2 image which boots and has SSH
+  acccess. If available, this should be the latest available cloud image,
+  but it may also invoke some other VM build tool or build service.
+  This script runs in the [cockpit/tasks](https://github.com/cockpit-project/cockpituous/tree/main/tasks/container)
+  container, and thus can only use the tools installed there. If necessary, add
+  them first.
+- `tux.setup`: The setup script runs inside the downloaded test image,
+  install all required build and test dependencies, and sets up an `admin`
+  user.
+
+For a new image it is recommended to follow the existing setup/bootstrap scripts,
+for example the `fedora` one.
+
+Run `./image-create -v tux` to build the image. If that succeeds, a new image
+is saved in `images` as `images/tux` (a symlink to the real qcow2 file).
+Test-boot it with `./vm-run tux`, and ensure you can:
+
+ - Log in with SSH as the `admin` and `root` user with our usual test SSH key
+ - Become root with `sudo` as admin works (password "foobar")
+
+To add that image to our CI, create a PR and follow the "Creating new
+images for a pull request" section. External contributors will need to ask a
+Cockpit team member to create a copy of the PR and follow this workflow.
+
+For the initial PR it is recommended to add the new image to the `_manual`
+testmap of `starter-kit` or other project to prove that the created image is
+functional.
+
 ### Updating CI to a new Fedora release
 
 `TEST_OS_DEFAULT` is usually set to the latest (stable) Fedora released,

external/source/HACKING.md

@@ -63,29 +63,27 @@ To re-create your development container from the latest image, run:
 ## Working on Cockpit's session pages
 
 Most contributors want to work on the web (HTML, JavaScript, CSS) parts of Cockpit.
+
+### Install Cockpit
+
 First, install Cockpit on your local machine as described in:
 
 <https://cockpit-project.org/running.html>
 
+### Build session pages
+
 Next, run this command from your top level Cockpit checkout directory, and make
 sure to run it as the same user that you'll use to log into Cockpit below.
 
     mkdir -p ~/.local/share/
     ln -s $(pwd)/dist ~/.local/share/cockpit
 
-This will cause cockpit to read JavaScript, HTML, and CSS files directly from the
+This will cause Cockpit to read JavaScript, HTML, and CSS files directly from the
 locally built package output directory instead of using the system-installed Cockpit
 files.
 
-Now you can log into Cockpit on your local Linux machine at the following
-address. Use the same user and password that you used to log into your Linux
-desktop.
-
-<http://localhost:9090>
-
-After every change to the source files, bundles need to be rebuilt. The
-recommended and fastest way is to do that is using the "watch" mode (`-w` or
-`--watch`) on the page that you are working on. For example, if you want to
+The recommended way to build bundles is to use the "watch" mode
+(`-w` or`--watch`) on the page you are working on. For example, if you want to
 work on anything in [pkg/systemd](https://github.com/cockpit-project/cockpit/tree/main/pkg/systemd/), run:
 
     ./build.js -w systemd
@@ -97,8 +95,16 @@ pkg/lib/), you can also build all pages:
 
     ./build.js -w
 
-Reload cockpit in your browser after page is built. Press `Ctrl`-`C` to
-stop watch mode once you are done with changing the code.
+Now you can log into Cockpit on your local Linux machine at the following
+address, using the same username and password as your desktop login:
+
+<http://localhost:9090>
+
+Watch mode automatically rebuilds when source files are modified. Once it
+finishes building, refresh your browser to see the changes in Cockpit.
+Press `Ctrl-C` to stop watch mode when you are done changing the code.
+
+### Testing
 
 You often need to test code changes in a VM. You can set the `$RSYNC` env
 variable to copy the built page into the given SSH target's
@@ -109,12 +115,14 @@ one of these commands:
     RSYNC=c ./build.js -w kdump
     RSYNC=c ./build.js -w
 
+### Returning to system packages
+
 To make Cockpit use system packages again, instead of your checkout directory,
 remove the symlink with the following command and log back into Cockpit:
 
     rm ~/.local/share/cockpit
 
-## Working on the non-web parts of Cockpit
+## Building and unit tests
 
 Cockpit uses autotools, so there are familiar `./configure` script and
 Makefile targets.
@@ -152,10 +160,26 @@ which will output a URL to connect to with a browser, such as
 <http://localhost:8765/qunit/base1/test-dbus.html>. Adjust the path for different
 tests and inspect the results there.
 
-You can also run individual tests by specifying the `TESTS` environment
-variable:
+QUnit tests are run as part of a pytest test called `test_browser`.  You can
+run individual tests via `pytest -k`, like so:
+
+    pytest -k test-fsinfo.html
+
+You can see JavaScript code coverage information for QUnit tests.  For a
+summary table:
+
+    pytest -k test_browser --js-cov
+
+And for detailed output on uncovered sections in a specific file, something
+like:
 
-    make check TESTS=qunit/base1/test-chan.html
+    pytest -k test-fsinfo.html --js-cov-files='*/fsinfo.ts'
+
+Coverage information is gathered into the pytest tmpdir, regardless of which
+coverage-related commandline flags are given, so it's also possible to drill
+down after the fact — without re-running tests — using something like:
+
+    test/common/js_coverage.py -m '*/fsinfo.ts' /tmp/pytest-of-*/pytest-current/js-coverage/*
 
 There are also static code and syntax checks which you should run often:
 
@@ -182,45 +206,47 @@ git submodules:
 Refer to the [testing README](test/README.md) for details on running the Cockpit
 integration tests locally.
 
-## Python bridge
+## Bridge
 
-Most distro releases now ship a replacement for the C bridge written in Python.
-It resides in `src/cockpit` with most of its rules in `src/Makefile.am`.  This
-directory was chosen because it matches the standard so-called "src layout"
-convention for Python packages, where each package (`cockpit`) is a
+The Cockpit bridge is the initial program launched in a Cockpit Linux session:
+Its stdin/out is connected to the web socket (and thus to JavaScript in the
+[pages](pkg/)), where it speaks a [JSON protocol](doc/protocol.md) that
+multiplexes "channels" -- abstractions of operating system APIs that the pages
+use to implement their functionality. This protocol is translated into
+operating system calls such as opening or writing files, D-Bus calls, or HTTP
+queries. Think of the bridge as the moral equivalent of "bash" in a human SSH
+session.
+
+The bridge resides in `src/cockpit` with most of its rules in `src/Makefile.am`.
+This directory was chosen because it matches the standard so-called "src
+layout" convention for Python packages, where each package (`cockpit`) is a
 subdirectory of the `src` directory.
 
 ### Running the bridge
 
-The Python bridge can be used interactively on a local machine:
+The bridge can be used interactively on a local machine out of the source tree:
 
     PYTHONPATH=src python3 -m cockpit.bridge
 
 To make it easy to test out channels without having to write out messages
 manually, `cockpit.misc.print` can be used:
 
-    PYTHONPATH=src python3 -m cockpit.misc.print open fslist1 path=/etc watch=False | PYTHONPATH=src python3 -m cockpit.bridge
+    PYTHONPATH=src python3 -m cockpit.misc.print open fsinfo path=/etc 'attrs=["type", "entries"]' | PYTHONPATH=src python3 -m cockpit.bridge
 
 These shell aliases might be useful when experimenting with the protocol:
 
     alias cpy='PYTHONPATH=src python3 -m cockpit.bridge'
     alias cpf='PYTHONPATH=src python3 -m cockpit.misc.print'
 
-When working with the Python bridge on test images, note that `rhel-8*` still
-uses the C bridge. So if you want to explicitly have the Python bridge on those
-images use:
-
-    ./test/image-prepare --python
-
 To enable debug logging in journal on a test image, you can pass `--debug` to
 `image-prepare`. This will set `COCKPIT_DEBUG=all` to `/etc/environment`, if
 you are only interested channel debug messages change `all` to
 `cockpit.channel`.
 
-### Testing the Python bridge
+### Testing the bridge
 
-There are a growing number of Python `unittest` tests being written to test
-parts of the new bridge code.  You can run these with `make pytest` or
+There are a growing number of [pytest](https://docs.pytest.org) tests being written to test
+the bridge code.  You can run these with `make pytest` or
 `make pytest-cov`.  Those are both just rules to make sure that the
 `systemd_ctypes` submodule is checked out before running `pytest` from the
 source directory.
@@ -273,10 +299,39 @@ During fast iterative development, you can also choose to not run stylelint, by
 running `./build.js` with the `-s`/`--no-stylelint` option. This speeds up the
 build and avoids build failures due to ill-formatted CSS or other issues.
 
+## Working on your local machine: systemd-sysext
+
+If you want to safely test your local changes directly on it, Cockpit supports
+installation as a [systemd-sysext](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html).
+This covers all parts of Cockpit (ws, tls, session, bridge, login page, systemd
+units, PAM configuration, and the session pages) except for the SELinux policy.
+It gets installed into `/run/extensions/`, so nothing ever hits the disk and
+this also works on read-only installations (CoreOS, OSTree, bootc).
+
+Just run:
+
+    tools/make-sysext
+
+This runs `./autogen.sh` if necessary, and then just re-`make`s your tree,
+re-installs it into `/run/extensions`, and reloads the sysext in systemd.
+Afterwards you can connect to http://localhost:9090 as usual.
+
+To remove this, reboot or run
+
+    tools/make-sysext stop
+
+**Attention**: This is not currently compatible with SELinux in enforcing mode.
+If you have that, you need to disable it:
+
+    sudo setenforce 0
+
 ## Working on your local machine: Web server
 
-To test changes to the login page or any other resources, you can bind-mount the
-build tree's `dist/static/` directory over the system one:
+If the above systemd-sysext approach does not work for you, you can also test
+changes with some bind mounts.
+
+To test changes to the login page, you can bind-mount the build tree's
+`dist/static/` directory over the system one:
 
     sudo mount -o bind dist/static/ /usr/share/cockpit/static/
 

external/source/test/README.md

@@ -25,8 +25,12 @@ You first need to build cockpit, and install it into a VM:
 
     test/image-prepare
 
-This uses the default OS image, which is currently Fedora 39. See `$TEST_OS`
-below how to select a different one.
+This uses the default OS image. See `$TEST_OS` below how to select a different one.
+
+See `test/image-prepare --help` for some special modes, like skipping unit
+tests, building overlays, or preparing an image with the
+[cockpit/ws container](https://quay.io/repository/cockpit/ws) instead of RPMs.
+See our scenarios in [test/run](./run) for how they are being used.
 
 In most cases you want to run an individual test in a suite, for example:
 
@@ -167,38 +171,23 @@ to push pixel tests.
 
 You can set these environment variables to configure the test suite:
 
-    TEST_OS    The OS to run the tests in.  Currently supported values:
-                  "centos-9-stream"
-                  "centos-10"
-                  "debian-stable"
-                  "debian-testing"
-                  "fedora-39"
-                  "fedora-40"
-                  "fedora-coreos"
-                  "rhel-9-4"
-                  "rhel4edge",
-                  "ubuntu-2204"
-                  "ubuntu-stable"
-               "fedora-39" is the default (TEST_OS_DEFAULT in bots/lib/constants.py)
-
-    TEST_JOBS  How many tests to run in parallel.  The default is 1.
-
-    TEST_CDP_PORT  Attach to an actually running browser that is compatible with
-                   the Chrome Debug Protocol, on the given port. Don't use this
-                   with parallel tests.
-
-    TEST_BROWSER  What browser should be used for testing. Currently supported values:
-                     "chromium"
-                     "firefox"
-                  "chromium" is the default.
-
-    TEST_SHOW_BROWSER  Set to run browser interactively. When not specified,
-                       browser is run in headless mode. When set to "pixels",
-                       the browser will be resized to the exact dimensions that
-                       are used for pixel tests.
-
-    TEST_TIMEOUT_FACTOR Scale normal timeouts by given integer. Useful for
-                        slow/busy testbeds or architectures.
+ * `TEST_OS`: The OS to run the tests in, like "fedora-coreos" or
+    "debian-stable". See the "cockpit-project/cockpit" section in the
+    [test map](https://github.com/cockpit-project/bots/blob/main/lib/testmap.py)
+    for all supported values. "fedora-40" is the default (`TEST_OS_DEFAULT` in
+    bots' [constants.py](https://github.com/cockpit-project/bots/blob/main/lib/constants.py)).
+
+ * `TEST_JOBS`:  How many tests to run in parallel.  The default is 1.
+
+ * `TEST_BROWSER`: What browser should be used for testing. Currently supported
+   values are "chromium" and "firefox". "chromium" is the default.
+
+ * `TEST_SHOW_BROWSER`: Set to run browser interactively. When not specified,
+   browser is run in headless mode. When set to "pixels", the browser will be
+   resized to the exact dimensions that are used for pixel tests.
+
+ * `TEST_TIMEOUT_FACTOR`: Scale normal timeouts by given integer. Useful for
+   slow/busy testbeds or architectures.
 
 See the [bots documentation]({{ site.baseurl }}/external/bots/README.html)
 for details about the tools and configuration for these.

external/wiki/Proxying-Cockpit-over-Apache-with-LetsEncrypt.md

@@ -101,6 +101,26 @@ Change this file so it looks like the following.
 </IfModule>
 ```
 
+With Apache 2.4.47 or later, this could also be written as:
+
+```apacheconf
+<IfModule mod_ssl.c>
+<VirtualHost *:443>
+  ServerName cockpit.your-domain.com
+  SSLCertificateFile /etc/letsencrypt/live/cockpit.your-domain.com/fullchain.pem
+  SSLCertificateKeyFile /etc/letsencrypt/live/cockpit.your-domain.com/privkey.pem
+  Include /etc/letsencrypt/options-ssl-apache.conf
+
+  ProxyPreserveHost On
+  ProxyRequests Off
+
+  # Proxy to your local cockpit instance
+  ProxyPass / http://127.0.0.1:9090/ upgrade=websocket
+  ProxyPassReverse / http://127.0.0.1:9090/
+</VirtualHost>
+</IfModule>
+```
+
 Save and close the file. Then restart Apache web server.
 
 `sudo systemctl restart apache2`