Local socket and review fixes

.github/workflows/ci.yml

@@ -36,13 +36,34 @@ jobs:
       - name: Build and run protocol vectors
         run: make -C server-c test
 
+  unbound-container:
+    # The only job that answers "does Unbound still build with the Python module,
+    # and does its pythonmod still call PFUI's entry points". Everything else
+    # tests PFUI against fakes; this builds the Unbound release the installer
+    # resolves and runs PFUI_Unbound inside it, so upstream API drift surfaces
+    # here rather than during someone's install.
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # 'latest' is what the installer builds by default. 'master' is what it
+        # builds on request, and is informational: upstream head is occasionally
+        # unstable, which is why the installer can be pinned to a release.
+        unbound: [latest, master]
+    continue-on-error: ${{ matrix.unbound == 'master' }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build Unbound with the Python module and run PFUI_Unbound in it
+        run: ./client-unbound/tests/container/run.sh ${{ matrix.unbound }}
+
   shell:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - name: Syntax-check the installers and ops scripts
         run: |
-          for f in *.sh client-unbound/tools/*.sh; do bash -n "$f"; done
+          for f in *.sh client-unbound/tools/*.sh \
+                   client-unbound/tests/container/*.sh; do bash -n "$f"; done
       - name: Syntax-check the rc.d scripts
         run: |
           sudo apt-get update -qq && sudo apt-get install -y -qq ksh

DECISIONS.md

@@ -16,6 +16,68 @@ now as a larger change to the daemon's startup path.
 On OpenBSD `/dev` is a static on-disk directory, so the mode persists across
 reboots; what resets it is `MAKEDEV` during a release upgrade.
 
+## RR TTLs are stored exactly as sent
+
+`db_push` records the TTL the client reported, with no floor. An earlier
+`max(ttl, 3600)` contradicted the expiry rule in `PROTOCOL.md`, and with the
+shipped `TTL_MULTIPLIER: 4` it turned a 60 second answer into four hours of
+authorised egress. It also raised the `ttl` of 0 that means do-not-cache, which
+`validate.extract` deliberately preserves.
+
+`TTL_MULTIPLIER` is the only knob for holding entries longer than the record
+says, which is the right place for it: the operator sets it knowing why (browsers
+caching past the TTL), rather than inheriting a hidden minimum. The Redis
+`EXPIRE` written alongside each key stays a backstop for the scan loop and is
+floored at one second, because Redis reads `EXPIRE 0` as "delete now".
+
+## Unbound is built from a release tag, resolved at install time
+
+`install-client-unbound.sh` asks NLnet Labs for its newest `release-*` tag rather
+than carrying a version. Three things follow from that choice:
+
+Tags, not branches. The `branch-<version>` heads are pruned upstream (nothing
+before 1.23 survives), so a version named in the script goes stale and then
+vanishes. Release tags are permanent.
+
+`git ls-remote`, not the GitHub releases API. It needs no token, has no rate
+limit, and uses the git the installer already requires. The API would also depend
+on a release being marked "latest" by hand.
+
+Git, not the signed release tarball. The tarball is PGP-signed, which the clone
+is not, and that is a real gap. It stays a clone because the OpenBSD build needs
+`Makefile.bsd-wrapper` dropped into a source tree, and because `configure` is
+committed upstream so no autoreconf is needed. Verifying the source is worth
+doing and is not done here.
+
+## The Unbound build is tested in a Linux container
+
+`client-unbound/tests/container/` builds Unbound with `--with-pythonmodule` and
+runs PFUI_Unbound inside it, on Debian. It cannot prove the OpenBSD build, and it
+is not trying to: what it protects is the pythonmod interface. No distribution
+ships Unbound with the Python module, so PFUI is the only consumer of that build,
+and an upstream API change used to surface as an operator's install failing.
+
+Two things it caught immediately, both of which the OpenBSD installer gets from
+base and would not have revealed: building from the git tree needs flex and
+bison, and `configure --with-pythonmodule` looks for `python`, not `python3`.
+
+It also settles what the TTL labelling in `read_rr` rests on: against 1.26.0 and
+against `master`, the reply path reports a relative TTL and the cache path an
+absolute unix timestamp, so `kind` means what `PROTOCOL.md` says it means. That
+was previously an assumption.
+
+Unbound really runs as `_unbound` in the container, and the local socket really is
+`0660 :_pfui` under a `0750` directory, so the container also exercises the group
+model that permits a same-host deployment rather than just the code paths.
+
+Known and benign: on shutdown the resolver logs `pythonmod: Exception occurred in
+function deinit` / `TypeError: 'NoneType' object cannot be interpreted as an
+integer`. It is not PFUI's — it persists with `deinit`'s body reduced to
+`return True`, so nothing PFUI executes can be setting it, and it happens after
+`service stopped`. The container prints any resolver `error:` line even when every
+check passes, which is how this was noticed; it is recorded here so it is not
+re-investigated.
+
 ## UDP has a message-size ceiling
 
 `UDP_DGRAM_CEILING` (1400 bytes) bounds a PFUI message over UDP. This is not a
@@ -25,6 +87,67 @@ message above the ceiling is logged and dropped rather than truncated silently.
 Accepted because UDP is lab-only and gated behind `ALLOW_INSECURE_UDP`; the fix
 for a real deployment is TCP.
 
+## A local socket for a same-host deployment
+
+When PFUI_Unbound runs on the firewall itself, `SOCKET_UNIX` on the server and a
+`SOCKET:` entry on the client replace loopback TCP. That is not a micro-
+optimisation: the client opens one connection per DNS answer (see below), so
+loopback TCP costs a handshake, a `TIME_WAIT` entry on the firewall and a slice of
+the ephemeral port range for every reply. A unix socket has none of those.
+
+Both listeners can run at once, and a CARP node with its own resolver wants
+exactly that: the firewall on this box over the socket, the peer over TCP. That is
+also why the transport is chosen **per `FIREWALLS` entry** on the client rather
+than by one global `SOCKET_PROTO` — one resolver genuinely needs both at the same
+time. `SOCKET_PROTO` now describes only the network entries.
+
+The server binds one accept loop per listener, in a thread each, rather than
+polling them together. Each loop already blocks only for `SOCKET_TIMEOUT` before
+re-checking for `SIGTERM`, and the worker pool, the slot semaphore and the shed
+path are shared and thread-safe, so this leaves each accept path as it was.
+
+### The filesystem is the access control
+
+There is no packet on this transport, so the `pf.conf` source restriction does not
+apply and cannot. The socket's own ownership and mode are the whole control on who
+may inject PF whitelist entries, which is a meaningfully different security model
+from the network listener's, in a different place, enforced by a different
+subsystem. A server serving both is only as restricted as the weaker one.
+
+`_pfui` is a dedicated group holding `_pfui_firewall` and `_unbound`, rather than
+reusing either account's own group. Membership means "may authorise egress", and
+that should not be implied by merely running as the resolver, or be acquired by
+anything later added to `_unbound`'s group for an unrelated reason.
+
+The bind path fails closed throughout, because every one of these leaves the
+socket reachable by more than intended:
+
+- `bind()` creates the node with the process umask, so the umask is narrowed
+  around it rather than the mode being fixed by a `chmod` afterwards. Otherwise
+  the socket is connectable by everyone for the moment in between.
+- A missing `SOCKET_UNIX_GROUP`, or a `chown`/`chmod` that does not take, exits
+  the daemon and unlinks the socket instead of serving on it.
+- A world-writable parent directory is refused unless it is sticky: anyone could
+  otherwise replace the socket and be handed the resolver's messages, whatever
+  mode the socket itself has. The parent is `0750 _pfui_firewall:_pfui`, so it is
+  a second gate in front of the socket.
+- A socket file left by an unclean stop is removed, but only after a probe
+  connect shows nothing is listening. Unlinking a live daemon's socket would
+  leave it running and unreachable.
+
+`SOCKET_UNIX` is length-checked at config load because `sockaddr_un.sun_path` is
+104 bytes on OpenBSD; without it the failure is an `AF_UNIX path too long` from
+`bind()` rather than a named configuration error.
+
+### EPIPE is visible here where TCP hid it
+
+A cache report is sent with `blocking=False`, so the client has closed by the time
+the server writes `ACKUPDATE`. Loopback TCP absorbs that write into a buffer
+nobody reads; a unix socket reports `EPIPE` at once. The tolerance in
+`disconnect()` was already there and is now load-bearing on this transport, and
+there is a test for it. Nothing is lost: the addresses are installed before the
+acknowledgement is attempted.
+
 ## Unauthenticated transport
 
 Neither the TCP nor the UDP transport authenticates or encrypts. IPs must reach

INSTALL.md

@@ -46,6 +46,19 @@ OpenBSD PF Firewall(s); Configure PFUI_Firewall `/etc/pfui_firewall.yml`
 Unbound DNS Resolver(s); Configure PFUI_Unbound `/var/unbound/etc/pfui_unbound.yml`
 ```
 
+### Transports
+The firewall needs at least one listener, and may run both:
+
+| Config | Serves |
+|--------|--------|
+| `SOCKET_LISTEN` + `SOCKET_PROTO` | Resolvers on other hosts. Restrict the port in `pf.conf` to the known resolvers. |
+| `SOCKET_UNIX` | A resolver on **this** host. No `pf.conf` rule; group `_pfui` and the socket's `0660` mode are the access control. |
+
+`SOCKET_LISTEN` is never defaulted to `0.0.0.0` — leaving it out is how a
+same-host deployment says "local socket only". On the resolver, each `FIREWALLS`
+entry picks its own transport with `HOST:` or `SOCKET:`. See
+[Same-host deployment](README.md#samehost).
+
 Warning; UDP mode (`SOCKET_PROTO: UDP`) is _not_ recommended (experimental) as Unbound's Python Module executes every DNS lookup, 
 using a unique network socket to PFUI_Firewall for each lookup. With UDP's default timers, the socket 
 remains (5mins) after the connection/PFUI_Firewall is updated, thus blocking subsequent connections until timeout.
@@ -83,10 +96,25 @@ pkg_add -i swig git bash cmake libconfig libiconv bison gawk mawk
 python3 -m pip install -r ./client-unbound/requirements.txt
 
 ### PFUI_Unbound - Download Unbound source
+Clone the release you mean to build, rather than the default branch. `--branch`
+accepts a tag, so a single-commit clone lands directly on it; cloning shallowly
+and then checking out another ref does not work, because `--depth` implies
+`--single-branch`.
+```
+# The tag the installer would resolve, newest release first
+git ls-remote --tags --refs https://github.com/NLnetLabs/unbound.git 'release-*' \
+  | sed 's#.*refs/tags/##' | grep -E '^release-[0-9.]+$' | sort -t. -k2,2n -k3,3n | tail -1
+
+git clone --depth 1 --branch release-1.26.0 https://github.com/NLnetLabs/unbound.git /tmp/unbound
+```
+`client-unbound/tools/unbound_release.sh` does this resolution for the installer,
+and prints the ref it picked:
 ```
-git clone --depth 20 https://github.com/NLnetLabs/unbound.git /tmp/unbound
-# --depth 20 helps with shallow clone errors
+./client-unbound/tools/unbound_release.sh          # latest release tag
+./client-unbound/tools/unbound_release.sh master   # passes through unchanged
 ```
+Note that `release-*` tags are permanent but the `branch-*` heads are not: NLnet
+Labs prunes them, and nothing before 1.23 still exists.
 
 #### Default Unbound build options in OpenBSD port, `unbound -V` (ref only);
 ```

README.md

@@ -91,8 +91,20 @@ adds latency). A fully recursive DNS query can take tens to hundreds of millisec
 
 ------
 ## PFUI Installation
-Tested from OpenBSD 7.0, Unbound 1.16, Python 3.8,
-up to OpenBSD 7.4, Unbound 1.18, Python 3.10
+Tested on OpenBSD from 7.0 (Unbound 1.16, Python 3.8) to 7.4 (Unbound 1.18, Python 3.10).
+
+`install-client-unbound.sh` builds the **latest Unbound release** by default: it
+resolves the newest `release-*` tag from NLnet Labs at install time rather than
+carrying a version in the script. Every release from 1.23 onwards is a candidate,
+so pin one if you need a known build:
+
+```
+UNBOUND_VERSION=release-1.25.2 doas ./install-client-unbound.sh   # a specific release
+UNBOUND_VERSION=master         doas ./install-client-unbound.sh   # upstream head
+```
+
+Unbound 1.26.0 with `--with-pythonmodule` is built and exercised on every commit;
+see [Tests](#tests).
 
 ### 1) Install PFUI_Firewall on OpenBSD PF Firewall(s)
 ```
@@ -223,6 +235,68 @@ Jordan's unbound-adblock installation guide for reference;\
 https://www.geoghegan.ca/pub/unbound-adblock/latest/install/openbsd.txt \
 https://www.geoghegan.ca/pub/pf-badhost/latest/man/man.txt
 
+------
+<a name="tests"></a>
+### Tests;
+
+```
+pytest                                        # protocol, client and server suites
+make -C server-c test                         # the C framing against the shared vectors
+./client-unbound/tests/container/run.sh       # builds Unbound and runs PFUI_Unbound in it
+```
+
+The PF ioctl suites skip unless they are run on OpenBSD, and
+`client-unbound/tests/test_unbound*.py`'s live cases skip unless `PFUI_FW_HOST`
+points at a running PFUI_Firewall.
+
+The container is the one test that builds Unbound from source with
+`--with-pythonmodule` and runs the real resolver against a local authoritative
+server and a stub firewall. No distribution packages Unbound with the Python
+module, so that build is PFUI's alone, and pythonmod API drift in a new Unbound
+release would otherwise only show up when someone ran the installer. It takes a
+few minutes and needs Docker; nothing runs on the host. CI runs it against the
+latest release, and against upstream `master` for information only.
+
+------
+<a name="samehost"></a>
+### Same-host deployment (local socket);
+
+When PFUI_Unbound and PFUI_Firewall run on the **same machine**, the resolver can
+reach the firewall over a unix domain socket instead of loopback TCP. The client
+opens one connection per DNS answer, so loopback TCP costs a handshake, a
+`TIME_WAIT` entry on the firewall and an ephemeral port for every reply; a local
+socket costs none of them, and needs no `pf.conf` rule because there is no packet
+to filter.
+
+On the firewall, in `/etc/pfui_firewall.yml`:
+```
+SOCKET_UNIX: /var/run/pfui/pfui_firewall.sock
+SOCKET_UNIX_GROUP: _pfui
+# SOCKET_LISTEN may be omitted entirely if no remote resolver needs to reach this
+# firewall. Keep it to serve both, which is what a CARP node wants.
+```
+On the resolver, in `/var/unbound/etc/pfui_unbound.yml`:
+```
+FIREWALLS:
+  - SOCKET: /var/run/pfui/pfui_firewall.sock   # this host, over the local socket
+  - HOST: 10.10.1.253                          # the CARP peer, over the network
+    PORT: 10001
+```
+The transport is per entry, so one resolver can use both at once. `SOCKET_PROTO`
+applies only to the `HOST` entries.
+
+**Access control moves from PF to the filesystem.** The socket is `0660`, owned by
+group `_pfui`, inside a directory only that group may traverse, and the resolver's
+account (`_unbound`) must be a member — so membership of `_pfui` is what authorises
+injecting PF whitelist entries. Both installers manage the group:
+`install-server-python.sh` creates it, and `install-client-unbound.sh` adds
+`_unbound` to it when it finds a firewall installed on the same host. **Unbound
+must be restarted** for a new group membership to take effect; a resolver that is
+not in the group fails to connect with `EACCES`.
+
+If the firewall is on a different machine, none of this applies: use `HOST` and
+restrict the listening port in `pf.conf` as before.
+
 ------
 ### Compatibility;