Restructure PFUI for C rewrite and other DNS servers

.gitattributes

@@ -1 +1 @@
-docs.html.pythonmod/* linguist-documentation
+client-unbound/docs.pythonmod/* linguist-documentation

.github/workflows/ci.yml

@@ -5,10 +5,9 @@ on:
   pull_request:
 
 jobs:
-  pure:
-    # The pure suites (wire, expiry, validation, persist files, PF struct
-    # layout) are host-independent. The PF ioctl and live-firewall suites skip
-    # themselves here and must be run on an OpenBSD host.
+  python:
+    # Host-independent suites: protocol, client and server. The PF ioctl and
+    # live-firewall suites skip here by design and need an OpenBSD host.
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -19,20 +18,32 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
-        run: python -m pip install --upgrade pip lz4 pyyaml pytest
+        run: |
+          python -m pip install --upgrade pip pytest
+          python -m pip install -r server-python/requirements.txt
+          python -m pip install -r client-unbound/requirements.txt
       - name: Run tests
-        run: python -m pytest tests/ -v
+        run: python -m pytest -v
       - name: Check both daemons parse
-        run: python -m py_compile pfui_firewall.py pfui_unbound.py
+        run: python -m py_compile server-python/pfui_firewall.py client-unbound/pfui_unbound.py
+
+  server-c:
+    # Builds the C framing implementation and runs the same conformance vectors
+    # as the Python one, so the two cannot drift apart unnoticed.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build and run protocol vectors
+        run: make -C server-c test
 
   shell:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - name: Syntax-check the install and ops scripts
+      - name: Syntax-check the installers and ops scripts
         run: |
-          for f in *.sh; do bash -n "$f"; done
+          for f in *.sh client-unbound/tools/*.sh; do bash -n "$f"; done
       - name: Syntax-check the rc.d scripts
         run: |
           sudo apt-get update -qq && sudo apt-get install -y -qq ksh
-          for f in rc.d/*; do ksh -n "$f"; done
+          for f in client-unbound/rc.d/* server-python/rc.d/*; do ksh -n "$f"; done

DECISIONS.md

@@ -62,6 +62,6 @@ mismatched pair fails closed in either direction.
 
 ## Unbound runs unchrooted
 
-`chroot: ""` in the shipped resolver config. `pfui_unbound.py` imports `lz4` and
+`chroot: ""` in the shipped resolver config. `client-unbound/pfui_unbound.py` imports `lz4` and
 `yaml` at module load, and those would have to exist inside the chroot. Revisit
 only alongside a plan for the module's dependencies.

INSTALL.md

@@ -34,7 +34,7 @@ export PKG_PATH=http://ftp.openbsd.org/pub/OpenBSD/%v/packages/%a/
 pkg_add -i bash
 
 # All edge OpenBSD Firewalls
-pfui_firewall_install.sh
+install-server-python.sh
 
 # All internal DNS Servers
 install_pfui_unbound.sh
@@ -80,7 +80,7 @@ pkg_add -i swig git bash cmake libconfig libiconv bison gawk mawk
 ```
 
 #### TODO: PFUI_Unbound Dependencies
-python3 -m pip install -r ./requirements-unbound.txt
+python3 -m pip install -r ./client-unbound/requirements.txt
 
 ### PFUI_Unbound - Download Unbound source
 ```

README.md

@@ -17,6 +17,24 @@ restricted to only trusted DNS resolvers (with corporate/community DNS Blocklist
 Ie, Users cannot bypass an administrator's DNS blocking attempts using 'DNS over TCP/HTTPS' (DoT/DoH), or other any other tunneling technology etc.
 
 
+------
+## Repository layout
+
+| Path | What it is |
+|------|------------|
+| [protocol/](protocol/) | The wire protocol: specification, conformance vectors, and the Python reference implementation shared by clients and servers |
+| [client-unbound/](client-unbound/) | PFUI client as an Unbound pythonmod plugin |
+| [server-python/](server-python/) | PFUI server for OpenBSD PF, in Python |
+| [server-c/](server-c/) | PFUI server in C. Framing only so far |
+| `install-client-unbound.sh` | Installs the Unbound client on a resolver |
+| `install-server-python.sh` | Installs the Python server on a PF firewall |
+| [examples/pf.conf](examples/pf.conf) | Example PF ruleset, applies to any server implementation |
+
+Clients and servers share only the protocol. Adding support for another resolver
+means a new `client-<resolver>/`; a second server implementation means a new
+`server-<language>/`. Both cases are conformance-tested against
+[protocol/vectors/](protocol/vectors/) rather than against each other.
+
 ------
 ## PFUI Firewall comprises two parts
 
@@ -80,7 +98,7 @@ up to OpenBSD 7.4, Unbound 1.18, Python 3.10
 ```
 pkg_add bash
 git clone https://github.com/andylemin/PFUI.git && cd PFUI
-doas ./pfui_firewall_install.sh
+doas ./install-server-python.sh
 ```
 * 1b) Now add IP Reputation Block Lists to PF Firewalls (optional/recommended);\
 https://www.geoghegan.ca/pfbadhost.html \
@@ -99,7 +117,7 @@ sysctl net.inet.tcp.ackonpush=1
 ```
 pkg_add bash
 git clone https://github.com/andylemin/PFUI.git && cd PFUI
-doas ./pfui_unbound_install.sh
+doas ./install-client-unbound.sh
 ```
 Note the following lines in the example Unbound `/var/unbound/etc/pfui_unbound.conf` file after install (copy these to your own Unbound config or use the example);
 ```
@@ -247,7 +265,7 @@ This will avoid issues with local firewall services, which are assumed to be tru
 
 ------
 ### Docs;
-The Unbound "Python Module" [documentation](docs.html.pythonmod/index.html) has been included here for reference 
+The Unbound "Python Module" [documentation](client-unbound/docs.pythonmod/index.html) has been included here for reference 
 (requires compiling from source) and all rights remain with Unbound arthor's Nlnetlabs.
 The Python Module documentation for Unbound was built with SWIG on: Sep 3 13:18 2019
 

client-unbound/README.md

@@ -0,0 +1,29 @@
+# PFUI_Unbound (client)
+
+The PFUI client for [Unbound](https://nlnetlabs.nl/projects/unbound/about/).
+Installed on the resolvers, it forwards successfully resolved IPs and their TTLs
+to every PFUI server.
+
+`pfui_unbound.py` is an Unbound **pythonmod plugin**, not a script. Unbound
+executes it in the embedded interpreter's `__main__` namespace, injects
+`log_info`, `log_err` and the `MODULE_*` constants, and calls `init`,
+`init_standard`, `deinit`, `inform_super`, `operate` and
+`inplace_cache_callback` itself. Nothing in PFUI calls those, which is why
+`tests/test_unbound_module.py` pins their signatures: a rename here would surface
+only on a running resolver.
+
+| Path | What it is |
+|------|------------|
+| `pfui_unbound.py` | The plugin |
+| `pfui_unbound.yml` | PFUI client configuration (firewall list, transport, timeouts) |
+| `examples/pfui_unbound.conf` | Example Unbound config with the module enabled |
+| `rc.d/pfui_unbound` | OpenBSD rc.d script for the Unbound build with pythonmod |
+| `tools/` | Root hints and DNS blocklist updaters, run from cron |
+| `docs.pythonmod/` | Vendored Unbound pythonmod documentation, from NLnet Labs |
+
+Install with `../install-client-unbound.sh` from the repository root. The wire
+format it speaks is specified in [../protocol/PROTOCOL.md](../protocol/PROTOCOL.md);
+the installer places `pfui_wire.py` beside the plugin.
+
+Adding a client for another resolver means a new `client-<resolver>/` directory
+that speaks the same protocol. Nothing in this directory is shared.

pfui_unbound.py → client-unbound/pfui_unbound.py

@@ -31,7 +31,7 @@
 # installer puts the shared pfui/ package beside this config, so derive it from
 # there: one hardcoded location, same as CONFIG_LOCATION itself.
 sys.path.insert(0, dirname(CONFIG_LOCATION))
-from pfui.wire import encode_payload, frame  # noqa: E402
+from pfui_wire import encode_payload, frame  # noqa: E402
 
 from socket import (
     AF_INET,

tests/test_unbound.py → client-unbound/tests/test_unbound.py

@@ -15,7 +15,7 @@
 
 import pytest
 
-from pfui.wire import encode
+from pfui_wire import encode
 
 HOST = os.environ.get("PFUI_FW_HOST")
 PORT = int(os.environ.get("PFUI_FW_PORT", 10001))

tests/test_unbound_faults.py → client-unbound/tests/test_unbound_faults.py

@@ -12,7 +12,7 @@
 import lz4.frame
 import pytest
 
-from pfui.wire import HEADER, MAX_MESSAGE, encode
+from pfui_wire import HEADER, MAX_MESSAGE, encode
 
 HOST = os.environ.get("PFUI_FW_HOST")
 PORT = int(os.environ.get("PFUI_FW_PORT", 10001))

tests/test_unbound_module.py → client-unbound/tests/test_unbound_module.py

@@ -19,7 +19,7 @@
 
 import pytest
 
-REPO = Path(__file__).resolve().parent.parent
+COMPONENT = Path(__file__).resolve().parent.parent
 
 # Unbound's pythonmod injects these; supply stubs so call-time lookups resolve
 INJECTED = {
@@ -37,9 +37,9 @@
 
 @pytest.fixture(scope="module")
 def plugin():
-    sys.path.insert(0, str(REPO))
+    sys.path.insert(0, str(COMPONENT))
     spec = importlib.util.spec_from_file_location(
-        "pfui_unbound_plugin", REPO / "pfui_unbound.py"
+        "pfui_unbound_plugin", COMPONENT / "pfui_unbound.py"
     )
     mod = importlib.util.module_from_spec(spec)
     spec.loader.exec_module(mod)
@@ -60,8 +60,8 @@ def test_module_loads_without_dunder_file():
     the fixture above cannot catch a dependency on it; exec the source in a bare
     namespace instead.
     """
-    source = (REPO / "pfui_unbound.py").read_text()
-    sys.path.insert(0, str(REPO))
+    source = (COMPONENT / "pfui_unbound.py").read_text()
+    sys.path.insert(0, str(COMPONENT))
     namespace = {"__name__": "pfui_unbound_no_file"}  # deliberately no __file__
     exec(compile(source, "pfui_unbound.py", "exec"), namespace)
     assert callable(namespace["operate"])

conftest.py

@@ -0,0 +1,18 @@
+"""Put each component on sys.path the same way it is laid out when installed.
+
+The installers place the shared pfui_wire module beside each daemon, so both a
+daemon and its tests import it as a top-level module. Mirroring that here keeps
+the test imports identical to production rather than merely equivalent.
+"""
+
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+
+for path in (
+    ROOT / "protocol" / "python",  # shared: pfui_wire
+    ROOT / "server-python",        # pfui.store / pfui.validate / pfui.pf_ioctl
+    ROOT / "client-unbound",
+):
+    sys.path.insert(0, str(path))

pfui_unbound_install.sh → install-client-unbound.sh

@@ -140,7 +140,7 @@ echo
 echo "PFUIDNS: Installing PFUI Python dependencies"
 # The resolver runs unchrooted (chroot: "" in pfui_unbound.conf), so the module
 # imports these from the system interpreter's site-packages
-python3 -m pip install -r "${DIR}/requirements-unbound.txt" \
+python3 -m pip install -r "${DIR}/client-unbound/requirements.txt" \
   || die "cannot install Python dependencies"
 
 if [[ "$OS" = "OpenBSD" ]]; then
@@ -205,26 +205,25 @@ if [[ "$OS" = "OpenBSD" ]]; then
   read -p "Would you like to install the example pfui_unbound.yml (existing will be backed up) y/n: " yn
   if [[ "$yn" = "y" ]]; then
     [ -f "${TARGET}/pfui_unbound.yml" ] && mv "${TARGET}/pfui_unbound.yml" "${TARGET}/pfui_unbound.yml.${HOUR}"
-    install -m 644 -o root -g wheel "${DIR}"/pfui_unbound.yml ${TARGET}/pfui_unbound.yml
+    install -m 644 -o root -g wheel "${DIR}"/client-unbound/pfui_unbound.yml ${TARGET}/pfui_unbound.yml
   fi
   echo "Default pfui_unbound config: ${TARGET}/pfui_unbound.yml"
 
   # Install PFUI_Unbound module script
-  install -m 644 -o root -g wheel "${DIR}"/pfui_unbound.py ${TARGET}/pfui_unbound.py
-  # Shared modules; pfui_unbound.py adds its own directory to sys.path to reach these
-  install -d -m 755 -o root -g wheel ${TARGET}/pfui
-  install -m 644 -o root -g wheel "${DIR}"/pfui/__init__.py "${DIR}"/pfui/wire.py ${TARGET}/pfui/
+  install -m 644 -o root -g wheel "${DIR}"/client-unbound/pfui_unbound.py ${TARGET}/pfui_unbound.py
+  # Shared protocol module; pfui_unbound.py adds its own directory to sys.path
+  install -m 644 -o root -g wheel "${DIR}"/protocol/python/pfui_wire.py ${TARGET}/
   # Install PFUI_Unbound RC script
   # root-owned: rcctl runs this as root, and a file's owner can always chmod it
-  install -m 555 -o root -g wheel "${DIR}"/rc.d/openbsd_pfui_unbound /etc/rc.d/pfui_unbound
+  install -m 555 -o root -g wheel "${DIR}"/client-unbound/rc.d/pfui_unbound /etc/rc.d/pfui_unbound
 
   echo
   echo "PFUIDNS: Installing Root Hints and example DNS-BL"
   [ -f "${TARGET}/update_root_hints.sh" ] && mv "${TARGET}/update_root_hints.sh" "${TARGET}/update_root_hints.sh.${HOUR}"
   # root-owned: these run from cron with the privilege to write /var/unbound and restart the service
-  install -m 755 -o root -g wheel "${DIR}"/update_root_hints.sh ${TARGET}/update_root_hints.sh
+  install -m 755 -o root -g wheel "${DIR}"/client-unbound/tools/update_root_hints.sh ${TARGET}/update_root_hints.sh
   [ -f "${TARGET}/update_dns_blocklist.sh" ] && mv "${TARGET}/update_dns_blocklist.sh" "${TARGET}/update_dns_blocklist.sh.${HOUR}"
-  install -m 755 -o root -g wheel "${DIR}"/update_dns_blocklist.sh ${TARGET}/update_dns_blocklist.sh
+  install -m 755 -o root -g wheel "${DIR}"/client-unbound/tools/update_dns_blocklist.sh ${TARGET}/update_dns_blocklist.sh
   echo "New scripts: ${TARGET}/update_root_hints.sh, ${TARGET}/update_dns_blocklist.sh"
 
   # Install Unbound example configuration with PFUI_Unbound enabled
@@ -233,7 +232,7 @@ if [[ "$OS" = "OpenBSD" ]]; then
   if [[ "$yn" = "y" ]]; then
     echo "Installing example ${TARGET}/pfui_unbound.conf"
     [ -f "${TARGET}/pfui_unbound.conf" ] && mv "${TARGET}/pfui_unbound.conf" "${TARGET}/pfui_unbound.conf.${HOUR}"
-    install -m 644 -o root -g wheel "${DIR}/examples/pfui_unbound.conf" \
+    install -m 644 -o root -g wheel "${DIR}/client-unbound/examples/pfui_unbound.conf" \
       "${TARGET}/pfui_unbound.conf" || die "cannot install pfui_unbound.conf"
   fi
   echo "Default pfui_unbound config: ${TARGET}/pfui_unbound.conf"

pfui_firewall_install.sh → install-server-python.sh

@@ -74,24 +74,28 @@ if [[ "$OS" = "OpenBSD" ]]; then
   # Not a virtualenv: the daemon runs under /usr/local/bin/python3 (see its
   # shebang), so the dependencies have to be importable there. The previous
   # venv was never wired into the daemon and was the only place these landed.
-  python3 -m pip install -r "${DIR}/requirements-firewall.txt" \
+  python3 -m pip install -r "${DIR}/server-python/requirements.txt" \
     || die "cannot install Python dependencies"
 
   echo "PFUIFW: Installing PFUI Firewall Service (will backup any existing pfui_firewall configuration)"
-  install -m 755 -o root -g wheel "${DIR}"/pfui_firewall.py /usr/local/sbin/pfui_firewall \
+  install -m 755 -o root -g wheel "${DIR}"/server-python/pfui_firewall.py /usr/local/sbin/pfui_firewall \
     || die "cannot install the daemon"
   # Shared modules must sit beside the daemon; a script's own directory is sys.path[0]
   install -d -m 755 -o root -g wheel /usr/local/sbin/pfui
-  install -m 644 -o root -g wheel "${DIR}"/pfui/__init__.py "${DIR}"/pfui/wire.py \
-    "${DIR}"/pfui/store.py "${DIR}"/pfui/validate.py /usr/local/sbin/pfui/ \
+  # Shared protocol module sits beside the daemon, as a top-level module
+  install -m 644 -o root -g wheel "${DIR}"/protocol/python/pfui_wire.py \
+    /usr/local/sbin/ || die "cannot install the shared wire module"
+  install -m 644 -o root -g wheel "${DIR}"/server-python/pfui/__init__.py \
+    "${DIR}"/server-python/pfui/store.py "${DIR}"/server-python/pfui/validate.py \
+    "${DIR}"/server-python/pfui/pf_ioctl.py /usr/local/sbin/pfui/ \
     || die "cannot install the pfui modules"
 
   [ -f /etc/pfui_firewall.yml ] && cp -p /etc/pfui_firewall.yml "/etc/pfui_firewall.yml.${HOUR}"
   # root-owned: the daemon only reads this, and CTL: PFCTL makes it a command source
-  install -m 644 -o root -g wheel "${DIR}"/pfui_firewall.yml /etc/pfui_firewall.yml
+  install -m 644 -o root -g wheel "${DIR}"/server-python/pfui_firewall.yml /etc/pfui_firewall.yml
   echo "PFUIFW: PFUI_Firewall default configuration file located at '/etc/pfui_firewall.yml' (please configure)"
   # root-owned: rcctl runs this as root, and a file's owner can always chmod it
-  install -m 555 -o root -g wheel "${DIR}"/rc.d/pfui_firewall /etc/rc.d/pfui_firewall \
+  install -m 555 -o root -g wheel "${DIR}"/server-python/rc.d/pfui_firewall /etc/rc.d/pfui_firewall \
     || die "cannot install the rc.d script"
 
   install -m 644 -o root -g wheel "${DIR}"/examples/pf.conf /etc/pf-pfui-example.conf