|
| 1 | +# Contributing |
| 2 | + |
| 3 | +How to build, test and extend erlang_python. Read this before your first |
| 4 | +change; each recipe lists every file a change of that kind touches, so |
| 5 | +nothing is left half done. Design background is in |
| 6 | +[architecture](architecture.md), file ownership in [code map](code-map.md), |
| 7 | +message shapes in [protocols](protocols.md). |
| 8 | + |
| 9 | +## Build |
| 10 | + |
| 11 | +You need OTP 27 or later, Python 3.12 or later with headers, CMake and a C |
| 12 | +compiler. |
| 13 | + |
| 14 | +```sh |
| 15 | +rebar3 compile # runs do_cmake.sh / do_build.sh, builds priv/py_nif.so |
| 16 | +PYTHON_CONFIG=python3.13-config rebar3 compile # pick an interpreter |
| 17 | +``` |
| 18 | + |
| 19 | +Notes: |
| 20 | + |
| 21 | +- `c_src/py_nif.c` includes the other `.c` files; there is no per-file |
| 22 | + compile. A change in any `c_src` file rebuilds the one NIF. |
| 23 | +- The interpreter used at build time is the one embedded at run time. |
| 24 | + `python3-config` on `PATH` is the default. |
| 25 | +- `rm -rf _build` when switching Python versions or build flags; CMake |
| 26 | + caches the interpreter. |
| 27 | + |
| 28 | +## Test |
| 29 | + |
| 30 | +```sh |
| 31 | +rebar3 ct --readable=compact # everything |
| 32 | +rebar3 ct --suite test/py_isolated_SUITE # one suite |
| 33 | +rebar3 ct --suite test/py_context_SUITE --case test_call # one case |
| 34 | +rebar3 dialyzer && rebar3 xref # required before a PR |
| 35 | +make lint-docs # snippets in README and docs/ |
| 36 | +``` |
| 37 | + |
| 38 | +Notes: |
| 39 | + |
| 40 | +- `test/test.config` turns memory limits on for the whole run; it is |
| 41 | + applied automatically. |
| 42 | +- The `test` profile pulls `iommap` for the shared memory suites; users |
| 43 | + add it to their own deps. |
| 44 | +- Free-threaded Python: build and run with `PYTHON_GIL=0` and a `3.13t` |
| 45 | + interpreter; cases that need the GIL skip themselves. |
| 46 | +- ASan: configure CMake by hand, then let rebar3 pick the objects up: |
| 47 | + |
| 48 | +```sh |
| 49 | +rm -rf _build && mkdir -p _build/cmake && cd _build/cmake |
| 50 | +cmake ../../c_src -DENABLE_ASAN=ON -DENABLE_UBSAN=ON && cmake --build . && cd ../.. |
| 51 | +rebar3 compile |
| 52 | +LD_PRELOAD=$(gcc -print-file-name=libasan.so) ASAN_OPTIONS=detect_leaks=0 rebar3 ct |
| 53 | +``` |
| 54 | + |
| 55 | +CI runs the matrix in `.github/workflows/ci.yml`: OTP 27 to 29 with Python |
| 56 | +3.12 to 3.14 on Ubuntu and macOS, FreeBSD 14 through `vmactions/freebsd-vm`, |
| 57 | +free-threaded 3.13t, and ASan. A PR is merged when all of them are green. |
| 58 | + |
| 59 | +### FreeBSD by hand |
| 60 | + |
| 61 | +CI covers FreeBSD, but the isolated mode (procctl, no cgroups, no |
| 62 | +`/dev/shm`) is easier to debug in a local VM. On an Apple Silicon Mac: |
| 63 | + |
| 64 | +```sh |
| 65 | +# 1. image (arm64 on Apple Silicon, amd64 elsewhere) |
| 66 | +curl -O https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/aarch64/Latest/FreeBSD-14.1-RELEASE-arm64-aarch64.qcow2.xz |
| 67 | +xz -d FreeBSD-14.1-RELEASE-arm64-aarch64.qcow2.xz |
| 68 | +qemu-img resize FreeBSD-14.1-RELEASE-arm64-aarch64.qcow2 +20G |
| 69 | + |
| 70 | +# 2. boot headless, ssh on host port 2222 |
| 71 | +qemu-system-aarch64 -M virt -accel hvf -cpu host -m 4096 -smp 4 \ |
| 72 | + -bios /opt/homebrew/share/qemu/edk2-aarch64-code.fd \ |
| 73 | + -drive file=FreeBSD-14.1-RELEASE-arm64-aarch64.qcow2,if=virtio,format=qcow2 \ |
| 74 | + -netdev user,id=n0,hostfwd=tcp::2222-:22 -device virtio-net-pci,netdev=n0 \ |
| 75 | + -nographic -serial mon:stdio |
| 76 | +``` |
| 77 | + |
| 78 | +On the console, log in as `root` (no password), set one, and enable ssh: |
| 79 | + |
| 80 | +```sh |
| 81 | +passwd |
| 82 | +sysrc sshd_enable=YES |
| 83 | +sed -i '' 's/^#PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config |
| 84 | +service sshd start |
| 85 | +pkg install -y erlang-runtime28 python313 py313-numpy cmake gmake git |
| 86 | +fetch -o /root/rebar3 https://github.com/erlang/rebar3/releases/download/3.25.0/rebar3 |
| 87 | +chmod +x /root/rebar3 |
| 88 | +``` |
| 89 | + |
| 90 | +Then from the host, ship the tree and run: |
| 91 | + |
| 92 | +```sh |
| 93 | +git archive --format=tgz -o /tmp/ep.tgz HEAD |
| 94 | +scp -P 2222 /tmp/ep.tgz root@127.0.0.1:/root/ |
| 95 | +ssh -p 2222 root@127.0.0.1 'rm -rf ep && mkdir ep && tar -C ep -xzf ep.tgz && cd ep \ |
| 96 | + && export PATH=/usr/local/lib/erlang28/bin:$PATH PYTHON_CONFIG=python3.13-config \ |
| 97 | + && /root/rebar3 compile && /root/rebar3 ct --readable=compact' |
| 98 | +``` |
| 99 | + |
| 100 | +`erlang-runtime28` installs outside the default `PATH`; `pkg info -l |
| 101 | +erlang-runtime28 | grep bin/erl` shows where. Use `-accel kvm` and the |
| 102 | +amd64 image on Linux hosts. |
| 103 | + |
| 104 | +## Recipes |
| 105 | + |
| 106 | +Every recipe ends with the same three steps: a test case, `rebar3 |
| 107 | +dialyzer && rebar3 xref`, and an entry in `CHANGELOG.md` under the |
| 108 | +unreleased version. |
| 109 | + |
| 110 | +### Add a NIF |
| 111 | + |
| 112 | +1. Implement `static ERL_NIF_TERM nif_x(ErlNifEnv*, int, const ERL_NIF_TERM[])` |
| 113 | + in the `c_src` file that owns the area (see `c_src/README.md`). |
| 114 | +2. Add `{"x", Arity, nif_x, Flags}` to `nif_funcs[]` at the end of |
| 115 | + `c_src/py_nif.c`. `Flags` is `ERL_NIF_DIRTY_JOB_CPU_BOUND` or |
| 116 | + `ERL_NIF_DIRTY_JOB_IO_BOUND` when the NIF can block or run Python, `0` |
| 117 | + otherwise. |
| 118 | +3. Add the stub, its `-spec` and a `@doc` to `src/py_nif.erl`, and the |
| 119 | + export. |
| 120 | +4. Follow the rules in `c_src/README.md`: only the context thread touches |
| 121 | + a context's Python objects; release the GIL around every blocking |
| 122 | + wait; respect the lock order on `py_context_t`. |
| 123 | +5. Test it from a suite through the Erlang API that uses it, not through |
| 124 | + `py_nif` directly, unless it is a test helper. |
| 125 | + |
| 126 | +### Add an `erlang.*` function for Python |
| 127 | + |
| 128 | +The `erlang` module has three implementations that must agree: |
| 129 | + |
| 130 | +1. Embedded modes, C: add `erlang_x_impl` and its entry in the method |
| 131 | + table of `c_src/py_callback.c` (`PyMethodDef erlang_methods`). Names |
| 132 | + starting with `_` are internal helpers for the Python package. |
| 133 | +2. Embedded modes, Python: if the function is written in Python, add it to |
| 134 | + `priv/_erlang_impl/__init__.py` and to `__all__`; the C module copies |
| 135 | + it onto `erlang` at import (see the bootstrap code near the end of |
| 136 | + `py_callback.c`). |
| 137 | +3. Isolated mode: add it to `install_erlang_module` in |
| 138 | + `priv/_erlang_impl/_isolated.py`. Anything the child cannot support |
| 139 | + goes through `_not_supported(name)` so the user gets a clear |
| 140 | + `RuntimeError`, never a silent difference. |
| 141 | +4. Document it in the guide of its area and in `README.md` (API |
| 142 | + reference section), and add the row to `test/coverage_audit.md`. |
| 143 | +5. Test it in a cross-mode suite so it runs in `worker` and `isolated` |
| 144 | + groups. |
| 145 | + |
| 146 | +### Add a context option |
| 147 | + |
| 148 | +Options arrive as the map given to `py_context:new/1`. Pool contexts are |
| 149 | +started by `py_context_sup:start_context/2` with the mode only, so an |
| 150 | +option that must apply pool-wide also needs `py_context_router` and the |
| 151 | +supervisor to carry it. |
| 152 | + |
| 153 | +1. Embedded modes: read it where the context is set up in |
| 154 | + `src/py_context.erl` (`init/4` and the helpers it calls; `memory_limit`, |
| 155 | + `preload`, `owner` and `start_timeout` are the existing examples). |
| 156 | + If the C side needs it, pass it to `nif_context_create` and extend the |
| 157 | + options parsing there. |
| 158 | +2. Isolated mode: read it in `src/py_isolated.erl` (`start_child/1` |
| 159 | + assembles the child command line and environment; `rlimits`, `cgroup`, |
| 160 | + `python`, `restart`, `max_restarts`, `kill_after` are the examples) and, |
| 161 | + if the child must know, add it to the `{init, Opts}` request handled by |
| 162 | + `_init` in `priv/_erlang_impl/_isolated.py`. |
| 163 | +3. If the option makes no sense in one mode, reject it there with an |
| 164 | + error that names the option rather than ignoring it. |
| 165 | +4. Document it in the options table of the relevant guide |
| 166 | + (`docs/isolated.md`, `docs/workers.md`, `docs/memory.md`) and test both |
| 167 | + the effect and the rejection. |
| 168 | + |
| 169 | +### Add a test suite |
| 170 | + |
| 171 | +1. Name it `test/py_<area>_SUITE.erl`; Python helpers go in |
| 172 | + `test/py_test_<area>.py` and are imported after |
| 173 | + `sys.path.insert(0, TestDir)` (see `py_reentrant_SUITE` for the |
| 174 | + pattern). |
| 175 | +2. Start and stop the application in `init_per_suite` / |
| 176 | + `end_per_suite`; the default pool is `py:start_contexts/0`. |
| 177 | +3. Behaviour every mode must share runs in groups: `groups/0` with |
| 178 | + `worker` and `isolated`, contexts created with |
| 179 | + `py_context:new(#{mode => Mode})` from `init_per_group` |
| 180 | + (`py_isolated_SUITE` shows the layout). |
| 181 | +4. Skip, do not fail, when a platform or interpreter cannot run a case: |
| 182 | + `{skip, Reason}` with the reason a human can act on. |
| 183 | +5. Add the suite to the table in `docs/code-map.md` and the cases that |
| 184 | + cover a documented API to `test/coverage_audit.md`. |
| 185 | +6. Cases that measure time or memory print their numbers with `ct:pal` |
| 186 | + and assert only on invariants, never on absolute timings. |
| 187 | + |
| 188 | +### Add or change a guide |
| 189 | + |
| 190 | +1. Create `docs/<name>.md` in the task-oriented form: one paragraph on |
| 191 | + what it is and when you need it, then steps with code, then short |
| 192 | + notes. Second person, Erlang snippets, no hype. |
| 193 | +2. Every Erlang snippet must call real exports at the right arity and |
| 194 | + every Python snippet must parse: `make lint-docs` checks both. |
| 195 | + `<!-- skip-lint -->` above a fence exempts it; say why in the prose. |
| 196 | +3. Register the page in both `extras` lists of `rebar.config` (the flat |
| 197 | + list and the grouped one) so `rebar3 ex_doc` builds it, and link it |
| 198 | + from the guide or README section that leads to it. |
| 199 | +4. If the page documents a new API, add its rows to |
| 200 | + `test/coverage_audit.md`. |
| 201 | + |
| 202 | +## Before opening a pull request |
| 203 | + |
| 204 | +- `rebar3 ct`, `rebar3 dialyzer`, `rebar3 xref`, `make lint-docs` all |
| 205 | + clean locally. |
| 206 | +- `CHANGELOG.md` updated under the unreleased version: `Added`, `Changed`, |
| 207 | + `Removed` or `Fixed`. Removing a public function is a major version. |
| 208 | +- The PR text says what the change intends and which path it takes; the |
| 209 | + diff already lists the files. |
| 210 | +- One squashed commit per PR. |
0 commit comments