Skip to content

Commit 8e42d42

Browse files
authored
Add the contributing guide (#79)
Build, test, the FreeBSD VM, and one recipe per kind of change (a NIF, an erlang.* function, a context option, a suite, a guide) listing every file that change touches, so a contributor can finish a change without reading the whole tree first.
1 parent ca7f8a7 commit 8e42d42

3 files changed

Lines changed: 216 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,10 @@ Key features:
4747
rebar3 compile
4848
```
4949

50+
To work on erlang_python itself (tests, CI matrix, FreeBSD VM, how to add
51+
a NIF, an `erlang.*` function, an option, a suite or a guide), see
52+
[docs/contributing.md](docs/contributing.md).
53+
5054
## Quick Start
5155

5256
### Erlang

‎docs/contributing.md‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
# Contributing
2+
3+
How to build, test and extend erlang_python. Read this before your first
4+
change; each recipe lists every file a change of that kind touches, so
5+
nothing is left half done. Design background is in
6+
[architecture](architecture.md), file ownership in [code map](code-map.md),
7+
message shapes in [protocols](protocols.md).
8+
9+
## Build
10+
11+
You need OTP 27 or later, Python 3.12 or later with headers, CMake and a C
12+
compiler.
13+
14+
```sh
15+
rebar3 compile # runs do_cmake.sh / do_build.sh, builds priv/py_nif.so
16+
PYTHON_CONFIG=python3.13-config rebar3 compile # pick an interpreter
17+
```
18+
19+
Notes:
20+
21+
- `c_src/py_nif.c` includes the other `.c` files; there is no per-file
22+
compile. A change in any `c_src` file rebuilds the one NIF.
23+
- The interpreter used at build time is the one embedded at run time.
24+
`python3-config` on `PATH` is the default.
25+
- `rm -rf _build` when switching Python versions or build flags; CMake
26+
caches the interpreter.
27+
28+
## Test
29+
30+
```sh
31+
rebar3 ct --readable=compact # everything
32+
rebar3 ct --suite test/py_isolated_SUITE # one suite
33+
rebar3 ct --suite test/py_context_SUITE --case test_call # one case
34+
rebar3 dialyzer && rebar3 xref # required before a PR
35+
make lint-docs # snippets in README and docs/
36+
```
37+
38+
Notes:
39+
40+
- `test/test.config` turns memory limits on for the whole run; it is
41+
applied automatically.
42+
- The `test` profile pulls `iommap` for the shared memory suites; users
43+
add it to their own deps.
44+
- Free-threaded Python: build and run with `PYTHON_GIL=0` and a `3.13t`
45+
interpreter; cases that need the GIL skip themselves.
46+
- ASan: configure CMake by hand, then let rebar3 pick the objects up:
47+
48+
```sh
49+
rm -rf _build && mkdir -p _build/cmake && cd _build/cmake
50+
cmake ../../c_src -DENABLE_ASAN=ON -DENABLE_UBSAN=ON && cmake --build . && cd ../..
51+
rebar3 compile
52+
LD_PRELOAD=$(gcc -print-file-name=libasan.so) ASAN_OPTIONS=detect_leaks=0 rebar3 ct
53+
```
54+
55+
CI runs the matrix in `.github/workflows/ci.yml`: OTP 27 to 29 with Python
56+
3.12 to 3.14 on Ubuntu and macOS, FreeBSD 14 through `vmactions/freebsd-vm`,
57+
free-threaded 3.13t, and ASan. A PR is merged when all of them are green.
58+
59+
### FreeBSD by hand
60+
61+
CI covers FreeBSD, but the isolated mode (procctl, no cgroups, no
62+
`/dev/shm`) is easier to debug in a local VM. On an Apple Silicon Mac:
63+
64+
```sh
65+
# 1. image (arm64 on Apple Silicon, amd64 elsewhere)
66+
curl -O https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/aarch64/Latest/FreeBSD-14.1-RELEASE-arm64-aarch64.qcow2.xz
67+
xz -d FreeBSD-14.1-RELEASE-arm64-aarch64.qcow2.xz
68+
qemu-img resize FreeBSD-14.1-RELEASE-arm64-aarch64.qcow2 +20G
69+
70+
# 2. boot headless, ssh on host port 2222
71+
qemu-system-aarch64 -M virt -accel hvf -cpu host -m 4096 -smp 4 \
72+
-bios /opt/homebrew/share/qemu/edk2-aarch64-code.fd \
73+
-drive file=FreeBSD-14.1-RELEASE-arm64-aarch64.qcow2,if=virtio,format=qcow2 \
74+
-netdev user,id=n0,hostfwd=tcp::2222-:22 -device virtio-net-pci,netdev=n0 \
75+
-nographic -serial mon:stdio
76+
```
77+
78+
On the console, log in as `root` (no password), set one, and enable ssh:
79+
80+
```sh
81+
passwd
82+
sysrc sshd_enable=YES
83+
sed -i '' 's/^#PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config
84+
service sshd start
85+
pkg install -y erlang-runtime28 python313 py313-numpy cmake gmake git
86+
fetch -o /root/rebar3 https://github.com/erlang/rebar3/releases/download/3.25.0/rebar3
87+
chmod +x /root/rebar3
88+
```
89+
90+
Then from the host, ship the tree and run:
91+
92+
```sh
93+
git archive --format=tgz -o /tmp/ep.tgz HEAD
94+
scp -P 2222 /tmp/ep.tgz root@127.0.0.1:/root/
95+
ssh -p 2222 root@127.0.0.1 'rm -rf ep && mkdir ep && tar -C ep -xzf ep.tgz && cd ep \
96+
&& export PATH=/usr/local/lib/erlang28/bin:$PATH PYTHON_CONFIG=python3.13-config \
97+
&& /root/rebar3 compile && /root/rebar3 ct --readable=compact'
98+
```
99+
100+
`erlang-runtime28` installs outside the default `PATH`; `pkg info -l
101+
erlang-runtime28 | grep bin/erl` shows where. Use `-accel kvm` and the
102+
amd64 image on Linux hosts.
103+
104+
## Recipes
105+
106+
Every recipe ends with the same three steps: a test case, `rebar3
107+
dialyzer && rebar3 xref`, and an entry in `CHANGELOG.md` under the
108+
unreleased version.
109+
110+
### Add a NIF
111+
112+
1. Implement `static ERL_NIF_TERM nif_x(ErlNifEnv*, int, const ERL_NIF_TERM[])`
113+
in the `c_src` file that owns the area (see `c_src/README.md`).
114+
2. Add `{"x", Arity, nif_x, Flags}` to `nif_funcs[]` at the end of
115+
`c_src/py_nif.c`. `Flags` is `ERL_NIF_DIRTY_JOB_CPU_BOUND` or
116+
`ERL_NIF_DIRTY_JOB_IO_BOUND` when the NIF can block or run Python, `0`
117+
otherwise.
118+
3. Add the stub, its `-spec` and a `@doc` to `src/py_nif.erl`, and the
119+
export.
120+
4. Follow the rules in `c_src/README.md`: only the context thread touches
121+
a context's Python objects; release the GIL around every blocking
122+
wait; respect the lock order on `py_context_t`.
123+
5. Test it from a suite through the Erlang API that uses it, not through
124+
`py_nif` directly, unless it is a test helper.
125+
126+
### Add an `erlang.*` function for Python
127+
128+
The `erlang` module has three implementations that must agree:
129+
130+
1. Embedded modes, C: add `erlang_x_impl` and its entry in the method
131+
table of `c_src/py_callback.c` (`PyMethodDef erlang_methods`). Names
132+
starting with `_` are internal helpers for the Python package.
133+
2. Embedded modes, Python: if the function is written in Python, add it to
134+
`priv/_erlang_impl/__init__.py` and to `__all__`; the C module copies
135+
it onto `erlang` at import (see the bootstrap code near the end of
136+
`py_callback.c`).
137+
3. Isolated mode: add it to `install_erlang_module` in
138+
`priv/_erlang_impl/_isolated.py`. Anything the child cannot support
139+
goes through `_not_supported(name)` so the user gets a clear
140+
`RuntimeError`, never a silent difference.
141+
4. Document it in the guide of its area and in `README.md` (API
142+
reference section), and add the row to `test/coverage_audit.md`.
143+
5. Test it in a cross-mode suite so it runs in `worker` and `isolated`
144+
groups.
145+
146+
### Add a context option
147+
148+
Options arrive as the map given to `py_context:new/1`. Pool contexts are
149+
started by `py_context_sup:start_context/2` with the mode only, so an
150+
option that must apply pool-wide also needs `py_context_router` and the
151+
supervisor to carry it.
152+
153+
1. Embedded modes: read it where the context is set up in
154+
`src/py_context.erl` (`init/4` and the helpers it calls; `memory_limit`,
155+
`preload`, `owner` and `start_timeout` are the existing examples).
156+
If the C side needs it, pass it to `nif_context_create` and extend the
157+
options parsing there.
158+
2. Isolated mode: read it in `src/py_isolated.erl` (`start_child/1`
159+
assembles the child command line and environment; `rlimits`, `cgroup`,
160+
`python`, `restart`, `max_restarts`, `kill_after` are the examples) and,
161+
if the child must know, add it to the `{init, Opts}` request handled by
162+
`_init` in `priv/_erlang_impl/_isolated.py`.
163+
3. If the option makes no sense in one mode, reject it there with an
164+
error that names the option rather than ignoring it.
165+
4. Document it in the options table of the relevant guide
166+
(`docs/isolated.md`, `docs/workers.md`, `docs/memory.md`) and test both
167+
the effect and the rejection.
168+
169+
### Add a test suite
170+
171+
1. Name it `test/py_<area>_SUITE.erl`; Python helpers go in
172+
`test/py_test_<area>.py` and are imported after
173+
`sys.path.insert(0, TestDir)` (see `py_reentrant_SUITE` for the
174+
pattern).
175+
2. Start and stop the application in `init_per_suite` /
176+
`end_per_suite`; the default pool is `py:start_contexts/0`.
177+
3. Behaviour every mode must share runs in groups: `groups/0` with
178+
`worker` and `isolated`, contexts created with
179+
`py_context:new(#{mode => Mode})` from `init_per_group`
180+
(`py_isolated_SUITE` shows the layout).
181+
4. Skip, do not fail, when a platform or interpreter cannot run a case:
182+
`{skip, Reason}` with the reason a human can act on.
183+
5. Add the suite to the table in `docs/code-map.md` and the cases that
184+
cover a documented API to `test/coverage_audit.md`.
185+
6. Cases that measure time or memory print their numbers with `ct:pal`
186+
and assert only on invariants, never on absolute timings.
187+
188+
### Add or change a guide
189+
190+
1. Create `docs/<name>.md` in the task-oriented form: one paragraph on
191+
what it is and when you need it, then steps with code, then short
192+
notes. Second person, Erlang snippets, no hype.
193+
2. Every Erlang snippet must call real exports at the right arity and
194+
every Python snippet must parse: `make lint-docs` checks both.
195+
`<!-- skip-lint -->` above a fence exempts it; say why in the prose.
196+
3. Register the page in both `extras` lists of `rebar.config` (the flat
197+
list and the grouped one) so `rebar3 ex_doc` builds it, and link it
198+
from the guide or README section that leads to it.
199+
4. If the page documents a new API, add its rows to
200+
`test/coverage_audit.md`.
201+
202+
## Before opening a pull request
203+
204+
- `rebar3 ct`, `rebar3 dialyzer`, `rebar3 xref`, `make lint-docs` all
205+
clean locally.
206+
- `CHANGELOG.md` updated under the unreleased version: `Added`, `Changed`,
207+
`Removed` or `Fixed`. Removing a public function is a major version.
208+
- The PR text says what the change intends and which path it takes; the
209+
diff already lists the files.
210+
- One squashed commit per PR.

‎rebar.config‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@
8181
<<"docs/glossary.md">>,
8282
<<"docs/protocols.md">>,
8383
<<"docs/state-machines.md">>,
84+
<<"docs/contributing.md">>,
8485
<<"docs/preload.md">>,
8586
<<"docs/owngil_internals.md">>,
8687
<<"docs/event_loop_architecture.md">>
@@ -121,6 +122,7 @@
121122
<<"docs/glossary.md">>,
122123
<<"docs/protocols.md">>,
123124
<<"docs/state-machines.md">>,
125+
<<"docs/contributing.md">>,
124126
<<"docs/preload.md">>,
125127
<<"docs/owngil_internals.md">>,
126128
<<"docs/event_loop_architecture.md">>

0 commit comments

Comments
 (0)