e2e: on-screen tests you can actually judge — ids, captions, and a review TUI - #133
Merged
Conversation
…well Watching the real-renderer suite, it was hard to tell what any given moment was supposed to look like, and impossible to report a fault precisely: the caption said "phase_cycles", many tests drew their stimulus and deleted it in the same breath, and the demo tests never showed the scene they had just built at all. Every test now carries an `onscreen(id, description)` marker. An autouse `stage` fixture puts `[GRAT-09] <description>` on screen for the whole test and `Stage.step()/hold()` keeps each state up for a multiple of --step-delay, so what a test claims is visible actually is. Ids are stable and unique (RECT, CIRC, ELLI, POLY, SHAPE, GRAT, TEXT, ANIM, SHARED, QUERY, SYS, CFG, VTL, DEMO, PSY), so a fault can be written down as "GRAT-09" and grepped back to the test. Tests that compare the whole scene declare the caption deferred and raise it once they are past that point — which is also what finally makes each demo visible, held for a few seconds before its teardown. Scenes are also cleaned up per test now: a `scene_reset` fixture clears stimuli, animations, VTL names, background and deferred mode after every test, so a failed assertion or a merely-disabled stimulus can no longer leave invisible clutter behind for the rest of the run. The PsychoPy suite gets a `flush_window` fixture with it, because its Window is deferred and would otherwise replay a staged command against a stimulus the reset has deleted. Null suites pin --step-delay to 0, so none of this costs a headless run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
…ausing Watching the suite run turned up three things. Text stimuli were sometimes invisible: `letter_height_px` and `box_size_px` both default to 0, so the opacity check in SHARED-09 created text that rendered at zero size — it now sets both. The caption could also be buried, because draw order is creation order and the caption was created before everything the test drew; it is now rebuilt on every step (new one first, so no frame goes without), sits higher up the frame clear of the demos' own titles, and carries a dim backing box so it stays readable over mid-grey backgrounds and over stimuli. Several animations were over before the caption could be read. The short flashes and flickers are lengthened where nothing asserts their frame count (ANIM-01/03/08/17/18/20/21/22/25/27/28), and `Stage.cue()` says what to watch for *before* the animation is armed rather than after it has started. Finally the suite can be paused: `--pause` waits for a keypress once per test, `--pause=step` at every caption change; Enter continues, `c` carries on without pausing, `q` stops the run. The prompt suspends pytest's capture so it is both visible and answerable, and a run with nothing on stdin turns pausing off instead of hanging. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
… e2e suite The suite's own server is fullscreen and covers the terminal the --pause prompt waits in. Pre-starting a windowed server is already enough — the fixture attaches to any reachable server rather than starting one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
…ging Watching the suite is only half the job; the other half is writing down what looked wrong, and ids on screen still left that to paper. The pause prompt is now a small review console: it names the test, what it is for, and what is on screen at that instant, and `f` records the test as problematic with a note before the run carries on. Flagged tests are listed in the terminal summary and written to e2e-review.md — each note, the caption it belongs to, and a ready-made command to re-run just those tests. Two make targets drive it, since the option combination is easy to forget: test-e2e-review pauses once per test, test-e2e-step at every caption change. Both take PYTEST_ARGS, and test-e2e/test-e2e-null now do too. The confirmation after flagging goes straight to the terminal rather than through print, which pytest's capture would otherwise swallow until (and unless) the test failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
… readout Watching the suite in order is not how a review actually goes: the thing that looked wrong was two tests ago, and there was no way back to it. The pause prompt now numbers every test, shows where in the suite it is, how the test came out, and takes vim's motions — j/k, 5j/5k, 42G, gg/G, r to replay, /text to search, l to list. Forward motions are handled inside the session by skipping ahead. Backwards cannot be: a pytest session runs its tests once, in order. So tests/e2e/browse.py owns the loop — pytest records the target and stops, the browser restarts it there, and the notes travel in the same file. It also starts a windowed server for the whole browse session, which keeps this terminal (and its prompt) visible, and reuses one that is already up. Stopping now goes through session.shouldstop rather than pytest.exit: the prompt runs inside a fixture teardown, and exiting from there abandoned the scene reset that follows it, leaving stimuli, a background colour or deferred mode behind for the next session to trip over. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
…creen Four things a real session on a real display turned up. The caption was pinned at y=420, which is near the top of a 1080-line screen and off the bottom of the 720-line window the review target opens — invisible, exactly as reported. Position, box and letter height are now fractions of what the server says the display is. The prompt came out as one long smear: readline treats an input() prompt as a single line and eats newlines in it. Everything multi-line is printed now and only the cursor line is the prompt, and both the caption and the list are folded to the terminal width instead of running off it. A test that lasts half a second was over before its description arrived, since the prompt only comes afterwards. Each test is now announced before it runs, so the description is on screen before the thing it describes happens. The animation captions that still quoted their old frame counts are corrected too. Taking the caption down before the scene reset clears it stops the server logging a handle-not-found warning per test. Also, four e2e targets were three too many: test-e2e runs start to finish, test-e2e-review is the one you drive by hand (now the browser, so it has the vim motions), test-e2e-null needs no display. While in the README, the e2e server address was documented as VSTIM_SERVER_ADDR in three places, which nothing reads — it is VSTIMD_SERVER. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
…le widgets Driving pytest's own terminal was the wrong shape for this. Single keypresses fought readline, the console had to be rebuilt out of print statements, and going backwards meant restarting pytest and handing state through a file. What the job actually wants is a UI: a list of every test with what it should put on screen, and keys to run, re-run, revisit and annotate. So the review target is now a Textual app with pytest underneath it. The plugin takes over pytest_runtestloop and runs whichever item the app asks for, in any order, as often as asked — passing a *next* item to the protocol keeps session-scoped fixtures (the server connection) alive between runs. Two settings keep the terminal ours: `-p no:terminal` silences pytest's reporting and `--capture=sys` captures at the Python level, leaving the real stdout to draw on. All the terminal-prompt machinery in conftest, and browse.py with it, is gone; what remains there is the caption, the scene reset and --step-delay. The panels are `vstimd.tui` widgets rather than app-local code, since a control program for the rig wants exactly these: StimulusList (what the scene holds), TriggerLines (VTL lines, with t to toggle and p to pulse — a trigger-driven scene exercised with no DAQ attached) and ServerStatus. They take a Connection and poll it; textual is an optional `tui` extra, so nothing about talking to a server depends on it. Verified headless with Textual's own pilot: moving, running, run-and-advance, flagging with a note, panel switching, and the report that comes out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
test-e2e said nothing about the display it needs, and test-e2e-review said nothing about being interactive. They are now test-e2e-visible and test-e2e-interactive, next to test-e2e-null. test-integration is deleted: it printed "not implemented yet: integration tests with MockServer", there is no MockServer anywhere in the tree, and the null renderer covers the ground a mock was meant to — it is a real server without a display, which is a more faithful thing to test against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
…he scene Found chasing an intermittent e2e failure: a mutation would take, and then a frame later read back as the value it had before. Roughly one full run in ten, never the same test twice, never reproducible in isolation. `end_deferred` scheduled a flip unconditionally. The flip promotes every copy slot over its live one, and outside deferred mode the copies are stale by construction — ordinary writes go to `live` alone. So a client that switches deferred mode off defensively, without ever having switched it on, armed a flip that undid every write made in the ~16 ms before the next frame. The animation path's END_DEFERRED did the same thing. Both now do nothing when there is nothing staged, and a test pins it. The e2e scene reset was the client that did this — it turned deferred mode off after every test, so the next test's first mutations were the ones reverted. It now asks for `cancel`, which drops staged state without scheduling a flip: the right request for "whatever the last test left, I do not want it", and correct against a server without this fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
The call returned a bare ack, so a client could not tell "flip queued" from
"there was nothing to end" — the distinction that made the reverted-scene bug
above so hard to see — and had no idea when the staged frame would be on screen.
The reply now carries `deferred` (the mode after), `was_deferred` (before, so a
no-op is visible), `flip_scheduled`, and `flip_frame`: the frame the staged
state is first drawn from. Together with the envelope's `frame_count` — the
frame the call was handled on — a client can say exactly where a batch started
and where it lands, and wait for that frame instead of sleeping on a guessed
vsync:
begun = conn.system.set_deferred_mode(True) # staging from begun.frame_count
...
ended = conn.system.set_deferred_mode(False)
conn.system.wait_for_frame(ended.flip_frame) # now it is drawn
The Python client returns a DeferredModeStatus with those fields, plus
`was_a_no_op` and `frames_staged`, and gains `wait_for_frame(n)` — the protocol
wait is relative, the interesting number is absolute, so the remaining distance
is recomputed until covered. The e2e deferred test now checks the reported
status rather than sleeping.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
Two sessions against one server wreck each other: every test resets the scene, so one session's teardown deletes the other's stimuli and animations, and what comes out is a handle-not-found in an unrelated test. That is what happened running the null suite next to an interactive review session. The null suites now start their own server on a free port, with --no-web so the web surface does not fight for 8080 either. Passing --server (or VSTIMD_SERVER) still opts into an existing one, which is what CI and a rig do; the default is simply no longer "whatever is on 5555". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
A REQ socket that times out waiting for a reply is stuck: it still owes one, and the next send fails with EFSM. `wait_until_ready` knew that and rebuilt the socket by hand; every other caller did not, so a single slow reply left the connection useless for the rest of the session. `_send` now does it once, centrally, and raises TimeoutError naming the address and the command rather than leaking zmq.Again. That is what makes a polling UI safe: the vstimd.tui widgets read from a connection on the UI thread, and a server that stops answering must not take the interface down with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD
67 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The on-screen e2e suite renders stimuli for a human to look at, but there was no
way to tell what any given moment was supposed to look like, and no way to
report a fault precisely. This branch makes the suite legible, then gives it a
front end — and, chasing a flake it exposed, fixes a real scene bug.
What changed
Every test says what it should show, and shows it long enough to judge.
Each test carries
@pytest.mark.onscreen("GRAT-09", "…"); an autouse fixturecaptions the display with
[GRAT-09] <what you should be seeing>while it runs.Ids are unique and stable across the 147 tests (
RECT CIRC ELLI POLY SHAPE GRAT TEXT ANIM SHARED QUERY SYS CFG VTL DEMO PSY), so a fault is reported as"GRAT-09 looked wrong" and found again with
grep -rn GRAT-09 tests/e2e.Descriptions are whole sentences — "a full square patch of stripes that becomes
a circular disc when the CIRCLE mask is applied — the corners are cut away" —
rather than the old
"mask".Tests that drew a stimulus and deleted it in the same breath now hold it up
first. The demo tests, which built a whole scene, diffed it and tore it down
without ever showing it, now hold each demo on screen for a few seconds. Short
animations are lengthened where nothing asserts their frame count, and
Stage.cue()says what to watch for before the animation is armed.Scenes are cleaned up per test. A
scene_resetfixture clears stimuli,animations, VTL names, background and deferred mode after every test — a failed
assertion, a disabled stimulus or one an animation hid used to accumulate
invisibly over a run.
A terminal UI for reviewing it by eye —
make test-e2e-interactive. Itlists every test with what it should put on screen; you pick one, run it, watch
the display, and flag it with a note if it looks wrong. Notes land in
e2e-review.mdwith a ready-made command to re-run just the flagged tests.pytest still does collection, fixtures and reporting, but the app decides what
runs and when — so a test can be replayed or gone back to, which a plain pytest
run cannot do. It starts its own windowed server so the terminal stays visible.
Reusable widgets in the client package (
vstimd.tui, behind atuiextra):StimulusList(the live scene),TriggerLines(VTL lines, withtto toggleand
pto pulse — a trigger-driven scene exercised with no DAQ attached) andServerStatus. The review app mounts all three, which is what proves theycompose; they are the beginnings of a terminal console for a rig you are on over
SSH.
Targets, renamed and pruned:
test-e2e-visible,test-e2e-interactive,test-e2e-null.test-integrationis gone — it printed "not implemented yet:integration tests with MockServer", no MockServer exists anywhere, and the null
renderer covers that ground as a real server without a display.
The server bug this turned up
About one full run in ten failed a single "mutate a property, query it straight
back" test — never the same test twice, never reproducible in isolation. The
cause is
end_deferredscheduling a flip unconditionally. The flip promotesevery copy slot over its live one, and outside deferred mode the copies are
stale by construction: ordinary writes go to
livealone. So a client thatswitches deferred mode off defensively, having never switched it on, armed a
flip that undid every write made in the ~16 ms before the next frame. The
animation path's
END_DEFERREDdid the same.Both now do nothing when nothing is staged, with a test pinning it, and the e2e
scene reset asks for
cancel— drop what is staged, schedule no flip — which isthe right request regardless of the server fix. Reproduced in three lines
against a live server before the fix, five consecutive full null runs clean
after it.
Fixes found along the way
letter_height_pxandbox_size_px;SHARED-09created one that way and was invisible.the review app opens. Position, box and letter height are fractions of the
real display now.
buried; it is rebuilt on each step, on top, with a dim backing box.
Windowis deferred: a queuedautoDraw = Falsewas replayingagainst a stimulus the next test's reset had deleted.
VSTIM_SERVER_ADDRin three places.Nothing reads that; it is
VSTIMD_SERVER.Verification
cargo testall green (49 incommands, including the new one),make test-e2e-null141 passed / 6 xfailed — five consecutive runs after the deferredfix, all clean —
make test79 passed,make typecheckclean. The fullon-screen suite was run for real on a display: 141 passed / 6 xfailed in 4:31.
The review app was driven headless through Textual's pilot harness — moving,
running, run-and-advance, flagging with a note, panel switching, report output.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Y8Epm1dhLes5amyezTRyTD