design: how parameters change over time — generic property animations, variables, or scripts

[joschaschmiedt]

The concrete trigger

There is no way to animate the outline width of a shape. Nor its fill colour, nor a
circle's radius, nor a grating's contrast, nor — now that it is a shared property —
its opacity.

The reason is structural. Every animation kind hardcodes the property it writes:

kind	writes
FlashForNFrames, FlickerForNFrames, CoupleVisibilityToTriggerLine, EnableOnTriggerEdge	flags.enabled
MoveAlongPath2D, MoveAlongSegments2D, ExternalPosition2D	transform.pos

Seven kinds, two properties. Adding "ramp the outline width" means a proto message, a
Rust variant, a serde tag, a Python method, a TypeScript method and tests — and then
the same again for the opacity fade, the contrast sweep, the looming radius, the
colour ramp. That is a million animations, and most of them differ only in which
field they assign.

The root cause

Which property and what behaviour over time are conflated into one enum. Separate
them and the count collapses: a handful of behaviours × any property.

behaviour  ×  property        →  today: one kind per pair
ramp          outline_width
waveform      opacity
table         contrast
external      radius
script        pos

So the design question is really two questions, and the first one is unavoidable
whichever way the second is answered.

Layer 1 — how is a property named?

Every option below needs a way to address "the outline width of stimulus 7". The
choices:

a. A schema enum. ScalarProperty { OPACITY, OUTLINE_WIDTH, CIRCLE_RADIUS, GRATING_CONTRAST, … }. Typed, discoverable, completable in clients, and per-type
validity is checkable at create time with the mechanism that already exists —
err_wrong_type / ERROR_CODE_WRONG_STIMULUS_TYPE, which is how SetGratingSf
rejects a rect. Costs: a closed list, extended by hand, and it cannot address a
custom shader uniform.

b. A path string. "appearance.outline_width". Open-ended, reaches shader
uniforms, no schema churn. Costs: typos become runtime errors, clients lose
completion, and the config format gains stringly-typed keys.

c. A named variable with a binding (the MWorks model, below). Properties are not
addressed by the animation at all; they are bound to a variable, and the animation
drives the variable.

A mix is plausible: an enum arm for built-ins and a string arm for shader uniforms,
in one oneof — the string path being genuinely unavoidable there, since only the
shader itself knows its uniform names.

Layer 2 — how does a value change over time?

Four mechanisms cover everything currently on file, if the property is a parameter
rather than part of the kind:

1. Ramp / waveform — from, to, duration, easing; or sine/square at a frequency.
2. Table playback — one value per frame, uploaded ahead of time. This already
   exists: MoveAlongPath2D { coords } is a precomputed table. The client can
   evaluate arbitrary Python and upload the result.
3. External — read the value each frame from shared memory or a device (feat(server/scene): device-driven animations — implement ExternalPosition2D, add DeviceDrivenTransform #79,
   feat(proto,client): input-device animations over the wire + Python shm writer #80; ExternalPosition2D is the unimplemented instance, ExternalPosition2D is a no-op stub: the server accepts it, never reads shm, and the stimulus never moves #84).
4. Script — evaluate user code per frame.

Precomputable vs closed-loop — the line that matters

Mechanisms 1 and 2 are precomputable: the value at frame N does not depend on
anything the server learns at run time. They can be evaluated off-device, uploaded,
and played back with no logic on the render thread at all.

Mechanisms 3 and 4 are closed-loop: the value depends on live state — a VTL line, a
treadmill, the previous frame's value. Only these need on-device evaluation, and only
these justify paying the price of running code next to the render thread.

Worth stating plainly because it bounds the scripting problem: most "I need a custom
animation" requests are precomputable, and for those, an arbitrary Python function
in the client plus a table upload is already the most flexible and the safest answer
available. The stimulus server should make that path good — arbitrary property, not
just position — before it makes the risky path good.

Could an animation simply be a script?

Yes, and it is attractive: Animation::Script { … } keeps everything the animation
system already provides — arming, waiting on a trigger edge, cancelling, REARM /
RESTART, pulsing a line on completion, DONE_LEVEL — and replaces only the fixed
per-frame behaviour with user code. One kind instead of a million.

The constraints it has to satisfy are the hard part, and they are the same ones
#109 (scriptable movie stimulus) is already weighing — language choice, sandboxing,
keeping a GC off the critical path. What this issue adds is where the script runs:

• advance_one executes on the render thread, once per frame, under the write
  lock. The rule for that thread is: never block, never heap-allocate. A tick that
  allocates, GCs, or runs unbounded turns a dropped frame into the failure mode this
  whole server exists to prevent.
• So a script animation needs: compilation to bytecode at create time (not per
  frame), a per-tick fuel/instruction budget, and a defined behaviour on budget
  exhaustion — kill the animation and log it, never stall the frame.
• Determinism matters for replay (dev/EVENT_LOGGING.md §11): a script that reads
  the clock or an unseeded RNG is not replayable. Either forbid both, or log the
  values it read.
• A script that only computes a curve is mechanism 2 with extra steps and extra
  risk. The case for evaluating on-device is closed loop.

The MWorks precedent — variables

Worth studying before committing, because it answers this question a different way
and it is the system our users are most likely to have come from.

In MWorks an experiment declares named variables, and stimulus parameters are
expressions over them rather than fixed values — a stimulus is declared with
x_size = stim_size, and when stim_size changes the stimulus follows. Variables are
assigned by the protocol's state system, by I/O devices, and from Python. Every
variable change is timestamped into the event file, so the same mechanism that drives
the display is the experimental record.

What that model buys, mapped onto vstimd:

• One addressing scheme for everything. Layer 1 is solved once, and the same
  names serve animations, scripts, external devices and logging.
• Scripts stop being a special case. "Modify parameters with a script" becomes
  "assign a variable", with no new API surface per property.
• It subsumes the external-drive mechanism. feat(server/scene): device-driven animations — implement ExternalPosition2D, add DeviceDrivenTransform #79 / feat(proto,client): input-device animations over the wire + Python shm writer #80 become "a device writes a
  variable", not a bespoke animation kind.
• It lines up with event logging. dev/EVENT_LOGGING.md wants a replayable
  record; variable assignments are exactly the right granularity for one.

What it costs here:

• A second, dynamically-typed namespace beside a protobuf API that is currently
  statically typed end to end — the trade discussed at length on API consistency pass before 0.1: full sizes, clear commands, shared opacity, honest queries, reproducible timing #118.
• An expression evaluator, if bindings are expressions rather than plain references,
  and expression evaluation on the render thread has the same real-time constraints
  as scripting.
• Config format: variables and bindings become part of the saved scene, and part of
  what config load has to restore consistently.
• A resolution order when both a variable binding and a direct SetOutlineWidth
  command write the same property. MWorks does not have this problem because there is
  no direct-write path; vstimd does, and it is the interesting failure mode.

Constraints any answer must satisfy

• Writes go through the existing setters, never straight to fields. cmd_set_outline_width
  routes to the deferred copy or live slot, and marks the stimulus dirty because
  stroke geometry depends on the width. The dirty rule is not uniform: shapes bake
  opacity into vertex colours and must re-tessellate; grating and text read it into
  push constants and must not (see the fix in API consistency pass before 0.1: full sizes, clear commands, shared opacity, honest queries, reproducible timing #118). A generic writer that assigns
  fields directly gets one of those two cases wrong.
• Deferred mode must keep working: a batch of property changes still has to flip
  on one frame.
• Clamping and validation live in the setters; a second write path must not skip
  them.
• Config round-trip: whatever is added is saved, loaded and re-serialized
  unchanged (demo_files_survive_a_reserialize_unchanged).
• Per-type validity is a real check, not a formality: contrast on a rect,
  radius on a text stimulus, a uniform name on a stimulus with no shader.

Relationship to existing issues

• feat: scriptable movie stimulus — an on-device script (DSL/Lua) driving image presentation and triggers #109 — scriptable movie stimulus. Same execution-environment question; that
  issue owns the language choice and the sandbox. If a script can be an animation,
  the two want one runtime, not two.
• feat(server/scene): device-driven animations — implement ExternalPosition2D, add DeviceDrivenTransform #79 / feat(proto,client): input-device animations over the wire + Python shm writer #80 / ExternalPosition2D is a no-op stub: the server accepts it, never reads shm, and the stimulus never moves #84 — device-driven animations, ExternalPosition2D. Mechanism 3
  above, currently position-only; a named property would generalise it.
• feat(animations): AnimationSequence — owning Animation variant for gapless sequences #49 — AnimationSequence. Composition of behaviours, orthogonal to which
  property they write, and cheaper to build once behaviours are generic.
• feat: condition system for switching active stimulus/animation groups #28 — condition system for switching stimulus/animation groups. Overlaps the
  variable question: conditions are state the scene branches on.
• feat(server/scene): make the camera an animation target; add 3-D navigation animations #74 — camera as an animation target. The target axis, as opposed to this
  issue's property axis.

Open questions

1. Do we settle Layer 1 (property addressing) on its own, before choosing between
   ramps, scripts and variables? It is required by all three, and it is the piece
   that ends up in the config format and therefore the hardest to change later.
2. Enum, path string, or variable name — or an enum with a string escape hatch for
   shader uniforms?
3. Is a variable/binding layer the right destination, or is it a bigger conceptual
   surface than a stimulus server needs, given the client is already a real
   programming environment and can precompute?
4. If both variables and direct commands can write a property, what wins?
5. Does a script animation share a runtime with feat: scriptable movie stimulus — an on-device script (DSL/Lua) driving image presentation and triggers #109, and does that runtime run on
   the render thread with a fuel budget, or one frame ahead on another thread?
6. What is the smallest useful step? A plausible answer: a generic
   RampScalar { property, from, to, duration_frames, easing } plus generalising
   table playback to any scalar property — both precomputable, both usable
   immediately, and neither foreclosing variables or scripts later.

[joschaschmiedt]

Direction

Settled, from discussion: a range of basic animations, plus frame-by-frame parameter
lists, plus scripting for what nobody anticipated. Not one mechanism — three, with a
clear division of labour:

	for	reproducible?
basic animations (ramp/waveform + easing)	the 80% that is "move this smoothly from A to B"	by construction
frame-by-frame parameter lists	anything the client can precompute — arbitrary Python, offline	by construction
script	closed-loop behaviour that depends on live state	only under the input rule below
pregenerated image movie (#108/#109)	stimuli that are not parametric at all	by construction

The rationale for scripting is extensibility, not expressiveness: we cannot enumerate
what researchers will want. The same reasoning is why the movie stimulus exists — an
escape hatch at the content level, where the script is the escape hatch at the
behaviour level.

Hard requirement: reproducible output

A saved config, replayed against the same recorded input-VTL trace, must produce the
same frames. This is what makes a config a record of what the animal saw rather than
an approximation of it, and it bounds every option in this issue.

The rule it implies: the only inputs an animation may read are the config, the frame
index, and recorded input state. Wall-clock time, measured frame rate, unseeded
randomness and un-recorded external data are not reproducible inputs.

We violate this today in two places — measured fps inside per-frame animation math —
filed as #120. Worth fixing before building on top of it.

Consequences for the three mechanisms:

• Basic animations and frame tables satisfy it for free, provided durations are
  counted in frames, not seconds. Seconds stay a client-side convenience (the
  clients already convert at create time).
• Scripts satisfy it if sandboxed to (frame_index, VTL input state, own prior state) and denied the clock, the measured fps and unseeded RNG. That is a sharp,
  checkable rule, and it makes scripting compatible with reproducibility rather than
  in tension with it — a seeded RNG with the seed in the config is fine.
• External/device-driven (feat(server/scene): device-driven animations — implement ExternalPosition2D, add DeviceDrivenTransform #79/feat(proto,client): input-device animations over the wire + Python shm writer #80/ExternalPosition2D is a no-op stub: the server accepts it, never reads shm, and the stimulus never moves #84) satisfies it only if the incoming stream is
  recorded into the event log alongside the VTL trace. Otherwise a replay has nothing
  to replay.

Prior art: Godot and Unity

Both engines solved the "million animations" problem the same way, and the convergence
is the useful signal.

A track is (target, property path, interpolation, keyframes). Godot's
AnimationPlayer addresses "Sprite2D:modulate:a" — node, property, sub-property.
Unity's AnimationClip binds curves to a component type plus a serialized property
path. Neither has a FadeAnimation and a MoveAnimation as separate types: the
property is data, and the animation is one mechanism. This is Layer 1 in the issue
body, and it is what both engines settled on independently.

A small, standard easing vocabulary goes a long way. Godot exposes transition ×
ease (linear, sine, quad, cubic, expo, elastic, back, bounce × in/out/in-out); Unity
uses keyframe tangents to the same effect. A handful of enum values covers nearly
every "basic animation" request, which is what makes the range of basic animations
finite and enumerable.

Discrete vs continuous tracks. Godot distinguishes value tracks that interpolate
from ones that step. Our existing kinds split exactly along that line — visibility is
discrete, position is continuous — and a generic mechanism needs the distinction
explicitly, not implied by the property.

Method/event tracks. Godot animations can call a method at a keyframe. That is
precisely what our action masks do, but pinned to start and end only: our
FINAL_ACTION_TRIGGER_LINE is a one-keyframe method track. Generalising it — a track
of VTL pulses at arbitrary frames of an animation — is directly useful for a stimulus
server, and it is what #109 wants for movies. Worth taking.

Sequencing (Unity Timeline, Godot animation tracks that play other animations) is
#49.

What deliberately does not transfer: blend trees, weighted state machines, skeletal
retargeting, and time-based (rather than frame-based) evaluation. Game engines
interpolate against a variable-length frame time because dropping a frame must not
change the outcome; for us a frame is the unit, and the whole point is that frame N
is exactly reproducible. That is the one place we should not copy them.

Revised open questions

1. Property addressing (still the first thing to settle, and now with two engines
   voting for a path/property model over an enum-per-kind).
2. Discrete vs continuous, and where that is declared — on the property, or on the
   track?
3. Do we adopt an event/method track for VTL pulses at arbitrary frames, replacing the
   start/final/cancel action bits with something general — or keep the bits and add
   tracks alongside?
4. Script sandbox: is the (frame, VTL, own state) input rule enforceable in the
   runtime feat: scriptable movie stimulus — an on-device script (DSL/Lua) driving image presentation and triggers #109 picks?
5. Does the frame-table mechanism reuse the movie stimulus' upload path (feat: image stimulus — load an image, position, rotate, set opacity #108), given
   both are "a big array uploaded once and played back by frame index"?