SWF v41 baseline: workflow episodes, fast processing v11, release notes

RELEASE_NOTES.md

@@ -1,5 +1,119 @@
 # Release Notes
 
+## v41 (2026-08-07)
+
+This release extends Snapper with site-level job histories and a campaign delivery view, introduces workflow
+episodes as a recorded unit across the stack, adds verified PanDA task pause and resume with paced bulk
+controls, restructures the TF slice record around its registered TF sample, and moves the heavy monitor pages
+to cached serving. The epicprod domain gains the delivered-data daily record and the campaign plan curation
+surface. These notes cover the coordinated baseline repositories and the relevant `main` work in their peer
+repositories.
+
+### Snapper: site job histories and campaign delivery (snapper-ai, swf-monitor)
+
+- Site-level job lifecycle histories: the PanDA component publishes cumulative terminal-outcome counters with
+  retrofill, and a Site focus renders per-site job curves in lifecycle order with application-standard state
+  colors. The site slice is a table with one row per curve; site facts drill down to the jobs pages, and
+  drilldowns are limited to persistent outcomes. The jobs-page graphics align with the selected Snapper
+  interval and use consistent semantic colors for failure classes.
+- Campaign delivery view: per-campaign tabs with delivery lens curves, a per-PC daily arrivals quilt, a
+  day-breakdown cut card covering the selected campaigns, and events and files as toggled quantities, served
+  from cached series. Delivery leaves account by progress rule, one leaf per configuration.
+- URL state: curve tick selection, provider-owned default-off curves, live-edge (`cut=now`) tracking URLs, and
+  sticky chosen cuts; display defaults never stamp the URL. Focus-sized series caching follows refreshes
+  promptly.
+- The numbered activity bars open run stories with machine-readable failed-file counts and application-standard
+  state fills; low-information curves are removed from the job and task plots.
+
+### Workflow episodes (swf-common-lib, snapper-ai, swf-monitor, swf-testbed)
+
+- The swf-common-lib episodes module is a generic episode-building engine: definitions specify event-fed
+  bounded records with start and end hooks; the builder adopts events, stamps arrivals, and reports each
+  participant once.
+- The testbed episode builder runs as a standing system service, capturing workflow episodes off the message
+  bus for all namespaces.
+- swf-monitor provides episode REST ingest and read endpoints backed by the snapper-ai episode store, and the
+  Snapper Time history adds an Episode view. An agentic workflow view over these records is planned, with plan
+  documents in swf-testbed and snapper-ai.
+
+### PanDA task operations and Capcom (swf-monitor, swf-epicprod)
+
+- Verified pause and resume for PanDA tasks: operator controls restricted to authorized users, executed and
+  verified against PanDA, with paced bulk operations rendered above the dynamic task table. An alarm-driven
+  automatic pause on failure-rate spikes is planned (swf-epicprod).
+- Capcom is an operations communication console: a page of state tiles showing current SWF operational state,
+  alongside a feed of operational notices. The console itself is currently external to SWF, implemented in a
+  separate operations system, and will probably be added to SWF. swf-monitor serves it: a state endpoint
+  provides tile-exact SWF state; alarm presence and the swf-bot dispatcher are state tiles; a nightly
+  campaign-delivery notice reports the day's arrivals; notices are buffered for consumer polling instead of
+  external push; the PanDA state includes the running task count and integer success percent.
+
+### Fast processing chain and TF slice restructure (swf-monitor, swf-testbed)
+
+- The fast processing pipeline v11 is the design of record: STF files slice directly and the separate FastMon
+  processing stage is retired; the TF sample persists as a registration record (FastMonFile) that parents the
+  slices.
+- The TFSlice record is keyed to its registered TF sample by foreign key and the flat tf_filename/stf_filename
+  columns are removed (monitor_app migrations 0005 and 0006). REST and MCP outputs are unchanged. The REST
+  create payload replaces the two filename fields with `fastmon_file` naming the registered sample; clients
+  that create slices must be updated accordingly. The example fast processing agent posts the new payload; its
+  worker queue messages are unchanged.
+- The fastmon-files list page 500 (unbound short_filename) is fixed.
+
+### PCS: delivered-data record, campaign plan, physics configuration (swf-epicprod, swf-monitor)
+
+- The campaign delivery daily record measures per-file events and accumulates per-PC arrivals over complete
+  days, backfilled from Rucio arrival history and produced nightly in the catalog_sync chain, bounded to
+  active and recorded campaigns. CAMPAIGN_DELIVERY.md documents the record and its views.
+- Campaign plan: a target-events curation surface (/pcs/plan/) with expected-events fields, a setter service,
+  and a REST endpoint; request-material target derivation; campaign pages link the campaign's delivery time
+  history.
+- The physics configuration is an entity (PhysicsConfig) and is documented as such. The physics page and
+  request composer serve cached products; an unknown tag type is a 404.
+- Catalog: forward-only rotation with server-side guards; the latest daily AI assessment appears inline on
+  current and producing campaign pages; assessment strips name their campaign and use human-addressed URLs.
+- EPICPROD_VALIDATION records the agreed production–validation loop. Assessment causal claims require evidence
+  and structured attribution.
+
+### Serving performance and deployment (swf-monitor)
+
+- The PanDA activity page, tasks list, compute-usage rollup, and log summary serve from cached products; the
+  tasks window cache renders cells at serve time. Typed REST filters reject malformed values with 400.
+  django-filter is enabled so filterset_fields declarations filter.
+- Deploys stage a unique release and never rebuild the current tree in place; the Apache maximum-requests
+  daemon recycling is removed.
+- The Mattermost bots keep a stable prompt-cache prefix and log usage per round.
+
+### Interface work (swf-monitor)
+
+- List pages: enumerated value walls are removed as filters; long names render middle-elided with the full
+  name on hover; the TF slices page is reorganized; latching table headers on all list pages with adaptive
+  two-axis scrolling for wide tables; visible checkbox borders application-wide.
+- The account page shows testbed activity lanes on the My Workflows tab; the testbed home places Snapper
+  namespace lanes above Workflows; hub routes are authoritative for navigation mode; robots.txt is served
+  outside the OIDC gate and disallows all crawling.
+
+### External access (swf-remote, site-canary)
+
+- swf-remote: crawlers are denied the whole proxied surface at the proxy choke point; a live-data access
+  contract with proxy identities; the aggregate System page is proxied; GitHub sign-in; a rolling session
+  window.
+- site-canary: queue assessments state their evaluation window, keep status evidence-current, and link to
+  Snapper and source documentation; Canary queues publish into Snapper site state with updated state colors.
+
+### Agents and CI (swf-testbed)
+
+- data_agent supports uploading to deterministic RSEs; prompt_processing adds a configurable mode for real
+  file processing; the integration test runs several workflows.
+- Run lifecycle: executions and announced runs terminalize on mid-run stop; status checks probe TCP directly
+  instead of shelling out; the testbed image installs site-canary before the meta-package.
+
+### Acknowledgments
+
+- Wen Guan: the TFSlice restructure (swf-monitor PR #45) and the fast processing workflow it serves.
+- Dmitry Kalinkin: data agent deterministic-RSE uploads, prompt processing real-file mode, and the
+  multi-workflow integration test (swf-testbed PRs #68, #66, #63).
+
 ## v40 (2026-07-25)
 
 This release adds Snapper, which records subsystem-published operational state as a queryable history, with a

agents.supervisord.conf

@@ -103,3 +103,4 @@ stopwaitsecs=10
 stopsignal=QUIT
 stdout_logfile=%(here)s/logs/%(program_name)s.log
 stderr_logfile=%(here)s/logs/%(program_name)s.log
+

agents/episode_builder_agent.py

@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+"""Episode builder agent: records workflow episodes from bus traffic.
+
+One more listener on the epictopic, in keeping with the open-listening
+messaging philosophy: it consumes every message, routes executions to
+the armed episode definitions (the episodes package), and drives each
+episode through open, append, completion, and close against the
+monitor's episode ingest REST (docs/agentic-workflow-view.md).
+
+Episodes are scope-wide, so unlike workflow agents this agent applies
+no namespace filtering: every namespace's executions are recorded.
+"""
+
+import json
+import logging
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from swf_common_lib.base_agent import BaseAgent
+from swf_common_lib.episodes import EpisodeBuilder, MonitorEpisodeIngest
+
+from episodes import ALL_DEFINITIONS
+
+
+class EpisodeBuilderAgent(BaseAgent):
+    def __init__(self):
+        super().__init__(agent_type='EPISODE_BUILDER',
+                         subscription_queue='/topic/epictopic')
+        # The builder identity is stable across restarts and instances:
+        # a restarted builder must resume the episodes its predecessor
+        # opened. The per-process agent name would strand them.
+        ingest = MonitorEpisodeIngest(
+            base_url=self.base_url,
+            token=self.api_token,
+            builder_identity='episode-builder',
+        )
+        self.builder = EpisodeBuilder(
+            [definition() for definition in ALL_DEFINITIONS], ingest)
+        logging.info('armed definitions: %s',
+                     [d.workflow_name for d in self.builder.definitions])
+        adopted = self.builder.adopt_open_episodes()
+        if adopted:
+            logging.info('adopted %d open episode(s) from a previous '
+                         'builder instance', adopted)
+
+    def on_message(self, frame):
+        try:
+            message = json.loads(frame.body)
+        except (ValueError, TypeError) as exc:
+            logging.error('unparseable message dropped: %s', exc)
+            return
+        self.builder.handle_message(message)
+
+    def send_heartbeat(self):
+        result = super().send_heartbeat()
+        # The heartbeat cycle is the completion-poll cadence.
+        self.builder.tick()
+        return result
+
+
+if __name__ == '__main__':
+    EpisodeBuilderAgent().run()

docs/agentic-workflow-view.md

@@ -0,0 +1,124 @@
+# Agentic workflow view
+
+A temporal view of one workflow execution: the agents that ran it, the
+workers it provisioned, the messages that passed between them, and the
+data that flowed through them — rendered as a live display while the
+workflow runs and as a replay of any past execution. The view is a
+Snapper surface; the engine mechanisms it requires are specified in
+snapper-ai `docs/EPISODES.md`.
+
+## Display
+
+The plot is a stack of horizontal lanes over a time axis, in the
+established Snapper Time history vocabulary (lanes, tiles, cut,
+floater, step arrows).
+
+- One lane per participant. The workflow runner and the agents open
+  the stack; worker lanes appear below as workers come into
+  existence. A prompt-processing execution opens with a handful of
+  lanes and fans out to tens of worker lanes, then converges as jobs
+  finish and the run tears down.
+- Lanes are dynamic: a lane begins when its participant registers and
+  ends when it exits. A finished participant's lane keeps its
+  vertical slot, dimmed, to the end of the episode, so the fanout and
+  convergence render as a stable shape and the vertical layout never
+  reshuffles during a replay.
+- Messages render as marks at their send time on the sender's lane,
+  with connectors to recorded consumers where consumption records
+  exist (see Data contract below).
+- Activity on a lane renders as tiles: an agent's processing spans, a
+  worker's created / started / finished phases.
+- A click on any element opens its detail card: message payload,
+  agent record, PanDA job record, file record.
+- The cut and step arrows carry their Time history meanings: a click
+  is a time slice, the arrows step the window. An execution-stepping
+  mode — arrows move between executions rather than time windows —
+  is a candidate addition.
+- Live mode: while the execution runs, the view follows it, either
+  pseudo-realtime as events land or promptly after completion. Both
+  are feasible with current record latencies (seconds for messages
+  and file records, minutes for PanDA job state).
+
+## Evidence audit
+
+Audited 2026-08-02 against prompt-processing execution
+`prompt_processing-zyang2-0845` (run 102827, 15 STFs, decision-box
+broadcast to E1_BNL and E1_JLAB) and the stf_datataking executions of
+the same day. The full event sequence is reconstructable from
+existing records, all in the system database with sub-second
+timestamps:
+
+| time (UTC) | event | record |
+|---|---|---|
+| 20:00:13 | execution starts | WorkflowExecution |
+| 20:00:18 | run_imminent | message log |
+| 20:00:26 | start_run | message log |
+| 20:00:27–20:01:02 | stf_gen ×15, pause/resume around standby | message log |
+| 20:00:29.6/.7 | stf_ready per site: the decision-box fanout | message log |
+| 20:00:34.6 | PanDA tasks created, one per site | PanDA tasks |
+| 20:05:55 | 15+15 jobs created | PanDA jobs |
+| 20:11:22–43 | jobs start | PanDA jobs |
+| 20:13–20:14:43 | jobs finish, tasks done | PanDA jobs, tasks |
+
+## Data contract
+
+Sources, all local database reads (no remote calls in the render
+path):
+
+- **WorkflowExecution** — episode identity, start and end, full
+  parameter set including agent roster and workflow configuration.
+- **WorkflowMessage** — sender agent, type, namespace, execution id,
+  run id, payload, sent-at to the microsecond.
+- **SystemAgent** — lane birth (`created_at` at registration), lane
+  death (`operational_state` EXITED, stamped by `updated_at`),
+  heartbeats, pid, hostname.
+- **STF files / TF slices** — file-level flow; TF slice records name
+  their `assigned_worker`.
+- **PanDA tasks** — the run number is embedded in the task name
+  (`user.<user>.swf.<run>.processed.<site>/`), giving a direct join
+  from execution to tasks; creation, start, and end times per task.
+- **PanDA jobs** — one record per worker with creation, start, and
+  end times, site, and output metadata; jobs join to tasks by
+  `jeditaskid`. For these workflows the worker lanes are the PanDA
+  jobs born within the execution's tasks.
+
+## Gaps to fill
+
+Verified in code at the executing commits, 2026-08-02:
+
+1. **Consumption records.** Messages record their sender only. The
+   messaging philosophy is open listening — any agent may subscribe —
+   so an addressee list recorded at send time would misstate the
+   model. Instead, consumers record consumption: an agent that receives a
+   message and acts on it records that fact with message id and
+   timestamp. This yields the message connectors for the view and,
+   independently, a workflow-integrity tool: a message no agent
+   consumed, or a message consumed by an unexpected agent, becomes a
+   detectable condition. Requires team discussion before
+   implementation.
+2. **Processing agent announcements.** The prompt-processing agent
+   emits only heartbeats; its PanDA task submissions are visible only
+   through the task records appearing. It should announce submission
+   events on the bus like its peers announce theirs.
+3. **Explicit agent exit stamp.** Lane death is currently inferred
+   from the EXITED transition's `updated_at`; an explicit exit
+   timestamp would remove the approximation.
+4. **Durable episode capture.** Messages and file records are
+   operational logs with retention policies. An episode is captured
+   into a durable Snapper record at (or promptly after) execution
+   end, so replay never depends on raw log retention.
+
+## Delivery phases
+
+1. **Episode capture** — a builder that joins the sources above into
+   a durable episode record for each execution, live or promptly
+   after completion.
+2. **Replay view** — the lane display for any captured episode.
+3. **Live mode** — the view follows a running execution.
+4. **Production scale** — the same mechanism applied to epicprod
+   workflows: campaign tasks fanning out across grid sites, with
+   PanDA jobs as worker lanes. The mechanism is generic; only the
+   provider data differs.
+
+Related documentation: `fast-processing-workflow.md`,
+`e0-e1-state-machine.md`, snapper-ai `docs/EPISODES.md`.

docs/architecture_and_design_choices.md

@@ -80,6 +80,17 @@ contributors and for future architectural reviews.
   `dedup_key` to avoid the duplicate-work race concurrency introduces; a send
   lock makes worker-thread sends safe and shutdown drains the pool.
 
+- **Concurrency is separate from responsiveness:** the bounded pool uses
+  `SWF_AGENT_MAX_WORKERS` and defaults to 4, so calls with different
+  `dedup_key`s may execute at the same time. Deduplication is not serialization.
+  The safe first migration of an existing serial handler is
+  `SWF_AGENT_MAX_WORKERS=1`: work leaves the receiver thread, but doers still run
+  one at a time. Use more workers only after per-message state has moved out of
+  shared `self.*` fields and every shared library, client, temporary path, and
+  process-global operation has been audited. A non-thread-safe subsection may
+  instead use a dedicated lock acquired inside the background doer; never wait
+  for that lock in `on_message`.
+
 - **API and consumers:** the API is documented in the `swf-common-lib` README;
   the first consumer is the epicprod ops agent
   (`swf-epicprod/docs/EPICPROD_OPS_AGENT.md`).