Skip to content

Commit 774f701

Browse files
author
t
committed
docs: accuracy fixes + clear-writing pass on README and docs/
- Correct overclaims surfaced while writing the site: holotype is a (specialized) backup, not 'not a backup'; deposits are automatic after setup; installing as a skill is optional, not required. - Fix stale principle 6 to reflect consent-driven auto-push (v2.4.0). - Add 'deliberately overkill', 'built Claude Code first', set-and-forget install model, and non-programmer-audience framing. - Remove all em dashes across README and docs/ (clear-writing pass).
1 parent bfa46ed commit 774f701

6 files changed

Lines changed: 97 additions & 83 deletions

File tree

README.md

Lines changed: 42 additions & 28 deletions
Large diffs are not rendered by default.

docs/ADDING_A_SOURCE.md

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
You are reading this because you are an LLM invoked through an agent CLI that holotype does not yet recognize. Your CLI stores session transcripts somewhere, in some format, and the user wants you to archive them with holotype's forensic guarantees alongside their other sessions.
44

5-
This document is your contract. If you follow it, the LLM who reads the archive in 5 years can verify, search, and cite your transcripts the same way they verify, search, and cite Claude Code or Codex transcripts. The rest of holotype does not need to change to accommodate your CLI only one new file in `holotype/sources/` does.
5+
This document is your contract. If you follow it, the LLM who reads the archive in 5 years can verify, search, and cite your transcripts the same way they verify, search, and cite Claude Code or Codex transcripts. The rest of holotype does not need to change to accommodate your CLI: only one new file in `holotype/sources/` does.
66

77
## The contract you're implementing
88

@@ -23,7 +23,7 @@ class MyCliSource(Source):
2323
def discover(source_root: Path) -> Iterator[DepositCandidate]:
2424
# Yield one DepositCandidate per transcript file under source_root.
2525
# MUST refuse to traverse outside source_root (e.g. symlinks
26-
# that resolve elsewhere see CodexSource.discover for the
26+
# that resolve elsewhere, see CodexSource.discover for the
2727
# pattern).
2828
...
2929

@@ -50,28 +50,28 @@ Don't guess from documentation. Inspect actual files on the user's machine. The
5050

5151
1. **Find where the CLI stores transcripts.** Look in `~/.<cli-name>/`, `~/Library/Application Support/<CLI>/`, `~/.config/<cli-name>/`. Look for files whose names include UUIDs, timestamps, "session", "conversation", "thread", "rollout". Run `ls -la` and read what's there. Confirm with the user before assuming.
5252

53-
2. **Categorize the sensitivity of nearby files.** If `auth.json`, `credentials`, `*.key`, `*.token`, or `config.toml` with API keys live near the transcripts, your `default_source_paths()` MUST point at the deepest dir that contains only transcripts. holotype enforces no-walk-up at the ingest level make that boundary explicit.
53+
2. **Categorize the sensitivity of nearby files.** If `auth.json`, `credentials`, `*.key`, `*.token`, or `config.toml` with API keys live near the transcripts, your `default_source_paths()` MUST point at the deepest dir that contains only transcripts. holotype enforces no-walk-up at the ingest level: make that boundary explicit.
5454

5555
3. **Pick the largest transcript file you can find** (`find <dir> -name '*.jsonl' -o -name '*.json' | xargs wc -l | sort -rn | head -5`) and read its first ~20 lines. Most agent CLIs use JSONL where each line is one event, but some use a single JSON document with a `messages` array, and a few use SQLite. Figure out which.
5656

5757
4. **Map your format's roles to the `MessageInfo` fields:**
58-
- `role` `"user"` / `"assistant"` / `"tool_result"` / etc. Whatever your CLI calls these.
59-
- `timestamp` ISO-8601 string preferred. If your CLI uses Unix epoch, convert.
60-
- `model`the model ID for assistant turns. None for user turns. Don't put the *provider* name here (e.g. "openai", "anthropic") those aren't model IDs and they pollute `manifest.models`.
61-
- `has_tool_use` True if this turn contains a tool call.
62-
- `has_thinking` True if this turn contains a reasoning / thinking block.
63-
- `fts_content` flat text of EVERYTHING in the turn that's searchable. **Do not filter for "noise".** Include tool inputs, tool outputs, thinking blocks, system reminders. Holotype's pitch is forensic completeness; a search index that drops content violates it.
64-
- `flags`set of source-specific markers. `{"compaction"}` if this line is a context-compaction summary. `{"header"}` if it's a header / control / telemetry pseudo-record that shouldn't count toward `message_count` (the timestamps on header-flagged lines still update `first_timestamp`/`last_timestamp` and `session_metadata`/`usage` on them are still harvested header just means "not a user-visible message").
65-
- `usage` *Optional, manifest v4.* When the host CLI exposes a per-turn token usage block (Claude Code's `message.usage`, Codex's `event_msg.token_count`), put the raw dict here. The manifest scanner aggregates `input_tokens`, `output_tokens`, `cache_creation_input_tokens`, `cache_read_input_tokens` / `cached_input_tokens` automatically. Leave `None` if your CLI doesn't report usage.
66-
- `token_count_kind` *Optional, manifest v4.* `"cumulative"` if the `usage` dict is a running total (Codex's `total_token_usage`), `"delta"` if it's per-turn billing (Claude Code's per-message `usage`), or `None` if no usage was attached. The aggregator prefers a final cumulative reading over summed deltas.
67-
- `session_metadata`*Optional, manifest v4.* For a header-flagged line that carries session-start metadata (Codex's `session_meta` carries the git block at session-start), put a dict here. Recognized keys: `cwd`, `cli_version`, `originator`, `model_provider`, `git_state: {commit, commit_short, branch, dirty, remote, captured_at}`. When `git_state.captured_at == "session-start"`, the manifest will prefer your header-derived state over the deposit-time probe more accurate, because it was captured by the host CLI at the moment the session ran.
58+
- `role`: `"user"` / `"assistant"` / `"tool_result"` / etc. Whatever your CLI calls these.
59+
- `timestamp`: ISO-8601 string preferred. If your CLI uses Unix epoch, convert.
60+
- `model`: the model ID for assistant turns. None for user turns. Don't put the *provider* name here (e.g. "openai", "anthropic"): those aren't model IDs and they pollute `manifest.models`.
61+
- `has_tool_use`: True if this turn contains a tool call.
62+
- `has_thinking`: True if this turn contains a reasoning / thinking block.
63+
- `fts_content`: flat text of EVERYTHING in the turn that's searchable. **Do not filter for "noise".** Include tool inputs, tool outputs, thinking blocks, system reminders. Holotype's pitch is forensic completeness; a search index that drops content violates it.
64+
- `flags`: set of source-specific markers. `{"compaction"}` if this line is a context-compaction summary. `{"header"}` if it's a header / control / telemetry pseudo-record that shouldn't count toward `message_count` (the timestamps on header-flagged lines still update `first_timestamp`/`last_timestamp` and `session_metadata`/`usage` on them are still harvested: header just means "not a user-visible message").
65+
- `usage`: *Optional, manifest v4.* When the host CLI exposes a per-turn token usage block (Claude Code's `message.usage`, Codex's `event_msg.token_count`), put the raw dict here. The manifest scanner aggregates `input_tokens`, `output_tokens`, `cache_creation_input_tokens`, `cache_read_input_tokens` / `cached_input_tokens` automatically. Leave `None` if your CLI doesn't report usage.
66+
- `token_count_kind`: *Optional, manifest v4.* `"cumulative"` if the `usage` dict is a running total (Codex's `total_token_usage`), `"delta"` if it's per-turn billing (Claude Code's per-message `usage`), or `None` if no usage was attached. The aggregator prefers a final cumulative reading over summed deltas.
67+
- `session_metadata`: *Optional, manifest v4.* For a header-flagged line that carries session-start metadata (Codex's `session_meta` carries the git block at session-start), put a dict here. Recognized keys: `cwd`, `cli_version`, `originator`, `model_provider`, `git_state: {commit, commit_short, branch, dirty, remote, captured_at}`. When `git_state.captured_at == "session-start"`, the manifest will prefer your header-derived state over the deposit-time probe: more accurate, because it was captured by the host CLI at the moment the session ran.
6868

6969
5. **Decide your archive layout** in `DepositCandidate.archive_subpath`. The convention is `<source-name>/<project-or-date>/<session-id>/`. Examples:
7070
- `claude-code/<project-dir>/<session-uuid>/`
7171
- `claude-code/<project-dir>/<parent-uuid>/subagents/<sub-id>/` (nested)
7272
- `codex/<YYYY>/<MM>/<DD>/<session-uuid>/` (date-partitioned)
7373

74-
Whatever's natural for your CLI's storage. The shape doesn't matter for correctness `holotype/archive.py`'s `iter_all_sessions` walks arbitrary depth but a layout that mirrors how the user thinks about their sessions makes manual inspection easier.
74+
Whatever's natural for your CLI's storage. The shape doesn't matter for correctness (`holotype/archive.py`'s `iter_all_sessions` walks arbitrary depth) but a layout that mirrors how the user thinks about their sessions makes manual inspection easier.
7575

7676
## How to verify your implementation works
7777

@@ -93,8 +93,8 @@ Before declaring done:
9393

9494
## What NOT to do
9595

96-
- **Do not** filter content for "noise" in `parse_line`. Tool calls, system reminders, hook outputs all of it. The whole point of holotype is preserving what other tools throw away.
97-
- **Do not** modify transcripts in place. Read-only with stable-mtime checks (see `read_with_stable_check` in `scripts/ingest.py` for the pattern). If your CLI's transcripts can be modified between reads, your source is automatically incompatible with holotype's hash-chain flag it to the user and stop.
96+
- **Do not** filter content for "noise" in `parse_line`. Tool calls, system reminders, hook outputs, all of it. The whole point of holotype is preserving what other tools throw away.
97+
- **Do not** modify transcripts in place. Read-only with stable-mtime checks (see `read_with_stable_check` in `scripts/ingest.py` for the pattern). If your CLI's transcripts can be modified between reads, your source is automatically incompatible with holotype's hash-chain: flag it to the user and stop.
9898
- **Do not** invent the on-disk schema. If your CLI doesn't document its transcript format and you can't infer it confidently from a real file, **defer**. A wrong source class will deposit garbage that passes hash checks but is unparseable to a reviewer 5 years from now.
9999
- **Do not** widen `default_source_paths` to include directories with credentials. Codex's `~/.codex/` contains `auth.json` next to `sessions/`; we point at `sessions/` specifically. Do the analogous thing for your CLI.
100100
- **Do not** push the archive to a remote without explicit user consent on this session, *every time*. The `--no-auto-push` discipline applies regardless of how the Source was added.

docs/LINUX_SYSTEMD.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Linux: background ingest via user systemd
22

3-
The macOS launchd background tick (`scripts/install-launchd.py`) catches long-running agent-CLI sessions whose `Stop` hook never fires. Linux users get the same behavior via a user-level systemd timer + service unit. This document is a reference template not yet installed by a script.
3+
The macOS launchd background tick (`scripts/install-launchd.py`) catches long-running agent-CLI sessions whose `Stop` hook never fires. Linux users get the same behavior via a user-level systemd timer + service unit. This document is a reference template, not yet installed by a script.
44

55
## Why a user-level service
66

@@ -21,7 +21,7 @@ Type=oneshot
2121
# (e.g. /home/you/Git/holotype) and <ARCHIVE> with the absolute path
2222
# returned by `cat ~/.config/holotype/archive-path`.
2323
ExecStart=/usr/bin/env python3 <HOLOTYPE_REPO>/scripts/ingest.py --archive <ARCHIVE> --quiet
24-
# Hold standard limits the ingest is mostly I/O bound and should
24+
# Hold standard limits: the ingest is mostly I/O bound and should
2525
# never exceed normal user resource caps.
2626
Nice=10
2727
IOSchedulingClass=best-effort
@@ -87,7 +87,7 @@ systemctl --user daemon-reload
8787
- **User must be logged in for the timer to fire** unless lingering is enabled (`loginctl enable-linger <user>`). For headless servers or auto-suspending laptops, enable lingering or accept that the timer pauses when you're logged out.
8888
- **Concurrency safety is handled inside ingest.py** via `flock` at `<archive>/.holotype/.lock`. If you accidentally run two ingest cycles back-to-back, the second exits cleanly with "another ingest is running, exiting."
8989
- **GPG-signed commits** (`--sign-commits`) need `GPG_TTY` available. For a systemd-driven invocation, that usually means using `gpg-agent` with `allow-loopback-pinentry` and an unlocked key. Test interactively first.
90-
- This template doesn't push to a remote same as the macOS launchd tick. Pushing is always an explicit user action.
90+
- This template doesn't push to a remote, same as the macOS launchd tick. Pushing is always an explicit user action.
9191

9292
## When to write `scripts/install-systemd.py`
9393

0 commit comments

Comments
 (0)