Project Vision.docx

Project Vision
We are building a Narrative Intelligence Engine – not just another chatbot or editor. This offline Python system must read and “understand” a very long novel (100k–500k+ words) the way a seasoned developmental editor does. In practice, as each new chapter is ingested, the system should update an evolving story memory so that by Chapter 100 it already “knows” everything important from Chapters 1–99 (without re-reading them). The focus is on structured narrative memory and reasoning, not just text retrieval. In other words, we emulate how a human editor remembers and reasons about characters, events, and themes, rather than relying solely on vector lookups.
Cognitive Subsystems (Editor’s Mental Model)
An expert editor juggles many mental models of the story. Key cognitive subsystems include:
•	Character Modeling: Who are the characters, including identities, aliases, traits, motivations, fears, goals, and how they grow or change. (E.g. “Alice now fears water,” “Bob’s goal is power.”)
•	Relationship Modeling: How characters relate to each other (family ties, friendships, rivalries, romances) and how those ties evolve over time.
•	Dialogue & Voice: Who speaks which lines, the tone and style of each character’s speech, and consistency of voice.
•	Scene and Plot Structure: Division of the book into scenes, each scene’s purpose/conflict, and the causal sequence of events.
•	Timeline: Temporal ordering of events (the chronology of what happens when, including flashbacks, chapters, and scene-to-scene transitions).
•	Worldbuilding & Setting: Details of the world (locations, objects, magic/technology rules, cultural lore) and maintaining consistency of the setting.
•	Themes & Symbols: Recurring motifs, symbols, themes or metaphors (e.g. “light vs dark,” “freedom”) and how they are set up or paid off.
•	Narrative Style & Tone: Pacing, voice, tense, descriptive density, and overall writing quality (e.g. sentence length, use of passive vs active voice).
These subsystems collectively form the “narrative memory” of the novel. For example, an editor might remember that Alice never cries, only swims when afraid, and loved her mother as a child. The system must store similar facts, not raw text – much like a knowledge graph of the story【21†L150-L159】.
Open-Source Tool Mapping
For each of the above cognitive tasks, we seek the best mature open-source library or model (each choice must be justified as “USE AS-IS”, “WRAP”, “EXTEND” or “BUILD”). Key mappings include:
•	Document Parsing: PyMuPDF (Apache-2.0) can extract text from PDFs; python-docx or pandoc can handle Word or Markdown files. These are industry standards for text ingestion (USE-AS-IS).
•	Sentence Segmentation & Tokenization: spaCy (MIT license) provides fast, production-ready tokenization, sentence splitting, POS and dependency parsing【10†L53-L61】. Use spaCy’s English models as the base (USE-AS-IS).
•	Named Entity Recognition (NER): GLiNER or spaCy’s NER can detect basic entities (persons, places, etc.). GLiNER in particular is flexible for custom labels (like “MagicItem” or “PoliticalFaction”), and is optimized for CPU. This covers standard entities well (USE-AS-IS for base extraction).
•	Coreference Resolution: FastCoref (GPU-accelerated) or AllenNLP’s coref module can cluster pronouns and noun phrases into entities. We will use one of these as-is to resolve “John…the detective…he” references (USE-AS-IS).
•	Quotation & Speaker Attribution: BookNLP (MIT license) includes a module for identifying quoted dialogue and attributing speakers【28†L262-L270】. We can integrate BookNLP’s speaker-attribution model to split dialogue vs narration (WRAP, possibly wrap in a Python call).
•	Scene Segmentation: No single library dominates. We will implement a heuristic or lightweight ML (potentially fine-tune a transformer) for detecting scene boundaries (e.g. location/time changes). This is CUSTOM – likely rule-based (look for heading changes, new locations, or significant time shifts), or use dynamic topic modeling【12†L150-L158】. (If a pre-trained model exists from research [14], we could fine-tune it; otherwise BUILD.)
•	Dialogue Detection: Heuristics or BookNLP’s quote detection can be used to identify dialogue segments. Rules for opening quotes (““”) or patterns from BookNLP (USE-AS-IS or WRAP).
•	POV Detection: Identify who is the viewpoint character of each scene (by majority references or first-person pronouns). Likely custom logic (BUILD-WRAP, since not a common open tool).
•	Relationship & Event Extraction: Use an Open Information Extraction library (like Stanford OpenIE or AllenNLP’s SRL/EIL). These extract subject-verb-object triples or event triggers, which we can interpret (e.g. “Alice kisses Bob” implies a “Romantic” relationship event). There’s no turnkey “novel relationship extractor,” so we will mostly wrap general relation/event extractors and then classify results (e.g. keyword matching or LLM prompting to label “family vs enemy vs friend”). (Likely WRAP+CUSTOM logic.)
•	Emotion/Sentiment Analysis: Off-the-shelf models (like HuggingFace’s transformers pipelines for emotion or Vader sentiment) can tag sentence-level mood or sentiment. These provide coarse checks (e.g. “scene is sad” or “Alice is angry”) (USE-AS-IS).
•	Theme/Motif Detection: No mature open tool specifically. We may use topic modeling (e.g. LDA) or transformer-based zero-shot classification on sets of chapters to detect recurring topics. This will be custom (BUILD or EXTEND), possibly guided by a small set of seed concepts.
•	Style Metrics: Use open libraries like LanguageTool for grammar/punctuation, and libraries like textstat for readability metrics. Also compute statistics (avg sentence length, dialogue ratio, etc.) with custom code (WRAP, EXTEND as needed).
•	Memory Storage/Graph: Use NetworkX or RDFLib (both Python libraries) to store a knowledge graph of entities/relations. Characters, Locations, Events, Themes will be nodes/edges in a graph (USE-AS-IS for graph structure). We can serialize this graph to JSON or an in-memory structure each chapter (so no heavy DB needed).
All tools above are chosen for maturity and permissive license. For example, BookNLP is specifically built for novels【28†L262-L270】 (it handles entity coreference, event tagging and quote attribution out of the box). spaCy and GLiNER are well-supported. For tasks lacking a clear library (scene splitting, dynamic themes, narrative consistency), we will custom-build only those modules.
Memory Architecture (Structured Story Memory)
The core of the system is structured memory, not raw text. After each chapter, the engine updates multiple interconnected memory stores. Unlike the initial design (which stored simple entity lists), we must model stateful, evolving facts about the story. The main memory components include (with example JSON fields):
•	Character Memory: One object per character (identified by name cluster). Fields include: name, aliases, traits, fears, goals, current_emotional_state, location, inventory, relationships, last_mentioned_chapter, arc_stage, secret_status, etc. For example:
 	"Alice": {
  "aliases": ["Ally", "Alicia"],
  "current_location": "Sea Port",
  "goal": "Find her mother",
  "fears": ["Water"],
  "inventory": ["OldKey"],
  "relationships": {"Bob": "Friend", "EvilKing": "Enemy"},
  "last_seen_chapter": 17,
  ...
}
Update Algorithm: For each coreferenced mention or NER: if it matches an existing character, update that entry; if new, create a character. When a chapter says “Alice took out a sword,” add Sword to Alice’s inventory. If it says “she is terrified,” update Alice.fears. This is mainly custom code that merges NLP output into the JSON state.
•	Relationship Memory: A graph (or adjacency list) of character-character relations (labels like friend/enemy/family/lovers). Each time two characters interact significantly, we update or add an edge. For example, if “Alice embraces Bob,” strengthen Alice–Bob: Lovers; if “They argued fiercely,” maybe change to Rivals. This can be stored either as part of Character objects or as separate triples.
•	World Memory: Details about locations, objects, magic systems, societies, etc. Example schema:
 	"locations": {"ForestCave": {"type": "Cave", "description": "damp and dark", "first_mentioned": 5}},
"objects": {"Excalibur": {"owner": "None", "power": "LegendarySword", "first_mentioned": 12}},
"lore": {"DragonMyth": {"summary": "...", "associated_characters": ["Dragonslayer"]}}
New world elements discovered in text are added. World rules (e.g. “no one can cross water alive”) are noted if described.
•	Timeline/Events Memory: A chronological list of major events, each tagged with chapter/scene. For example: {"chapter": 7, "scene": 2, "event": "Alice obtains sword", "characters": ["Alice"], "time": "Night"}. This timeline also notes unresolved events (cliffhangers) and completed arcs.
•	Scene Memory: Each scene (within chapters) is an entry, with fields: scene_id, characters_present, setting, conflict, outcome, emotion_tone, purpose. Scenes link into the chapter and timeline. (E.g. Scene 7.3: In the Goblin Lair, Alice confronts goblins – Conflict: “Rescue her brother” – Outcome: “Escapes with goblin key”.)
•	Theme/Motif Memory: Track high-level themes (e.g. “betrayal,” “redemption”). Count how often each appears or is referenced. We might list specific motifs (e.g. a “broken mirror” symbol).
•	Promise/Foreshadow Tracker: Every time a promise or prophecy is made (“I will return for you”), add an entry. Each has a speaker, listener, promise_text, chapter_made, status. If the promise is later broken or fulfilled, update status. Unresolved promises by Chapter 100 can be flagged.
•	Consistency/Fact Database: A special store of “established facts,” such as static traits (hair color, age, physical description). After each chapter, we cross-check any new mention against this database to detect contradictions (e.g. hair changing).
•	Editorial Notes Memory: This is where the system accumulates issues it has found so far (like "Alice’s hair color changed from black to brown between Ch.5 and Ch.12"). These notes feed into the final review.
All memory components are continuously updated. For example, after reading Chapter N:
•	We parse the chapter → extract entities/events.
•	Character Memory Update: We match any names/aliases to existing characters. If “Ally” appears and maps to “Alice”, we update Alice’s state. We merge coreference clusters so “he/she/they” updates the right profile.
•	World Memory Update: If a new location or object is mentioned, add it. If an object changes hands, update the owner.
•	Relationship Update: If a scene shows two characters interacting in a new way, adjust their relationship edge.
•	Theme/Promise Update: If a new motif or foreshadow occurs, log it; if a previous motif is paid off, mark it resolved.
•	Timeline/Scene Update: Record the new scene(s) as entries. Link them to previous ones.
By structuring memory this way (essentially a story knowledge graph), the system can answer questions like “What has Alice done recently?” or “Who does Alice trust?” without scanning raw text. This approach echoes how knowledge-graph-backed storytelling has been shown to improve coherence【21†L150-L159】.
Pipeline Redesign
Rather than a simple linear pipeline, we implement a rich, multi-stage cognitive pipeline. Each stage can USE an existing tool (or wrap it) or require custom logic. An example pipeline is:
1.	Document Parsing & Cleaning: Read file (with PyMuPDF, docx) → extract plain text. Normalize whitespace and fix encoding issues. (USE-AS-IS)
2.	Sentence Segmentation & Tokenization: Use spaCy to split text into sentences, tokens, POS tags and basic parsing【10†L53-L61】. (USE-AS-IS)
3.	NER & Coreference: Run spaCy/GLiNER NER to extract named entities; run FastCoref to cluster mentions into entities. (USE-AS-IS)
4.	Alias Resolution: Merge coref clusters with NER: e.g. if FastCoref links “Mr. Sawyer” with spaCy’s “Tom Sawyer”, unify them. Maintain an alias map to join different names. (WRAP: some custom merging logic.)
5.	Dialogue & Speaker Attribution: Detect quoted speech (e.g. using regex or BookNLP). For each quote, identify the speaker (using BookNLP’s model or pronoun heuristics) (WRAP). Output: list of (speaker, utterance) pairs.
6.	Scene Segmentation: Determine where one scene ends and the next begins. This may use cues (e.g. location/time change). We can build a simple ML model (or use dynamic topic shifts【12†L150-L158】) to propose cuts. (BUILD or fine-tune if corpus available.)
7.	Entity Linking: (Optional) If we have any external knowledge base (Wikipedia, etc.), we could link famous names. For fiction, probably skip. (Likely SKIP or minimal custom entity normalization.)
8.	Event Extraction: For each sentence or clause, extract an event triple (subject-verb-object) using an OpenIE tool or a transformer model (like a fine-tuned T5 for event extraction【28†L262-L270】). Label the event type (e.g. “action:kiss”, “transaction”). (WRAP or EXTEND an existing event extraction system.)
9.	Emotion/Sentiment Analysis: Classify each sentence or scene for emotion or sentiment (e.g. using DistilRoBERTa on Ekman’s 6 emotions【8†L247-L256】). Tag “anger, fear, joy, sadness, surprise, disgust” or polarity. (USE-AS-IS with HuggingFace model.)
10.	Style & Pacing Analysis: Compute metrics like average sentence length, paragraph length, dialogue ratio, grammar issues (via LanguageTool), readability score. Store these in Style Memory. (USE/WRAP textstat, LanguageTool.)
11.	Theme/Motif Detection: Run a topic modeling pass on the chapter’s text (or the chapter plus memory) to see if known themes appear. (BUILD/WRAP a simple LDA or transformer-based classifier.)
12.	Memory Update (Character/World/Timeline): Using the extracted entities, events, scenes: update Character Memory, World Memory, etc. (BUILD the logic described above.)
13.	Consistency Checking: Compare updated memory against what was known. E.g.: “Did Alice’s hair change color?” If a fact has changed unexpectedly, record a potential inconsistency. (BUILD custom rule-checkers.)
14.	Editorial Reasoning: Finally, run higher-level analysis. This might involve prompting an LLM (e.g. Gemini or GPT) with a summary of the chapter plus relevant memory (characters’ states, unresolved threads). The model can then output summary notes and critique (e.g. “Alice’s motivation is unclear here,” “Dialogue feels stiff”). (WRAP a chat completion model, passing structured memory as context.)
15.	Review Output: Compile all findings into JSON: character profiles, timeline, scene list, editorial notes, consistency issues, and chapter summary.
Each stage’s input/output is clearly defined. For example, after “Dialogue & Speaker Attribution” we have a list of (character, quote) pairs that feed into character memory and emotion analysis. After “Event Extraction”, we have structured event records that update timeline and relationships. Whenever possible, we note if a stage is USE (pure existing tool), WRAP (existing tool plus glue), EXTEND (modify an existing model), or BUILD (custom code).
Throughout, the focus is on mature, single-purpose tools. For instance, BookNLP already bundles NER, coref, event tagging and quote attribution【28†L262-L270】 – we can use it as a reference or for some parts. spaCy serves most lower-level NLP needs. Only novel areas (scene splitting, narrative consistency rules) require fresh implementation.
Scene Engine
A strong Narrative Engine must explicitly model scenes (sub-chapter units). Scenes are “segments where time and discourse align, one location, and a stable cast of characters”【14†L26-L34】. We will build a Scene Engine that:
•	Detects Scene Boundaries: Using cues (chapter headings, changes in setting phrases, time jumps). We can start with simple heuristics (e.g. a new line starting with a location name). Research suggests scene segmentation is hard, but even a baseline helps【14†L63-L72】.
•	Identifies Scene Properties: For each scene, record its setting, time_of_day, characters_present (from character memory), emotional_tone (aggregate from emotion analysis), conflict (highest-stakes goal), and outcome. For example, Scene 3.2 might be “Battle at the Dock” – characters “Alice, Bob”; conflict “defend the ship”; outcome “pirates repelled”.
•	Links Scenes into Chapters and Timeline: Each scene gets a unique ID like “Chap7.Scene2”. Scenes within a chapter are ordered. Scenes carry pointers to memory: the events that occur and characters who enter/exit.
Having explicit scenes lets the system reason like an editor: “Scene 4 was too long and lost focus” or “this scene has no clear conflict”. It also enables localized memory updates (e.g. coreference is easier within one scene). We can optionally visualize the scene graph (the Reddit thread noted visualizing timelines and character networks as helpful【4†L186-L194】).
Character Engine
Instead of treating characters as static, each character’s state must evolve. Every time a chapter is read, we update each character’s dynamic profile. Key elements of a character’s state include:
•	Current Goals and Motivations: What are they actively trying to do in this chapter? E.g. Alice’s goal might shift from “escape” to “confront Bob” as the plot evolves.
•	Emotional State: Mood or feelings at chapter start/end (happy, anxious, furious, etc.). These come from emotion analysis and context.
•	Knowledge/Beliefs: What the character knows. If Alice learns a secret in Ch.5, record “Alice knows King is traitor.” This can be updated by scanning factual statements that are directed to or observed by the character.
•	Possessions/Status: Items carried, alive/dead status, injuries. If Alice picks up a key, add to inventory. If a character dies, mark them accordingly.
•	Relationships: Current stance toward others (friend/enemy) which may be updated if they reconcile or have a falling out.
•	Character Arc Stage: Track which act of the story the character is in (e.g. “Introduction,” “Crisis,” “Resolution”). We could infer this by how many significant events they’ve had.
All these fields are part of the Character Memory JSON, but they are dynamic subfields that change. For example, if the chapter shows Alice overcoming her fear of water, we remove “Water” from fears. If Bob reveals he loved Alice, we update Alice’s relationships["Bob"] = "LoveInterest" and vice versa.
By tracking dynamic state, the engine can flag issues like character regression (e.g. Alice suddenly forgetting her fear without reason) or consistency errors (Alice cannot simultaneously love and hate Bob without explanation). This goes far beyond a simple name list – it’s a full profile for each character.
Editorial Engine
This is the reasoning and critique layer – the heart of the project. It should look at the updated narrative memory and the current chapter and act like an expert editor. It will check for problems and suggest improvements, such as:
•	Consistency Checks: Compare chapter facts against memory (e.g. “Alice’s eye color changed,” “Magic rule violation: objects can’t teleport in this world,” “Timeline gap: Bob was seen across town a minute ago”). Contradictions are flagged.
•	Character Consistency: Ensure characters act in line with their arcs and traits. E.g. if Alice is cowardly until chapter 50, going “Rambo-mode” early triggers a warning.
•	Relationship Plausibility: If two characters suddenly become lovers with no buildup, note that. If a promise was made (e.g. “We will run away together”) and not yet fulfilled or broken, remind the author.
•	Plot and Pacing Issues: Identify scenes or chapters with too much or too little action, unclear conflict, or misplaced digressions. (E.g. “Ch.10 spends 5 pages describing a market that doesn’t affect the plot.”)
•	Dialogue & Style Suggestions: E.g. “Alice’s dialogue sounds monotone compared to her angry inner voice” or “We’ve not seen anyone use slang until now.” Check for overused words or broken prose.
•	Emotional/Theme Consistency: “The chapter starts very dark but ends cheerfully – consider smoothing the transition.” Or “This scene doesn’t reinforce any of the main themes.”
•	Structural Advice: If an important subplot is introduced late or left hanging, note it. If a chapter or scene doesn’t serve the story arc, point that out.
The Editorial Engine will likely use a mix of rule-based checks (e.g. simple flagging of contradictions) and a small LLM for more nuanced feedback. For instance, we can craft a prompt: “Given Character Memory (goals, fears, relationships) and the chapter text, list any inconsistencies or weak motivations.” Because it has the structured memory as context, the model can be more precise. We keep these suggestions in editorial.json.
Implementation Roadmap
We break development into phases, each delivering a working subset. Every phase is tested on example chapters and has clear acceptance criteria.
1.	Setup & Basic Pipeline: Goal: Ingest text and run spaCy. Install and test PyMuPDF, python-docx, spaCy (USE-AS-IS). Deliverables: Script that reads a chapter file and outputs tokenized sentences with POS tags. Test: Verify extraction on sample PDF/DOCX. (Est. 2 days)
2.	Entity & Coref Extraction: Goal: Extract named entities and coref clusters. Install FastCoref or AllenNLP coref. Deliverables: List of characters with their mentions in the chapter. Test: On a short story, check that pronouns link to correct names. (Est. 3 days)
3.	Dialogue & BookNLP Integration: Goal: Identify quoted dialogue and speakers. Integrate BookNLP or implement regex-based quote splitting and simple speaker heuristics. Deliverables: (Speaker: Utterance) pairs for chapter. Test: Given a chapter with dialogue, verify each quote is tagged with a speaker (or “Unknown”). (Est. 5 days)
4.	Initial Memory Structures: Goal: Define JSON schemas for Character, Relationship, World, Scene, Timeline, etc. Initialize an empty memory. Deliverables: Schema files and code to load/save them. Test: JSON validity, no missing fields. (Est. 2 days)
5.	Basic Memory Update: Goal: Write code to take NER/coref/dialogue output and update Character Memory (names, aliases, last seen). Also add Locations/Objects to World Memory. Deliverables: After processing a chapter, characters.json and world.json show new entries or updates. Test: Manually inspect that new characters are added and facts merged correctly. (Est. 5 days)
6.	Scene Segmentation & Scene Memory: Goal: Implement scene boundary detection (e.g. by blank lines or chapter subheadings) and create scene entries. Deliverables: A list of scenes per chapter with basic metadata (characters present, setting if obvious). Test: On a chapter with two settings, pipeline should split into two scenes. (Est. 4 days)
7.	Relationship & Timeline Update: Goal: Use events (subject-verb-object) to infer simple relations (e.g. “Alice hugs Bob” → add/strengthen Alice–Bob: “Lover”). Append major events to timeline. Deliverables: relationships.json and timeline.json updated per chapter. Test: Known interactions in text produce correct edges or timeline entries. (Est. 5 days)
8.	Style & Emotion Analysis: Goal: Add grammar/style metrics and emotion tags. Use LanguageTool API or library and an emotion classifier. Deliverables: style.json and emotions noted in scene/character profiles. Test: Flag a grammatical error in a test sentence; classify obvious emotions correctly. (Est. 3 days)
9.	Consistency Checker: Goal: Implement checks (e.g. contradiction detection) using the growing memory. Example: if Character Memory says “blue eyes” and new text says “green eyes,” flag it. Deliverables: consistency.json listing any found issues. Test: Create a test chapter with a deliberate inconsistency and ensure it’s caught. (Est. 5 days)
10.	Editorial Output (LLM Integration): Goal: Integrate an LLM (like GPT-4, Claude or Gemini) to generate chapter summaries and critique based on memory. Deliverables: editorial.json with summary of chapter and list of suggested improvements. Test: Prompt the LLM with a known problem case and see if it points it out. (Est. 5 days)
11.	Refinement & Optimization: Goal: Polish JSON schemas, add error handling, and improve any modules (like refining scene splits or alias merging). Deliverables: Clean, documented code and final memory + review outputs. Test: Run on a multi-chapter example novel; review output for completeness. (Est. 7 days)
Each phase’s Acceptance Criteria means the pipeline runs end-to-end for that phase’s scope without errors, and manual inspection confirms correctness on a small test. All data is stored locally (JSON files); no complex databases or cloud.
Summary
Version 2 of the architecture centers on persistent narrative memory and editorial reasoning, not just extraction. We significantly expand the initial design by introducing explicit memory stores (character profiles, scene list, event timeline, etc.) and engines for scenes, character arcs, and editorial checks. We leverage mature OSS for as much as possible (spaCy, BookNLP, FastCoref, PyMuPDF, etc.)【28†L262-L270】【14†L26-L34】. Crucially, we link story elements into a knowledge graph: characters, events, locations and themes are interconnected, much as “story elements” are stored in a graph to enforce coherence【21†L150-L159】. Only in the truly novel parts (like the Promise Tracker or custom consistency rules) do we build new modules.
This design reads like a specification for a solo engineer: it uses only local Python libraries, splits work into self-contained stages, and avoids distributed complexity. By the end, the engine will operate “like an editor” – summarizing chapters in context, catching continuity errors, and remembering the entire novel’s world as a living structure.
Sources: We consulted state-of-the-art literature and tools for narrative processing. For instance, Cognee’s analysis highlights that LLMs need an external memory layer to overcome “context window exhaustion” and retain knowledge【2†L62-L69】. Similarly, research on scene segmentation shows the importance of breaking novels into coherent parts to simplify long-text analysis【14†L26-L34】【14†L67-L74】. We incorporated these insights and only resort to custom coding where no mature open-source solution exists.
________________________________________