Skip to content

Document high-level ingestion methods #261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bmerkle merged 1 commit into from
Apr 30, 2026
+28 −3
Merged

Document high-level ingestion methods #261

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions docs/high-level-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ class ConversationMessage(
text_chunks: list[str], # Text of the message, 1 or more chunks
tags: list[str] = [], # Optional tags
timestamp: str | None = None, # ISO timestamp in UTC with 'z' suffix
metadata: ConversationMessageMeta, # See below
metadata: ConversationMessageMeta, # See below
Comment on lines 22 to +23
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs say the timestamp uses a UTC 'z' suffix (lowercase), but the implementation uses an explicit 'Z' suffix (uppercase) for UTC (see format_timestamp_utc and ConversationMessage.timestamp docs). This example format should be updated to avoid generating/accepting the wrong timestamp format.

Copilot uses AI. Check for mistakes.
)
```

Expand Down Expand Up @@ -64,7 +64,32 @@ extracted and indexed knowledge thereof.
It is constructed by calling the factory function
`typeagent.create_conversation` described below.

It has one public method:
It has these public methods:
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc now says ConversationBase “has these public methods”, but ConversationBase also exposes other non-underscored methods (e.g., create and add_metadata_to_index). Either document these as well or adjust the wording to avoid implying the list is exhaustive.

Suggested change
It has these public methods:
Some of its public methods are:

Copilot uses AI. Check for mistakes.

- `add_messages_with_indexing`
```py
async def add_messages_with_indexing(
messages: list[TMessage],
*,
source_ids: list[str] | None = None,
) -> AddMessagesResult
```

Adds messages and updates all indexes in a single transaction.
For SQLite storage this is all-or-nothing.
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add_messages_with_indexing is documented as “all-or-nothing” for SQLite, but the implementation explicitly notes that in-memory storage has no rollback support and partial changes may remain on error. Consider adding a brief note about the in-memory behavior here so callers don’t assume transactional semantics when dbname=None.

Suggested change
For SQLite storage this is all-or-nothing.
For SQLite storage this is all-or-nothing. When using in-memory
storage (`dbname=None`), rollback is not supported, so partial
changes may remain if an error occurs.

Copilot uses AI. Check for mistakes.

- `add_messages_streaming`
```py
async def add_messages_streaming(
messages: AsyncIterable[TMessage],
*,
batch_size: int = 100,
on_batch_committed: Callable[[AddMessagesResult], None] | None = None,
) -> AddMessagesResult
```

Adds messages from an async stream, committing each batch separately.
Useful for very large ingestions where one large transaction is impractical.
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add_messages_streaming also performs source-id based skipping (already-ingested messages are silently skipped when source_id is set). This affects messages_added vs. the number of streamed messages; consider documenting this behavior since it’s an important semantic difference from naive “stream + commit” ingestion.

Suggested change
Useful for very large ingestions where one large transaction is impractical.
Useful for very large ingestions where one large transaction is impractical.
When source-id based deduplication is enabled, messages that were already
ingested may be silently skipped during streaming. As a result,
`AddMessagesResult.messages_added` may be smaller than the total number of
messages yielded by the async iterable.

Copilot uses AI. Check for mistakes.

- `query`
```py
Expand All @@ -80,7 +105,7 @@ It has one public method:

## Functions

There is currently only one public function.
There is currently only one public top-level function.

#### Factory function

Expand Down
Loading