Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 43 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,18 @@ Archive a lifetime of email. Analytics and search in milliseconds, entirely offl

Your messages are yours. Decades of correspondence, attachments, and history shouldn't be locked behind a web interface or an API. By default, msgvault downloads a complete local copy and then everything runs offline. Search, analytics, and the MCP server all work against your msgvault archive with no mailbox network access required. If you configure a remote deployment, the archive lives on your own server rather than a hosted msgvault service.

Currently supports Gmail, Google Calendar, and IMAP sync, plus offline imports from MBOX exports and Apple Mail (.emlx) directories.
Currently supports Gmail, Google Calendar, Microsoft Teams, and IMAP sync, plus
offline imports from MBOX exports, Apple Mail (.emlx) directories, PST archives,
and common chat/text export formats.

## Features

- **Full Gmail backup**: raw MIME, attachments, labels, and metadata
- **Google Calendar sync**: archive events, organizers, and attendees; searchable alongside email
- **Microsoft Teams sync**: archive delegated Graph chats, channels, replies, and inline media with `message_type = teams`
- **IMAP sync**: archive mail from any standard IMAP server
- **MBOX / Apple Mail import**: import email from MBOX exports or Apple Mail (.emlx) directories
- **Incremental backup snapshots**: verifiable `msgvault backup` repositories for the SQLite archive and attachments
- **MBOX / Apple Mail / PST import**: import email from local export formats
- **Interactive TUI**: drill-down analytics over your entire message history, powered by DuckDB over Parquet — connects to a remote `msgvault serve` instance or runs locally
- **Full-text search**: FTS5 with Gmail-like query syntax (`from:`, `has:attachment`, date ranges)
- **MCP server**: access your full archive at the speed of thought in Claude Desktop and other MCP-capable AI agents
Expand Down Expand Up @@ -90,8 +94,11 @@ msgvault tui
| `sync EMAIL` | Sync only new/changed messages |
| `add-calendar EMAIL` | Authorize read-only Google Calendar access and register calendars |
| `sync-calendar NAME\|EMAIL` | Sync Google Calendar events (full first run, then incremental) |
| `add-teams EMAIL` | Authorize delegated Microsoft Graph access for Teams |
| `sync-teams EMAIL` | Sync Microsoft Teams chats and channels |
| `backup` | Initialize, create, list, verify, and restore backup snapshots |
| `tui` | Launch the interactive TUI (`--account` to filter, `--local` to bypass HTTP) |
| `search QUERY` | Search messages (`--account` to filter, `--json` for machine output) |
| `search QUERY` | Search messages (`--account` and `--message-type` to filter, `--json` for machine output) |
| `show-message ID` | View full message details (`--json` for machine output) |
| `mcp` | Start the MCP server for AI assistant integration |
| `serve` | Run the API/scheduler or manage the background daemon (`start`, `status`, `stop`, `restart`) |
Expand Down Expand Up @@ -201,6 +208,39 @@ By default only calendars you own or can write to are synced (add `--all-calenda

Msgvault stores Google OAuth refresh tokens under the Msgvault home directory with file permissions restricted to the current user. Tokens and client secrets are not written into `config.toml`, logs, README examples, or exported fixtures.

### Microsoft Teams

Archive Microsoft Teams chats and channels through delegated Microsoft Graph
sync. Teams uses the `[microsoft]` OAuth app config but stores a separate
`teams_<email>.json` token from Outlook/IMAP OAuth.

```bash
msgvault add-teams user@example.com
msgvault sync-teams user@example.com
msgvault search "incident review" --message-type teams
```

See the [Microsoft Teams guide](https://msgvault.io/usage/teams/) for Graph
permissions, scheduling, channel sync behavior, and inline media backfill.

### Backup Snapshots

Create an append-only backup repository, take incremental snapshots, and verify
or restore them later:

```bash
msgvault backup init --repo ~/Backups/msgvault
msgvault backup create --repo ~/Backups/msgvault
msgvault backup verify --all --quick --repo ~/Backups/msgvault
msgvault backup restore --target ~/msgvault-restored --repo ~/Backups/msgvault
```

Set `repo` under `[backup]` in `config.toml` to omit `--repo` from every
command after `init`.

See the [Backup guide](https://msgvault.io/usage/backup/) for repository format,
secret-handling flags, restore proof, and operating recommendations.

## Configuration

All data lives in `~/.msgvault/` by default (override with `MSGVAULT_HOME`).
Expand Down
74 changes: 74 additions & 0 deletions docs/api-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ Paginated message list.
{
"id": 12345,
"subject": "Q4 Planning",
"message_type": "email",
"from": "alice@example.com",
"to": ["bob@example.com"],
"cc": ["carol@example.com"],
Expand All @@ -170,6 +171,67 @@ Paginated message list.

---

### Filter messages {#get-apiv1messagesfilter}

**Endpoint:** `GET /api/v1/messages/filter`

List messages with structured filters backed by the query engine. This is the
API equivalent of drilling into aggregate/search results and is the endpoint to
use for message-type filtering when you do not need full-text ranking.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `sender` / `sender_name` | string | — | Sender address/phone or display-name filter |
| `recipient` / `recipient_name` | string | — | Recipient address/phone or display-name filter |
| `domain` | string | — | Participant domain filter |
| `label` | string | — | Label filter |
| `message_type` | string | — | Stored message type, for example `email`, `teams`, `calendar_event`, or `sms` |
| `source_id` | int | — | Restrict to one source |
| `conversation_id` | int | — | Restrict to one conversation/thread |
| `after` / `before` | date | — | RFC3339 or `YYYY-MM-DD` bounds |
| `attachments_only` | bool | `false` | Only include messages with attachments |
| `hide_deleted` | bool | `false` | Exclude messages marked deleted at the source |
| `offset` | int | `0` | Zero-based row offset |
| `limit` | int | `500` | Maximum rows to return; capped at 500 outside conversation fetches |
| `sort` | enum | `date` | `date`, `size`, or `subject` |
| `direction` | enum | `desc` | `asc` or `desc` |

**Response:**

```json
{
"count": 1,
"has_more": false,
"offset": 0,
"limit": 500,
"messages": [
{
"id": 12345,
"subject": "Incident review",
"message_type": "teams",
"from": "alice@example.com",
"to": [],
"sent_at": "2026-07-01T15:30:00Z",
"snippet": "Follow-up from the channel discussion...",
"labels": [],
"has_attachments": false,
"size_bytes": 0
}
]
}
```

The companion `GET /api/v1/messages/gmail-ids` endpoint returns matching Gmail
source message IDs for email workflows such as deletion staging. It honors a
subset of these parameters: `sender` / `sender_name`, `recipient` /
`recipient_name`, `domain`, `label`, `source_id`, and `limit`. Results are
always restricted to Gmail sources, exclude deleted messages, and are ordered
newest-first; the remaining `/messages/filter` parameters (`message_type`,
`conversation_id`, `after` / `before`, `attachments_only`, `hide_deleted`,
`offset`, `sort`, `direction`) are ignored.

---

### Message details {#get-apiv1messagesid}

**Endpoint:** `GET /api/v1/messages/{id}`
Expand All @@ -182,6 +244,7 @@ Full message details including body and attachment metadata.
{
"id": 12345,
"subject": "Q4 Planning",
"message_type": "email",
"from": "alice@example.com",
"to": ["bob@example.com"],
"cc": ["carol@example.com"],
Expand Down Expand Up @@ -251,8 +314,17 @@ to `mode=fts` instead.
| `mode` | enum | `fts` | `fts`, `vector`, or `hybrid` |
| `page` | int | `1` | Page number (FTS only — vector/hybrid reject `page>1`) |
| `page_size` | int | `20` | Results per page (max 100 for FTS, max `[vector].search.max_page_size_hybrid` for vector/hybrid) |
| `message_type` | string | — | Message-type filter; repeat or comma-separate for multiple values |
| `account` | string | — | Restrict to one account/source |
| `collection` | string | — | Restrict to one collection |
| `explain` | 0/1 | `0` | When `1` and `mode=vector|hybrid`, include per-signal scores |

`message_type` uses the same values as local search: `email`,
`calendar_event`, `teams`, `sms`, `mms`, `whatsapp`, `imessage`,
`fbmessenger`, `synctech_sms_call`, `google_voice_text`,
`google_voice_call`, and `google_voice_voicemail`. The query string can also
carry `message_type:` / `message_type=` operators inside `q`.

**Response (mode=fts, default):**

```json
Expand All @@ -265,6 +337,7 @@ to `mode=fts` instead.
{
"id": 12345,
"subject": "Q4 Planning",
"message_type": "email",
"from": "alice@example.com",
"to": ["bob@example.com"],
"cc": ["carol@example.com"],
Expand Down Expand Up @@ -298,6 +371,7 @@ to `mode=fts` instead.
{
"id": 12345,
"subject": "Q2 planning offsite agenda",
"message_type": "email",
"from": "alice@example.com",
"to": ["team@example.com"],
"sent_at": "2024-01-15T10:30:00Z",
Expand Down
11 changes: 10 additions & 1 deletion docs/architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Architecture Overview
description: Package structure and data flow.
---

msgvault syncs your Gmail, IMAP, and Microsoft 365 accounts to a local SQLite database by default and can import local PST/MBOX archives, Apple Mail exports, and chats/texts from WhatsApp, iMessage, Google Voice, Facebook Messenger, and SMS Backup & Restore. PostgreSQL is available as an opt-in backend for new archives. Keyword search, analytics, the TUI, and the MCP server run against the configured archive database, Parquet metadata exports where available, and local attachment files. Optional vector search calls the embedding endpoint configured in `[vector.embeddings]` to build/query semantic vectors, then stores those vectors in `vectors.db` on SQLite or pgvector tables on PostgreSQL.
msgvault syncs your Gmail, Google Calendar, IMAP, Microsoft 365 mail, and Microsoft Teams accounts to a local SQLite database by default and can import local PST/MBOX archives, Apple Mail exports, and chats/texts from WhatsApp, iMessage, Google Voice, Facebook Messenger, and SMS Backup & Restore. PostgreSQL is available as an opt-in backend for new archives. Keyword search, analytics, the TUI, and the MCP server run against the configured archive database, Parquet metadata exports where available, and local attachment files. Optional vector search calls the embedding endpoint configured in `[vector.embeddings]` to build/query semantic vectors, then stores those vectors in `vectors.db` on SQLite or pgvector tables on PostgreSQL.

<img src="/assets/static/how-it-works.svg" alt="msgvault architecture: Gmail API syncs to SQLite, then offline Parquet analytics, FTS5 search, TUI, and MCP Server" style="width: 100%; max-width: 960px; margin: 1.5rem auto; display: block;" />

Expand All @@ -17,8 +17,12 @@ msgvault/
│ ├── tui/ # Bubble Tea TUI
│ ├── query/ # DuckDB/SQL query engines
│ ├── store/ # SQLite/PostgreSQL database access
│ ├── backup/ # Snapshot repository capture/verify/restore
│ ├── pack/ # Backup pack-file encoding
│ ├── deletion/ # Deletion staging and manifest
│ ├── gmail/ # Gmail API client
│ ├── gcal/ # Google Calendar API client
│ ├── calsync/ # Google Calendar sync orchestration
│ ├── sync/ # Sync orchestration
│ ├── imap/ # IMAP client (go-imap/v2)
│ ├── importer/ # Local email import orchestration
Expand All @@ -34,6 +38,7 @@ msgvault/
│ ├── fbmessenger/ # Facebook Messenger DYI import
│ ├── synctechsms/ # SMS Backup & Restore import
│ ├── microsoft/ # Microsoft 365 OAuth
│ ├── teams/ # Microsoft Teams Graph ingestion
│ ├── oauth/ # OAuth2 flows (browser + device)
│ └── mime/ # MIME parsing
├── go.mod
Expand All @@ -46,7 +51,10 @@ msgvault/
|---|---|
| `cmd/` | Cobra CLI commands, config loading |
| `internal/store` | SQLite and PostgreSQL database operations, schema management |
| `internal/backup` | Backup repository snapshots, manifests, verification, and restore |
| `internal/pack` | Backup pack-file framing, compression, and object integrity |
| `internal/sync` | Sync orchestration, MIME parsing, checkpoint management |
| `internal/gcal` / `internal/calsync` | Google Calendar API client and event sync |
| `internal/imap` | IMAP client, connection management, credential storage |
| `internal/importer` | Local email import orchestration and message ingestion |
| `internal/mbox` | MBOX format reader (mboxo/mboxrd) |
Expand All @@ -66,6 +74,7 @@ msgvault/
| `internal/fbmessenger` | Facebook Messenger Download Your Information parsing and import |
| `internal/synctechsms` | SMS Backup & Restore XML/ZIP parsing and Drive source sync |
| `internal/microsoft` | Microsoft 365 OAuth flow |
| `internal/teams` | Microsoft Teams Graph client mapping, sync cursors, and importer |
| `internal/mime` | MIME message parsing, charset detection |

## Design Decisions
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ All message data (metadata, labels, participants, and raw MIME) lives in the con
| Column | Type | Description |
|---|---|---|
| `id` | INTEGER PK | Auto-increment |
| `source_type` | TEXT | `gmail`, `imap`, `mbox`, `pst`, `apple-mail`, `whatsapp`, `apple_messages`, `google_voice`, `facebook_messenger`, `synctech_sms` |
| `source_type` | TEXT | `gmail`, `imap`, `gcal`, `teams`, `mbox`, `pst`, `apple-mail`, `whatsapp`, `apple_messages`, `google_voice`, `facebook_messenger`, `synctech_sms` |
| `identifier` | TEXT | Email address or phone number |
| `display_name` | TEXT | Account display name |
| `sync_cursor` | TEXT | Sync cursor (Gmail history ID for Gmail accounts) |
Expand All @@ -49,7 +49,7 @@ All message data (metadata, labels, participants, and raw MIME) lives in the con
| `conversation_id` | INTEGER FK | References `conversations` |
| `source_id` | INTEGER FK | References `sources` |
| `source_message_id` | TEXT | Source-specific message ID |
| `message_type` | TEXT | `email`, `whatsapp`, `imessage`, `google_voice_text`, `teams` |
| `message_type` | TEXT | `email`, `calendar_event`, `teams`, `sms`, `mms`, `whatsapp`, `imessage`, `fbmessenger`, `synctech_sms_call`, `google_voice_text`, `google_voice_call`, `google_voice_voicemail` |
| `sent_at` | DATETIME | Send timestamp |
| `sender_id` | INTEGER FK | References `participants` |
| `subject` | TEXT | Message subject |
Expand Down
Loading