Add reusable schema templates for pg and clickhouse#1
Merged
Conversation
Templates are plain functions that register column groups on a table and return a struct of typed *Col[T] handles, keeping schema declarations DRY without giving up type safety. Ships Timestamps, SoftDelete, Audit[T], and UUIDPrimaryKey for both dialects; external libraries follow the same recipe to expose their own. https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Introduces opt-in ORM-style primitives shared by pg and clickhouse: InsertHook / UpdateHook / DeleteHook (pg only — CH has no builder-side mutations), Table.DefaultFilter, and Unscoped() on Select/Update/Delete to bypass the default scope. The Mixin interface (Apply(*Table)) lets templates contribute columns, indexes, hooks, and filters in one shot. TimestampsMixin bumps updated_at on UPDATE; SoftDeleteMixin rewrites DELETE into UPDATE deleted_at = now() and applies a "deleted_at IS NULL" scope to every Select/Update/Delete. User-supplied values always win over hook contributions to keep backfills and overrides predictable. Tables without hooks render SQL unchanged, so the feature is backwards-compatible. https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
NewEntity[T any](*Table) precomputes the column ↔ struct-field
mapping and (for pg) the PK column, then exposes type-safe CRUD on
the bound type:
- pg.Entity[T]: Get / Create / Update / Save / Delete /
Query.All/One. Save branches on the PK's zero value;
Get/Update/Delete short-circuit through the PK column.
- clickhouse.Entity[T]: Create / CreateMany / Query.All/One —
the narrow surface CH allows (no RETURNING, no per-row
UPDATE/DELETE builders).
Composes with the Phase-1 hooks: SoftDelete still rewrites
Delete into UPDATE, Timestamps still bumps updated_at on Update,
and EntityQuery.Unscoped() bypasses default scopes.
Zero-valued fields are omitted from INSERT when the column has a
DEFAULT (or is the implicit serial-family default for pg PKs);
explicit values pass through unchanged.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Two opt-in extensions to Entity[T]:
- Validator[T any] = func(*T) error registered via
Entity.Validate(...). Runs in order before Create / Update /
Save (pg) or Create / CreateMany (clickhouse) — first failure
aborts the operation before any SQL is issued.
- OptimisticLock() on a pg column marks it as the version column.
Entity.Update auto-emits SET version = version + 1 and
AND version = current; ErrStaleObject signals a missed write.
Both features are detected per-entity at NewEntity time, so tables
without them are unaffected. Polymorphic relations and cmd/dropsgen
are deferred to a separate iteration — they are large surfaces that
benefit from their own design pass.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
MorphTo wires a child whose (type_col, id_col) pair points at one of
several parent tables, registered via a MorphMap. Each child is
loaded as a *Parent via type assertion on the relation field
(declared `any`):
morphs := pg.NewMorphMap()
pg.RegisterMorph[User](morphs, "users", Users)
pg.RegisterMorph[Post](morphs, "posts", Posts)
pg.NewRelations(Comments).MorphTo("commentable",
CommentTypeCol, CommentIDCol, morphs)
MorphMany is the inverse: every child whose type_col equals a fixed
discriminator becomes a member of the parent's slice. Reuses the
HasMany loader with an extra WHERE on the type column.
The loader bucketises children by morph_type and runs one query per
distinct type, plus the root query — same cost profile as
HasMany/BelongsTo. Nested With() on MorphTo is rejected because the
loaded parent's Go type varies row-by-row.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
dropsgen reads a Go source file, finds structs whose doc comment
carries a `//drops:entity table=<TableVar>` directive, and emits a
sibling `<file>_drops_gen.go` with three helpers per entity:
- Cols<T>() []string — column names in declared order.
- Bind<T>(*T) []any — field values in column order, ready for
INSERT / UPDATE parameter lists.
- Scan<T>(drops.Rows, *T) error — row scan into the struct
without reflection.
The generator is a small AST + text/template combo (~150 lines)
with golden-file tests under testdata/. Relations, validators, and
polymorphism are intentionally out of scope for this first
iteration — the goal is to provide a fastpath for the hot
row-binding loop, not to subsume the runtime Entity API.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
dropsgen now emits a Register<T>(e *pg.Entity[T]) helper alongside Cols/Bind/Scan, and pg.Entity[T] gains SetFastScan to accept it. The SELECT executors (Get, EntityQuery.All, EntityQuery.One) take the zero-reflection path when a fast scanner is registered, and transparently fall back to the reflection scanner when relations have been queued via With/WithRel (the loader needs the reflection-populated parent slice). This closes the gap left by the previous dropsgen commit: the generated helpers are now consumed by the runtime, so `go generate` on a struct delivers the SELECT-side speedup directly. INSERT/UPDATE/Save fastpaths are still reflection — bind requires column-aware codegen we haven't built yet, and the cost is amortised across far fewer calls than reads in typical workloads. https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
The single biggest source of boilerplate in drops was the
table-plus-column declaration that always shadowed the Go struct:
var (
Users = pg.NewTable("users")
UserID = pg.Add(Users, pg.BigSerial("id").PrimaryKey())
UserName = pg.Add(Users, pg.Text("name").NotNull())
...
)
AutoTable[T] derives the *Table directly from extended `db:` tags
on T, and NewAutoEntity[T] bundles AutoTable + NewEntity so an
entire entity collapses to one line:
type User struct {
ID int64 `db:"id,pk,autoinc"`
Name string `db:"name,notnull"`
Email string `db:"email,notnull,unique"`
}
var UserEntity = pg.NewAutoEntity[User]("users")
Supported tag options: pk, autoinc, notnull, unique, default=<sql>,
version. Pointer fields default to nullable. Go types map to the
canonical SQL types via the existing simple/parametrised type
machinery. Unknown options or unsupported field types panic at
startup so misconfiguration fails loudly rather than at query time.
The row scanner now also splits comma-separated `db:` tags so
manual column declarations and AutoTable-declared ones see the
same field map.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
The library is called drops, so the schema-aware struct tag now matches the brand: - db:"col,pk,..." → drop:"col,pk,..." - db_rel:"name" → drop_rel:"name" Updated everywhere the tag is read (pg/scan, pg/autotable, pg/find for relation lookup, clickhouse/scan, cmd/dropsgen/parse) and everywhere it is written (entity tests, autotable tests, relation tests, morph tests, examples, readme, generated golden). ClickHouse scanner also gained the comma-split honoured by AutoTable, so multi-option drop tags work uniformly across dialects. Breaking change: existing code with db: tags must be updated. The project is pre-1.0 and the rename buys a clean, branded source of truth for schema declarations. https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Aligns the schema-aware tag options with Go's idiomatic camelCase:
- pk → primaryKey
- autoinc → autoIncrement
- notnull → notNull
- unique → unique (unchanged)
- default=X → default=X (unchanged)
- version → version (unchanged)
Updates the parser, doc comments, tests, package doc, and the
dropsgen helper text.
Example:
type User struct {
ID int64 `drop:"id,primaryKey,autoIncrement"`
Email string `drop:"email,notNull,unique"`
Name string `drop:"name,notNull"`
}
var UserEntity = pg.NewAutoEntity[User]("users")
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Aligns the relation-field tag with the camelCase convention adopted
for the drop tag's options. Updated everywhere:
- pg/find.go: relation-field lookup reads dropRel
- pg/relations_test.go, pg/morph_test.go, pg/entity_fastpath_test.go
- readme.md
Example:
type User struct {
ID int64 `drop:"id,primaryKey,autoIncrement"`
Name string `drop:"name,notNull"`
Posts []Post `dropRel:"posts"`
}
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Sweeps the project for snake_case identifiers used as column / table
/ constraint names and converts them to camelCase, matching the
struct-tag rename. Affects:
- Template column names: created_at→createdAt, updated_at→updatedAt,
deleted_at→deletedAt, created_by→createdBy, updated_by→updatedBy.
Applied to pg and clickhouse Templates, Mixins, hooks and tests.
- Constraint name builders in pg/snapshot.go: fkName /
uniqueName now emit camelCase (usersEmailUnique,
postsUserIdUsersIdFk) instead of the drizzle-kit
snake_case form.
- Migration tracking table column: applied_at → appliedAt.
- Untagged-field scanner fallback: snakeCase() → camelCase().
UserID now matches "userId" (not "user_id"), HTTPStatus matches
"httpStatus". Same in pg/scan.go and clickhouse/scan.go.
- All test fixtures, sample tables and example code (user_id →
userId, commentable_type → commentableType, user_groups →
userGroups, etc.) updated to keep tests passing.
Breaking change for any existing database with snake_case column
or constraint names. Pre-1.0 trade for end-to-end brand
consistency: Go side and SQL side speak the same language.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Entity.WithCache(cache.Cache, ttl) plugs the drops/cache backend
into the typed CRUD shortcuts:
- Get(id) reads through: cache hit returns immediately, miss runs
the SELECT and populates the cache. Single-flight dedupes
concurrent misses on the same key so a thundering herd of
"give me user 42" collapses to one DB query.
- EntityQuery.All / .One cache by sha256(SQL+args). Identical
queries within the TTL window are served from the cache.
Queries with eager-loaded relations bypass the cache because
relation loaders need the reflection-populated slice.
- Create / Update / Save write-through: the post-RETURNING values
land back into the PK cache so the next Get hits.
- Delete invalidates the PK entry.
Encoding via encoding/gob, keys formatted "drops:<table>:pk:<id>"
or "drops:<table>:q:<hash>". No new external dependencies — the
single-flight group is a minimal in-package implementation.
Composes with the dropsgen fast-scan path (zero reflection on
cache miss too), the lifecycle hooks (UpdatedAt still bumps on
Update + cache refresh), and the optimistic-lock guard
(ErrStaleObject still propagates; the cache is not refreshed on
a failed Update).
Tests cover hit/miss, write-through, invalidate-on-delete,
single-flight under 25-goroutine fan-out, and query-hash caching.
Suite passes under -race.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Two production-polish additions:
* SQLSTATE-classified errors. db.Exec and db.Query now wrap
driver errors in *PgError whose Sentinel field points at the
matching package-level Err* value, so callers can branch
cleanly:
if errors.Is(err, pg.ErrUniqueViolation) { ... }
Recognised codes: 23505/23503/23514/23502 (constraint
violations), 42P01/42703 (undefined table/column),
40001/40P01 (serialization / deadlock). Constraint names are
exposed when the driver supplies them (pgx-style
ConstraintName(); falls back to bare *PgError otherwise).
Drivers that don't surface SQLSTATE pass through unchanged.
* pg.TestTx(t, db, ctx, fn) opens a transaction, hands the
scoped *DB to fn, and rolls back unconditionally — even on
panic — via t.Cleanup. Drops the manual setup/teardown from
integration tests.
Both are additive: nothing pre-existing changes behaviour.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Three scale-oriented additions on top of Entity[T]:
* Page builder — UserEntity.Page(db).OrderBy(...).Limit(20)
.After(cursor).All(ctx) returns a typed Page[T] with Items,
HasMore, and an opaque base64 NextCursor. Cursors carry the
ordering columns' values in a gob blob; the WHERE guard uses
PostgreSQL row-comparison ((c1, c2) > ($1, $2)) for stable
homogeneous direction, or a tie-break disjunction for mixed
asc/desc. One extra row (limit+1) detects HasMore without a
follow-up COUNT.
* EntityQuery.Stream(ctx, func) iterates one row at a time,
memory-bounded, suitable for exports and batch jobs.
Returning an error from fn aborts and propagates. Bypasses
the cache (streaming results would defeat the purpose). The
fast-scan path is honoured when registered. Eager-loaded
relations are explicitly rejected because the loader needs
the populated parent slice.
* Entity.CreateMany(rs) batches a multi-row INSERT in one
round-trip; Entity.UpsertMany(rs) extends it with
ON CONFLICT (pk) DO UPDATE SET col = EXCLUDED.col for every
non-PK column — idempotent ingestion in one statement.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Budget bounds the cost of an Entity operation:
- MaxArgs caps the parameter count rendered by the builder.
Catches accidentally-huge IN clauses before they hit the
driver's prepared-statement arg ceiling.
- MaxRows is enforced by applying an upper-bound LIMIT to
EntityQuery.All. The user's own Limit wins when tighter — the
budget is a ceiling, not an override.
- MaxDuration wraps the caller's context with context.WithTimeout
so the operation aborts cleanly when it exceeds the budget.
Get / One / Update / Delete are single-row by construction and
skip MaxRows; Stream / Page are explicitly paginated and bypass
it too. Composes with the cache (budgets apply before the
cache-hit check), the optimistic-lock guard, and the fast-scan
path.
Errors surface as ErrBudgetExceeded for MaxArgs / MaxRows;
MaxDuration produces the usual context.DeadlineExceeded.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
WithN1Detector(ctx) wraps a context with a tracker. Pair it with
pg.N1Hook attached to the DB:
db := pg.New(drv).WithHook(pg.N1Hook)
func handler(...) {
ctx, finish := pg.WithN1Detector(ctx)
defer func() {
if r := finish(5); !r.IsClean() {
log.Warn("N+1 candidates", "patterns", r.Patterns)
}
}()
...
}
drops queries are parametrised, so the same SQL skeleton fired
with different args is the unambiguous N+1 signature. The
detector counts per skeleton in a per-context tracker; the
report lists patterns whose count >= threshold, ordered by
count desc.
The hook is a no-op on contexts without a tracker, so attaching
N1Hook to the DB carries no cost for untracked traffic. Use this
in dev/staging or behind a debug feature flag — production left
unmonitored if the overhead matters.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
pg.JSONField[T any](col, path...) returns a typed handle that
walks a jsonb path and casts the final text accessor to the SQL
type matching T:
beta := pg.JSONField[bool](UserMeta, "flags", "beta")
db.Select(...).Where(beta.Eq(true))
// SELECT ... WHERE (("users"."meta" -> 'flags' ->> 'beta')::boolean = $1)
Operators provided: Eq / Ne / Gt / Gte / Lt / Lte / In / IsNull /
IsNotNull / Like. Type-to-cast mapping: string→text, int/int32→
integer, int64→bigint, float32→real, float64→double precision,
bool→boolean, time.Time→timestamptz, fallback→text.
Containment / key-existence operators (JSONContains, JSONHasKey,
JSONHasAnyKey, JSONHasAllKeys) work on the raw jsonb column —
they don't need a typed leaf. The previous untyped JSONPath
function was renamed to JSONGetPath to free the namespace.
Path key literals are single-quoted and any embedded apostrophe
is doubled, so weird keys like "isn't" round-trip safely.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
pg.NewReplicated(primary, replicas...) returns a drops.Driver
that routes Exec / Begin to the primary and Query round-robin
across replicas. Plug it into a regular *pg.DB:
db := pg.New(pg.NewReplicated(primary, r1, r2))
The read-your-writes window guarantees the caller observes
their own writes even on a stale replica:
ctx = pg.WithReadYourWrites(ctx, 2*time.Second)
_ = UserEntity.Update(db, ctx, &u)
got, _ := UserEntity.Get(db, ctx, u.ID) // served by primary
Every Exec on a tagged ctx re-arms the window; reads inside the
window go to the primary, reads outside fall back to a replica.
Replica routing is atomic round-robin (sync/atomic counter, no
mutex) and degrades gracefully to primary-only when no replicas
are configured. Transactions always live on the primary because
the read/write split would otherwise break consistency.
Implements drops.Driver, so nothing in pg internals needs to
change. Close() walks every underlying driver that exposes one.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
pg.Tracer / pg.Span are minimal interfaces mirroring a subset of
OpenTelemetry's trace.Tracer / trace.Span. drops imports nothing
extra; callers write a tiny adapter once and inject it:
type otelAdapter struct{ tracer trace.Tracer }
func (a otelAdapter) Start(ctx context.Context, name string) (context.Context, pg.Span) {
ctx, span := a.tracer.Start(ctx, name)
return ctx, otelSpan{span}
}
db := pg.New(drv).WithTracer(otelAdapter{tracer: otel.Tracer("myapp")})
DB.Exec / DB.Query open a span around the driver call, attach
db.system=postgresql, db.operation, db.statement,
db.args.count, and record any error before End. nil tracer is
the default; the hot path emits a zero-allocation noopSpan, so
attaching the wiring to a global DB carries no cost when
tracing is off.
The attribute keys are exported as constants (AttrStatement,
AttrArgsCount, AttrSystem, AttrOperation, AttrSystemPG) so
downstream dashboards and alerts can match deterministically.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
pg.MermaidDiagram(schema) renders an erDiagram block from the
table + column + relation metadata. Plug the output into any
Mermaid renderer (mermaid.live, GitHub markdown, VS Code) to
keep the schema diagram up to date without manual upkeep:
fmt.Println(pg.MermaidDiagram(pg.NewSchema(Users, Posts)))
Cardinality marks follow standard Mermaid notation:
HasMany → ||--o{, HasOne → ||--o|, ManyToMany → }o--o{,
MorphTo → }o--|| per registered morph entry. BelongsTo is
suppressed because it is the inverse of HasMany and would
duplicate the line.
pg.Seeder accumulates fixture data and applies it in
declaration order, wrapped in a single transaction by default:
seeder := pg.NewSeeder(db)
pg.SeedAdd(seeder, UserEntity, alice, bob)
pg.SeedAddCreate(seeder, PostEntity, post1, post2) // populates PKs
pg.SeedDo(seeder, func(db *pg.DB, ctx context.Context) error {
_, err := db.Exec(ctx, "ANALYZE")
return err
})
seeder.Apply(ctx)
SeedAdd / SeedAddCreate / SeedDo are free functions because Go
disallows generic methods. WithoutTransaction() opts out when
the surrounding migration tool wants to own the tx boundaries.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
The outbox pattern solves "publish only if the DB write committed"
by writing the event into a co-resident table inside the same
transaction; a separate worker drains the table and publishes.
ob := pg.NewOutbox(db, "outbox")
// Emit inside the business transaction
db.InTx(ctx, func(tx *pg.DB) error {
if err := UserEntity.Create(tx, ctx, &u); err != nil { return err }
return ob.Emit(tx, ctx, "user.created", u)
})
// Drain in a worker
worker := pg.NewOutboxWorker(ob).
WithInterval(time.Second).
OnEvent(func(ctx context.Context, e pg.OutboxEvent) error {
return publisher.Publish(ctx, e.Kind, e.Payload)
})
go worker.Run(ctx)
pg.NewOutboxTable("outbox") returns the canonical table definition
(BigSerial id, kind, jsonb payload, createdAt, publishedAt) so
the migrator / Push manages the DDL.
Drain uses SELECT ... FOR UPDATE SKIP LOCKED, so multiple workers
can drain in parallel without collisions. Handler returning nil
marks the row published; returning an error leaves the row in
the queue for the next tick. OnError forwards drain / mark
failures to the host's logger.
Payloads are encoded as json.RawMessage so the wire format is
controllable by the caller (RawMessage passes through unchanged).
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
The bind/scan generator goes struct → helpers; introspect goes
the other way: snapshot.json → annotated Go structs ready for
AutoEntity.
dropsgen -snapshot meta/0001_snapshot.json -out models/
For each table the generator emits a <name>_drops.go file with
a struct whose fields carry the full drop: tag — primaryKey,
autoIncrement, notNull, unique, default=... — derived from the
snapshot's columns and single-column unique constraints.
SQL → Go type mapping covers the canonical set (bigserial→int64,
text→string, timestamptz→time.Time, jsonb→json.RawMessage, etc.)
with a string fallback for exotic types the user can edit by
hand. Identifier folding preserves acronyms (id→ID, http→HTTP)
so generated names read naturally.
Onboarding a legacy database is now: introspect the live DB
(pg.IntrospectSchema) → snapshot.json → dropsgen introspect →
models/. Drop into your app, point AutoEntity at the structs, go.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
drops is the operational counterpart of cmd/dropsgen: where
dropsgen generates Go code, drops operates on the snapshot
JSON itself — offline tooling that needs neither a DB
connection nor running user code.
drops diagram --snapshot meta/0001_snapshot.json --out schema.mmd
drops version
The diagram subcommand parses a drops snapshot v7 file and
emits a Mermaid `erDiagram` block: tables become entities,
foreign keys become parent ||--o{ child edges, primary-key
columns carry the PK marker. Pipe the result through any
Mermaid renderer (mermaid.live, GitHub, VS Code) for an
always-current ER diagram of the actual schema.
cmd/dropsgen stays the canonical home for codegen
(bind/scan, struct introspection). The two binaries divide
the surface cleanly: dropsgen → Go code; drops → schema
artefacts.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
db.WithRetry(policy) wraps InTx with a retry loop that catches
transient failures and re-runs the callback up to
policy.MaxAttempts times. Standard production wiring:
db := pg.New(drv).WithRetry(pg.DefaultRetryPolicy())
DefaultRetryPolicy retries on ErrSerializationFailure and
ErrDeadlock with 3 attempts and 10ms-base exponential backoff
capped at 1s. For custom behaviour, build the policy explicitly:
db := pg.New(drv).WithRetry(pg.RetryPolicy{
MaxAttempts: 5,
Errors: []error{pg.ErrSerializationFailure},
Backoff: pg.ExponentialJitter(20*time.Millisecond, 2*time.Second),
})
The retry is at the transaction boundary — the entire callback
re-runs inside a fresh transaction each time, so it must be
idempotent across retries. Side effects that shouldn't replay
(emails, HTTP) belong in the outbox so the rollback also rolls
them back.
Context cancellation short-circuits both the sleep between
attempts and the loop itself; the original ctx.Err() surfaces.
ExponentialJitter implements the standard formula:
base * 2^(attempt-1), clipped at max, plus [0, base) random
jitter so concurrent retries don't synchronise. Without a
policy InTx behaves exactly as before — additive change only.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
IdempotencyStore stamps a (response, completed) record per key so
retries of the same logical operation observe the original result
instead of executing again.
store := pg.NewIdempotencyStore(db, "idempotency_keys", 24*time.Hour)
raw, err := store.Run(ctx, requestKey, func(tx *pg.DB) ([]byte, error) {
if err := PaymentEntity.Create(tx, ctx, &p); err != nil {
return nil, err
}
return json.Marshal(map[string]int64{"paymentId": p.ID})
})
The standard Stripe / Bolt pattern: every POST that touches money
or emits events carries an Idempotency-Key header; the server
runs the body once and caches the response keyed by that header.
Concurrency is handled with INSERT ... ON CONFLICT (key) DO
NOTHING + SELECT ... FOR UPDATE inside the surrounding
transaction. Late arrivals queue behind the in-flight closure;
when the closure commits they observe the cached response. A
failed closure rolls back the claim so the next caller retries.
RunJSON[T] is the typed sugar: closure returns T, the store
handles marshalling. NewIdempotencyTable declares the canonical
DDL (key PK, response bytea, completed bool, createdAt,
expiresAt). Cleanup() reclaims expired rows; SweepEvery launches
a background sweeper.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
pg.Secret[T] wraps a value so it round-trips as encrypted bytea
through the driver. AutoTable recognises it and emits a bytea
column regardless of T:
type User struct {
ID int64 `drop:"id,primaryKey,autoIncrement"`
Email pg.Secret[string] `drop:"email"`
Card pg.Secret[Card] `drop:"card"` // any encodable type works
}
Encrypt-on-write (Value()) and decrypt-on-read (Scan()) use the
keyring registered via pg.SetKeyring. AESGCMKeyring is the
zero-dep default — AES-256-GCM with random 96-bit nonces
prepended to the ciphertext. For production, plug in your own
KMS-backed implementation of the Keyring interface; the wire
format is opaque to drops, so rotation is a SetKeyring swap
plus a re-encryption batch.
Wire format is AES-GCM(gob(T)), so any gob-encodable type works
out of the box: strings, integers, time.Time, even structs with
their own encoders. Tampered ciphertext fails to decrypt with a
non-nil error so a corrupted row never silently leaks plaintext.
The Keyring registry is process-global by design — column
encryption is rarely scoped per-DB. SetKeyring(nil) clears it;
all subsequent reads/writes return ErrNoKeyring until the next
SetKeyring.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Two-step flow that makes the difference between an ORM you can
deploy in a GDPR jurisdiction and one you can't:
1. Mark the column. Either at column-build time:
pg.Add(Users, pg.Text("email").NotNull().AsPII())
or via the drop tag option:
Email string `drop:"email,notNull,pii"`
2. Hooks / loggers / tracers see "<redacted>" instead of the
value — automatically. Entity binders wrap PII values in
pg.PIIParam before they reach the builder; db.Exec /
db.Query unwraps before calling the driver so the wire
form is unchanged, but the args slice retained for the
hook event keeps the markers.
The marker (piiArg) implements fmt.Formatter so every verb
(%v / %s / %q / %+v / %#v) prints "<redacted>". A defensive
log.Printf("%+v", args) can no longer leak the value.
For raw queries that don't have an entity column to tag, wrap
the literal manually:
db.Exec(ctx, "UPDATE users SET password = $1 WHERE id = $2",
pg.PII(hashedPw), id)
Composes with pg.Secret[T]: encrypted on disk, redacted in
logs / traces — the column is invisible end-to-end.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
pg.WithAudit(entity, auditLog) wires an entity so every mutation
writes an audit row in the SAME transaction as the business
write — a rollback rolls both back, required for any compliance
regime where the audit trail must be authoritative.
audit := pg.NewAuditLog(db, "audit_events")
pg.WithAudit(UserEntity, audit)
pg.WithAudit(PostEntity, audit)
ctx := pg.WithActor(ctx, currentUserID)
UserEntity.Update(db, ctx, &u)
// audit_events row: entity=users, op=update, pk=42,
// payload={...}, actor=currentUserID, createdAt=now()
NewAuditTable declares the canonical DDL: id, entity, op, pk
(jsonb), payload (jsonb), actor, createdAt. Op values are
"create" / "update" / "delete". Payload carries the post-RETURNING
row for create/update; delete only records the PK.
Actors are propagated via pg.WithActor(ctx, anything) — string,
int, or any value that fmt.%v prints sensibly. Absent actors
record as empty string.
Create / Update / Delete auto-wrap in InTx when audit is
attached, so the audit row and the mutation commit atomically.
Without audit attached the methods take the existing fast path
— zero overhead for entities that don't opt in.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Multi-tenant SaaS without explicit data isolation is a bug
factory: a single forgotten WHERE tenantId = $1 leaks rows across
customers. ScopeByTenant + WithTenant make the isolation a
property of the entity rather than the call site:
var Projects = pg.NewAutoEntity[Project]("projects").
ScopeByTenant(ProjectsTable.Col("tenantId"))
ctx = pg.WithTenant(ctx, currentTenant)
Projects.Get(db, ctx, id)
Projects.Query(db).Where(...).All(ctx)
Projects.Update(db, ctx, &p)
Projects.Delete(db, ctx, id)
// every call AND-s WHERE "tenantId" = $ctx-tenant
Forgetting WithTenant on the request ctx errors out with
ErrTenantMissing — the bad code path fails closed, not open.
Create stamps the tenant onto r automatically when its tenant
field is the zero value; a non-zero value that disagrees with
ctx returns ErrTenantMismatch, catching the "background job
inserted into the wrong tenant" class of bug at the seam.
Tenant-scoped entities skip the cache-by-PK fast path because
the cache key isn't tenant-aware; the tenant-aware SELECT runs
instead. Cache-aware variants land in a follow-up.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
PostgreSQL's LISTEN/NOTIFY is a built-in pub/sub channel that
nobody exposes well in Go. drops makes it first-class:
// Producer
pg.Notify(db, ctx, "invoice_paid", InvoicePaidEvent{ID: 7})
// Consumer
ch, err := pg.Listen[InvoicePaidEvent](db, ctx, "invoice_paid")
for ev := range ch {
// push WebSocket, refresh dashboard, kick a job
}
Notify uses pg_notify($1, $2) so the channel is parameterised
and the payload travels as a parameter — no string
concatenation, no injection. Payloads are JSON-encoded on the
producer and decoded into T on the consumer; malformed
messages are dropped (a misaligned producer is a deployment
bug, not a per-message error).
LISTEN requires a sticky connection, so drops dispatches via
the Listener interface — same duck-typing pattern as Copier.
Wire pgx's PgConn.WaitForNotification or lib/pq's
pq.Listener into an adapter; drivers without a notify path
return ErrListenNotSupported and the caller falls back to
polling.
Combined with the existing outbox, drops now has the
ingredients for a Kafka-less real-time fan-out for small /
mid-scale services: write a row → outbox → NOTIFY → typed
channel on the consumer side.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
PostgreSQL advisory locks let N replicas elect a single
executor without Redis, etcd, or Zookeeper. drops makes them
ergonomic:
// Blocking — every member runs the work eventually
err := pg.WithAdvisoryLock(db, ctx, "queue-drain", func(tx *pg.DB) error {
return drainQueue(tx, ctx)
})
// Non-blocking — only one member runs per tick
err := pg.TryWithAdvisoryLock(db, ctx, "nightly-report", func(tx *pg.DB) error {
return generateReport(tx, ctx)
})
if errors.Is(err, pg.ErrLockNotAcquired) {
// another replica took it; we're done
}
Both flavours use pg_advisory_xact_lock /
pg_try_advisory_xact_lock under the hood, so the lock is
released automatically when the wrapping transaction ends —
no leaked lock when the application crashes mid-run.
Keys are int64s in PostgreSQL; drops hashes string keys via
FNV-1a so callers write human-readable identifiers. The hash
function is exposed via AdvisoryLockKey for callers that need
to match a key generated outside drops.
Replaces an entire distributed-coordination service for cron
dedup, queue draining, and leader-elected background jobs.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
pg.NewSaga(name).Step(name, forward, compensate)... runs a
sequence of transactional steps where each step commits its
own tx. If step N fails, drops invokes the compensations of
steps [0, N-1] in reverse order — the canonical
"distributed transaction without 2PC" pattern.
saga := pg.NewSaga("checkout").
Step("charge", chargeFn, refundFn).
Step("ship", shipFn, cancelShipmentFn).
Step("email", emailFn, nil) // no comp needed
state := &pg.SagaState{}
state.Set("orderId", orderID)
if err := saga.Run(db, ctx, state); err != nil {
var se *pg.SagaError
if errors.As(err, &se) {
// earlier steps have been compensated
log.Warn("saga failed", "step", se.FailedStep,
"comp_failures", se.CompFailures)
}
}
State flows between steps as a typed key/value bag —
pg.SagaStateGet[T] for type-safe lookups. Compensations are
best-effort: a compensation failure is recorded in
SagaError.CompFailures but does not stop the remaining
compensations from running. Compensations run with a detached
context (same shape as InTx rollback) so a cancelled parent
doesn't block cleanup.
The implementation is in-memory; durable saga state across
process crashes is a future commit. For the current shape,
combine with the outbox and idempotency keys to make each
step crash-resilient.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Float for money is a bug waiting to ship. pg.Money stores
amounts as a signed int64 of minor units (cents for USD,
eurocents for EUR, satoshi-units for BTC at exponent 8). All
arithmetic is integer; floats only enter MulRate (taxes,
fees, interest) and the result is rounded half-to-even
(banker's rounding) to keep bias out of long sums.
type Payment struct {
ID int64 `drop:"id,primaryKey,autoIncrement"`
Amount pg.Money `drop:"amount,notNull"`
}
p := Payment{Amount: pg.MoneyFromString("12.34")}
p.Amount = p.Amount.
Add(pg.MoneyFromCents(50)).
MulRate(1.08) // apply 8% tax with banker's rounding
AutoTable maps pg.Money → bigint. driver.Valuer / sql.Scanner
serialise as int64; JSON renders as a string ("12.34") so
JavaScript clients don't truncate values > 2^53. Exponent
defaults to 2; override for non-2-decimal currencies via
MoneyWithExponent. Add / Sub / Compare panic on exponent
mismatch — a 12.34 (2 places) accidentally added to 12.345 (3
places) would corrupt totals silently otherwise.
Real fintech with sub-cent precision over many rate ops still
wants a dedicated decimal library; Money is the right tool
when "exact integer cents" is the precision contract.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
dropsgen now compiles .sql files into typed Go functions. Each query is a directive-comment block parsed into signature, arguments, kind and result type: -- queries/users.sql -- name: GetUserByEmail(email string) :one User SELECT id, name, email FROM users WHERE email = $1; -- name: ListByOrg(orgID int64, limit int64) :many User SELECT id, name, email FROM users WHERE orgId = $1 LIMIT $2; -- name: DeactivateUser(id int64) :exec UPDATE users SET active = false WHERE id = $1; Compile: dropsgen -sql queries/ -out queries/ -pkg queries generates queries/queries_gen.go with: func GetUserByEmail(db *pg.DB, ctx context.Context, email string) (User, error) func ListByOrg(db *pg.DB, ctx context.Context, orgID, limit int64) ([]User, error) func DeactivateUser(db *pg.DB, ctx context.Context, id int64) (drops.Result, error) The functions route through db.Query / db.Exec and use pg.ScanOne / pg.ScanAll (newly exported for code generators) for the reflection-based scan. Result types live in the user's package — pair with `dropsgen introspect` when you need the struct emission too. :one returns ResultType, :many returns []ResultType, :exec returns drops.Result. Files are gofmt'd; functions are emitted in alphabetical order for deterministic diffs. Composes naturally with the runtime — generated code is just typed wrappers around db.Query, so cache / tracer / hooks all fire as they would for ad-hoc queries. https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
The companion to Replicated: where Replicated splits reads
across replicas of one dataset, Sharded splits the dataset
itself across N independent primaries. Implements drops.Driver
so wiring is unchanged from the rest of the runtime.
shards := []drops.Driver{db1, db2, db3, db4}
sharded := pg.NewSharded(shards, func(key any) int {
return int(key.(int64) % int64(len(shards)))
})
db := pg.New(sharded)
ctx = pg.WithShardKey(ctx, userID)
got, err := UserEntity.Get(db, ctx, userID)
// routes to userID's shard
Missing shard key returns ErrShardKeyMissing — silently
defaulting to shard 0 would silently leak data across
customers, so the bad code path fails closed. Begin opens the
tx on the ctx's shard and stays there; drops does not model
cross-shard 2PC.
ForEachShard is the explicit scatter for admin tasks and
cross-shard reads, returning a per-shard error slice the
caller aggregates. HashShardKey provides a default picker for
string / int keys (FNV-1a + modulo).
Replaces an entire Vitess-style sharding layer for use cases
where the partitioning function fits in one Go func — typical
B2B SaaS sharded by tenant, social network sharded by user,
ride-hailing sharded by region.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
drops gains first-class geographic types so Bolt-style dispatch
("nearest 10 drivers"), Tesla vehicle telemetry, and any
location-aware service can stay in the entity API instead of
dropping to raw SQL.
type Driver struct {
ID int64 `drop:"id,primaryKey,autoIncrement"`
Position pg.Point `drop:"position,notNull"`
}
// AutoTable maps Point → geography(Point,4326)
nearby, _ := DriverEntity.Query(db).
Where(pg.Within(DriverPos, pg.Box{
SW: pg.Point{Lat: 41.85, Lon: 12.40},
NE: pg.Point{Lat: 41.95, Lon: 12.55},
})).
Where(pg.WithinRadius(DriverPos, userLoc, 1500)).
OrderBy(pg.NearestFrom(DriverPos, userLoc)).
Limit(10).
All(ctx)
// ORDER BY position <-> '...'::geography uses the KNN index
Point.Value emits SRID-tagged WKT; Point.Scan accepts both WKT
and the standard EWKB hex form most drivers return on a
geography column (little-endian + optional SRID flag, the
PostGIS default). Drivers that pre-parse geometry into custom
types are unsupported — wrap in ST_AsText if needed.
SQL helpers:
Within(col, Box) -> ST_Within(col::geometry, ST_MakeEnvelope(...))
WithinRadius(col, p, metres) -> ST_DWithin(col, p::geography, metres)
DistanceFrom(col, p) -> ST_Distance(col, p::geography)
NearestFrom(col, p) -> col <-> p::geography (KNN-index friendly)
For ST_Intersects / ST_Buffer / fancier shapes, drops.Raw +
the geography column type lets PostGIS resolve naturally — the
helpers above cover the high-frequency 80%.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
High-traffic counters (tweet likes, view counts, engagement
scores) hit the same row from thousands of goroutines.
"SELECT count, UPDATE count = count + 1" round-trips and races;
"UPDATE count = count + 1" inline is correct but tedious to
write through Entity (which wants a struct).
Patch sidesteps both:
TweetEntity.Patch(db, ctx, tweetID,
pg.Inc(TweetLikes, 1),
pg.Inc(TweetEngagement, 1),
pg.SetIfGreater(TweetMaxScore, currentScore),
pg.SetIfChanged(TweetSlug, newSlug),
)
// UPDATE "tweets" SET
// "likes" = "tweets"."likes" + $1,
// "engagement" = "tweets"."engagement" + $2,
// "maxScore" = GREATEST("tweets"."maxScore", $3),
// "slug" = CASE WHEN "tweets"."slug" IS DISTINCT FROM $4 THEN $5 ELSE "tweets"."slug" END
// WHERE "id" = $6
// AND <tenant predicate>
// AND <guard predicate>
All ops are SQL-side: concurrent patches against the same row
serialise on the row lock, no lost updates. The PK cache (when
attached) is invalidated post-patch because the server computes
the new value and we don't have it locally.
Op catalogue:
Inc / Dec — counter mutation (typed numeric)
Set — typed plain assignment
SetIfGreater — GREATEST(col, $1) — high-watermark
SetIfLess — LEAST(col, $1) — low-watermark
SetIfChanged — CASE WHEN IS DISTINCT FROM — no-op when
equal (triggers still fire on the row)
Honours tenant scope, authorisation guards, and the audit log
(opens an InTx wrapper when audit is attached). Returns
drops.Result so callers can read rows-affected to distinguish
"row missing" from "row updated".
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Time-series ingestion at scale demands partitioned tables —
without them the heap turns to soup within days. PostgreSQL has
native PARTITION BY RANGE; the operational burden is managing
the lifecycle. TimeSeriesTable encodes that pattern:
ts := pg.NewTimeSeriesTable("vehicle_events", "ts").
PartitionEvery(24 * time.Hour).
Retain(90 * 24 * time.Hour).
WithBrinIndex("ts")
// Once
ts.Bootstrap(db, ctx)
// Scheduled every hour
ts.Maintain(db, ctx) // creates upcoming + drops expired
Maintain calls EnsureNext(2) followed by DropExpired so a daily
partition is always present a day ahead of traffic, and any
partition older than retention is reclaimed. Bootstrap is
idempotent (CREATE TABLE IF NOT EXISTS) so re-running on
deployment is safe.
The parent table must be declared with PARTITION BY RANGE
(timeCol) — drops does not generate it because column
definitions belong to the user's schema. Partition discovery
for DropExpired uses pg_inherits + pg_class so it skips the
parent and any manually-managed siblings; names that don't
match the deterministic format ("<parent>_YYYYMMDD[_HH[MM]]")
are also skipped, so a human's "events_special" partition is
never dropped.
WithBrinIndex emits a BRIN index per partition — the canonical
time-series index: tiny, fast to build, great for range scans
on naturally-ordered columns. Tesla telemetry, WhatsApp message
logs, and X event streams all want this shape.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
SREs don't deploy databases without /metrics. drops exposes
pool health via PoolStatsProvider — any driver that
implements Stats() PoolStats gets read-through metrics pushed
to a sink at a chosen interval.
stop := db.StartPoolMetrics(ctx, 5*time.Second, func(s pg.PoolStats) {
prom.DropsPoolInUse.Set(float64(s.InUse))
prom.DropsPoolIdle.Set(float64(s.Idle))
prom.DropsPoolWaitDuration.Observe(s.WaitDuration.Seconds())
})
defer stop()
PoolStats fields mirror database/sql.DBStats (MaxOpenConnections,
OpenConnections, InUse, Idle, WaitCount, WaitDuration,
MaxIdleClosed, MaxIdleTimeClosed, MaxLifetimeClosed) so the
adapter is one line: the stdlib driver via the existing
drops/stdlib bridge satisfies it natively; pgx pools wrap their
own Stats() shape into the same struct.
StartPoolMetrics emits one snapshot immediately so the metric
pipeline shows current state without waiting for the first
tick, then ticks every interval until ctx is cancelled or the
returned stop() is called. nil sink is a no-op; unsupported
driver is also a no-op (no goroutine spawned), so attaching
the wiring globally costs nothing for non-pool drivers.
SupportsPoolStats is the predicate for code paths that want to
register metrics only when introspection is available.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Sweep the package comments and replace illustrative
"company X / Y / Z" examples with the technical scenarios they
were standing in for. The functionality is unchanged; only the
prose loses the brand callouts.
Changes:
pg/idempotency.go — "Bolt / Stripe call these" →
generic "payment APIs"
pg/shard.go — drops the named shard-axis examples,
keeps the technical (user / chat /
region) summary
pg/timeseries.go — "Tesla / WhatsApp / X" → generic
"IoT telemetry, message logs, event
streams"
pg/geo.go — "Bolt-style dispatch" → "Dispatch query"
pg/patch.go — "tweet likes" → "like counts"; example
uses PostEntity / posts table
pg/patch_test.go — table fixture renamed tweets → posts
pg/diagram.go — "GitHub markdown, VS Code" → generic
"Markdown viewers, IDE previews"
References to the OSS project drizzle-kit stay — that's
compatibility credit, not a brand callout. Test fixtures that
happen to model real-world domains (payments, posts, users)
also stay; they're generic enough.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
DiffDown(prev, cur, opts) returns the SQL that reverses the
migration from cur back to prev — semantically equivalent to
Diff(cur, prev, opts) but provided as a distinct entry point so
generated migration sets can carry both directions without the
caller having to swap arguments.
GenerateMigration gains a WithDown option that emits a paired
<tag>.down.sql file alongside the up SQL:
res, err := pg.GenerateMigration(pg.GenerateOptions{
Schema: currentSchema,
Dir: "migrations",
WithDown: true,
})
// Writes:
// migrations/0003_warm_iron.sql
// migrations/0003_warm_iron.down.sql
The down direction is best-effort: column ADD ↔ DROP, type ↔
type swap, NOT NULL ↔ DROP NOT NULL, constraints ADD ↔ DROP all
roundtrip cleanly. DROP COLUMN can never be reversed losslessly
because the data is gone — operators should review generated
down scripts before relying on them in prod.
The naming convention matches the existing Migrator's
expectations (<tag>.down.sql), so the native pg.Migrator picks
up the rollback file automatically; DrizzleMigrator continues
to ignore it (drizzle-kit has no down direction natively).
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
drizzle's "with: { posts: { limit: 5 } }" shape — each parent
receives at most N children — now ships natively. The relation
loader rewrites the child query as a ROW_NUMBER() window over
the join key so the cap applies per parent, not globally.
db.Find(Users).
WithRel("posts", func(c *pg.RelConfig) {
c.OrderBy(PostsTable.Col("createdAt").Desc()).
Limit(5).
Offset(10) // optional skip
}).
All(ctx, &users)
Generated SQL:
SELECT * FROM (
SELECT <cols>, ROW_NUMBER() OVER (
PARTITION BY "posts"."userId"
ORDER BY "posts"."createdAt" DESC
) AS _rn
FROM "posts"
WHERE "posts"."userId" IN ($1, $2, ...) [AND <user wheres>]
) AS _ranked
WHERE _rn > $offset AND _rn <= $offset + $limit
Applies to HasMany, MorphMany (the multi-row relations);
HasOne / BelongsTo already cap at one. Without Limit the
original IN-list path runs unchanged — no perf regression for
existing call sites. The default ORDER BY (when OrderBy isn't
configured) falls back to the child's primary key so the
window partition is stable.
The extra _rn projection column is harmless: the row scanner
discards unmatched columns, so it doesn't leak into the
target struct.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
constraints to the snapshot+diff generator
The Snapshot machinery now models four table-scoped features
that previously sat as map[string]any placeholders:
- Indexes — captured from Table.AddIndex; diff emits CREATE
INDEX / DROP INDEX (and drop+recreate when an existing
index's shape changes — uniqueness, method, columns,
WHERE).
- Composite primary keys — declared via the new
Table.PrimaryKey(cols...) method (single-column PKs
continue to live on the column). Diff emits ALTER TABLE
ADD/DROP CONSTRAINT ... PRIMARY KEY.
- Composite uniques — declared via the new
Table.AddUnique(name, cols...) method. Reuses the existing
UniqueSnapshot type (which already supports multi-column
Columns []string) and rides the existing diff path.
- CHECK constraints — declared via the new
Table.AddCheck(name, expr) method. Diff emits ALTER TABLE
ADD/DROP CONSTRAINT ... CHECK.
Typed snapshot structs (IndexSnapshot, CompositePKSnapshot,
CheckSnapshot) replace the old map[string]any placeholders so
JSON round-trips, diff comparisons, and introspection all
share one source of truth.
New-table emission stays clean: columns + inline UNIQUE
render via CREATE TABLE; composite PKs and CHECK constraints
follow as ALTER TABLE statements right after, so the order
stays cross-table-safe and matches drizzle-kit's PG snapshot
layout.
Enums, sequences, views and RLS policies are next.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL
Schema-level objects (pg.Schema): - AddEnum / AddSequence / AddView register typed descriptors alongside tables - Diff emits CREATE TYPE ... AS ENUM (idempotent via DO/EXCEPTION block), ALTER TYPE ... ADD VALUE for forward-only enum evolution, DROP TYPE - CREATE / DROP SEQUENCE honour Start / Increment / Min / Max / Cache / Cycle - Regular views diff via CREATE OR REPLACE VIEW; materialised views drop + recreate because PG can't alter the SELECT in place Table-level RLS (pg.Table): - EnableRLS toggles ALTER TABLE ... ENABLE/DISABLE ROW LEVEL SECURITY - AddPolicy + pg.NewPolicy builder cover PERMISSIVE/RESTRICTIVE, FOR command, TO roles, USING and WITH CHECK expressions - Diff emits CREATE / DROP POLICY and drop+recreate when the expression or scope changes (PG has no ALTER POLICY ... USING) Snapshot JSON preserves drizzle-kit v7 field names (isRLSEnabled, policies, enums, sequences, views) so external tooling keeps round-tripping.
…keup
Production-grade transactional outbox for game / real-time backends.
Schema (NewOutboxTable):
- aggregateType / aggregateID for per-aggregate ordering scope
- headers jsonb for tracing context propagation
- availableAt / attempts / lastError / failedAt for retry handling
- Partial indexes on (availableAt,id) and (aggregateType,aggregateID,id)
WHERE publishedAt IS NULL AND failedAt IS NULL — drain stays O(log n)
even with millions of published rows pending cleanup
API:
- EmitWith(tx, ctx, kind, payload, EmitOptions{AggregateType, AggregateID,
Headers}) — extended emit; Emit is the zero-options shortcut
- WithNotifyChannel(ch) — Emit also pg_notify; worker LISTENs for
sub-second wakeup, falls back to polling when driver lacks Listener
- DrainAggregate / PendingAggregates for per-aggregate workers
- MarkFailed(id, attempts, nextRetryAt, lastErr) — zero nextRetryAt
parks the row as terminally failed
- Cleanup(retainAfter) — janitor for published rows; failed rows are
kept as poison-message audit log
Worker:
- WithMaxAttempts(n) + WithBackoff(fn) — exponential-jitter default
- WithOrdering(OrderingPerAggregate) — advisory-lock-keyed processing
so events sharing (AggregateType, AggregateID) are delivered in
emission order across a parallel worker pool. Events without an
aggregate fall through to the unordered drain.
- OnBatch(fn) alternative to OnEvent — single call per drain batch
for brokers with high per-publish overhead (Kafka producer)
… sets Keyset / cursor pagination is the only correct way to page deep into a result set; OFFSET scans every row up to the offset so the 1000th page costs 1000× more than the first. API: - CursorSpec + OrderKey declare the (column, direction, nulls) shape - OrderByCursor(spec) wires the ORDER BY in lock-step with the spec - AfterCursor(spec, c) / BeforeCursor(spec, c) append the row-wise expansion of "(k1, ..., kn) > (v1, ..., vn)" with per-key direction honoured — handles mixed ASC/DESC cursors that the simple tuple comparator can't express - EncodeCursor returns base64 URL-safe opaque blobs safe to ship in query strings; the on-wire format is typed JSON so int64 / time.Time / []byte round-trip to their original Go types instead of float64 / string defaults - Empty cursor is a no-op (first page); malformed cursor degrades to WHERE FALSE so the caller sees an empty result instead of a query that silently fans out
Factory[T] wraps Entity[T] with a template callback that receives a monotonic sequence so generated identifiers stay distinct without extra bookkeeping. Build / BuildN produce values without touching the database; Create / CreateN insert via Entity (CreateN batches into a single INSERT for cheap fixture seeding). With(mutate) spins off variant factories — admins, banned players, expired sessions — that share the parent's sequence counter so identifiers remain unique across the family. Reset / Sequence expose the counter for test setup boundaries; the counter is atomic so factories are safe to share across goroutines.
Static rules over generated migration SQL catch the most common foot-guns before they hit production. Pass GenerateResult.SQL through AnalyzeMigration; iterate the SafetyWarning slice for human-readable findings, severity, and a remediation suggestion. Rules (severity in parens): - add-not-null-column-without-default (error) — table rewrite + lock - add-not-null-column-with-volatile-default (error) — rewrites every row - alter-column-type (warn) — typical full-table rewrite - alter-column-set-not-null (warn) — full table scan under ACCESS EXCL - create-index-not-concurrent (warn) — blocks writes; suppressed when the index is inside a fresh CREATE TABLE - drop-table (error) — irreversible - drop-column (warn) — irreversible + breaks rolling deploys - alter-type-drop-value (error) — PG cannot do this at all - rename-column / rename-table (warn) — breaks rolling deploys - truncate-table (error) — bypasses triggers, irreversible SafetyOptions.Ignore suppresses individual rules by ID for known-safe migrations (small tables, scheduled downtime).
Single biggest visibility gap in ORMs is the moment a plan flips overnight (index dropped, statistics stale, distribution shifted) and P99 quietly doubles. drops captures EXPLAIN as structured nodes, derives a stable fingerprint over plan shape — operators, relations, indexes, join types, without cost / timing noise — and surfaces diffs so plan regressions ring instead of paging. API: - Explain(db, ctx, sql, args...) — planner-only, no execution - ExplainWith(opts) — Analyze / Buffers / Verbose toggles - ExplainPlan.Fingerprint() — stable hash for change detection - ExplainPlan.SeqScans / UsedIndexes / JoinTypes — derived shortcuts - DiffPlans(before, after) — SeqScansAdded/Removed, IndexesAdded/Removed, CostDelta, RowsDelta + Same shortcut The fingerprint deliberately ignores costs and rows so equivalent plans always hash the same across runs — change detection sees shape, not noise. Persist Fingerprint() next to the query when recording slow-query samples; compare on every capture.
The time-based stickiness on Replicated unconditionally pins every read to the primary for the configured window — correct but wasteful. WithLSNTracking captures pg_current_wal_lsn() at write time and routes follow-up reads to the first replica that has replayed past that point, only falling back to primary when none have caught up. API: - Replicated.WithLSNTracking(ttl) — opt in; TTL caps cached replica LSN samples - ParseLSN / FormatLSN — convert PG's "X/Y" hex pair to / from the underlying byte position - rywState carries writeLSN alongside the existing time window so the time-based fallback still works when LSN queries fail
The only safe way to mutate hundred-million-row tables without downtime is to walk them in chunks under throttle, persist progress so a crash resumes from the last commit, and (optionally) back off when replication lag widens. drops drives the loop; the caller supplies the Fetch (next batch of IDs) and Process (per-chunk work) callbacks. Each chunk runs inside a transaction so partial failures roll back cleanly. State persists between chunks to a backfillJobs table so a crash resumes from the last successful commit. Already-completed jobs skip the loop entirely — Run is idempotent. API: - NewBackfill(db, name) — name is the unique state key for resume - ChunkSize / Throttle — load shaping - Fetch(fn) — return next batch + new lastID - Process(fn) — per-chunk work inside a transaction - OnProgress(fn) — progress callback for logging / ETA - PauseIfLag(repl, bytes) — block before each chunk while replica lag exceeds threshold; reuses Replicated's LSN plumbing - Status / Reset for operational queries - NewBackfillStateTable declares the canonical schema
InstallChangeFeed emits the DDL for a PL/pgSQL AFTER INSERT/UPDATE/
DELETE trigger that calls pg_notify with {op, id}. The payload stays
small (well under PG's 8KB cap) and is the same regardless of row
width — wide tables don't blow up the channel.
Subscribe[T] is the typed consumer: opens LISTEN on the channel via
the driver's Listener implementation, decodes each notification into
a ChangeEvent, and optionally hydrates insert/update events with the
fetched row body via Entity.Get. Delete events carry only the ID
since the row is already gone.
Drivers without Listener support fail subscribe with the existing
ErrListenNotSupported. The returned channel closes when ctx ends or
the underlying LISTEN terminates. Events are dropped (not buffered
unboundedly) when a slow consumer can't keep up.
Single-column PKs only for v1; composite PKs would need a different
payload shape and are not yet supported.
Append-only log of domain events per aggregate with optimistic concurrency on (aggregateType, aggregateID, version). Pairs naturally with the outbox for transactional publishing and the change feed for projection fan-out. API: - NewEventStoreTable / NewSnapshotTable declare the canonical schemas; the (type, id, version) unique index is what enforces concurrency at commit time - EventStore.Append(tx, type, id, expectedVersion, events...) — optimistic concurrency; unique-violation surfaces as ErrConcurrencyConflict so callers branch cleanly on the retry path instead of pattern-matching driver errors - Load / LatestVersion for replay; Stream for projection workers that watermark on the store-wide append offset - SaveSnapshot / LoadSnapshot persist the materialised state so reads avoid replaying from the beginning of time on large aggregates Events are appended in batches inside a transaction so partial failures roll back, and headers carry tracing context the same way the outbox does so observability flows from emitter to projection.
EnvelopeCipher does AES-256-GCM with a fresh per-call data encryption key (DEK) wrapped by the configured KMS. Each ciphertext embeds its own wrapped DEK so key rotation only re-wraps DEKs; existing ciphertexts don't need rewriting. The wire format is fixed-prefix versioned: [magic | wrappedDEK length (uint32 BE) | wrappedDEK | nonce | ct+tag] so the decryptor can split without scanning, and future format revisions can coexist with old data. KMS is the user-supplied wrap / unwrap contract — no SDK dependencies in drops. LocalKMS ships as the dev / test path: a 32-byte master key in process memory, plenty for tests, never appropriate for production. Tampering surfaces as AEAD-tag failure; KMS errors propagate without obfuscation so observability stays straight.
DetectDrift takes the repo's canonical Snapshot and a live introspection, and returns a two-way diff so CI can fail on both directions of drift: - PendingMigrations: statements that would bring live UP TO repo — non-empty means production hasn't applied every change in the repo yet - UnauthorizedChanges: statements that would bring repo UP TO live — non-empty means someone applied DDL directly without going through the repo. Headline alert. InSync is the boolean shortcut both lists are empty. Pair with Introspect (live PG → Snapshot) in CI to gate merges on a clean diff in either direction.
Register each materialised view with its upstream dependencies and
an optional periodic interval; drops drives the refresh order so a
view only ever rebuilds after its inputs have caught up.
API:
- NewMatViewManager(db) — empty manager
- Add(MatView{Name, DependsOn, Mode, Every}) — register a view
- Refresh(ctx, name) — single view
- RefreshDownstream(ctx, upstream) — every view transitively
dependent on upstream, in topological order; used when a base
table changes and you want the cascade
- RefreshAll(ctx) — full graph in topological order
- Start(ctx) — background scheduler honouring per-view Every
- WithPollInterval(d) — tune the scheduler tick (defaults 250ms)
- Mode picks REFRESH vs REFRESH CONCURRENTLY; the latter avoids
the exclusive lock at the cost of requiring a UNIQUE index on
the view
Schema views can now be declared with the same typed SelectBuilder
that drives runtime queries instead of hand-written SQL strings:
pg.View("activeUsers").As(
db.Select(Users.ID, Users.Name).
From(Users).
Where(Users.Active.Eq(true)))
pg.MaterializedView("playerStats").As(query)
Parameter placeholders ($1, $2, ...) are inlined as SQL literals
since CREATE VIEW doesn't bind parameters at install time. Common
types (string with quote-doubling, int / float / bool, time.Time,
[]byte) are supported; unsupported types panic at declaration so
the schema fails loudly at startup.
NewView(name, def) and NewMaterializedView(name, def) remain the
raw-SQL fast path. AsSQL on the builder is the escape hatch for
SELECT shapes (recursive CTEs, dialect-specific functions) the
builder doesn't cover.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Templates are plain functions that register column groups on a table
and return a struct of typed *Col[T] handles, keeping schema
declarations DRY without giving up type safety. Ships Timestamps,
SoftDelete, Audit[T], and UUIDPrimaryKey for both dialects; external
libraries follow the same recipe to expose their own.
https://claude.ai/code/session_01QT2i7GCtK7zn6cd4kPH7gL