pg-lakebase

A lakehouse database built on PostgreSQL

pg-lakebase is building a lakehouse database on PostgreSQL. The goal is to bring PostgreSQL's SQL interface, transaction model, and ecosystem to open lakehouse table formats.

The long-term vision includes first-class support for the Apache Iceberg, Delta Lake, and Apache Hudi lakehouse table formats.

Beyond table formats, one future goal is to add time-series database capabilities to pg-lakebase, using lake tables as the storage foundation for ingesting, managing, and analyzing time-series data at scale. Another future goal is to support vector data and exact and approximate nearest-neighbor search through the PostgreSQL SQL interface, with lake tables in object storage as the durable data layer. Neither the time-series nor vector capabilities are implemented today.

Apache Iceberg is the first and currently the only implemented lakehouse table format. The other formats and capabilities are planned product directions, not current capabilities.

Warning

This project is under active development and is not recommended for production workloads. pg-iceberg-am is the primary runnable extension. pg-delta-am is only a framework skeleton, not a Delta Lake implementation.

Why pg-lakebase?

• PostgreSQL as the database interface. Use ordinary PostgreSQL SQL, transactions, drivers, and tools instead of adopting a separate interface for lakehouse data.
• Writable lakehouse tables. Work with Iceberg-backed PostgreSQL tables using common DML, transaction control, and supported schema changes, rather than treating lakehouse data as read-only external files.
• Local and object storage. Keep Iceberg metadata and Parquet data on the local filesystem for simple deployments, or place object-backed tables in S3-compatible object storage through storage volumes while using the same PostgreSQL SQL interface. GCS and Azure providers are experimental.
• A unified database direction. Build from the current Iceberg support toward additional open lakehouse formats, time-series database capabilities backed by lake tables, and vector search with lake tables in object storage as the durable data layer.

Current Iceberg capabilities

The following capabilities are backed by implementation and regression or isolation tests in this repository. They do not imply complete coverage of every Iceberg specification feature or cross-engine interoperability. Format-version terminology follows the Apache Iceberg table specification, and isolation terminology follows the PostgreSQL 17 transaction isolation documentation.

Area	What works today	Current boundary
Iceberg format versions	Create Iceberg v1, v2, and v3 tables	Feature coverage varies by version. This is not a blanket claim that every feature in each specification is implemented; for example, row-level UPDATE and DELETE are rejected for v1 tables.
SQL operations	SELECT, INSERT, UPDATE, DELETE, MERGE, and COPY	Row-level changes use Iceberg delete semantics and therefore depend on the selected format version.
Transactions and isolation	Statement-level Iceberg metadata visibility under READ COMMITTED, read-your-own-writes, commit, rollback, and savepoints	SERIALIZABLE currently strengthens Iceberg write-conflict validation but does not yet provide full PostgreSQL SSI semantics. REPEATABLE READ is not supported.
Schema evolution	ADD COLUMN, DROP COLUMN, RENAME COLUMN, and DROP NOT NULL	Other ALTER TABLE schema changes are rejected.
Storage	Local filesystem and S3-compatible object storage through storage volumes	S3-compatible storage has repository end-to-end coverage. GCS and Azure providers exist but remain experimental.
Partitioned tables	PostgreSQL declarative partitioning with partition-routed INSERT, UPDATE, DELETE, MERGE, and COPY	Each Iceberg leaf is managed as its own relation.
Scan optimization	Predicate pushdown plus Iceberg file and Parquet row-group pruning for supported expressions	PostgreSQL retains residual predicates when required for correctness; some comparisons are deliberately not pushed when semantics could differ.
Maintenance	VACUUM, VACUUM FULL, and scheduled automatic Iceberg maintenance	Maintenance remains subject to operational limits while the project is under development.

External Iceberg catalogs and interoperability validation with engines such as Spark, Flink, and Trino are planned. The current SQL-facing implementation uses a PostgreSQL-backed Iceberg metadata catalog.

Quick start

The current quick start builds the extensions from source. See Build from source for pgrx setup details and the full installation variants.

Initialize PostgreSQL 17 with pgrx using an existing pg_config:

cargo pgrx init --pg17=/path/to/pg_config

Install the shared runtime and the Iceberg access method into that PostgreSQL installation:

cargo pgrx install --package pg-lakebase-runtime --pg-config /path/to/pg_config
cargo pgrx install --package pg-iceberg-am --pg-config /path/to/pg_config

pg-iceberg-am depends on pg-lakebase-runtime. cargo pgrx install installs the package named by --package, so both commands are required; installing the access method does not install the runtime artifacts.

Add both extensions to postgresql.conf and restart PostgreSQL:

shared_preload_libraries = 'pg_lakebase_runtime,pg_iceberg_am'

Then connect to a database and run:

CREATE EXTENSION IF NOT EXISTS pg_lakebase_runtime;
CREATE EXTENSION IF NOT EXISTS pg_iceberg_am;

CREATE TABLE events (
    event_time  timestamptz NOT NULL,
    device_id   bigint      NOT NULL,
    temperature double precision
) USING iceberg;

BEGIN;

INSERT INTO events VALUES
    (now(), 101, 21.5),
    (now(), 102, 22.0);

UPDATE events
SET temperature = 22.1
WHERE device_id = 101;

COMMIT;

SELECT *
FROM events
WHERE device_id = 101;

This creates an Iceberg-backed table that applications can access through ordinary PostgreSQL SQL.

Use object storage

Object-backed Iceberg tables are configured through a storage volume and a PostgreSQL tablespace. Storage-volume administration requires a superuser and is a nontransactional operation. Invoke the administration function as the only expression in a standalone top-level SELECT; do not call it from an explicit transaction, function, procedure, trigger, DO block, CTE, subquery, or pipelined batch.

The following example uses an S3 bucket and the provider's default credential chain. Replace the bucket, prefix, region, and local tablespace path for the deployment. The LOCATION directory must be an existing, empty absolute path that PostgreSQL can use for tablespace metadata.

SELECT lakebase.create_storage_volume(
    'events-lake',
    's3://my-lake-bucket/pg-lakebase',
    '{"type":"default_chain"}'::jsonb,
    '{"region":"us-east-1"}'::jsonb
);

CREATE TABLESPACE lake_s3
LOCATION '/path/to/local/tablespace'
WITH (lakebase_storage_volume = 'events-lake');

CREATE TABLE object_events (
    event_time timestamptz NOT NULL,
    device_id bigint NOT NULL,
    payload text
) USING iceberg TABLESPACE lake_s3;

The same storage-volume API includes experimental providers for gs:// locations in Google Cloud Storage and az:// locations in Azure Blob Storage. These providers do not yet have the same end-to-end test coverage as the S3-compatible path. Credentials and provider options are validated by the runtime and persisted in the PostgreSQL data directory's protected storage-volume configuration. They are not encrypted by PostgreSQL; use the deployment's credential and filesystem security controls.

Roadmap

Current — Reliable Iceberg tables

Make writable Iceberg tables reliable, interoperable, and straightforward to deploy from PostgreSQL, with stronger format coverage, object-storage reliability, compatibility testing, packaging, and performance validation.

Next — Broader lakehouse format support

Expand the database beyond Iceberg with first-class Delta Lake and Apache Hudi implementations while preserving a consistent PostgreSQL experience.

Future — Time-series and vector capabilities

Use lake tables as the storage foundation for time-series ingestion and analytics, and add vector data and similarity search through the PostgreSQL SQL interface, with lake tables in object storage as the durable data layer.

These roadmap items describe intended product outcomes. Their implementation designs will be documented separately as they are validated.

Architecture

At a high level, pg-lakebase integrates lakehouse table implementations with PostgreSQL's table access, planning, execution, and transaction lifecycle, then routes table data to local or object storage.

                 PostgreSQL SQL and transactions
                               |
                               v
             +--------------------------------------+
             | pg-iceberg-am                        |
             | Iceberg table access method          |
             | + custom scan paths                  |
             +------------------+-------------------+
                                |
              +-----------------+------------------+
              |                                    |
              v                                    v
     Local filesystem                         Object storage
                                            storage volumes
                                                   |
                                                   v
                                          pg-lakebase-storage
                                         S3 / GCS / Azure

• The PostgreSQL Table Access Method integration makes USING iceberg tables PostgreSQL relations rather than a separate query API.
• Custom scan paths push supported predicates into Iceberg scans for pruning while preserving PostgreSQL evaluation wherever required for correctness.
• Transaction-local state provides statement-consistent reads and stages data and schema changes until the PostgreSQL transaction boundary.
• The shared runtime and storage service route object-backed tables through configured storage volumes.

PostgreSQL support

PostgreSQL 17 is the only currently supported version. Support for PostgreSQL 16, 18, and 19 is planned.

Project components

• pg-iceberg-am is the current SQL-facing Iceberg table implementation.
• pg-lakebase-core provides the reusable PostgreSQL Table Access Method and CustomScan framework, lifecycle adapters, and transaction boundaries.
• pg-lakebase-runtime provides shared workers, runtime coordination, and storage-volume administration.
• pg-arrow-conv provides Arrow/PostgreSQL value conversion.
• iceberg-lite is the synchronous, PostgreSQL-oriented Iceberg library derived from iceberg-rust.
• pg-lakebase-storage provides the local cache and object-storage service used by object-backed tables.

iceberg-lite is adapted for PostgreSQL's synchronous execution model and custom I/O path. Changes to it should preserve a manageable merge path from the upstream iceberg-rust project.

Documentation and development

• Build from source
• Contributing and test commands
• pg-iceberg-am details
• pg-iceberg-am testing design
• pg-lakebase-core framework
• pg-lakebase-storage service
• iceberg-lite adaptation

License

This project is licensed under the Apache License 2.0. See LICENSE for details.