Added support for Postgres database (#1)

* New architecture WP 1

* New architecture WP 2

* WIP 3

* Optimized cloudsync_table_context interaction

* New architecture WP 4

* Refactored begin/commit ALTER

* New architecture WP 6

* New architecture (WP 5)

* Minor changes

* New architecture WP 6

* New architecture WP 7

* Small compilation issue fixed

* New architecture WP 9

* fix: minor compilation issue

* fix: database_* functions must call cloudsync_memory_* functions instead of direct sqlite3_* memory function to support the memory debugger module

call cloudsync_memory_mprintf and cloudsync_memory_free instead of direct sqlite3_mprintf and sqlite3_free memory functions from database_create_insert_trigger otherwise the memory debugger would report "Pointer being freed was not previously allocated."

* fix(memdebug): fix pointer returned by memdebug_zeroalloc

* fix(unittest): avoid a memory leak from do_test_dbutils

* test: add unittest target to Makefile

Introduces a 'unittest' target to run only unit tests and updates the help output accordingly.

* Update unit.c

* Refactored SQL in dbutils (related to settings)

* Updated static statements in cloudsync.c (WP 1)

* Replaced name escape with a function (WP 2)

* Completed SQL refactoring

* fix: minor compilation issue

* test: re-add integration test

use `make unittest` to run only the unittest

* fix(network): fix the network layer after database-api refactoring

* Several compilation warnings fixed

* Cleaned-up 64bit types

* fix(lcov): exclude sql_sqlite.c from code coverage, it just contains query strings

* Update cloudsync_sqlite.c

* Use int64_t for version variables in network.c

Replaces sqlite3_int64 with int64_t for new_db_version and new_seq variables to standardize integer type usage and improve portability. Was giving a compile error on linux musl

* Update vtab.c

* chore: remove warnings

* Update vtab.c

* test: fix compile errors on linux musl

* Replaced all %lld (except one)

* fix: minor compilation error

* refactor: new directory structure to separate multi-platform code from database-specific implementations

Refactor the codebase to separate multi-platform code from database-specific implementations, preparing for PostgreSQL extension development.
vtab.c/h has been renamed to sqlite/cloudsync_changes_sqlite.c/h

* fix(workflow): add CFLAGS for CURL Android builds and clean up Android build files

* fix(android): add -fPIC to CFLAGS

* fix(android): use OpenSSL specific version

* fix(android): update OpenSSL install path to a local one, instead of a system wide path

* db_version must be int64_t in network.c

* fix: avoid crash on postgres when these functions are called with NULL values for db and data

* improved error logs, fix some debug messages

* New postgresql extension WIP 1

* Updated SQL_DATA_VERSION and added a new sql_build_select_nonpk_by_pk function to database.h

* fix: remove obsolete property

* chore

* fix: improved SQL queries (WIP)

* implement SQL_SCHEMA_VERSION with app_schema_version table and event trigger

* fix: fix PG SQL queries used in cloudsync_init

* fix: avoid a segfault crash in cloudsync_init

Allocate in TopMemoryContext to survive SPI cleanup

* test: calling twice the cloudsync_init function on postgresql is failing (WIP)

* Code simplification and memory cleanup (wp)

* Minor fixes

* test: add debug PostgreSQL devcontainer and Docker setup

Introduces a VS Code devcontainer configuration for PostgreSQL development with CloudSync, including a debug Dockerfile, docker-compose file, and Makefile changes to support debug builds. This setup enables easier debugging and development of the CloudSync extension for PostgreSQL in a containerized environment.

* chore: update docker/README.md with VS Code Dev Container Debugging instructions

* test: add .vscode/launch.json with the "Attach to Postgres (gdb)" configuration

* Added more explicit string_dup functions

* Fixed network

* Error returned by sqlite3_extension_init function must be dynamically allocated

* test: add debug symbols and src code of postgresql server to dev container

* fix: add SPI_connect and SPI_finish to _PG_init function

* dbutils.c removed db_t

* Minor changes

* Refactoring (wp)

* Refactoring (wp)

* Refactoring (wp)

* Refactoring (wp)

* Refactoring (wp)

* Refactoring (wp)

* Refactoring (wp)

Removed cloudsync_private.h and db_t

* Refactoring (pg wp)

* Various PostgreSQL fixes

* Refactoring PG code

* chore: minor fixes for when debug macros are enabled

* fix: use global memory when the DBFLAG_PERSISTENT flag is set to avoid crash for double free

* fix(cloudsync): guard against errmsg aliasing in error formatting

It was fixed by snapshotting db_error into a local buffer when it aliases data->errmsg, then formatting from the copy. This avoids undefined behavior.

* fix(postgresql): fix error message in cloudsync_init_internal, the error message must be copied before database_rollback_savepoint reset the error message

* fix(postgresql): skip SPI_cursor_open for non-cursorable plans

 - avoid trying to open a cursor on INSERT/UPDATE/DELETE plans without RETURNING
 - use SPI_is_cursor_plan to decide whether to use a portal or execute once
 - fixes “cannot open INSERT query as cursor” during cloudsync_init paths (for example with SQL_INSERT_SITE_ID_ROWID)

* Minor fixes

* Improved  SPI’s memory ownership rule and prevents accumulation across large result sets

* dbmem apis now use pg native memory functions

* More memory related fixes

* Apparently repalloc doesn't like a NULL ptr

* Update database_postgresql.c

* get_cloudsync_context must be allocated in a global context

* Revert "get_cloudsync_context must be allocated in a global context"

This reverts commit 4e614f1.

* Revert "Update database_postgresql.c"

This reverts commit 13367c1.

* Revert "Apparently repalloc doesn't like a NULL ptr"

This reverts commit ffd587a.

* Revert "More memory related fixes"

This reverts commit f44c161.

* Revert "dbmem apis now use pg native memory functions"

This reverts commit 4523e42.

* Several memory related issues fixed

* Fixed memory allocations in PG BLOB functions

* pgvalue_vec_push can fails (it now return a bool)

* Improved memory handling for dbvalue_t

* fix(sql_postgresql): fix placeholder for cloudsync_memory_mprintf in SQL_PRAGMA_TABLEINFO_LIST_NONPK_NAME_CID

* fix(database_postgresql): refactor error handling in PostgreSQL database functions to always call PG_END_TRY, this fix a SIGSEGV error for corrupted stack

* fix(dbutils): dbutils_table_settings_get_value and dbutils_settings_get_value must return NULL in case no rows or error

Caller code can check if the return value is NULL or not

* fix: remove unnecessary switch to top memory context for text_to_cstring

* fix: if databasevm_bind_text size argurmnt is negative, then the length of the string is the number of bytes up to the first zero terminator

* refactor(pgvalue): always alloc values owned by pgvalue struct in the PG memory context to simplify the memory management

* chore

* chore: remove unused SQL queries from sql_postgresql.c

* fix(sql_postgresql): fix SQL_PRAGMA_TABLEINFO_LIST_NONPK_NAME_CID

* fix(sql_postgresql): SQL_PRAGMA_TABLEINFO_LIST_NONPK_NAME_CID

* fix(postgresql): implement triggers and functions called by triggers, fix metatable's schemas, fix cloudsync_pk_encode/decode functions (use bytea instead of text for pk col values)

* test(pg smoke test): add tests for cloudsync_pk_encode and for insert/delete triggers and for the content of the metatable

* Minor changes

* Payload encoding sanity check added

* Improved dbutils_settings_get_value

* fix(cloudsync_postgresql): cloudsync_init returns the site_id as bytea type

* chore

* fix(postgresql): lazy-init cloudsync context per call

Avoid relcache/snapshot leaks during CREATE EXTENSION by moving SPI-dependent init to normal function calls.
Add cloudsync_pg_ensure_initialized helper, drop SPI work from _PG_init, and wire initialization into SQL entry points so context loads on demand.

* Improved dbutils_table_settings_get_value

* fix: solve a compile error. cloudsync_context is opaque in this compilation unit, so data->site_id isn’t accessible, use the public accessor instead.

* feat(postgresql): implement cloudsync_update

* chore

* fix(postgresql): fix the PG_TRY/PG_CATCH exception stack, must not return inside PG_TRY block

* Various improvements to encoding/decoding functions

* Improved cloudsync_pg_context_init

* Better network memory management

* Implemented cloudsync_changes

* fix: sql_build_rekey_pk_and_reset_version_except_col has different parameters for sqlite and postgresql

* fix(cloudsync_postgresql.c): fix implementation of cloudsync_update aggregate function and cloudsync_delete

* ci: add new target to build and run the debug version for vscode and the standalone asan version

* fix(cloudsync_postgresql): fix memory context for cloudsync_changes (read) and change the col_value type to bytea to store pk_encoded value

preserve SQLite-compatible payloads by encoding `col_value` with the same pk wire format before it reaches the SRF/view layer.
With the bytea value for col_value we can reuse the same existing columns from the sqlite extension to encode/decode the type of the value and the value itself, and reuse the same query `SELECT cloudsync_payload_encode(tbl, pk, col_name, col_value, ...) FROM cloudsync_changes`, otherwise we should add a new column for the type and use a cast to the specific type

* Added new pk_encode_value

* fix(sql_postgresql): fix SQL_BUILD_UPSERT_PK_AND_COL  query used for col_merge_stmt

* add a skip_decode_idx argument to pk_decode, used in postgresql

* fix(cloudsync_postgresql): fix cloudsync_changes_insert_trg to use col_value as bytea with the pk encoded value

* test: add read and write tests for cloudsync_changes

* Renamed PG changes functions and removed unused variable

* No real changes

* Improved cloudsync_changes SELECT

* Updated cloudsync_changes (wp)

* Finished implementing cloudsync_changes

* fix(pk): the original blob value for the undecoded col_value for the skipped column was missing the first type byte

* fix(pk): add the skip_idx argument for the pk_encode and pk_encode_size just like I already added to pk_decode, needed by cloudsync_payload_encode on postgresql

* fix(cloudsync): fix the buffer len value (blen) after decompressing a compressed payload

* fix(sql_postgresql): fix the  SQL_CLOUDSYNC_UPSERT_RAW_COLVERSION for postgresql

* fix(sql_postgresql): fix placeholder from ? to $<n> notation for postgresql

* refactor(postgresql): add pgvalue_free function to make it clear how to free pgvalue object from internal functions

* chore

* fix(postgresql/coydsync--1.0.sql): fix arguments for cloudsync_payload_encode aggregate function

* Update cloudsync--1.0.sql

* fix(postgresql): fix cloudsync_changes_insert_trigger for TOMBSTONE rows

* fix(postgresql): trying to fix relcache/plancache/snapshot leaks that occurs when an exception is thrown and catched inside cloudsync_changes_insert_trigger (WIP)

* test(postgresql/smoke_test): add a test for payload roundtrip to another database

* test: minor changes to smoke_test.sql

* Added bounds check to pk and checksum to payload

* test: update include directive for integration test when run with CLOUDSYNC_LOAD_FROM_SOURCES from Xcode project

* Checksum is checked only if payload version is >= 2

* Fixed compilation issue

* fix(android): renamed endian.h to cloudsync_endian.h to avoid android ndk clang to have conflicts with sys/endian.h

* ci: update the dockerfile configurations to use postgresql 17 instead of 16, the same version used by supabase

* build(supabase): build a custom supabase/postgres:17.6.1.071 docker image to be used from the `supabase start` stack

* test(postgres/smoke_test): update the test to create different databases to simulate different peers

* Several issues fixed and optimizations added (reported by Claude)

* skip the schema hash check for now, we cannot compare the hash between sqlite and postgres like we were doing for sqlite

* feat(network)!: support the v2 endpoints exposed by the new sqlite-sync-server

* fix: free SPI_tuptable (if exists) after each invocation of SPI_execute to avoid memory leaks and to optimize memory usage

* fix: update the return type for the cloudsync_payload_apply function, it returns the number of applied rows

* Update database_postgresql.c

* fix(supabase): prevent a unhealthy status during the restart of the supabase stack

The error occurs because the event trigger function inserts into app_schema_version without schema qualification, failing due to missing table in the search_path used by Supabase realtime (which connects to the "postgres" database with a different schema context).
Always create app_schema_version in public and updating the function’s insert statement to reference public.app_schema_version

* docs(docker/README.md): added a troubleshooting note about the app_schema_version/Realtime migration error

* Several memory related issues fixed

* fix(network): token size when calling cloudsync_network_set_token before cloudsync_network_init, if the token was greater that 256 chars it was truncated

* fix: bind null values for col_value column in INSERT INTO cloudsync_changes with type bytea to avoid "failed to find conversion function from unknown to bytea" error

* fix(postgresql): return raw column in SQL_BUILD_SELECT_COLS_BY_PK_FMT instead of the encoded value

This query is used during merge conflict resolution to compare local values against incoming changes. Returning encoded bytea caused type mismatches and order-dependent winners in multi-db tests, failing 03_3db_multiple_roundtrip.sql.

* test(postgresql): move the smoke test to the test/postgresql dir

Also split the smoke test into different test files, all these test files are called by smoke_test.sql

* bump version

* test(postgresql): new multi-db tests

* chore: add docs with analysis on the open issues

* test(postgresql): make 02_roundript.sql test executable as a standalone test

* Update ISSUE_WARNING_resource_was_not_closed.md

* Added support for schema

* release wip-pg-extension branch node and expo packages to npmjs with the "pg" tag

* renamed workflows for OIDC publishing issues, to revert before merging to main

* Added new database_internal_table_exists function to make sure to check for system tables in the public schema (PG only)

* fix(workflow): node packages to use pg tagged version

* fix(packages/node): wrong fat binary artifact folder

* Bump version to 0.9.63

* fix(database_postgresql): fix database_select1_value to make it work with text result of type name (for example the result of SELECT current_schema();)

* fix(cloudsync_postgresql): only free quoted identifiers if they're different from the input

* test(postgresql): add tests for multi-schema scenario

* fix(postgresql): prevent duplicate primary keys when tables exist in multiple schemas

When a table name exists in multiple schemas (e.g., public.users and auth.users), SQL queries joining information_schema.table_constraints with information_schema.key_column_usage were returning duplicate primary key columns.

Solution:
Added "AND tc.table_schema = kcu.table_schema" to all JOIN conditions to ensure primary key information is only retrieved from the target schema specified by cloudsync_schema() or current_schema().

* fix(cloudsync): avoid a crash when setting the cloudsync_set_schema to the the same previous pointer

* test(postgresql): improved tests

* Added new define for schema literal

* fix: skip the decode step regardless of the data type (for the col idx specified by skip_decode_idx)

we still need to parse the value depending on the data type to increment the bseek value for the next loop

* Several minor issues fixed

* Several other issues fixed

* Update network.m

* fix: preserve prepared plan across databasevm_reset() in PostgreSQL backend

Previously, databasevm_reset() called databasevm_clear_bindings() which
destroyed the SPIPlanPtr on every reset, forcing a full SPI_prepare on
each bind/step cycle. This negated the benefit of caching statements in
cloudsync_table_context. Now reset() only clears parameter values while
keeping the plan, types, and nparams intact for reuse.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Added new CLOUDSYNC_CHANGES_NCOLS constant

* database_value_text returns NULL also in PostgreSQL implementation

* fix(postgres): fix current memory context to avoid crashes on PG_CATCH, the CopyErrorData func must not be called from a error context

* test(postgres): add a test similar to the sport_tracker app

* Fixed allocation in value returned after SPI_finish

* Updated databasevm_step0, databasevm_step  and databasevm_clear_bindings. Fixed warnings in test (due to uint64_t usage in a signed BIGINT)

* Fix for 11_multi_table_rounds.sql

* Several minor issues fixed

* fix(packages/node): broken dynamic import for platform specific package in ESM

* test(postgresql): improved test for multi_table_multi_columns_rounds and move the test for repeated_table on multiple schemas to a separated test file

* Fixed cloudsync_network_logout

* fix(network): cleanup the network configuration during network logout

* fix: use the correct schema for previously initialized tables on new connections to the database

* test(postgres): build the debug image with no-optimization flag

* Fixed some issues related to escaping

* Quoting and memory issues fixed

Several quoting issues fixed.
Added pfree(elems) and pfree(nulls) after the loop to free memory allocated by deconstruct_array.
Moved CStringGetTextDatum allocations before PG_TRY and pfree calls after PG_END_TRY. A need_schema_param flag determines
  whether 1 or 2 Datums are allocated/freed. This ensures the Datum memory is cleaned up on both the success path and the PG_CATCH error path.

* Removed unused files

* Delete .github/workflows/main.yml

* Rename rename_to_main_before_merge_to_main_branch.yml to main.yml

---------

Co-authored-by: Andrea Donetti <andinux@gmail.com>
Co-authored-by: Gioele Cantoni <gioele.cantoni@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

Author: 4 people

.devcontainer/devcontainer.json

@@ -0,0 +1,17 @@
+{
+  "name": "cloudsync-postgres-dev",
+  "dockerComposeFile": [
+    "../docker/postgresql/docker-compose.debug.yml"
+  ],
+  "service": "postgres",
+  "workspaceFolder": "/tmp/cloudsync",
+  "overrideCommand": false,
+  "postStartCommand": "pg_isready -U postgres",
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "ms-vscode.cpptools"
+      ]
+    }
+  }
+}

.github/workflows/main.yml

@@ -454,4 +454,4 @@ jobs:
           files: |
             cloudsync-*-${{ steps.tag.outputs.version }}.*
             CloudSync-*-${{ steps.tag.outputs.version }}.*
-          make_latest: true
+          make_latest: true

.gitignore

@@ -41,7 +41,6 @@ jniLibs/
 *.dex
 
 # IDE
-.vscode
 .idea/
 *.iml
 *.swp

.vscode/launch.json

@@ -0,0 +1,37 @@
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Attach to Postgres (gdb)",
+      "type": "cppdbg",
+      "request": "attach",
+      "program": "/usr/lib/postgresql/17/bin/postgres",
+      "processId": "${command:pickProcess}",
+      "MIMode": "gdb",
+      "miDebuggerPath": "/usr/bin/gdb",
+      "stopAtEntry": false,
+      "setupCommands": [
+        {
+          "description": "Enable pretty-printing for gdb",
+          "text": "-enable-pretty-printing",
+          "ignoreFailures": true
+        },
+        {
+          "description": "Add PostgreSQL source dir",
+          "text": "dir /usr/src/postgresql-17/src",
+          "ignoreFailures": true
+        },
+        {
+          "description": "Map Postgres build paths to source",
+          "text": "set substitute-path /build/src /usr/src/postgresql-17/src",
+          "ignoreFailures": true
+        },
+        {
+          "description": "Map Postgres build paths (relative) to source",
+          "text": "set substitute-path ./build/src /usr/src/postgresql-17/src",
+          "ignoreFailures": true
+        }
+      ]
+    }
+  ]
+}