From e71a55c17a046017085ea5e76ef9daf5645a080d Mon Sep 17 00:00:00 2001 From: vishalm0509 Date: Tue, 2 Dec 2025 12:38:25 +0530 Subject: [PATCH 1/4] doc: add external postgres database for OLake UI --- docs/install/olake-ui/index.mdx | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/docs/install/olake-ui/index.mdx b/docs/install/olake-ui/index.mdx index c53bef59..7afa2159 100644 --- a/docs/install/olake-ui/index.mdx +++ b/docs/install/olake-ui/index.mdx @@ -132,6 +132,73 @@ x-app-defaults: This will create and use `/custom/path/to/olake-data` instead of the default `./olake-data` directory. +### External PostgreSQL Configuration + +OLake supports using an **external PostgreSQL instance** instead of the built-in Postgres service included in the docker-compose file. PostgreSQL stores all job data, configurations, sync state, and Temporal workflow data. + +**Configure `docker-compose.yml`** + +You can download the compose file from the OLake UI repository: +```bash +curl -o docker-compose.yml https://raw.githubusercontent.com/datazip-inc/olake-ui/refs/heads/master/docker-compose.yml +``` + +Inside the file, you will find an extension field called `x-db-envs`, which contains all database configuration values used by OLake and Temporal. + +```yaml +x-db-envs: + DB_HOST: &DBHost postgresql + DB_PORT: &DBPort 5432 + DB_USER: &DBUser temporal + DB_PASSWORD: &DBPassword temporal + DB_SSLMODE: &DBSSLMode disable + OLAKE_DB_NAME: &olakeDBName postgres + TEMPORAL_DB_NAME: &temporalDBName temporal +``` + +These values use YAML anchors `(&name)`, allowing them to be reused across multiple services. +Update these values to point to your external PostgreSQL instance. All relevant services (olake-ui, temporal-worker, and temporal) will automatically pick them up. + +**TLS settings for external Temporal database** + +The compose file also includes optional TLS-related variables for Temporal's SQL connection: + +```yaml + # SQL_TLS: true + # SQL_TLS_DISABLE_HOST_VERIFICATION: true + # SQL_TLS_ENABLED: true + # SQL_HOST_VERIFICATION: false +``` + +These are commented out by default because the built-in PostgreSQL container does not require TLS. +If you are connecting Temporal to an external PostgreSQL instance (especially one with SSL/TLS enabled), simply uncomment and configure these variables as needed. + +**Separate database for Temporal** + +If you want to run Temporal with its own dedicated database, you can override Temporal-specific environment variables directly in the compose file: + +```yaml +temporal: + environment: + DB: postgres12 + POSTGRES_SEEDS: + DB_PORT: + POSTGRES_USER: + POSTGRES_PWD: + DBNAME: + ENABLE_ES: true + ES_SEEDS: elasticsearch + ES_VERSION: v7 + TEMPORAL_ADDRESS: temporal:7233 # temporal service address + TEMPORAL_CLI_ADDRESS: temporal:7233 # temporal cli address + + # for external postgres database + SQL_TLS: true + SQL_TLS_DISABLE_HOST_VERIFICATION: true + SQL_TLS_ENABLED: true + SQL_HOST_VERIFICATION: false +``` + ### Service Environment Variables Key environment variables that can be customized within specific services: From 8c26c5a1c0e2caae27c158a5c7f7f50365757fc5 Mon Sep 17 00:00:00 2001 From: vishalm0509 Date: Tue, 2 Dec 2025 13:26:02 +0530 Subject: [PATCH 2/4] chore: translate to passive voice --- docs/install/olake-ui/index.mdx | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/install/olake-ui/index.mdx b/docs/install/olake-ui/index.mdx index 7afa2159..0805f794 100644 --- a/docs/install/olake-ui/index.mdx +++ b/docs/install/olake-ui/index.mdx @@ -138,12 +138,12 @@ OLake supports using an **external PostgreSQL instance** instead of the built-in **Configure `docker-compose.yml`** -You can download the compose file from the OLake UI repository: +The compose file can be downloaded from the OLake UI repository: ```bash curl -o docker-compose.yml https://raw.githubusercontent.com/datazip-inc/olake-ui/refs/heads/master/docker-compose.yml ``` -Inside the file, you will find an extension field called `x-db-envs`, which contains all database configuration values used by OLake and Temporal. +Within the compose file, an extension field named `x-db-envs` defines all database configuration values used by OLake and Temporal: ```yaml x-db-envs: @@ -156,12 +156,12 @@ x-db-envs: TEMPORAL_DB_NAME: &temporalDBName temporal ``` -These values use YAML anchors `(&name)`, allowing them to be reused across multiple services. -Update these values to point to your external PostgreSQL instance. All relevant services (olake-ui, temporal-worker, and temporal) will automatically pick them up. +These values use YAML anchors `(&name)`, enabling reuse across multiple services. +To connect to an external PostgreSQL instance, the values in this section can be updated accordingly. All dependent services (olake-ui, temporal-worker, and temporal) will automatically read and use the updated configuration. **TLS settings for external Temporal database** -The compose file also includes optional TLS-related variables for Temporal's SQL connection: +The compose file includes optional TLS-related variables for Temporal's SQL connection: ```yaml # SQL_TLS: true @@ -170,12 +170,12 @@ The compose file also includes optional TLS-related variables for Temporal's SQL # SQL_HOST_VERIFICATION: false ``` -These are commented out by default because the built-in PostgreSQL container does not require TLS. -If you are connecting Temporal to an external PostgreSQL instance (especially one with SSL/TLS enabled), simply uncomment and configure these variables as needed. +These values remain commented by default because the built-in PostgreSQL container does not require TLS. +When Temporal is configured to use an external PostgreSQL instance particularly one with SSL/TLS enabled, these variables can be uncommented and configured as needed. **Separate database for Temporal** -If you want to run Temporal with its own dedicated database, you can override Temporal-specific environment variables directly in the compose file: +Temporal can also be configured to use a dedicated PostgreSQL database, separate from the database used for storing OLake metadata and job-related information. This configuration is achieved by overriding the Temporal-specific environment variables in the service definition: ```yaml temporal: From dae391a9b3d41aa6267f3a9b55a2c0ed8f2b7e3c Mon Sep 17 00:00:00 2001 From: vishalm0509 Date: Tue, 2 Dec 2025 14:24:02 +0530 Subject: [PATCH 3/4] chore: shortened the doc --- docs/install/olake-ui/index.mdx | 56 +++++++-------------------------- 1 file changed, 12 insertions(+), 44 deletions(-) diff --git a/docs/install/olake-ui/index.mdx b/docs/install/olake-ui/index.mdx index 0805f794..cfe31c70 100644 --- a/docs/install/olake-ui/index.mdx +++ b/docs/install/olake-ui/index.mdx @@ -138,12 +138,7 @@ OLake supports using an **external PostgreSQL instance** instead of the built-in **Configure `docker-compose.yml`** -The compose file can be downloaded from the OLake UI repository: -```bash -curl -o docker-compose.yml https://raw.githubusercontent.com/datazip-inc/olake-ui/refs/heads/master/docker-compose.yml -``` - -Within the compose file, an extension field named `x-db-envs` defines all database configuration values used by OLake and Temporal: +Within the compose file, an extension field named `x-db-envs` defines all database configuration values used by OLake and Temporal. These values can be modified to point to the external PostgreSQL instance. ```yaml x-db-envs: @@ -156,49 +151,22 @@ x-db-envs: TEMPORAL_DB_NAME: &temporalDBName temporal ``` -These values use YAML anchors `(&name)`, enabling reuse across multiple services. -To connect to an external PostgreSQL instance, the values in this section can be updated accordingly. All dependent services (olake-ui, temporal-worker, and temporal) will automatically read and use the updated configuration. - -**TLS settings for external Temporal database** - -The compose file includes optional TLS-related variables for Temporal's SQL connection: - -```yaml - # SQL_TLS: true - # SQL_TLS_DISABLE_HOST_VERIFICATION: true - # SQL_TLS_ENABLED: true - # SQL_HOST_VERIFICATION: false -``` - -These values remain commented by default because the built-in PostgreSQL container does not require TLS. +The compose file includes optional TLS-related variables for Temporal's SQL connection. When Temporal is configured to use an external PostgreSQL instance particularly one with SSL/TLS enabled, these variables can be uncommented and configured as needed. -**Separate database for Temporal** - -Temporal can also be configured to use a dedicated PostgreSQL database, separate from the database used for storing OLake metadata and job-related information. This configuration is achieved by overriding the Temporal-specific environment variables in the service definition: - ```yaml -temporal: - environment: - DB: postgres12 - POSTGRES_SEEDS: - DB_PORT: - POSTGRES_USER: - POSTGRES_PWD: - DBNAME: - ENABLE_ES: true - ES_SEEDS: elasticsearch - ES_VERSION: v7 - TEMPORAL_ADDRESS: temporal:7233 # temporal service address - TEMPORAL_CLI_ADDRESS: temporal:7233 # temporal cli address - - # for external postgres database - SQL_TLS: true - SQL_TLS_DISABLE_HOST_VERIFICATION: true - SQL_TLS_ENABLED: true - SQL_HOST_VERIFICATION: false + # for external postgres database + SQL_TLS: true + SQL_TLS_DISABLE_HOST_VERIFICATION: true + SQL_TLS_ENABLED: true + SQL_HOST_VERIFICATION: false ``` +:::info **Note** (optional) +A separate PostgreSQL container (`postgresql` service) is included by default. +When an external database is used, this service is not needed and can be removed after all the services are started and healthy. +::: + ### Service Environment Variables Key environment variables that can be customized within specific services: From 0c2e0295a055e89ce843bfa6a780edc27ee7ed43 Mon Sep 17 00:00:00 2001 From: Schitiz Sharma Date: Tue, 2 Dec 2025 18:09:16 +0530 Subject: [PATCH 4/4] chore: simplify doc and add x-envs section --- docs/install/olake-ui/index.mdx | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/docs/install/olake-ui/index.mdx b/docs/install/olake-ui/index.mdx index cfe31c70..96a1ff81 100644 --- a/docs/install/olake-ui/index.mdx +++ b/docs/install/olake-ui/index.mdx @@ -134,11 +134,9 @@ This will create and use `/custom/path/to/olake-data` instead of the default `./ ### External PostgreSQL Configuration -OLake supports using an **external PostgreSQL instance** instead of the built-in Postgres service included in the docker-compose file. PostgreSQL stores all job data, configurations, sync state, and Temporal workflow data. +OLake supports using an **external PostgreSQL instance** instead of the built-in Postgres service included in the docker-compose file. This PostgreSQL stores all job data, configurations, sync state, and Temporal workflow data. -**Configure `docker-compose.yml`** - -Within the compose file, an extension field named `x-db-envs` defines all database configuration values used by OLake and Temporal. These values can be modified to point to the external PostgreSQL instance. +Within the [compose file](https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml), an extension field named `x-db-envs` defines all database configuration values used by OLake and Temporal. These values can be modified to point to the external PostgreSQL instance. ```yaml x-db-envs: @@ -151,33 +149,34 @@ x-db-envs: TEMPORAL_DB_NAME: &temporalDBName temporal ``` -The compose file includes optional TLS-related variables for Temporal's SQL connection. -When Temporal is configured to use an external PostgreSQL instance particularly one with SSL/TLS enabled, these variables can be uncommented and configured as needed. +If connection to external PostgreSQL instance is over TLS, the following variables need to be uncommented under `services.temporal.env` section in the compose file: ```yaml - # for external postgres database + # for TLS enabled external postgres database SQL_TLS: true SQL_TLS_DISABLE_HOST_VERIFICATION: true SQL_TLS_ENABLED: true SQL_HOST_VERIFICATION: false ``` -:::info **Note** (optional) +:::info **Optional** A separate PostgreSQL container (`postgresql` service) is included by default. -When an external database is used, this service is not needed and can be removed after all the services are started and healthy. +When an external database is used, this service is not needed and can be stopped after all the services are started and healthy. ::: ### Service Environment Variables -Key environment variables that can be customized within specific services: +Key environment variables that can be customized within `x-envs` section. These variables are automatically injected into the `olake-ui` and `olake-worker` containers: ```yaml -services: - olake-ui: - environment: - PERSISTENT_DIR: *hostPersistencePath - POSTGRES_DB: "postgres://olake:olake@postgresql:5432/olakedb" - OLAKE_SECRET_KEY: *encryptionKey +x-envs: + shared: &sharedEnvs + CONTAINER_REGISTRY_BASE: ${CONTAINER_REGISTRY_BASE:-registry-1.docker.io} + OLAKE_SECRET_KEY: *encryptionKey + PERSISTENT_DIR: *hostPersistencePath + FOO: bar + KEY: value + NAME: example ``` ### Data Persistence