KDB-X DB Service - Public Preview 1

The DB Service is a self-contained variation of the KDB-X tick-based architecture, with many useful built-in features accessible via a well-documented API. It is a simple way to quickly deploy a fully-implemented KDB-X database to capture and query streaming and batch data.

Full documentation available at code.kx

Prerequisites

Before starting the DB Service, you must have your local environment ready. Ensure the following:

• License. You have a valid KDB-X license. Set in the .env file. If it is a community edition license it should be assigned to KDB_LICENSE_B64. For a k4 license, use KDB_K4LICENSE_B64.

• Docker, with docker compose available. Refer to the Docker installation guide and the docker compose installation instructions.

• KDB-X installed, if you'd like to interact with the service via q. Refer to the KDB-X installation guide for instructions.

Configuring usage restrictions

DB Service is configured by default to enforce the usage restrictions of the KDB-X Community Edition license. If you are not using Community Edition you should edit .env and delete the indicated variables (DS_SM_MEM_LIMIT, DS_DA_MEM_LIMIT and DS_OTHER_MEM_LIMIT) to disable the restrictions.

If you are using Community Edition you are permitted to edit these restrictions so long as you remain within the 16GB total limit. If you are running on a typical Linux system, the systemd slice approach outlined below is more flexible.

Starting the DB Service

The DB Service runs as a containerized service and is distributed using Docker Compose.

To start the DB service:

1. Clone this repo.

2. Log in to the KX docker registry. If you don't already have an account, you can sign up for free at the KX Developer Center. A login token can be created at https://portal.dl.kx.com/auth/token, if you need to generate a new one.

EMAIL=email@example.com
BEARER=BEARERTOKEN
docker login -u $EMAIL -p $BEARER portal.dl.kx.com

3. Run the initialization script - this command initializes the database directories, and copies sample data to the import path:

./init-db.sh

4. Start the service:

docker compose up -d

Using the DB Service

You can connect to the DB Service using one of the following interfaces. There is a basic import and query workflow available in the documentation's quickstart. Example notebooks are also bundled with the service.

q client

Download the q client from the q client repo and load it into your q session.

dbs:use`kx.dbservice_client

Full client documentation and usage examples are available in the q client repo.

Python client

The Python client can be installed directly from https://portal.dl.kx.com.

pip install --pre --extra-index-url https://portal.dl.kx.com/assets/pypi/ kdbx_db_service_client

Full client documentation and usage examples are available in the Python client repo. For additional KDB-X Python installation and environment setup, refer to the KDB-X Python install guide.

REST API

You can interact with the DB Service using HTTP requests. Send requests to the service endpoint on port 8080. For example:

curl -X GET "http://localhost:8080/api/v0/tables"

For full request and response examples, refer to the OpenAPI documentation.

Example notebooks

Notebooks are included in this repo with end-to-end examples using the q client, Python client, and cURL.

• notebooks/qClient_notebook.ipynb
• notebooks/python_notebook.ipynb
• notebooks/curl_notebook.ipynb

Notebook setup

1. Install the requirements into a virtual environment

cd notebooks
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Launch Jupyter

jupyter lab

Ensure DB Service is running at http://localhost:8080, then open the URL printed by Jupyter.

Sample data feed

There is a sample data feed in the samples/ directory, demonstrating how to use the Python RT library to send streaming data to the DB Service. To run the sample feed:

1. Install dependencies

cd samples
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Start the feed

python3 fxfeed.py

Using systemd slices for Community Edition usage limits

If you have root access and are on a systemd-based Linux system you can use its slices functionality to restrict the total memory usage of the whole service rather than the default approach of configuring individual limits. To enable it, run:

sudo cp kx-db-service.slice /etc/systemd/system/

Then edit .env and uncomment the line

#export DS_CGROUP_PARENT=kx-db-service.slice

You should also disable the individual CE memory limits, as they are unnecessary when using the slice.

Resetting the service

⚠️ Resetting the service permanently deletes all data and RT client logs, and re-initializes directories. If you need to reset the database, run:

 docker compose down 
./reset-db.sh