docker-compose.dev.yml hangs on startup, leaves Docker engine unresponsive; containers cannot be removed

[SofienRogue]

Description:

I can successfully run the all-in-one HyperDX image with:

docker run -p 8080:8080 -p 4317:4317 -p 4318:4318 docker.hyperdx.io/hyperdx/hyperdx-all-in-one

It works perfectly on Ubuntu 25.04 with Docker 29.0.3 (build 511dad6).
However, when using the dev compose setup:

git clone https://github.com/hyperdxio/hyperdx.git
cd hyperdx
docker-compose -f docker-compose.dev.yml up

The following happens:

1. hdx-oss-dev-ch-server-1 (ClickHouse) starts successfully.
2. The other containers (db-1, otel-collector-1, otel-collector-json-1) appear to start (Docker shows "Created"), but never become healthy.
3. The process hangs indefinitely, and eventually Docker itself becomes unresponsive—commands like docker ps, docker stop, or docker-compose down hang or fail.
4. After this, the only recovery is to restart the Docker engine.
5. Even after restart, the stuck containers cannot be removed normally—they persist in a limbo state until a full systemctl restart docker and manual cleanup.

The only log output before failure is:

ch-server-1  | /entrypoint.sh: create new user 'default' instead 'default'
ch-server-1  | Processing configuration file '/etc/clickhouse-server/config.xml'.
ch-server-1  | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
dependency failed to start: container hdx-oss-dev-ch-server-1 is unhealthy

System Info:

OS: Ubuntu 25.04
Docker: 29.0.3, build 511dad6
CPU: 4 cores
RAM: 15.62 GiB
Compose file: docker-compose.dev.yml from main branch

Expected behavior:
All services start cleanly, or at least fail gracefully without destabilizing Docker.

Actual behavior:
Docker engine hangs or crashes; containers become undeletable; full Docker reset required.

Impact:
This makes local development using docker-compose.dev.yml currently unusable for me, despite the all-in-one image working fine.

Request:
Investigate possible race conditions or init loops in MongoDB or OTel collector containers

Happy to provide full logs if helpful!