An Observability Data Lake engine designed to be fast, easy-to-use, cost-effective, scalable, and fault-tolerant.
IceGate supports open protocols and APIs compatible with standard ingesting and querying tools. All data is persisted in Object Storage, including WAL for ingested data, catalog metadata, and the data layer.
- Highly Scalable: Scale compute resources independently based on workload demands of specific components
- ACID Transactions: Full transaction support without requiring a dedicated OLTP database
- Exactly-Once Delivery: Reliable data ingestion with no data loss or duplication
- Real-Time Queries: Access live data through WAL while maintaining historical query capabilities
- Open Standards: Built on Apache Iceberg, Arrow, and Parquet with OpenTelemetry protocol support
- Cost-Effective: Object storage-based architecture minimizes infrastructure costs
- Fault-Tolerant: Designed for resilience and high availability
IceGate employs a compute-storage separation architecture, allowing independent scaling of processing and storage resources. This design enables cost-effective scaling where compute resources (Ingest, Query, Maintain, Alert) can be scaled independently based on workload demands, while all data resides in object storage.
The system consists of five core components for handling observability data (metrics, traces, logs, and events):
- Technology: Apache Iceberg
- Purpose: Organizes the data lake with ACID transaction support
- Key Feature: Custom catalog implementation that doesn't require a dedicated OLTP database while still supporting transactions
- Protocol: OpenTelemetry
- Purpose: Accept observability data and persist it in Object Storage
- Implementation: WAL using Parquet files organized in a special way to be compatible with the Storage data layer
- Delivery Guarantee: Exactly-once delivery
- Note: WAL files can be used by the Query layer to provide real-time data access
- Technology: Apache DataFusion and Apache Arrow
- Purpose: Query engine for processing logs, metrics, traces, and events
- Implementation: Rust-native query engine built on Apache Arrow, providing a foundation to build query engines using various protocols
- Data Format: Apache Parquet
- Features:
- Statistics and bloom filters for efficient querying
- Additional custom statistics and filters for optimization
- Data optimization operations: merge, TTL support, manifest optimization, and orphan resource cleanup
- Purpose: Maximize query efficiency and maintain data lake health
- Purpose: Provides management of alerting rules, analyzing observability data, and generating alert events
- Features:
- Rule management for defining alert conditions
- Real-time analysis of observability data (logs, metrics, traces)
- Event generation and delivery based on rule evaluation
- Data Type: Events are treated as a dedicated data type alongside logs, metrics, and traces
- Convention: Follows OpenTelemetry Events Semantic Conventions
- Implementation: Leverages the Query layer for querying observability data to evaluate alert rules
- Docker and Docker Compose - for running the application
- Git - for cloning the repository
-
Clone the repository:
git clone https://github.com/icegatetech/icegate.git cd icegate -
Start the core services in docker compose:
make run-docker-core-release
This command builds and runs all core services in Docker containers. The first run will take a few minutes to build the images and start all services.
- Orbstack, kubectl and Kustomize - for running the application
- Git - for cloning the repository
-
Clone the repository:
git clone https://github.com/icegatetech/icegate.git cd icegate -
Start the core services in Kubernetes on Orbstack (Mac):
make run-kubernetes-core-release
This command builds and runs all core services in Kubernetes. The first run will take a few minutes to build the images and start all services.
Once running, the following services are available:
- OTLP gRPC:
http://localhost:4317- Ingest observability data using OpenTelemetry gRPC protocol - OTLP HTTP:
http://localhost:4318- Ingest observability data using OpenTelemetry HTTP protocol - Ingest Metrics:
http://localhost:9091/metrics- Prometheus metrics for ingest service
- Loki API:
http://localhost:3100- Query logs using Loki-compatible API ✅ Currently Supported - Prometheus API:
http://localhost:9090- Query metrics (planned) - Tempo API:
http://localhost:3200- Query traces (planned)
- Grafana:
http://localhost:3000- Dashboard and visualization (no login required)
- MinIO Console:
http://localhost:9001- Object storage web interface (login: minioadmin/minioadmin) - MinIO API:
http://localhost:9000- S3-compatible object storage API - Nessie API:
http://localhost:19120- Iceberg catalog REST API
Run with additional services for testing and monitoring:
# Include load generator for testing
make run-docker-load-release
# Pure runtime services
make run-docker-core-release
# Include analytics tools (Trino SQL query engine at http://localhost:8082)
make run-docker-analytics-release- Send telemetry data to
localhost:4317(gRPC) orlocalhost:4318(HTTP) using any OpenTelemetry SDK - View data in Grafana at
http://localhost:3000 - Query data using Loki, Prometheus, or Tempo APIs
For developers who want to build from source and contribute:
cargo build # Build the project
cargo test # Run tests
make dev # Start development stack with hot-reloadFor detailed development setup, build commands, and code quality guidelines, see CONTRIBUTING.md.
Contributions are welcome! See CONTRIBUTING.md for guidelines.
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Built with:
- Apache Iceberg - Table format for data lakes
- Apache Arrow - Columnar memory format
- Apache Parquet - Columnar storage format
- OpenTelemetry - Observability framework
IceGate is currently in prototype development. APIs and features are subject to change.