This project is a distributed, append-only commit log built from scratch in Go, designed as both a learning exercise and a production-grade infrastructure component. It mirrors the architecture of systems like Kafka, NATS Streaming, and WAL implementations in modern databases.
- In-memory log for fast prototyping and tests
- File-backed
Storewith length-prefixed framing - Protobuf-encoded
Recordpersistence - Indexed segments with offset-to-position mapping
- Segment rotation and compaction
- Multi-segment
Logorchestration - gRPC and HTTP API layers
- TLS-secured transport with authentication
- Manual leader-based clustering (Raft optional)
- Append-only storage and data immutability
- File I/O with
os.File, buffering, and flushing - Binary encoding with
encoding/binary - Structured data with Protocol Buffers
- TDD methodology and
testifyassertions - Mutexes and thread-safe concurrency
internal/
log/
store.go # Append-only file abstraction
store_test.go # Full coverage of append, read, flush, close
... # (Coming soon: segment, index, log layers)
api/
record.proto # Protobuf schema for `Record`
record.pb.go # Generated Go types for gRPC and disk encoding
Ensure you have Go installed (Go 1.22+).
go test ./internal/log -v# Install plugins if not already installed
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
# Generate Go code from .proto definition
protoc --go_out=. --go_opt=paths=source_relative \
--go-grpc_out=. --go-grpc_opt=paths=source_relative \
api/record.proto- Milestone 0 – In-Memory Log
- Milestone 1 – File-backed Store (framed append + read)
- Milestone 2 – Index (offset → position mapping)
- Milestone 3 – Segment (store + index per chunk)
- Milestone 4 – Log abstraction (multi-segment controller)
- Milestone 5 – gRPC, TLS, and clustering
Kimba SABI N'GOYE
MIT License