[INFRA] Centralized log aggregation with Grafana Loki and Promtail

[KingFRANKHOOD]

Repo Avatar
KingFRANKHOOD/Amana

Feature Description
Deploy a Grafana Loki + Promtail stack for centralized log aggregation across all services (backend, frontend, contracts event listener, PostgreSQL, Redis). Replace ad-hoc docker logs and kubectl logs with structured, searchable, and alertable logging.

Problem Statement
Currently, debugging production issues requires SSH-ing into pods/containers and running docker logs or kubectl logs. There is no centralized log search, no log-based alerting, and no way to correlate logs across services. Pino logs are structured JSON but sit on individual instances.

Proposed Solution

• Add Loki + Promtail to docker-compose.yml staging profile:

  loki:
    image: grafana/loki:3.0
    ports: ["3100:3100"]
  promtail:
    image: grafana/promtail:3.0
    volumes: ["/var/log:/var/log", "/var/lib/docker/containers:/var/lib/docker/containers"]

• For K8s: deploy Loki via Helm chart (grafana/loki-stack) with S3 storage backend
• Promtail config:
  • Scrape Docker container logs (JSON file driver)
  • Parse Pino-structured JSON: extract level, service, trace_id, trade_id, error_code as labels
  • Add compose_service label for service discovery
• Grafana datasource: add Loki, create "Logs" explore view
• Log-based alerting (via Loki Ruler):
  • rate({service="backend"} | json | level="error" [5m]) > 10 → warning
  • rate({service="backend"} | json | error_code="DB_CONNECTION_FAILED" [5m]) > 0 → critical
  • rate({service="backend"} | json | level="fatal" [5m]) > 0 → critical
• Retention: 30 days (Loki), cold storage after 7 days

Alternatives Considered

1. ELK stack (Elasticsearch, Logstash, Kibana) — heavier, more resource-intensive than Loki
2. Datadog/Splunk — paid SaaS; not suitable for OSS self-hosted deployment

Impact Area

• Infrastructure
• Observability

Priority
P2 - Medium

Implementation Complexity
Moderate

[pimaster900]

@pimaster900 has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

I would sincerely love to work on this issue because I have strong experience in working and dealing with complex workflow logic.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @pimaster900 to this issue.

[coderolisa]

@coderolisa has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

Hello maintainer let me work on this
please assign

ℹ️ Repo Maintainers: To accept this application, review their application or assign @coderolisa to this issue.

[samueldoo]

@samueldoo has applied to work on this issue as part of the Stellar Wave Program's 7th wave.

Hello Maintainer, I’d love to take ownership of this issue and submit a high-quality pull request once completed.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @samueldoo to this issue.

[laurentketterle-hub]

@laurentketterle-hub has applied to work on this issue