Microservice architecture / Observability (Elasticsearch/Prometheus)

[alexeysp11]

Description

Elasticsearch, Prometheus, and Grafana form the so-called Observability Stack. They are responsible for collecting, storing, analyzing, and visualizing your logs and metrics.

Here are their roles in the architecture:

• Elasticsearch (Log Collection and Storage)
  • Purpose: Storage for your structured logs.
  • How it works: Serilog will be configured to send logs (sink) directly to Elasticsearch or through an intermediary, such as Seq (a simpler option to get started) or Logstash/Filebeat. Elasticsearch will index the logs, making them available for quick searching and filtering.
• Prometheus (Metrics Collection and Storage)
  • Purpose: Collection of metrics (digital data about the system state), not text logs.
  • How it works: Prometheus periodically pulls your ASP.NET Core services at a special address (the /metrics endpoint). Your services (using libraries like App.Metrics or built-in .NET metrics) provide data on requests per second, execution time, memory usage, CPU load, and more.
  • Note: Prometheus does not store logs; it only stores metrics.
• Grafana (Visualization and Monitoring)
  • Purpose: A unified dashboard for displaying logs and metrics.
  • How it works: Grafana connects to both Elasticsearch (for displaying logs) and Prometheus (for displaying metrics graphs). You can see a real-time dashboard of service load and, if needed, drill down into specific error logs.