Transition Grafana Data Source from Pull to Push for Enhanced Security

[lxcmyf]

Background

The current monitoring setup for TRON Fullnode and its system utilizes Grafana + Prometheus. Fullnode nodes expose specific metrics ports, allowing Prometheus to actively pull data. However, exposing these ports poses security risks. This proposal explores secure push-based alternatives to eliminate the need for exposed ports.

Evaluated Solutions

Pushgateway (Not Recommended)

• ​Mechanism: Acts as intermediary for metrics push from ephemeral jobs

• ​Key Limitations:

  • Single point of failure & performance bottleneck
  • No automatic health monitoring via up metric
  • Metrics persist indefinitely without manual cleanup
  • Only suitable for service-level batch jobs, not long-running node monitoring

Thanos Remote Write (Complex Alternative)

• ​Architecture:

  • Prometheus instances write to Thanos Receive via remote-write

  • Thanos Query aggregates data for Grafana

• ​Challenges:

  • Multi-component deployment complexity

  • Requires maintaining Prometheus instances

  • Steeper learning curve for configuration

VictoriaMetrics (Recommended Solution)

• ​Advantages:

  • Full Prometheus API compatibility

  • Simple single-node deployment with Docker

  • 7x storage efficiency over Prometheus

  • Native support for metrics push via HTTP API

  • Horizontal scaling capabilities

• Security Benefit: Eliminates need for exposed metrics ports

Implementation Plan

Phase 1: VictoriaMetrics Deployment

docker-compose.yml (Single-node)

services:

  victoriametrics:

    image: victoriametrics/victoria-metrics:latest

    ports:

      - "8428:8428"

      - "8089:8089"

    volumes:

      - ./vm-data:/victoria-metrics-data

    command:

      - --storageDataPath=/victoria-metrics-data

      - --retentionPeriod=365d

Phase 2: Metrics Push Configuration

#!/bin/bash

# push_metrics.sh

METRICS_URL="http://localhost:9527/metrics"

VM_URL="http://victoriametrics:8428/api/v1/import/prometheus"

EXTRA_LABELS="extra_job=fullnode&env=production"


while true; do

  curl -s $METRICS_URL | \

  curl -X POST --data-binary @- -H "Content-Type: text/plain" \

  "$VM_URL?$EXTRA_LABELS"

  sleep 1

done

Phase 3: Grafana Integration

Add new datasource in Grafana:

Type: Prometheus

URL: http://victoriametrics:8428/

Update dashboards to use new datasource

Maintain legacy Prometheus instance for historical data access