prometheus-render

Draw RRDtool / MRTG / Munin style graphs straight from Prometheus or VictoriaMetrics.

繁體中文說明見 README.md。

prometheus-render -q 'node_load1' --from -6h -o load.png

Why this exists

There is no server-side image renderer in the Prometheus ecosystem. Both the Prometheus UI and VictoriaMetrics' vmui draw with uPlot, a browser canvas library, so getting a PNG out of either means running a headless browser.

What makes the RRD look recognisable is specific: the Cur/Min/Avg/Max legend table, the bevelled frame, a grey page around a white canvas, and gridlines whose density follows how many seconds one pixel covers.

This repo carries tsgraph, a pure-Go drawing library. No rrdtool process, no cairo, no cgo — go build cross-compiles it to any platform as a single binary.

Features

The four MRTG timescales, named after their averaging interval, with spans that abut without a gap:

File	Averaging	Span	MRTG calls it
`5m`	5 minutes	32 hours	Daily
`30m`	30 minutes	8 days	Weekly
`2h`	2 hours	5 weeks	Monthly
`1d`	1 day	13 months	Yearly

MRTG's four colour roles. The peaks are the point of the scheme, because averaging hides spikes:

Colour		Series
`#00CC00`	green	RX, filled
`#0000FF`	blue	TX, line
`#006600`	dark green	peak of the RX
`#FF00FF`	magenta	peak of the TX

In the graph above the 30-minute average tops out at 41 Mbps, while the busiest 5-minute sample in the same window reached 68 Mbps — the average understates the real peak by two thirds.

The dark theme keeps MRTG's green and blue and lightens the two peak colours instead: on a dark canvas a peak has to sit lighter than its own average to separate from it, the opposite of darker-is-peak on white.

Layering. The grid and the labelled time rules are drawn over the data, dashed and blended — an opaque line over a filled area cuts it into bands. A red rule marks a tick that carries a label, so it never lands where there is nothing to read.

Real resolution. --zoom redraws rather than upscales. The canvas, the font, the line widths, the dash lengths and the grid thickness all scale together; missing any one of them shows up as a washed-out colour rather than as a thin line.

A gap is a gap. NaN breaks the line instead of being drawn as zero.

Install

make build          # -> bin/prometheus-render
make install        # -> $GOPATH/bin
make test

Nothing else to install.

Usage

prometheus-render -q <promql> [flags]
prometheus-render --config site.yml --serve :8080

Run prometheus-render -h for the full list.

Flag	Meaning
`-u, --url`	Data source base URL (env `PROMETHEUS_URL`), default `http://localhost:9090`
`-q, --query`	PromQL expression, repeatable
`-l, --legend`	Series name, with `{{label}}` placeholders, repeatable
`--from` / `--until`	Window: `-1h`, `-7d`, `now-90min`, a Unix timestamp, RFC3339
`--step`	Resolution (`60`, `5min`). Default: about one point per pixel
`-t, --theme`	`mrtg`, `dark`, `munin`
`-w/-H`	Plot canvas size, default 400x175
`--area`	`none`, `first`, `all`, `stacked`
`--zoom`	Redraw at a multiple of the nominal size
`--behind-from`	Draw series N onwards first, i.e. behind the rest
`--tz`	Timezone, e.g. `Asia/Taipei`
`-o, --output`	Output file, `-` for stdout

VictoriaMetrics

Point --url at the vmselect Prometheus-compatible prefix:

prometheus-render -u http://vmselect:8481/select/0/prometheus -q 'node_load1'

Examples

The classic MRTG traffic graph — filled RX, TX as a line:

prometheus-render -t mrtg --area first --vtitle 'Mbps' \
  -q 'rate(node_network_receive_bytes_total{device="eth0"}[5m])*8/1000000'  -l 'rx' \
  -q 'rate(node_network_transmit_bytes_total{device="eth0"}[5m])*8/1000000' -l 'tx' \
  --title 'eth0' -o traffic.png

Munin-style stacked CPU:

prometheus-render -t munin --area stacked --from -1d --title CPU \
  -q 'sum by (mode) (rate(node_cpu_seconds_total[5m]))' -l '{{mode}}' -o cpu.png

Scheduled graphs and an HTML site

--config takes a YAML file, draws every graph in it over MRTG's four timescales, writes the pages that present them, and redraws on the interval the file names:

prometheus-render --config site.yml

source:
  url: http://localhost:9090
  resolution: 15s   # how often it scrapes; $step is measured against this

output:
  dir: site
  title: Network
  listen: ":8080"   # empty writes the files and nothing more, for nginx
  interval: 5m      # 0 draws once and exits, which is what cron wants
  workers: 0        # 0 means one per CPU

defaults:
  theme: mrtg
  dark_theme: dark  # switched on the page; both share one query
  peak: true        # MRTG's peak traces, offered as a button
  area: first
  tz: Asia/Taipei
  # Omit ranges to get MRTG's four: 1d / 1w / 1m / 1y, each redrawn at its step

graphs:
  - name: traffic
    title: Traffic
    vtitle: Mbps
    series:
      - {expr: 'sum(rate(nginx_http_in_bytes_total[$step])) * 8 / 1e6', legend: RX}
      - {expr: 'sum(rate(nginx_http_bytes_total[$step])) * 8 / 1e6',    legend: TX}

A full example is in site.example.yml.

Many places

A regions block makes the same graphs readable both ways round: everything about one place, or one thing across every place.

regions:
  label: region              # split on this; it also names the placeholder
  match: '{job="nginx"}'     # discover values from these series only
  titles: {tnn: core-tnn1, tyo: core-tyo1}   # used in pages, URLs and captions

graphs:
  - name: traffic
    series:
      - {expr: 'sum(rate(nginx_http_in_bytes_total{region="$region"}[$step]))', legend: RX}

  - name: sensor
    only_regions: [tnn]      # something that lives in one place

  - name: total
    global: true             # not split at all

The name under titles is what the site calls a region everywhere -- pages, URLs and the captions on the drawings -- while the label value stays in the query. Because it becomes a path segment, a name is letters, digits, dot, dash or underscore; anything else is refused at load rather than quietly becoming something else in the URL.

Redraws are pinned to the clock: a five-minute interval draws at :00, :05, :10, rather than from whenever the process started. A page works out the same boundaries from the same numbers, so neither side has to ask the other -- no version file, no polling. The numbers are the interval and, per drawing, how often that timescale is redrawn.

Each pass begins five seconds early, so the drawings are in place when the boundary arrives. A pass that outruns that lead says so in the log.

The header counts down to the next boundary and swaps in the drawings that are due at it. Image URLs are stable (no version query), so every cache in front of them keeps hitting.

The inline scripts carry data-cfasync="false" so Cloudflare's Rocket Loader leaves them alone: it rewrites the type attribute to something the browser will not run and executes the script itself, later and out of order, which leaves the page without its timer, its toggles or its theme.

Neither a new element nor a fetch can do this on its own. A browser keeps a second cache for images, above HTTP and keyed by URL, which deliberately ignores expiry -- the HTML spec calls it the list of available images, and it exists for compatibility. An <img> pointed at a URL it already holds renders from there without a request, while a fetch of that URL fills the HTTP cache, which is a different store.

So the bytes are handed to the element directly: fetch → blob → createObjectURL. That sidesteps the URL-keyed entry while the address on the wire never changes, so nginx and the CDN keep their entries and their hit rate.

Every drawing arrives that way, including the first -- no <img> in the HTML carries a src -- so a page just opened cannot be showing a stale copy either.

Each drawing carries the time it was made in its bottom right corner, in the same zone -- so a picture that has been saved or passed on still says how old it is.

The values are discovered from the data, so a node arriving or leaving changes the pages without editing the file. A graph with nothing to show in a region is left off the pages; one whose query failed is kept, because that means unknown rather than absent.

A label value that could reshape a query -- one carrying a quote, a backslash or a brace -- is refused rather than escaped. Real infrastructure labels do not look like that.

Redraw cadence

A timescale is redrawn at its own step, because one plot column is one step wide. The yearly graph gains a column tomorrow, so redrawing it every five minutes spends 288 queries a day returning the picture already on disk, and redrawing it every fifteen seconds spends 5,760.

output:
  interval: 5m

defaults:
  ranges:
    - {name: 1d, from: -1d,   step: 5m}             # every pass
    - {name: 1y, from: -365d, step: 1d, every: 6h}  # or say otherwise

graphs:
  - name: requests
    every: {1w: 5m, 1m: 5m, 1y: 5m}   # the one people leave open

every on a range sets the ladder. every on a graph overrides one timescale of it, keyed by range name, so the graph worth watching closely can say so without restating the ladder; a name that is not one of that graph's own timescales is refused at load rather than quietly doing nothing.

Each interval is rounded up to a whole multiple of output.interval, since a pass is the only moment anything is redrawn. That is what keeps the boundary a page counts down to exactly the boundary a pass lands on: each <img> knows its own cadence, refetches only itself and only when it is due, while the countdown in the header runs to the soonest drawing on the page.

Over MRTG's four timescales that comes to 1.2 queries per graph per pass rather than 4. A timescale whose query failed is not counted as drawn, so the next pass tries it again instead of leaving a yearly graph stale until tomorrow.

Resolution

$step in an expression is the sampling interval of the trace being drawn, so one line of config is read at the resolution it is drawn at: the daily graph asks for rate(x[5m]) where the yearly one asks for rate(x[1d]).

A window written out as a constant flattens all four timescales instead, and it flattens them badly. Through rate(x[5m]), five seconds of 100 Mbps arrive as +1.7 Mbps -- sixty times smaller than they were, which on the canvas is nothing at all. The same burst through rate(x[10s]) is still around 50 Mbps.

source.resolution is how often the source scrapes. rate() needs two samples to report anything, so it is also the floor: a step under twice it is refused at load when the expressions use $step, because a graph that comes out empty reads as an outage rather than as a setting. A range asking for more points than a source will return is refused there too, rather than having its step quietly widened out from under the window it just described.

Themes and peaks

Every graph is drawn in a light and a dark palette, and a graph with peak: true is also drawn with MRTG's peak traces. The page switches between them with buttons; the theme is remembered in localStorage, and each drawing remembers its own peak setting.

Peaks follow MRTG: the highest value in each sample bucket rather than the mean, drawn behind the averages so they show only where they rise above them. Each timescale peaks at the resolution of the one below it -- the yearly graph at the monthly graph's -- which is both what MRTG does and what keeps the subquery from asking for a hundred thousand points.

The finest timescale peaks at the source's own resolution, since there is nothing below it to borrow: with resolution: 5s, five-minute buckets are peaked over ten-second samples. A peak reads $step as the interval its inner samples are taken at rather than the bucket they are reduced into, so it looks as close to the source's raw resolution as rate() allows -- which is the whole reason it is worth drawing a second trace. Peaking at its own step instead puts one sample in each bucket and draws a peak identical to the average behind it.

The ladder runs out of budget at the long end. A year of ten-second samples is three million points per series, so the yearly graph peaks at the monthly graph's step, and a five-second burst reaches that trace hundreds of times smaller than it was. peak_expr draws it from somewhere else instead -- a recording rule that took the maximum once, at the source's own resolution:

series:
  - expr: sum(rate(nginx_http_bytes_total[$step])) * 8 / 1e6
    peak_expr: max_over_time(nginx:bytes:peak30s[$step]) * 8 / 1e6
    peak_ranges: [1w, 1m, 1y]   # omit for every timescale

$step is the bucket there rather than the samples inside it: what the rule holds is already a maximum, and a subquery over it would step past the maxima it exists to keep. The other way out is to raise the ceiling. source.max_subquery_points (100000 by default, matching -search.maxPointsSubqueryPerTimeseries) is what the ladder is spent out of, and a subquery returns one point per bucket whatever it walked to get there -- so a year peaked every 30s costs the source a million points and the render twelve hundred. Raise it here and at the source, and the long timescales read the raw samples they always had, with no rule to keep and no history to wait for.

peak_ranges leaves the fine end on the ladder, where it already reads at the source's own resolution and has nothing to gain. A timescale drawn this way is not held to the subquery budget, because it no longer asks for one.

One query feeds all four images: a palette and a peak trace change how samples are drawn, not which are read.

Serving the drawn pages

--serve serves the pages that were drawn, overriding output.listen in the config. It requires --config:

prometheus-render --config site.yml --serve :8080

What is served is what the config drew; no endpoint accepts a query. That is deliberate. A URL parameter carrying PromQL would let whoever can reach the page decide what this process reads and how much it costs to read -- an injection surface rather than a feature. To publish another graph, add another entry under graphs.

/healthz returns ok.

Using it as a library

tsgraph stands on its own and knows nothing about Prometheus — it takes samples and returns a PNG:

import "github.com/ExpTechTW/prometheus-render/tsgraph"

theme := tsgraph.LookupTheme("mrtg")
png, err := tsgraph.Render([]tsgraph.Series{{
    Name:   "rx",
    Start:  start.Unix(),
    Step:   300,
    Values: values,           // NaN marks a gap
    Colour: theme.Colour(0),
    Kind:   tsgraph.Area,
}}, tsgraph.Options{
    Title: "eth0", VLabel: "Mbps",
    Width: 500, Height: 150, Theme: theme,
})

Its only dependency is golang.org/x/image. The axis algorithms are described in tsgraph/DESIGN.md.

Example images

Everything under out/ is rendered from testdata/sample.db — offline, reproducible, and needing no data source:

make examples

The layout is out/<metric>/<host>/<theme>/<variant>/<tier>.png, where the variant is peak (averages plus the peak traces) or plain (averages alone).

Notes

Time ranges accept the rrdtool spellings: -1h, -90min, -7d, -2w, now-1d, plus Unix timestamps and RFC3339.
Sample ceiling is 11000 points per query, the Prometheus limit. Past that the step widens rather than the query failing -- except in a config file, where the range is refused at load instead, since a widened step would leave $step describing a resolution the drawing is no longer at.
Bucket convention, as in RRD and MRTG: a sample taken at time T is drawn in the bucket ending at T.

Layout

tsgraph/                the drawing library, usable on its own
tsgraph/fonts           the embedded face, Maple Mono cut to ASCII
cmd/prometheus-render   CLI
internal/promapi        query_range client, time parsing, densifying
internal/query          window and step resolution, parallel fetch
internal/params         settings shared by the CLI and the config
internal/render         joins a query to the library
internal/config         the YAML config file
internal/site           scheduling, the worker pool and the HTML
examples/gallery        renders out/ from SQLite (its own module)
testdata/sample.db      the sample dataset
hack/                   checks that read the rendered pixels back

License

Apache 2.0. See LICENSE and NOTICE. The embedded face is Maple Mono, under the SIL Open Font License 1.1 — see tsgraph/fonts/OFL.txt.

FilesExpand file tree

README-EN.md

Latest commit

History