Skip to content

New Relic

Jump to bottom
David Aguiar edited this page Jul 28, 2026 · 1 revision

We use New Relic for APM, browser/RUM monitoring, and log aggregation across all three Data.gov apps. Data.gov's account runs on New Relic's

For log-review conventions and known-acceptable error patterns, see Log Review. For dataset usage metrics specifically, see Dataset metrics.

How it's wired up per app

All three apps are instrumented with the New Relic Python APM agent, started via newrelic-admin run-program wrapping Gunicorn, pointed at a config/newrelic.ini. App names follow <app>-<environment>, which is how each deployed instance maps to an entity in the New Relic UI:

App Prod Staging Dev
catalog.data.gov datagov-catalog-prod datagov-catalog-staging datagov-catalog-dev
harvest.data.gov datagov-harvest-prod datagov-harvest-staging datagov-harvest-dev
inventory.data.gov inventory-prod inventory-staging inventory-development

Only catalog.data.gov runs browser/RUM monitoring (it's the one heavily user-facing UI); the other two are admin/API-oriented and don't inject it. Where RUM is active, page-level events (PageView, AjaxRequest, etc.) automatically carry IP-geolocated countryCode/regionCode attributes with no extra integration — useful to know before building a new data pipeline for geographic traffic questions.

NEW_RELIC_LICENSE_KEY is never a plain env var or repo secret — it's stored on a cloud.gov user-provided service (<app>-secrets) and read out of VCAP_SERVICES at container startup. See How to Rotate New Relic License Key for the rotation procedure.

Querying via the API (NerdGraph)

New Relic's GraphQL API, NerdGraph, is useful for scripting anything about account configuration — alert policies/conditions, dashboards, entity search — rather than clicking through the UI by hand.

Two kinds of API key, easy to mix up:

  • User key — authenticates NerdGraph (config read/write). Create one at one.newrelic.com → user icon → API keys → Create a key → type: User.
  • Ingest key (License/Event) — used to send data in, not to configure the account. Different purpose from NEW_RELIC_LICENSE_KEY above, though conceptually similar.

Example: run NRQL through NerdGraph with curl:

curl https://api.newrelic.com/graphql \
  -H 'Content-Type: application/json' \
  -H "API-Key: $NEW_RELIC_API_KEY" \
  -d '{"query": "{ actor { account(id: 1601367) { nrql(query: \"SHOW EVENT TYPES SINCE 1 week ago\") { results } } } } "}'

List existing alert policies/conditions before creating something new (avoid duplicating coverage):

query($accountId: Int!) {
  actor {
    account(id: $accountId) {
      alerts {
        policiesSearch { policies { id name incidentPreference } }
        nrqlConditionsSearch {
          nrqlConditions { id name enabled nrql { query } policyId }
        }
      }
    }
  }
}

NRQL gotchas specific to alert conditions

Alert-condition NRQL is a stricter subset of what works in ad-hoc/dashboard queries:

  • LIMIT is rejected outright in alert-condition NRQL ("Query contains disallowed component: LIMIT"). Fine in dashboards, not in conditions.
  • FACET without LIMIT defaults to the top 10 values by count, descending — this is the built-in way to cap facet/signal cardinality when a field has many possible values, without an explicit LIMIT.
  • Facet/signal overflow silently produces "no data," not an error. If a saved condition's Signal History panel shows nothing regardless of date range, check whether the condition has ever produced real evaluation data before assuming it's a UI/display issue: 
    SELECT count(*) FROM NrAiIncident WHERE conditionId = '<id>' SINCE 30 days ago
SELECT count(*) FROM NrAiSignal WHERE conditionId = '<id>' SINCE 30 days ago
    Both returning 0 means the condition never evaluated — check facet cardinality (SELECT uniqueCount(<field>) FROM ...) as a likely cause.
  • Anomaly/Baseline threshold mode has no literal "% increase" field. It expresses sensitivity as a standard deviation from a learned per-facet baseline, plus a minimum duration outside that band (e.g. "N standard deviations, for at least M minutes"). This mode fits well when different facet values have very different normal scales, where a flat static threshold wouldn't make sense applied uniformly.
  • The condition-builder page's URL encodes the visible time range as duration=<milliseconds> — useful for confirming a date-range picker actually applied, if a chart seems stuck on stale data.

Alerts: policy vs. condition

New Relic separates policies (grouping + notification routing) from conditions (the actual threshold logic). A condition must belong to a policy.

  • Policies (Alerts & AI → Policies) set an incident preference: by policy (all conditions share one incident), by condition (each condition gets its own stream), or by condition and signal (each faceted signal — e.g. each value of a FACET — opens its own incident; useful when you need to know which facet value triggered).
  • Conditions (Alerts & AI → Alert conditions) hold the NRQL, signal window, and threshold, and attach to a policy. Save new conditions disabled first to observe behavior before they can page anyone.

Dashboards

Dashboard/ad-hoc NRQL is less restrictive than alert-condition NRQL — LIMIT works fine here even though it's rejected in conditions. Common widget patterns:

  • Line chart, faceted with TIMESERIES, to visualize a trend per facet value over time.
  • Billboard (big number), for an at-a-glance current-state figure.
  • Table, for exact current numbers per facet value.

See also

Clone this wiki locally