Skip to content

feat(telemetry): implement app-extended-heartbeat event#5531

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 13 commits intomasterfrom
ayan.khan/add-extended-heartbeat
Apr 1, 2026
Merged

feat(telemetry): implement app-extended-heartbeat event#5531
gh-worker-dd-mergequeue-cf854d[bot] merged 13 commits intomasterfrom
ayan.khan/add-extended-heartbeat

Conversation

@khanayan123
Copy link
Copy Markdown
Contributor

@khanayan123 khanayan123 commented Mar 31, 2026

Summary

Implement the `app-extended-heartbeat` telemetry event for the Ruby tracer.

Motivation

Long-running services (24h+) currently only report their configuration state via the initial `app-started` event. If the backend misses or loses that event, there's no way to recover visibility into the SDK's configuration. The `app-extended-heartbeat` event solves this by re-sending the full configuration payload every 24h, ensuring reliable state reporting for long-running instances.

Implementation

`AppExtendedHeartbeat` inherits `AppStarted` to reuse its configuration-building logic. Rather than snapshotting configuration at startup, the worker recomputes the full payload fresh at each firing by passing `settings` and `agent_settings` directly — this means remote config changes are correctly reflected in the periodic heartbeat.

The interval is configurable via `DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL` (in seconds) for system test parity validation; the default (86400s) is defined solely in the config DSL. The tick-counter approach mirrors the existing heartbeat pattern and resets correctly after fork via `initialize_state`.

Change log entry

Yes. Add `app-extended-heartbeat` telemetry event emitted every 24h with full configuration payload (including remote config changes) for long-running service visibility.

Related

Add support for the app-extended-heartbeat telemetry event per the
telemetry v2 API spec. The event fires periodically (default 24h) and
includes the full configuration payload, matching app-started.

The interval is configurable via DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL
(in seconds) to enable system testing with shorter intervals.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions github-actions bot added the core Involves Datadog core libraries label Mar 31, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 31, 2026

Thank you for updating Change log entry section 👏

Visited at: 2026-04-01 14:13:45 UTC

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 31, 2026

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 1 steep:ignore comment, and clears 1 steep:ignore comment.

steep:ignore comments (+1-1)Introduced:
lib/datadog/core/telemetry/worker.rb:289
Cleared:
lib/datadog/core/telemetry/worker.rb:269

Untyped methods

This PR introduces 1 untyped method and 2 partially typed methods, and clears 1 untyped method. It decreases the percentage of typed methods from 61.69% to 61.68% (-0.01%).

Untyped methods (+1-1)Introduced:
sig/datadog/core/telemetry/worker.rbs:67
└── def buffer_klass: () -> untyped
Cleared:
sig/datadog/core/telemetry/worker.rbs:61
└── def buffer_klass: () -> untyped
Partially typed methods (+2-0)Introduced:
sig/datadog/core/telemetry/event/app_extended_heartbeat.rbs:8
└── def initialize: (settings: untyped, agent_settings: Core::Configuration::AgentSettings) -> void
sig/datadog/core/telemetry/worker.rbs:32
└── def initialize: (?enabled: bool, heartbeat_interval_seconds: Float, extended_heartbeat_interval_seconds: Integer, metrics_aggregation_interval_seconds: Float, emitter: Emitter, metrics_manager: MetricsManager, settings: untyped, agent_settings: Core::Configuration::AgentSettings, ?shutdown_timeout: Float | Integer, ?buffer_size: Integer, dependency_collection: bool, logger: ::Logger) -> void

Untyped other declarations

This PR introduces 1 untyped other declaration. It increases the percentage of typed other declarations from 77.73% to 77.75% (+0.02%).

Untyped other declarations (+1-0)Introduced:
sig/datadog/core/telemetry/worker.rbs:25
└── @settings: untyped

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

khanayan123 and others added 4 commits March 31, 2026 00:42
Add DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL to
supported-configurations.json (implementation A, type int, default
86400) to satisfy the EnvStringValidationCop lint check.

Add RBS type signature for AppExtendedHeartbeat to pass steep
typecheck.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL to the generated
  SUPPORTED_CONFIGURATION_NAMES set to fix EnvStringValidationCop
- Update Worker RBS signature with new constructor param and instance
  variables for steep typecheck

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add ENV_EXTENDED_HEARTBEAT_INTERVAL to ext.rbs for steep typecheck
- Update Worker.new mock in component_spec.rb to include the new
  extended_heartbeat_interval_seconds parameter

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Verify the event type is 'app-extended-heartbeat' and the payload
includes the configuration array passed at construction.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@datadog-prod-us1-5
Copy link
Copy Markdown

datadog-prod-us1-5 bot commented Mar 31, 2026

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 93.40%
Overall Coverage: 95.36% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d1ab17a | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@khanayan123 khanayan123 marked this pull request as ready for review March 31, 2026 14:35
@khanayan123 khanayan123 requested a review from a team as a code owner March 31, 2026 14:35
@khanayan123 khanayan123 requested a review from vpellan March 31, 2026 14:36
Copy link
Copy Markdown
Member

@p-datadog p-datadog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean implementation — the tick-counter approach mirrors the existing heartbeat pattern nicely, and the initialize_state placement of @extended_heartbeat_ticks correctly resets the 24h timer after fork.

A couple of things caught my eye below.

Comment thread supported-configurations.json
Comment thread lib/datadog/core/telemetry/worker.rb Outdated
@khanayan123 khanayan123 added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Mar 31, 2026
Use attr_reader instead of reaching into payload hash to access
configuration in the extended heartbeat worker. This avoids coupling
to the payload structure and makes the dependency explicit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@khanayan123 khanayan123 requested a review from p-datadog March 31, 2026 17:42
khanayan123 and others added 3 commits March 31, 2026 13:46
… attr_reader

attr_reader :configuration conflicts with the existing private
def configuration(settings, agent_settings) method. Use a separate
public method name to avoid Lint/DuplicateMethods and RBS conflicts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@vpellan vpellan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting change to include configs changed through remote config.

Comment thread lib/datadog/core/telemetry/worker.rb Outdated
Comment thread lib/datadog/core/telemetry/worker.rb Outdated
Comment thread spec/datadog/core/telemetry/event/app_extended_heartbeat_spec.rb Outdated
Comment thread supported-configurations.json
…ttings

Pass settings and agent_settings to the worker so app-extended-heartbeat
recomputes the full configuration payload at each firing instead of
snapshotting it from app-started, ensuring remote config changes are
reflected. Remove the default for extended_heartbeat_interval_seconds
from the worker constructor since the config DSL is the single source of truth.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@khanayan123 khanayan123 requested a review from vpellan April 1, 2026 14:10
- Add extended_heartbeat! worker tests: no-op before initial event,
  fires after configured interval, resets counter and fires repeatedly
- Prove freshness in extended heartbeat spec by modifying a setting
  and verifying the change is reflected in a new event instantiation
- Use Datadog::Core::Configuration::Settings.new instead of the global
  Datadog.configuration to avoid coupling to global state
- Fix CI failure: inline worker_class.new in perform spec was missing
  the new required settings/agent_settings/extended_heartbeat_interval_seconds params
- Remove verbose class comment per project minimal-comment style

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Member

@p-datadog p-datadog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 LGTM

@khanayan123
Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 bot commented Apr 1, 2026

View all feedbacks in Devflow UI.

2026-04-01 18:26:34 UTC ℹ️ Start processing command /merge


2026-04-01 18:26:47 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2026-04-01 18:35:21 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 35m (p90).


2026-04-01 19:14:51 UTC ℹ️ MergeQueue: This merge request was merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos core Involves Datadog core libraries mergequeue-status: done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants