Internal logs architecture document #1741

jmacd · 2026-01-08T00:32:22Z

Document the approach we will take for routing internal logs, see #1736.

codecov · 2026-01-08T00:35:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.20%. Comparing base (a2b3698) to head (313db2f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1741      +/-   ##
==========================================
+ Coverage   84.09%   84.20%   +0.10%     
==========================================
  Files         470      473       +3     
  Lines      136558   137454     +896     
==========================================
+ Hits       114836   115738     +902     
+ Misses      21188    21182       -6     
  Partials      534      534

Components	Coverage Δ
otap-dataflow	`85.47% <ø> (+0.09%)`	⬆️
query_abstraction	`80.61% <ø> (ø)`
query_engine	`90.53% <ø> (+0.14%)`	⬆️
syslog_cef_receivers	`∅ <ø> (∅)`
otel-arrow-go	`53.50% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lalitb · 2026-01-08T18:18:31Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+`otap_df_pdata::views::logs::LogsDataView`, our zero-copy accessor. We
+refer to this most-basic form of printing to the console as raw
+logging because it is a safe configuration early in the lifetime of a
+process. Note that the views implementation 


nit - not related to the proposed architecture, but this sentence seems incomplete :)

lalitb · 2026-01-08T18:22:14Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+the log record on the same core. When this fails, the configurable
+telemetry router will support options to use global logs collection
+thread, a raw logger, or do nothing (dropping the internal log
+record).


What should be the default for production, if the user doesn't select any of these options. Good to mention that.

lalitb · 2026-01-08T18:31:34Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+## OTLP-bytes first
+
+As a key design decision, the OTAP-Dataflow internal telemetry data
+path produces OTLP-bytes first. Because OTLP bytes is one of the


"OTLP-byte first" approach would need OTLP byte encoding of the message - will this be done in the hot-path? As this can be issue given the df_engine’s per-core, non-blocking requirement.

I believe I need to review #1735 first, to understand this better :)

lquerel

I made a series of proposals that I hope go in the direction of what you want to put in place.

lquerel · 2026-01-08T22:13:04Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+pipeline to be the standard configuration for all OpenTelemetry
+signals.
+
+Consuming self-generated telemetry presents a potential a kind of


Suggested change

Consuming self-generated telemetry presents a potential a kind of

Consuming self-generated telemetry presents a potential

lquerel · 2026-01-08T22:13:39Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+the connected processor and exporter components reachable from ITR
+source nodes.
+
+To begin with, every OTAP-Dataflow comonent is configured with an


Suggested change

To begin with, every OTAP-Dataflow comonent is configured with an

To begin with, every OTAP-Dataflow component is configured with an

lquerel · 2026-01-08T22:14:02Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+third-party instrumentation.
+
+We use an intermediate representation in which the dynamic elements of
+the `tracing` event are encoded while primtive fields and metadata


Suggested change

the `tracing` event are encoded while primtive fields and metadata

the `tracing` event are encoded while primitive fields and metadata

lquerel · 2026-01-08T22:24:48Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+- Option to configure internal telemetry multiple ways, including the
+  no-op implementation, multi-threaded subscriber, routing to the
+  same-core ITR, and/or raw logging.


Multi-threaded subscriber? Or multi-channel subscriber?

lquerel · 2026-01-08T22:36:57Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+  no-op implementation, multi-threaded subscriber, routing to the
+  same-core ITR, and/or raw logging.
+
+## OTLP-bytes first


In our recent telemetry guidelines, for events we distinguish between entity-based attributes (stable context) and other event-specific attributes (dynamic context). Using this terminology, attributes belonging to the stable context do not need to be emitted with every event instance; instead, they are identified by a unique numeric ID that references the attributes in an attribute registry.

The "dynamic" attributes are the ones that should travel as OTLP bytes from the hot path to the cold path. If I recall our recent conversation correctly, you were arguing that building this dynamic map would take roughly the same amount of time in a classic representation as in OTLP bytes form. That seems plausible to me, provided we are careful with this micro-serialization and keep attribute values simple.

In any case, I think we should run a few benchmarks to validate all of this.

lquerel · 2026-01-08T22:55:36Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+  these use the configured internal telemetry SDK and for ordinary
+  components (not ITR-downstream) these are routed through the ITR the
+  same core. These are always non-blocking APIs, the internal SDK must
+  drop logs instead of blocking the pipeline.


and keep track of the number of logs dropped.

lquerel · 2026-01-08T23:02:34Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+- For raw logging, format directly for the console
+- Finish the full OTLP bytes encoding for the `LogRecord`
+- Sort and filter before combining into a `LogsData`.


The id of the attribute set corresponding to the entite-based attributes will not be serialized I guess because it needs to be interpreted into the ITR. Also in many situations, I believe the dynamic attribute set will be an empty set so the OTLP-bytes will be more or less the timestamp serialized and the severity number.

lquerel · 2026-01-08T23:06:57Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+This OTLP-bytes-to-human-readable logic will be used to implement raw
+logging.
+
+### Global logs collection thread


Rather than using the term global logs collection, I would find it more explicit to refer to our internal telemetry system or internal telemetry pipeline. This internal telemetry system can be deployed primarily in two modes:

One thread per process (as today)

One thread per NUMA node, to eliminate any inter-NUMA-node communication

lquerel · 2026-01-08T23:09:10Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+could call Tokio `tracing` APIs, we arrange to explicitly disallow
+these threads from logging. The macros are disabled from executing.
+
+### Global and Per-core Event Router


I think a small diagram would help clarify all of this.

lquerel · 2026-01-08T23:11:11Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+  telemetry:
+    logs:
+      level: info
+      internal_collection:


I think that some (if not all) of these parameters should also be overridable at the level of each deployed pipeline. That would make it a truly multi-tenant system.

Internal logs architecture document

1b42ce6

github-project-automation bot added this to OTel-Arrow Jan 8, 2026

github-actions bot added the rust Pull requests that update Rust code label Jan 8, 2026

jmacd marked this pull request as ready for review January 8, 2026 00:32

jmacd requested a review from a team as a code owner January 8, 2026 00:32

jmacd mentioned this pull request Jan 8, 2026

Internal logging code path: Raw logger support #1735

Open

edits

313db2f

lalitb reviewed Jan 8, 2026

View reviewed changes

jmacd mentioned this pull request Jan 8, 2026

Internal telemetry routing ** WIP DRAFT ** #1672

Closed

lquerel reviewed Jan 8, 2026

View reviewed changes

	Consuming self-generated telemetry presents a potential a kind of
	Consuming self-generated telemetry presents a potential

	To begin with, every OTAP-Dataflow comonent is configured with an
	To begin with, every OTAP-Dataflow component is configured with an

	the `tracing` event are encoded while primtive fields and metadata
	the `tracing` event are encoded while primitive fields and metadata

Internal logs architecture document #1741

Are you sure you want to change the base?

Internal logs architecture document #1741

Conversation

jmacd commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lquerel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jmacd commented Jan 8, 2026 •

edited

Loading

codecov bot commented Jan 8, 2026 •

edited

Loading