Skip to content

Make analytics multi-backend, add native solution#1090

Draft
gmarull wants to merge 6 commits intocoredevices:mainfrom
teslabs:analytics-multi-backend
Draft

Make analytics multi-backend, add native solution#1090
gmarull wants to merge 6 commits intocoredevices:mainfrom
teslabs:analytics-multi-backend

Conversation

@gmarull
Copy link
Copy Markdown
Member

@gmarull gmarull commented Apr 13, 2026

This transforms the analytics service into multi-backend capable. It also adds a native backend option that does not rely on third-party services. Compared to the old native implementation, here we should not rely on any fixed blob layout. Instead, the ELF file can be used to obtain the actual blob layout/fields/sizes, etc. (see the last commit with a tool that shows how to do that).

Needs some testing...!

@gmarull gmarull requested review from ericmigi and jplexer April 13, 2026 19:26
@gmarull gmarull force-pushed the analytics-multi-backend branch 3 times, most recently from fe1ee92 to fce9770 Compare April 13, 2026 19:51
gmarull added 3 commits April 13, 2026 22:01
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
So we can use multiple backends at the same time.

Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull gmarull force-pushed the analytics-multi-backend branch from fce9770 to 5290603 Compare April 13, 2026 20:01
@ericmigi
Copy link
Copy Markdown
Collaborator

just updated eng-dash to match this format. @sjp4 can you add this to mobile pls?


PBL_ASSERTN(dls_initialized());

s_dls_session = dls_create(DlsSystemTagAnalyticsDeviceHeartbeat, DATA_LOGGING_BYTE_ARRAY,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use a different/new tag for this? The existing tag will be reporting from legacy firmwares with a different format, so maybe we should separate this format

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm gonna fix this

@gmarull
Copy link
Copy Markdown
Member Author

gmarull commented Apr 14, 2026

just updated eng-dash to match this format. @sjp4 can you add this to mobile pls?

The new mechanism should work as follows:

  1. Add a new eng-dash endpoint where we can start uploading ELF files; this way, you can parse (see example tool): (1) the build ID and (2) the blob layout
  2. Mobile is the dumb part: it should just get the blob and post it to the server
  3. The server can now match the blob ID with any previously uploaded ELF ID. If there's a match, simply take data following blob layout.

This makes sure the backend will always store new metrics without any changes, and that non-released builds will be ignored (no ELF match).

Add the native analytics backend that logs heartbeat records via DLS.
Metrics are stored in flat arrays indexed by key enum and serialized
into a packed struct on each heartbeat.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull gmarull force-pushed the analytics-multi-backend branch from 5290603 to f01df02 Compare April 14, 2026 09:02
gmarull and others added 2 commits April 14, 2026 11:04
Add a pyelftools-based script that extracts the native_heartbeat_record
struct layout from ELF DWARF debug info, printing field names, offsets,
sizes, and types. Uses a fast binary search in .debug_str/.debug_info
to locate the right CU, avoiding full DWARF parsing. Useful for building
parsers for the analytics DLS blob.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull gmarull force-pushed the analytics-multi-backend branch from f01df02 to afee44a Compare April 14, 2026 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants