Skip to content

feat(data_library): add journald sample package - #294

Open
chelseawright7 wants to merge 1 commit into
observIQ:mainfrom
chelseawright7:feat/process-metrics-and-journald-samples
Open

feat(data_library): add journald sample package#294
chelseawright7 wants to merge 1 commit into
observIQ:mainfrom
chelseawright7:feat/process-metrics-and-journald-samples

Conversation

@chelseawright7

@chelseawright7 chelseawright7 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Proposed Change

Adds data_library/journald/journal_json.log — 16 journalctl -o json records across the syslog, stdout, journal, kernel and driver transports: sshd auth success/failure/lockout, nginx worker exit and upstream refusal, systemd unit failure, sudo (authorized and NOT-in-sudoers), a UFW block, an OOM kill, cron, dockerd, postgres, and journald rate-limit suppression.

Journald had no Blitz coverage at all before this.

Fields use raw journald names (_PID, _EXE, MESSAGE, PRIORITY, SYSLOG_IDENTIFIER, _HOSTNAME) rather than OpenTelemetry-conventional ones, which is what the OTel journald receiver actually delivers — and what makes the sample useful to a pipeline whose job is renaming them.

One limitation, documented in docs/generator/filegen.md: journald's canonical timestamp __REALTIME_TIMESTAMP is epoch microseconds, which has no ctime directive, so those values are static. Syslog-transport records also carry SYSLOG_TIMESTAMP, which does use a directive and advances per emission.

Checklist
  • Changes are tested
  • CI has passed

go build ./..., go test ./..., make lint and make security pass, as does go build/go test under -tags embed_library.

Smoke-tested through the CLI:

blitz --generator-type filegen --generator-filegen-source package:journald \
      --output-type stdout --generator-count 3

Confirms the package resolves by name and that SYSLOG_TIMESTAMP directive substitution advances per emission.


Per review: the hostmetrics process scraper that was originally in this PR has been split out into #295. This PR is now journald only.

@chelseawright7
chelseawright7 requested review from a team as code owners August 13, 2026 11:30
@Dylan-M Dylan-M self-assigned this Aug 13, 2026

@Dylan-M Dylan-M left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the process scraper. PRs should be for a single feature/capability. The JournalD work looks pretty good, but cannot be co-merged with the process work.

Adds data_library/journald/journal_json.log, 16 `journalctl -o json` records
across the syslog, stdout, journal, kernel and driver transports: sshd auth
success/failure/lockout, nginx worker exit and upstream refusal, systemd unit
failure, sudo (authorized and NOT-in-sudoers), a UFW block, an OOM kill, cron,
dockerd, postgres, and journald rate-limit suppression.

Fields use raw journald names (_PID, _EXE, MESSAGE, PRIORITY,
SYSLOG_IDENTIFIER, _HOSTNAME) rather than OpenTelemetry-conventional ones,
which is what the OTel journald receiver actually delivers.

journald's canonical timestamp __REALTIME_TIMESTAMP is epoch microseconds and
has no ctime directive, so those values are static; syslog-transport records
also carry SYSLOG_TIMESTAMP, which does use a directive and advances per
emission. Both noted in the filegen docs.
@chelseawright7
chelseawright7 force-pushed the feat/process-metrics-and-journald-samples branch from ff9428d to 1ba9426 Compare August 13, 2026 12:51
@chelseawright7 chelseawright7 changed the title feat(generator): add process scraper and journald data library package feat(data_library): add journald sample package Aug 13, 2026

@Dylan-M Dylan-M left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Details inline

@@ -0,0 +1,16 @@
{"__REALTIME_TIMESTAMP":"1783087320123456","__MONOTONIC_TIMESTAMP":"884512300","_BOOT_ID":"8f3c2a1b4d5e6f708192a3b4c5d6e7f8","_MACHINE_ID":"a1b2c3d4e5f60718293a4b5c6d7e8f90","_HOSTNAME":"linux-host01.test","_TRANSPORT":"syslog","SYSLOG_FACILITY":"4","SYSLOG_IDENTIFIER":"sshd","SYSLOG_TIMESTAMP":"%b %d %H:%M:%S","PRIORITY":"6","_UID":"0","_GID":"0","_COMM":"sshd","_EXE":"/usr/sbin/sshd","_CMDLINE":"sshd: deploy [priv]","_PID":"1842","_SYSTEMD_UNIT":"ssh.service","_SYSTEMD_CGROUP":"/system.slice/ssh.service","MESSAGE":"Accepted publickey for deploy from 10.0.4.12 port 51322 ssh2: ED25519 SHA256:abcd1234"}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

__REALTIME_TIMESTAMP is hardcoded to fixed epoch-microsecond values. It doesn't advance per emission the way the ctime directives everywhere else in the library do. The journald receiver keys record time off this field, so every run replays the same 2026 timestamps. Once the %EPOCH_US directive lands (see #296), I'd use it here so this renders live epoch microseconds. The other 15 lines need the same change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants