Add StaticTimestamped generator that uses timestamps from a file to send logs by TheSafo · Pull Request #1687 · DataDog/lading

TheSafo · 2026-01-07T22:32:22Z

What does this PR do?

Adds a new "static" generator that parses timestamps from a log file, and chunks each second worth of logs into a single block.

Motivation

This allows you to e.g. download an hour worth of logs, parse their timestamps, and then (in conjunction w/ prior block-based throttling) send 1 block per second to get a "realistic" load pattern as opposed to generating X bytes/second.

Related issues

Follow up of #1683

TheSafo · 2026-01-07T22:32:51Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 739872f122

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f1c39bb28f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 75583e6697

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 40e6e498ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b5501b16b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b48880d37

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-23T18:04:59Z

lading_payload/src/static_timestamped.rs

+            let (ts, remainder) = NaiveDateTime::parse_and_remainder(line, timestamp_format)
+                .map_err(|_| Error::Timestamp(line.to_string()))?;


Support chrono formats that omit a full date-time

parse_line_with_format() only falls back to DateTime<FixedOffset> and NaiveDateTime, so formats that do not describe a complete date-time (for example common syslog prefixes like %b %e %H:%M:%S or time-only %H:%M:%S) are rejected as Timestamp errors even though timestamp_format is documented as a general chrono format. In practice this makes the new generator unusable for many real log files unless they already include a full date, which is a surprising limitation for replaying existing logs.

Useful? React with 👍 / 👎.

This is more of a theoretical issue / feature request than a bug.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d1e52d1a91

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-26T20:23:43Z

lading_payload/src/static_timestamped.rs

+            let (ts, remainder) = NaiveDateTime::parse_and_remainder(line, timestamp_format)
+                .map_err(|_| Error::Timestamp(line.to_string()))?;
+            let body = remainder


Avoid leaking full log lines in parse errors

When timestamp parsing fails, parse_line_with_format builds Error::Timestamp(line.to_string()), and that string is later propagated as crate::Error::Validation(err.to_string()). In practice, a misconfigured timestamp_format causes startup errors that include the entire source log line, which can expose sensitive payload content (tokens, user data, etc.) in logs and error surfaces; returning a line number or truncated sample is safer.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be80891c6e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-26T20:34:37Z

lading_payload/src/static_timestamped.rs

+        let block = self
+            .next_block
+            .take()
+            .expect("fill_next_block guarantees a block");


Preserve next second when block probe yields empty output

In fixed-cache mode, block construction intentionally retries many sampled sizes (construct_block_cache_inner treats empty output as SpinError::EmptyBlock and probes again), but this serializer consumes stream state before knowing whether anything can be written. Because self.next_block is unconditionally take()n here, a probe with max_bytes == 0 (or any size smaller than every line in that second) returns an empty block and permanently drops that entire second on retry, which corrupts the replay timing profile.

Useful? React with 👍 / 👎.

blt

One small nit comment, not blocking.

Cargo.toml

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 79b6e1e3e0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

This was referenced Jan 7, 2026

Update generator throttler to support block-based strategy #1675

Merged

Modify throttle errors to not assume capacity issue #1679

Merged

Add flush_every config option to BufWriter generators #1683

Merged

TheSafo mentioned this pull request Jan 7, 2026

Add deserialization tests for BytesThrottleConfig and LoadProfile #1684

Merged

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from 78768d6 to b3de2e0 Compare January 7, 2026 22:59

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from 78c5594 to fa19c37 Compare January 15, 2026 21:34

TheSafo force-pushed the jsaf/flush-every-n branch from 8071239 to ff8a57f Compare January 15, 2026 21:34

TheSafo mentioned this pull request Jan 15, 2026

Resize block buffer after allocation if significant wasted space #1674

Merged

TheSafo force-pushed the jsaf/staic-timestamped branch from fa19c37 to 7e1a732 Compare January 15, 2026 21:35

TheSafo force-pushed the jsaf/flush-every-n branch 2 times, most recently from 1be6e74 to 30363c3 Compare January 21, 2026 15:48

TheSafo force-pushed the jsaf/staic-timestamped branch from 7e1a732 to 1098a47 Compare January 21, 2026 15:48

TheSafo force-pushed the jsaf/flush-every-n branch from 30363c3 to 4aba943 Compare January 21, 2026 18:03

TheSafo force-pushed the jsaf/staic-timestamped branch from 1098a47 to 02c10d3 Compare January 21, 2026 18:03

TheSafo force-pushed the jsaf/flush-every-n branch from 4aba943 to b413062 Compare February 4, 2026 19:38

TheSafo force-pushed the jsaf/staic-timestamped branch from 02c10d3 to 9aa2aee Compare February 4, 2026 19:38

TheSafo force-pushed the jsaf/flush-every-n branch from b413062 to 85dfb43 Compare February 9, 2026 15:24

TheSafo force-pushed the jsaf/staic-timestamped branch from 9aa2aee to 49cdb8f Compare February 9, 2026 15:24

TheSafo force-pushed the jsaf/flush-every-n branch from 85dfb43 to fac862b Compare March 5, 2026 15:52

TheSafo force-pushed the jsaf/staic-timestamped branch from 49cdb8f to 9e8ecc9 Compare March 5, 2026 15:52

TheSafo force-pushed the jsaf/flush-every-n branch from fac862b to c5eee9b Compare March 5, 2026 15:59

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from 6bc7260 to b173aed Compare March 5, 2026 16:21

TheSafo force-pushed the jsaf/flush-every-n branch from 8e1b11c to 84e040f Compare March 5, 2026 16:38

TheSafo force-pushed the jsaf/staic-timestamped branch from b173aed to 9e7b813 Compare March 5, 2026 16:38

TheSafo changed the title ~~static-timestamped~~ Add generator to parse timestamps from a log-file and send at specified rate Mar 5, 2026

TheSafo force-pushed the jsaf/staic-timestamped branch from 9e7b813 to 72668f1 Compare March 5, 2026 17:35

TheSafo marked this pull request as ready for review March 18, 2026 16:59

TheSafo requested a review from a team as a code owner March 18, 2026 16:59

TheSafo changed the title ~~Add static generator that uses timestamps from a file to send logs at "realistic" rate~~ Add StaticTimestamped generator that uses timestamps from a file to send logs Mar 18, 2026

chatgpt-codex-connector bot reviewed Mar 18, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from efc7cc8 to f1c39bb Compare March 18, 2026 23:05

chatgpt-codex-connector bot reviewed Mar 18, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from 6c49d5b to 75583e6 Compare March 18, 2026 23:14

chatgpt-codex-connector bot reviewed Mar 18, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch from 75583e6 to 40e6e49 Compare March 23, 2026 17:08

chatgpt-codex-connector bot reviewed Mar 23, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Show resolved Hide resolved

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch from 40e6e49 to 7b5501b Compare March 23, 2026 17:43

chatgpt-codex-connector bot reviewed Mar 23, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Show resolved Hide resolved

lading_payload/src/static_timestamped.rs Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch from 7b5501b to 2b48880 Compare March 23, 2026 17:57

chatgpt-codex-connector bot reviewed Mar 23, 2026

View reviewed changes

TheSafo force-pushed the jsaf/staic-timestamped branch 3 times, most recently from f1090e9 to d1e52d1 Compare March 26, 2026 20:17

chatgpt-codex-connector bot reviewed Mar 26, 2026

View reviewed changes

remove stale RUSTSEC-2026-0049 ignore in deny.toml

be80891

TheSafo force-pushed the jsaf/staic-timestamped branch from d1e52d1 to be80891 Compare March 26, 2026 20:29

chatgpt-codex-connector bot reviewed Mar 26, 2026

View reviewed changes

blt approved these changes Mar 27, 2026

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Mar 30, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

lading_payload/src/static_timestamped.rs Show resolved Hide resolved

remove emdash

cdc67f0

TheSafo force-pushed the jsaf/staic-timestamped branch from 79b6e1e to cdc67f0 Compare March 30, 2026 14:06

TheSafo enabled auto-merge (squash) March 30, 2026 14:08

TheSafo merged commit 96bc973 into main Mar 30, 2026
27 checks passed

TheSafo deleted the jsaf/staic-timestamped branch March 30, 2026 14:14

		let (ts, remainder) = NaiveDateTime::parse_and_remainder(line, timestamp_format)
		.map_err(\|_\| Error::Timestamp(line.to_string()))?;

Conversation

TheSafo commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Related issues

Uh oh!

TheSafo commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

TheSafo Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

blt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

TheSafo commented Jan 7, 2026 •

edited

Loading

TheSafo commented Jan 7, 2026 •

edited

Loading