Skip to content

[Fix] Preserve buffered samples across packed batch flushes - #308

Open
primorLee wants to merge 1 commit into
ByteDance-Seed:mainfrom
primorLee:fix/278-preserve-buffered-samples
Open

[Fix] Preserve buffered samples across packed batch flushes#308
primorLee wants to merge 1 commit into
ByteDance-Seed:mainfrom
primorLee:fix/278-preserve-buffered-samples

Conversation

@primorLee

Copy link
Copy Markdown

Summary

  • return an overflow sample to the buffer when it was popped from that buffer but still cannot fit the current packed batch
  • add a deterministic regression test for sample conservation across a packed-batch flush

Problem

PackedDataset prefers samples from its overflow buffer while the current pack is short. If a popped sample still exceeds max_num_tokens when combined with the current pack, the iterator yields that pack and resets its state. Because the sample was already removed from the buffer and the existing buffering branch only accepts newly sampled items, the popped sample was silently discarded.

This changes the effective training distribution: the underlying dataset iterator has advanced, but the buffered sample appears in no emitted batch.

Fix

After the current packed batch is yielded, put the popped sample back into the buffer before resetting. The next iteration can retry it against a fresh pack without changing the maximum buffer occupancy, since the same sample was removed from the buffer earlier.

Validation

The regression uses token lengths 6, 6, 3, 6 with a 10-token hard limit. Before the fix, the first two emitted packs are ('a',) and ('c', 'd'), so b disappears. After the fix they are ('a',) and ('b', 'c').

  • python -m unittest data.test_dataset_base -v
  • python -m ruff check data/dataset_base.py data/test_dataset_base.py
  • python -m compileall -q data/dataset_base.py data/test_dataset_base.py
  • repeated the unit test and compile check in a clean Linux Python 3.10 container

Closes #278

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dataset_base会丢失训练数据

1 participant