Skip to content

[FIX] Clear stale CEA-708 window rows in no-rollup mode - #2318

Open
nlp201 wants to merge 1 commit into
CCExtractor:masterfrom
nlp201:fix/cea708-no-rollup-stale-rows
Open

[FIX] Clear stale CEA-708 window rows in no-rollup mode#2318
nlp201 wants to merge 1 commit into
CCExtractor:masterfrom
nlp201:fix/cea708-no-rollup-stale-rows

Conversation

@nlp201

@nlp201 nlp201 commented Aug 12, 2026

Copy link
Copy Markdown

In raising this pull request, I confirm the following (please check boxes):

Reason for this PR:

  • This PR adds new functionality.
  • This PR fixes a bug that I have personally experienced or that a real user has reported and for which a sample exists.
  • This PR is porting code from C to Rust.

Sanity check:

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • This is a bug fix, so I have not added it to the changelog.
  • I am not adding new C code except to fix this existing, reproducible bug.

Repro instructions:

The input is a 35.1-second, video-only H.264 MPEG-TS sample. To avoid distributing the original broadcast pictures, every decoded frame was replaced with black before H.264 re-encoding. FFmpeg preserved the A53/CEA-708 caption side data in the new H.264 SEI messages. The sample contains Korean CEA-708 service 1 and no audio.

Sample URL: https://sampleplatform.ccextractor.org/sample/188

Sample properties:

size:       344,792 bytes
duration:   35.101733 seconds
video:      H.264, 1280x720, 30000/1001 fps
frames:     1,049
other A/V:  no audio and no original broadcast imagery
captions:   CEA-708 service 1
SHA-256:    36d67699a84883d901d3eef089e6bbca5e72a603f55925a406e9513dccbebbfe

The Sample Platform identifies the uploaded caption stream as EIA-708 and reports the same SHA-256 as the locally verified redacted artifact.

Baseline commit:

6077cf5c617e5db63b7068f2c06eae2fabe6e8c9

Build from the checked-out source:

docker build \
  --build-arg USE_LOCAL_SOURCE=1 \
  --build-arg BUILD_TYPE=minimal \
  -f docker/Dockerfile \
  -t ccextractor:local .

Run the same command against the baseline and this change:

docker run --rm \
  -v "$PWD:/data" -w /data \
  ccextractor:local \
  sample-redacted-black.ts \
  --input ts \
  --out srt \
  --service '1[EUC-KR]' \
  --utf8 \
  --no-rollup \
  --no-bom \
  --trim \
  --autodash \
  -o result.srt

For this service, CCExtractor writes result.p1.svc01.srt.

Description

When a CEA-708 carriage return reaches a roll-up boundary, CCExtractor first emits the current window. With --no-rollup, it then clears only the current pen row. Text in previously emitted upper rows remains in the backing window and is emitted again with later cues.

This change clears every row in the active window after emission when --no-rollup is enabled, then marks the window empty. It deliberately preserves the current pen location, pen color, and pen attributes. Normal roll-up behavior is unchanged.

The C and Rust CEA-708 implementations are updated together. The Rust regression test exercises process_cr(..., no_rollup = true), verifies that all emitted rows are cleared, verifies is_empty, and verifies that active pen styling is preserved.

Before

The second cue repeats two lines already emitted by the first cue:

2
00:00:03,671 --> 00:00:06,339
<font color="#ffff00">내뿜으며 순식간에 폭발했고 그 </font>
<font color="#ffff00">충격으로 건물 파편들이 사방으로 쏟아져 </font>
<font color="#ffff00">같은 시각 근처를 지나던 차량에서도 </font>

Those stale lines recur in later cues. The baseline also emits a cue at 00:00:21,255 --> 00:00:27,327 containing only stale text.

After

The second cue contains only its new text:

2
00:00:03,671 --> 00:00:06,339
<font color="#ffff00">같은 시각 근처를 지나던 차량에서도 </font>

The original two lines remain together in cue 1, where they are valid, but do not reappear. The stale-only cue disappears. The redacted sample produces 12 baseline cues and 11 fixed cues; all fixed lines at shared timestamps are the expected ordered suffix of the baseline lines.

Cross-broadcaster validation

Representative 10-minute CEA-708 service-1 samples from five Korean broadcasters were compared with identical options:

Broadcaster Baseline Fixed Result
MBC 67 cues 67 cues byte-identical, including 40 color spans
KBS1 148 cues 148 cues byte-identical
SBS 29 cues 28 cues stale previous rows removed; one stale-only cue removed
KBS2 162 cues 162 cues byte-identical
EBS1 153 cues 153 cues byte-identical

All five samples use three-row CEA-708 windows. SBS starts at row 0 and relies on CR to advance through the window. Once the pen reaches the final row, clearing only that row leaves rows 0 and 1 available for re-emission. The other four samples set the pen directly to the final row for each cue, so their upper rows are already empty.

An initial candidate used clear_text(). That fixed SBS but reset the active pen patterns and removed all 40 yellow color spans from MBC. The final implementation clears row contents and sets is_empty = 1 without resetting active pen color or attributes. This restores byte-for-byte output parity for the four unaffected broadcasters.

Tests

The focused regression was first confirmed to fail on the old one-row behavior. It was then strengthened after the rejected clear_text() candidate to require preservation of a non-default foreground color and italic state. The unit test drives the real process_cr(..., no_rollup = true) path and fixes the decoder-state contract; the redacted-sample A/B run below separately verifies the user-visible SRT output across consecutive emissions.

cargo fmt --all -- --check                  passed
focused no-rollup regression               1 passed
cargo test --lib                           402 passed; 0 failed
official minimal Docker build              passed (C and Rust compiled)
redacted black-video sample A/B             passed; baseline 12 cues, fixed 11 cues
five-broadcaster 10-minute comparison      passed
RCWT command analyzer                      12 passed
git diff --check                           passed

The redacted-sample output comparison confirms that the first valid multi-line cue is preserved, later cues no longer repeat its stale upper rows, no fixed-only text or timestamp interval is introduced, and the stale-only baseline cue disappears.

cargo clippy --lib -- -D warnings reports four existing unnecessary_cast findings in libccxr_exports/time.rs and libccxr_exports/util.rs. The same four findings occur on the clean baseline; this change introduces no new clippy findings.

No public checks existed for the fork commit before this PR. The local results above are exact-artifact results; repository CI for this branch should run on the PR.

Production validation

The fixed binary was also exercised by the production caption pipeline against seven archived days from the same five broadcasters. The normal transactional publication and verifier paths completed successfully:

broadcaster-days verified                  35 (7 days × 5 broadcasters)
programme SRT artifacts verified           1,091
captioned programmes                       1,026
verified no-caption programmes             65
total cues                                 669,258
failed or unknown programme outcomes       0

All 35 publication manifests verified the expected binary path, version 0.96.5, extraction policy, and current generation. This operational run supplements—but does not replace—the focused regression, reproducible sample, or repository CI.

Scope

This PR does not change CEA-608, the no_rollup = false path, public APIs, or dependencies. It does not include the original broadcast video. The provided reproduction sample contains black video plus the caption packets and timing needed to reproduce the decoder bug.

Checklist

  • I have tested these changes locally.
  • I have added a focused regression test.
  • I have updated both C and Rust CEA-708 implementations.
  • I have included exact reproduction instructions and before/after output.
  • I have compared representative CEA-708 output from five broadcasters.
  • I have removed the original broadcast imagery and audio from the shareable sample.
  • I have uploaded the redacted sample to the CCExtractor Sample Platform and linked it above.

@ccextractor-bot

ccextractor-bot commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator
CCExtractor CI platform finished running the test files on linux. Below is a summary of the test results, when compared to test for commit 6077cf5...:
Report Name Tests Passed
Broken 9/13
CEA-708 2/14
DVB 0/7
DVD 3/3
DVR-MS 2/2
General 22/27
Hardsubx 1/1
Hauppage 3/3
MP4 3/3
NoCC 10/10
Options 69/86
Teletext 0/21
WTV 12/13
XDS 31/34

Your PR breaks these cases:

NOTE: The following tests have been failing on the master branch as well as the PR:


It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you).

Check the result page for more info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nlp201 @ccextractor-bot