Skip to content

Unnest compound mode search loops and clean up inter search - #5343

Merged
yeqing-wu merged 4 commits into
AOMediaCodec:av2-encfrom
kslu-aom:av2enc-hcip
Sep 1, 2026
Merged

Unnest compound mode search loops and clean up inter search#5343
yeqing-wu merged 4 commits into
AOMediaCodec:av2-encfrom
kslu-aom:av2enc-hcip

Conversation

@kslu-aom

@kslu-aom kslu-aom commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

This change includes the following cleanups

  • Unnest JMVD scale factor, CWP, and RefineMV loops in handle_compound_inter_prediction to avoid redundant motion search and skip invalid tool combinations. The new search order is 1) base mode, 2) refine MV, 3) CWP, 4) JMVD scale. This search order is intended to allow potential early termination in the future
  • Remove unused variables, unnecessary checks, redundant mbmi field re-assignments, and move the update of common variable outside of search loops.
  • Use reset_inter_mode_info helper to unify mode_info field resets

STATS_CHANGED

Results tested with RA 33 frames based on 18a0096

    1) Speed 0
      +------------+-------+-------+-------+-------+-------+-------+
      | Class      |     Y |    Cb |    Cr |  wAvg |  Enc% |  Dec% |
      +------------+-------+-------+-------+-------+-------+-------+
      | A1         | -0.01 | +0.02 | +0.12 | -0.01 | 100.8 |  99.4 |
      | A2         | -0.00 | +0.06 | +0.01 | +0.00 | 100.2 |  99.4 |
      +------------+-------+-------+-------+-------+-------+-------+
    
    2) Speed 1
      +------------+-------+-------+-------+-------+-------+-------+
      | Class      |     Y |    Cb |    Cr |  wAvg |  Enc% |  Dec% |
      +------------+-------+-------+-------+-------+-------+-------+
      | A1         | +0.00 | +0.02 | +0.03 | +0.01 | 100.9 | 101.4 |
      | A2         | +0.01 | +0.01 | -0.07 | +0.01 | 100.7 | 100.5 |
      +------------+-------+-------+-------+-------+-------+-------+
    
    3) Speed 2
      +------------+-------+-------+-------+-------+-------+-------+
      | Class      |     Y |    Cb |    Cr |  wAvg |  Enc% |  Dec% |
      +------------+-------+-------+-------+-------+-------+-------+
      | A1         | -0.01 | +0.00 | -0.02 | -0.01 | 101.0 | 101.6 |
      | A2         | +0.01 | +0.02 | -0.01 | +0.01 | 100.5 | 100.7 |
      +------------+-------+-------+-------+-------+-------+-------+

@kslu-aom

Copy link
Copy Markdown
Contributor Author

Please review this refactoring patch and let me know if anything can be improved. I have a follow up patch to clean up more variables, search state structs, and function signatures, but I think it is better to separate this unnesting change for code review. I will mark the PR as ready when speed 0 tests are finished.

@yeqing-wu yeqing-wu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kslu-aom, thanks very much for helping refactor these code, it looks good to me. Just have one question that need your confirm.

Comment thread av2/encoder/rdopt.c Outdated
@kslu-aom
kslu-aom marked this pull request as ready for review August 31, 2026 19:05
@kslu-aom

Copy link
Copy Markdown
Contributor Author

Results have been updated. This PR has been marked as ready.

@yeqing-wu
yeqing-wu enabled auto-merge (squash) August 31, 2026 19:20
Unnest the JMVD scale factor, CWP, and RefineMV loops in
handle_compound_inter_prediction into sequential evaluation stages:
1. Base compound mode (scale 0, CWP EQUAL, RefineMV 0)
2. RefineMV mode (scale 0, CWP EQUAL, RefineMV 1)
3. CWP modes (scale 0, CWP != EQUAL, RefineMV 0)
4. JMVD scale factor modes (scale 1..N-1, CWP EQUAL, RefineMV 0)

Avoid redundant motion search and skip invalid tool combinations.
…prediction

- Hoist invariant mbmi initializations, rate computations, and loop invariants
  out of search loops in handle_single_inter_prediction and
  handle_compound_inter_prediction.
- Use reset_inter_mode_info helper to unify mode_info field resets.
- Use base template struct assignment in compound search instead of
  re-assigning individual invariant fields per iteration.
- Clean up BAWP search in single prediction by simplifying scalar BAWP MV
  tracking and mode_info access.
- Remove redundant dead checks in CWP and JMVD scaling stages.
@yeqing-wu
yeqing-wu merged commit 2128e7e into AOMediaCodec:av2-enc Sep 1, 2026
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants