Skip to content

Support multi-partition groupby standard deviation aggregations#21944

Open
Matt711 wants to merge 5 commits intorapidsai:mainfrom
Matt711:fea/polars/std-decomp
Open

Support multi-partition groupby standard deviation aggregations#21944
Matt711 wants to merge 5 commits intorapidsai:mainfrom
Matt711:fea/polars/std-decomp

Conversation

@Matt711
Copy link
Copy Markdown
Contributor

@Matt711 Matt711 commented Mar 30, 2026

Description

Supports streaming groupby std and var aggregations.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@Matt711 Matt711 added the feature request New feature or request label Mar 30, 2026
@Matt711 Matt711 requested a review from a team as a code owner March 30, 2026 12:58
@Matt711 Matt711 added the non-breaking Non-breaking change label Mar 30, 2026
@github-actions github-actions bot added Python Affects Python cuDF API. cudf-polars Issues specific to cudf-polars labels Mar 30, 2026
@GPUtester GPUtester moved this to In Progress in cuDF Python Mar 30, 2026
@wence-
Copy link
Copy Markdown
Contributor

wence- commented Mar 30, 2026

This is mathematically correct but numerically tremendously unstable. We really want the grouped aggregation machinery to expose the "intermediate" value before apply finalisation. The one-pass approach uses welford's algorithm iirc. If we xan get that out then we could use it as an input to merge two grouped objects and then continue

@Matt711
Copy link
Copy Markdown
Contributor Author

Matt711 commented Mar 30, 2026

This is mathematically correct but numerically tremendously unstable. We really want the grouped aggregation machinery to expose the "intermediate" value before apply finalisation. The one-pass approach uses welford's algorithm iirc. If we xan get that out then we could use it as an input to merge two grouped objects and then continue

Yeah I even had to switch around some of the OOOs to get the test to pass (trying to avoid small numerical precision differences).

The alternative you're suggesting is interesting. Reading variance algorithims...

@Matt711
Copy link
Copy Markdown
Contributor Author

Matt711 commented Mar 30, 2026

Interesting, so we already have M2 and MERGE_M2 aggregations in libcudf. I wonder if we can reuse them here.

#21962 computes M2 on each partition and MERGE_M2 on the partial results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cudf-polars Issues specific to cudf-polars feature request New feature or request non-breaking Non-breaking change Python Affects Python cuDF API.

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

3 participants