Add "emitEmptyBuckets" parameter to the "Bucket" function. #131609

przemekwitek · 2025-07-21T09:40:06Z

-- DRAFT --

This PR extends the BUCKET function by adding the ability to emit empty buckets (i.e. buckets with no data).

Currently, the BUCKET function can work in two modes and can have 2 or 4 arguments (see docs).
With this PR, the 5th parameter (emitEmptyBuckets, name to be finalized) is introduced. The emitEmptyBuckets parameter is of boolean type and has the following semantics:

false: empty buckets will not be emitted (this is the current behavior)
true: empty buckets will be emitted. In this case, from and to parameters are required, even when the desired bucket size is provided explicitly.
The default value for emitEmptyBuckets is false for BWC.

Relates #111483

cla-checker-service · 2025-07-21T15:09:45Z

💚 CLA has been signed

ivancea

Hey, it looks quite good!
Mixed some high level comments with minor things. But I would forget later otherwise, so feel free to ignore if this will evolve!

I would like somebody else from the team check it, as they may have ideas on the critical topics here:

Doing this in the coord node
Finding the Bucket
Maybe other validations and things to do

ivancea · 2025-08-01T11:28:31Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/RailRoadDiagram.java

+                            assert toArg.optional();
+                            var emitEmptyBucketsArg = args.get(++i);
+                            assert emitEmptyBucketsArg.optional();
+                            // TODO: Should it be possible to be able to specify "emitEmptyBuckets" but not "from" and "to"?


Maybe as a continuation? After the coordinator reduced all the datanode aggs, we have the min and max in theory.
Yet, I'm not sure if that would be clear to the users, or if there's any use case. Maybe worth a discussion

ivancea · 2025-08-01T11:42:33Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Bucket.java

@@ -212,13 +221,20 @@ public Bucket(
            type = { "integer", "long", "double", "date", "keyword", "text" },
            optional = true,
            description = "End of the range. Can be a number, a date or a date expressed as a string."
-        ) Expression to
+        ) Expression to,
+        @Param(


I wonder if this should be a MapParam instead. Basically, a named parameter

ivancea · 2025-08-01T13:14:16Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Bucket.java

@@ -410,9 +475,8 @@ protected TypeResolution resolveType() {

    private TypeResolution checkArgsCount(int expectedCount) {
        String expected = null;
-        if (expectedCount == 2 && (from != null || to != null)) {


Why was this case removed?

ivancea · 2025-08-01T13:22:08Z

...n/esql/compute/src/main/java/org/elasticsearch/compute/operator/HashAggregationOperator.java

    @Override
    public void addInput(Page page) {
+        if (isInitialPage.compareAndSet(true, false)
+            && (aggregators.size() == 0 || AggregatorMode.INITIAL.equals(aggregators.get(0).getMode()))) {


For later: Would be nice if we could do this on output() on the coordinator, on FINAL aggs only.
I remember the reason to not do this was that Bucket wouldn't be on the coord physical plan, as it's moved to an early eval

ivancea · 2025-08-01T13:25:38Z

...n/esql/compute/src/main/java/org/elasticsearch/compute/operator/HashAggregationOperator.java

+                                // TODO: DocBlock doesn't allow appending nulls
+                                ((DocBlock.Builder) blockBuilders[channel]).appendShard(0).appendSegment(0).appendDoc(0);


This feels dangerous. Adding a comment to not forget it

ivancea · 2025-08-01T13:27:57Z

...n/esql/compute/src/main/java/org/elasticsearch/compute/operator/HashAggregationOperator.java

@@ -289,6 +310,42 @@ protected Page wrapPage(Page page) {
        return page;
    }

+    private Page createInitialPage(Page page) {
+        // If no groups are generating bucket keys, move on
+        if (groups.stream().allMatch(g -> g.emptyBucketGenerator() == null)) {


I would add an assert on the constructor (probably?) that either all have a generator, or none have. If there's a mix, I guess we'll have a problem

ivancea · 2025-08-01T13:31:49Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Bucket.java

+        @Override
+        public void generate(Block.Builder blockBuilder) {
+            for (double bucket = round(Math.floor(from / roundTo) * roundTo, 2); bucket < to; bucket = round(bucket + roundTo, 2)) {
+                ((DoubleBlock.Builder) blockBuilder).appendDouble(bucket);


Let's move this cast outside the for. Maybe an:

if (blockBuilder instanceof DoubleBlock.Builder doubleBlockBuilder) { // Code... } else { throw ...; }

Same for the other generator.

ivancea · 2025-08-01T13:45:55Z

...l/src/main/java/org/elasticsearch/xpack/esql/planner/AbstractPhysicalOperationProviders.java

+            if (ne.id().equals(bucketId)) {
+                if (ne.children().size() > 0 && ne.children().get(0) instanceof Bucket bucket) {
+                    foundBucket.set(bucket);
+                    // TODO: Hack used when BUCKET is wrapped with ROUND. How to generalize it?


Bucket technically can be used within any expression I think? Is this only for Round?

przemekwitek added the WIP label Jul 21, 2025

elasticsearchmachine added the v9.2.0 label Jul 21, 2025

przemekwitek force-pushed the emit_empty_buckets branch 5 times, most recently from c150e9c to 38b19d3 Compare July 21, 2025 14:58

przemekwitek force-pushed the emit_empty_buckets branch from e6e6729 to db7accd Compare July 25, 2025 10:36

Add "emitEmptyBuckets" parameter to the "Bucket" function.

3cae3af

przemekwitek force-pushed the emit_empty_buckets branch from 6ef0feb to 3cae3af Compare July 28, 2025 07:51

github-actions bot deployed to docs-preview July 28, 2025 07:52 View deployment

ivancea reviewed Aug 1, 2025

View reviewed changes

markov00 mentioned this pull request Aug 13, 2025

[ES|QL] The chart starts with the bucket that has data and doesn't respect the timepicker start time elastic/kibana#189597

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add "emitEmptyBuckets" parameter to the "Bucket" function. #131609

Add "emitEmptyBuckets" parameter to the "Bucket" function. #131609

Uh oh!

przemekwitek commented Jul 21, 2025 •

edited

Loading

Uh oh!

cla-checker-service bot commented Jul 21, 2025 •

edited

Loading

Uh oh!

ivancea left a comment

Uh oh!

ivancea Aug 1, 2025

Uh oh!

ivancea Aug 1, 2025

Uh oh!

ivancea Aug 1, 2025

Uh oh!

ivancea Aug 1, 2025

Uh oh!

ivancea Aug 1, 2025

Uh oh!

ivancea Aug 1, 2025

Uh oh!

ivancea Aug 1, 2025

Uh oh!

ivancea Aug 1, 2025

Uh oh!

Uh oh!

		// TODO: DocBlock doesn't allow appending nulls
		((DocBlock.Builder) blockBuilders[channel]).appendShard(0).appendSegment(0).appendDoc(0);

Add "emitEmptyBuckets" parameter to the "Bucket" function. #131609

Are you sure you want to change the base?

Add "emitEmptyBuckets" parameter to the "Bucket" function. #131609

Uh oh!

Conversation

przemekwitek commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cla-checker-service bot commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ivancea left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

przemekwitek commented Jul 21, 2025 •

edited

Loading

cla-checker-service bot commented Jul 21, 2025 •

edited

Loading