Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pkg/cel/environment.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ func DefaultEnvironment(options ...EnvOption) (*cel.Env, error) {
cel.OptionalTypes(),
ext.Encoders(),
library.Random(),
library.Maps(),
}

opts := &envOptions{}
Expand Down
106 changes: 106 additions & 0 deletions pkg/cel/library/maps.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
package library
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: missing license header

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@a-hilaly we should maybe write a linter for new files to avoid this. WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we had one! :) go generate should get that done. cc @tomasaschan

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


import (
"math"

"github.com/google/cel-go/cel"
"github.com/google/cel-go/common/types"
"github.com/google/cel-go/common/types/ref"
"github.com/google/cel-go/common/types/traits"
)

// Maps returns a cel.EnvOption to configure extended functions for map manipulation.
//
// # Merge
//
// Merges two maps. Keys from the first map take precedence over keys in the second map.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keys from the first map take precedence over keys in the second map. - is this a common behaviour? i have the impression that most languages/libraries do the opposite - where the second map's values overwrite the first one. i.e in python

a = {'x':1}
b= {'x':2}
print(**a, **b) # prints {'x':2}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was thinking the same, when picking this up again a few weeks back. It was motivated by the implementation, which it shouldn't be (cel map implementation doesn't seem to allow to replace values). I can fix that easily.

//
// map(string, int).merge(map(string, int)) -> map(string, int)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this mean that .merges won't work with map(string, string) types?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it takes string keys, but values can be anything. I'll change int to any, unless you have a better idea.

//
// Examples:
//
// {}.merge({}) == {}
// {}.merge({'a': 1}) == {'a': 1}
// {'a': 1}.merge({}) == {'a': 1}
// {'a': 1}.merge({'b': 2}) == {'a': 1, 'b': 2}
// {'a': 1}.merge({'a': 2, 'b': 2}) == {'a': 1, 'b': 2}
func Maps(options ...MapsOption) cel.EnvOption {
l := &mapsLib{version: math.MaxUint32}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this common to version it to MAX_INT ?

for _, opt := range options {
opt(l)
}
return cel.Lib(l)
}

type mapsLib struct {
version uint32
}

type MapsOption func(*mapsLib) *mapsLib

// LibraryName implements the cel.SingletonLibrary interface method.
func (l *mapsLib) LibraryName() string {
return "cel.lib.ext.kro.maps"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a way to indicate this is still experimental/alpha.

cel.lib.ext.experimental.maps ?
or
cel.lib.ext.kro.experimental.maps ?

}
Comment on lines +56 to +59
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we edit the library in name for random to match this one and similar patterns in cel-go? https://github.com/kro-run/kro/blob/main/pkg/cel/library/random.go#L53-L55 - https://github.com/google/cel-go/blob/master/ext/lists.go

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just matched the patterns ;). So, I think changing random would make sense from that perspective.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for both of these. Let's change up the random lib name in a separate mini PR. Shouldnt block anything here though since its mostly for conflict resolution

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we need to rename the folder from library to ext ?
Even a separate repo under kro-run ?


// CompileOptions implements the cel.Library interface method.
func (l *mapsLib) CompileOptions() []cel.EnvOption {
mapType := cel.MapType(cel.TypeParamType("K"), cel.DynType)
// mapDynType := cel.MapType(cel.DynType, cel.DynType)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
mapType := cel.MapType(cel.TypeParamType("K"), cel.DynType)
// mapDynType := cel.MapType(cel.DynType, cel.DynType)
mapType := cel.MapType(cel.TypeParamType("K"), cel.TypeParamType("V")

I'm wondering if we should restrict this function to only act on maps of similar types? Like map[string]int only merging with map[string]int. Users could leverage macros to transform map types if needed. My highlevel toughts here is that i'd rather restrict this now and relax it later, than try to make a generic function and hit tricky type situations.

Copy link
Contributor Author

@gfrey gfrey Aug 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It gets worse. It allows memory to grow, which undermines some of the core CEL assumptions, if I understand that correctly.
Despite all of this, my motivation is to get from a list of strings (say a list of availability zones to a list of subnet configurations that I can use for setting up a VPC. This would require this feature, but also the ability to merge maps of arbitrary types (not sure the suggested cel.TypeParamType("V") would allow for that).
Upon reflecting on that for some more, I agree. Let's start simple and extend if necessary. We know it is possible, but not need to go there immediately. I'll try to build it with pinned types.

opts := []cel.EnvOption{
cel.Function("merge",
cel.MemberOverload("map_merge",
[]*cel.Type{mapType, mapType},
mapType,
cel.BinaryBinding(func(arg1, arg2 ref.Val) ref.Val {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: expose this function separately instead of inlining, and then unit test this function as such instead of having to go through the environment

self, ok := arg1.(traits.Mapper)
if !ok {
return types.ValOrErr(arg1, "no such overload: %v.merge(%v)", arg1.Type(), arg2.Type())
}
other, ok := arg2.(traits.Mapper)
if !ok {
return types.ValOrErr(arg1, "no such overload: %v.merge(%v)", arg1.Type(), arg2.Type())
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wonder if the !ok checks are redundant here - doesn't CEL already validate argument types?

PS: I see what we're doing in https://github.com/kro-run/kro/blob/main/pkg/cel/library/random.go#L53-L55 and i wonder if we should fix that too :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're right, at least in the test added, the error is found prior to any execution in the env.Check call.

result, err := merge(self, other)
if err != nil {
return types.WrapErr(err)
}
return result
}),
),
),
}
return opts
}

// ProgramOptions implements the cel.Library interface method.
func (l *mapsLib) ProgramOptions() []cel.ProgramOption {
return []cel.ProgramOption{}
}

// merge merges two maps. Keys from the first map take precedence over keys in
// the second map.
func merge(self traits.Mapper, other traits.Mapper) (traits.Mapper, error) {

Check failure on line 83 in pkg/cel/library/maps.go

View workflow job for this annotation

GitHub Actions / lint

merge - result 1 (error) is always nil (unparam)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

golangcilint is complaining here because merge never returns a non-nil error (which makes it a redundant parameter)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

D'oh, my bad. Removed the unused error return value.

var result traits.MutableMapper

if mapVal, ok := self.Value().(map[ref.Val]ref.Val); ok {
result = types.NewMutableMap(types.DefaultTypeAdapter, mapVal)
} else {
result = types.NewMutableMap(types.DefaultTypeAdapter, nil)
for i := self.Iterator(); i.HasNext().(types.Bool); {
k := i.Next()
v := self.Get(k)
result.Insert(k, v)
}
}

for i := other.Iterator(); i.HasNext().(types.Bool); {
k := i.Next()
if result.Contains(k).(types.Bool) {
continue
}
v := other.Get(k)
result.Insert(k, v)
}
return result.ToImmutableMap(), nil
}
77 changes: 77 additions & 0 deletions pkg/cel/library/maps_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
package library
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto


import (
"fmt"
"strings"
"testing"

"github.com/google/cel-go/cel"
)

func TestMaps(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: using testify can make your life a lot easier here probably (we already use it in other tests in this package):

package library

import (
	"fmt"
	"testing"

	"github.com/google/cel-go/cel"
	"github.com/stretchr/testify/require"
)

func TestMaps(t *testing.T) {
	tests := []struct {
		expr string
		err  require.ErrorAssertionFunc
	}{
		{expr: `{}.merge({}) == {}`},
		{expr: `{}.merge({'a': 1}) == {'a': 1}`},
		{expr: `{}.merge({'a': 2.1}) == {'a': 2.1}`},
		{expr: `{}.merge({'a': 'foo'}) == {'a': 'foo'}`},
		{expr: `{'a': 1}.merge({}) == {'a': 1}`},
		{expr: `{'a': 1}.merge({'b': 2}) == {'a': 1, 'b': 2}`},
		{expr: `{'a': 1}.merge({'a': 2, 'b': 2}) == {'a': 2, 'b': 2}`},
		{expr: `{}.merge([])`, err: func(t require.TestingT, err error, i ...interface{}) {
			require.ErrorContains(t, err, "ERROR: <input>:1:9: found no matching overload for 'merge' applied to 'map(dyn, dyn).(list(dyn))'")
		}},
	}

	env := testMapsEnv(t)
	for i, tc := range tests {
		t.Run(fmt.Sprintf("%d", i), func(t *testing.T) {
			r := require.New(t)
			ast, iss := env.Compile(tc.expr)
			if tc.err != nil {
				tc.err(t, iss.Err())
				return
			}
			r.NoError(iss.Err(), "compile failed for expr: %s", tc.expr)

			prg, err := env.Program(ast)
			r.NoError(err, "program failed for expr: %s", tc.expr)

			out, _, err := prg.Eval(cel.NoVars())
			if tc.err != nil {
				r.Error(err, "expected error for expr: %s", tc.expr)
				r.Contains(err.Error(), tc.err)
			} else {
				r.NoError(err, "unexpected error for expr: %s", tc.expr)
				r.IsType(cel.BoolType, out.Type(), "unexpected type for expr: %s", tc.expr)
				r.True(out.Value().(bool), "unexpected result for expr: %s", tc.expr)
			}
		})
	}
}

func testMapsEnv(t *testing.T, opts ...cel.EnvOption) *cel.Env {
	t.Helper()
	env, err := cel.NewEnv(append([]cel.EnvOption{Maps()}, opts...)...)
	require.NoError(t, err, "cel.NewEnv(Maps()) failed")
	return env
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha, I had started converting the tests to using testify (back when I originally wrote this code, I didn't see that testify was available). Thanks, learnt a few tricks from this (r := require.New(t) f.ex. and usage of ErrorAssertionFunc)

mapsTests := []struct {
expr string
err string
}{
{expr: `{}.merge({}) == {}`},
{expr: `{}.merge({'a': 1}) == {'a': 1}`},
{expr: `{'a': 1}.merge({}) == {'a': 1}`},
{expr: `{'a': 1}.merge({'b': 2}) == {'a': 1, 'b': 2}`},
{expr: `{'a': 1}.merge({'a': 2, 'b': 2}) == {'a': 1, 'b': 2}`},
{expr: `{'a': 1}.merge({'b': 2.1}) == {'a': 1, 'b': 2.1}`},
{expr: `{'a': 1}.merge({'b': 'foo'}) == {'a': 1, 'b': 'foo'}`},
{expr: `{'a': 1}.merge({'b': {'c': 2}}) == {'a': 1, 'b': {'c': 2}}`},

// {expr: `{}.merge([])`, err: "ERROR: <input>:1:9: found no matching overload for 'merge' applied to 'map(dyn, dyn).(list(dyn))'"},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: do we want dynamic support in here? If so, either uncomment the test, add todo for the comment if outstanding, or remove the comments

}

env := testMapsEnv(t)
for i, tc := range mapsTests {
t.Run(fmt.Sprintf("%d", i), func(t *testing.T) {
var asts []*cel.Ast
pAst, iss := env.Parse(tc.expr)
if iss.Err() != nil {
t.Fatalf("env.Parse(%v) failed: %v", tc.expr, iss.Err())
}
asts = append(asts, pAst)
cAst, iss := env.Check(pAst)
if iss.Err() != nil {
t.Fatalf("env.Check(%v) failed: %v", tc.expr, iss.Err())
}
asts = append(asts, cAst)

for _, ast := range asts {
prg, err := env.Program(ast)
if err != nil {
t.Fatalf("env.Program() failed: %v", err)
}
out, _, err := prg.Eval(cel.NoVars())
if tc.err != "" {
if err == nil {
t.Fatalf("got value %v, wanted error %s for expr: %s",
out.Value(), tc.err, tc.expr)
}
if !strings.Contains(err.Error(), tc.err) {
t.Errorf("got error %v, wanted error %s for expr: %s", err, tc.err, tc.expr)
}
} else if err != nil {
t.Fatal(err)
} else if out.Value() != true {
t.Errorf("got %v, wanted true for expr: %s", out.Value(), tc.expr)
}
}
})
}
}

func testMapsEnv(t *testing.T, opts ...cel.EnvOption) *cel.Env {
t.Helper()
baseOpts := []cel.EnvOption{
Maps(),
}
env, err := cel.NewEnv(append(baseOpts, opts...)...)
if err != nil {
t.Fatalf("cel.NewEnv(Maps()) failed: %v", err)
}
return env
}
Loading