Implement Receiver resource filtering with CEL #948

bigkevmcd · 2024-10-10T05:37:38Z

This implementation allows filtering of wildcarded resources for receivers using CEL expressions.

Closes: #491

bigkevmcd · 2024-10-10T05:39:57Z

@stefanprodan @matheuscscp I reworked it to filter the resources using CEL.

I'm not quite sure that this is really that valuable, I suspect label filtering is also more efficient?

There's some optimisation of the CEL that could be done within requestReconciliation to reduce the creation of CEL values and I can do that but I thought I'd get this out for comment.

internal/server/receiver_handlers.go

matheuscscp

Thanks again for all the work here, @bigkevmcd! I'm very excited for this feature 😄

This is just a preliminary review. I will try to review this again soon.

internal/server/receiver_handlers.go

stefanprodan

LGTM

Thanks @bigkevmcd 🥇

matheuscscp

LGTM! Thanks @bigkevmcd! Pretty cool contribution 👍

internal/server/cel.go

internal/server/receiver_handlers.go

api/v1/receiver_types.go

stefanprodan · 2024-11-01T20:31:01Z

@bigkevmcd this is good to go, please signoff all your commits and force push.

internal/controller/receiver_controller.go

internal/server/cel.go

bigkevmcd · 2024-11-13T16:43:36Z

@darkowlzz It'll be a couple of weeks before I can devote more time to these things, but I will get to them.

darkowlzz

Almost concluding my CEL research with this. I don't think there's any hurry. This can be addressed whenever you're back to it. 🙂

internal/server/cel.go

pjbgf

@bigkevmcd really good stuff, I am looking forward to seeing this landing. 🙇

internal/server/receiver_handlers.go

internal/server/cel.go

bigkevmcd · 2024-11-18T19:40:35Z

@pjbgf

+1 on leaning on k8s' default limits as a starting point. It might be worth just confirming whether the payload and expression sizes for this use case are equivalent, so that those limits would not be too forgiving.

https://github.com/kubernetes/kubernetes/blob/cf480a3a1a9cb22f3439c0a7922822d9f67f31b5/staging/src/k8s.io/apiserver/pkg/apis/cel/config.go#L40

That's 3MiB, I've definitely seen GitHub hook notifications that were > 1.5MiB.

matheuscscp

New batch of comments ❤️

api/v1/receiver_types.go

docs/spec/v1/receivers.md

api/v1/condition_types.go

matheuscscp

Still need to go over internal/server/receiver_handlers.go once again

docs/spec/v1/receivers.md

internal/controller/receiver_controller.go

internal/server/cel.go

internal/controller/receiver_controller.go

internal/server/cel.go

internal/server/cel_test.go

matheuscscp · 2025-01-31T14:48:49Z

@bigkevmcd we have released the fluxcd/pkg packages, please pull them in with:

go get github.com/fluxcd/pkg/apis/[email protected] for InvalidCELExpressionReason
go get github.com/fluxcd/pkg/[email protected] for the runtime/cel library

🙏

Edit: Like Stefan said below, just need to rebase.

stefanprodan · 2025-02-01T07:20:13Z

@bigkevmcd if you rebase with main, you'll find runtime/cel and the InvalidCELExpressionReason.

Signed-off-by: Kevin McDermott <[email protected]> Co-authored-by: Matheus Pimenta <[email protected]>

stefanprodan

🚀

matheuscscp · 2025-02-11T08:39:55Z

Thanks very much for this contribution @bigkevmcd ! 🚀 🚀 🚀

bigkevmcd force-pushed the cel-resource-filtering branch from 4969c46 to 1fe996b Compare October 10, 2024 05:40

github-advanced-security bot found potential problems Oct 10, 2024

View reviewed changes

internal/server/receiver_handlers.go Fixed Show fixed Hide fixed

bigkevmcd force-pushed the cel-resource-filtering branch 3 times, most recently from d87dbfd to b1c4e7e Compare October 10, 2024 19:03

stefanprodan changed the title ~~CEL Resource Filtering~~ Implement Receiver resource filtering with CEL Oct 11, 2024

bigkevmcd force-pushed the cel-resource-filtering branch from 45bd724 to 1e6929c Compare October 13, 2024 14:57

bigkevmcd marked this pull request as ready for review October 13, 2024 15:18

bigkevmcd marked this pull request as draft October 16, 2024 07:30

bigkevmcd force-pushed the cel-resource-filtering branch from 1e6929c to d658d4a Compare October 17, 2024 08:46

bigkevmcd marked this pull request as ready for review October 17, 2024 08:54

matheuscscp reviewed Oct 17, 2024

View reviewed changes

internal/server/receiver_handlers.go Show resolved Hide resolved

github-advanced-security bot found potential problems Oct 17, 2024

View reviewed changes

internal/server/receiver_handlers.go Fixed Show fixed Hide fixed

bigkevmcd force-pushed the cel-resource-filtering branch 4 times, most recently from 5a22c4f to bddfd56 Compare October 18, 2024 13:06

stefanprodan approved these changes Oct 28, 2024

View reviewed changes

matheuscscp approved these changes Oct 28, 2024

View reviewed changes

darkowlzz reviewed Oct 31, 2024

View reviewed changes

internal/server/cel.go Outdated Show resolved Hide resolved

internal/server/receiver_handlers.go Outdated Show resolved Hide resolved

api/v1/receiver_types.go Show resolved Hide resolved

bigkevmcd force-pushed the cel-resource-filtering branch 6 times, most recently from 8e892ef to 18c1f31 Compare November 1, 2024 14:34

stefanprodan reviewed Nov 1, 2024

View reviewed changes

api/v1/receiver_types.go Outdated Show resolved Hide resolved

bigkevmcd force-pushed the cel-resource-filtering branch from 8a9e435 to 57c36b0 Compare November 1, 2024 20:58

darkowlzz added the area/api API related issues and pull requests label Nov 4, 2024

github-advanced-security bot found potential problems Nov 10, 2024

View reviewed changes

internal/controller/receiver_controller.go Fixed Show fixed Hide fixed

bigkevmcd force-pushed the cel-resource-filtering branch from 348c595 to 2636e01 Compare November 10, 2024 16:52

darkowlzz reviewed Nov 13, 2024

View reviewed changes

bigkevmcd force-pushed the cel-resource-filtering branch from 891a42c to 757f96d Compare November 13, 2024 17:44

darkowlzz reviewed Nov 14, 2024

View reviewed changes

internal/server/cel.go Outdated Show resolved Hide resolved

pjbgf reviewed Nov 15, 2024

View reviewed changes

bigkevmcd force-pushed the cel-resource-filtering branch from 757f96d to 1645796 Compare December 12, 2024 20:03

bigkevmcd force-pushed the cel-resource-filtering branch from 1645796 to f79599e Compare January 12, 2025 17:55

This was referenced Jan 28, 2025

[RFC-0009] Add InvalidCELExpressionReason to apis/meta fluxcd/pkg#859

Merged

[RFC-0009] Add healthcheck expressions to kustomize types fluxcd/pkg#857

Merged

[RFC-0009] Add CEL library with custom healthchecks to runtime fluxcd/pkg#858

Merged

bigkevmcd force-pushed the cel-resource-filtering branch from f79599e to e438ca1 Compare January 28, 2025 13:45

matheuscscp self-requested a review January 28, 2025 17:12

matheuscscp reviewed Jan 28, 2025

View reviewed changes

matheuscscp reviewed Jan 29, 2025

View reviewed changes

bigkevmcd force-pushed the cel-resource-filtering branch from 15d0f16 to eb6f1ce Compare February 3, 2025 08:32

matheuscscp mentioned this pull request Feb 3, 2025

runtime: downgrade github.com/google/cel-go and cel.dev/expr fluxcd/pkg#865

Closed

bigkevmcd force-pushed the cel-resource-filtering branch from 7183c3c to fc59218 Compare February 3, 2025 19:01

matheuscscp force-pushed the cel-resource-filtering branch 2 times, most recently from 592c82e to be22b2b Compare February 10, 2025 16:03

Implement Receiver resource filtering with CEL

28deef9

Signed-off-by: Kevin McDermott <[email protected]> Co-authored-by: Matheus Pimenta <[email protected]>

matheuscscp force-pushed the cel-resource-filtering branch from be22b2b to 28deef9 Compare February 10, 2025 16:22

stefanprodan approved these changes Feb 11, 2025

View reviewed changes

stefanprodan merged commit 2eb3c35 into fluxcd:main Feb 11, 2025
6 checks passed

Implement Receiver resource filtering with CEL #948

Implement Receiver resource filtering with CEL #948

Uh oh!

Conversation

bigkevmcd commented Oct 10, 2024 • edited by stefanprodan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bigkevmcd commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

matheuscscp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stefanprodan left a comment

Choose a reason for hiding this comment

Uh oh!

matheuscscp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefanprodan commented Nov 1, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bigkevmcd commented Nov 13, 2024

Uh oh!

darkowlzz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pjbgf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bigkevmcd commented Nov 18, 2024

Uh oh!

matheuscscp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matheuscscp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matheuscscp commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stefanprodan commented Feb 1, 2025

Uh oh!

stefanprodan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matheuscscp commented Feb 11, 2025

Uh oh!

bigkevmcd commented Oct 10, 2024 •

edited by stefanprodan

Loading

bigkevmcd commented Oct 10, 2024 •

edited

Loading

matheuscscp left a comment •

edited

Loading

matheuscscp commented Jan 31, 2025 •

edited

Loading