tetragon: Add support to retrieve environment variables #4184

olsajiri · 2025-10-13T16:18:17Z

storing environment variables into process event data, enabled with --enable-process-environment-variables option,
and we allow to redact info from the variables

reworked a bit the exec data parsing and added the environment variables parsing contributed by @slntopp,
I left the SOB info in some changes, @slntopp please shout if you want to make some changes

netlify · 2025-10-13T16:18:52Z

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	`e0d7b24`
🔍 Latest deploy log	https://app.netlify.com/projects/tetragon/deploys/69283ad5cf26ef00083b442d
😎 Deploy Preview	https://deploy-preview-4184--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

FedeDP

Left some comments but the PR is egregious :) Great job!

bpf/process/bpf_execve_event.c

kkourt

Thanks!

Please find some comments below.

bpf/process/bpf_execve_event.c

kkourt · 2025-11-20T12:51:41Z

pkg/sensors/exec/exec_linux.go

-		var desc dataapi.DataEventDesc
-
-		dr := bytes.NewReader(args)
+	if exec.SizePath != 0 {


Are there legitimate cases where this can be zero? (similarly for the other added sizes)

yes, if probe read fails on the envs data, then EVENT_ERROR_ARGS should be set

But we continue reading everything else if I understand correctly from the code?

kkourt · 2025-11-20T12:55:06Z

bpf/process/bpf_execve_event.c

 }

+#ifdef __LARGE_BPF_PROG
+FUNC_INLINE __u32 read_envs(void *ctx, struct msg_execve_event *event)


Will this work correctly for execs happening from inside the kernel (e.g, call_usrmodehelper and friends).

yep, environment is stored the same way, so we get it.. like:

{"process_exec":{"process":{"exec_id":"cWVtdS0xOjUyNzE3MDc3ODA3MjoxMjYxOA==","pid":12618,"uid":0,"cwd":"/","binary":"/usr/lib/systemd/systemd-coredump","arguments":"12617 1000 1000 11 1763675427 102400000000 qemu-1 1 3","flags":"execve rootcwd clone","start_time":"2025-11-20T21:50:27.329868374Z","auid":4294967295,"parent_exec_id":"cWVtdS0xOjI5NzAwMDAwMDA6Mg==","tid":12618,"in_init_tree":false,"environment_variables":"HOME=/ TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin"},"parent":{"exec_id":"cWVtdS0xOjI5NzAwMDAwMDA6Mg==","pid":2,"uid":0,"binary":"[kthreadd]","flags":"procFS","start_time":"2025-11-20T21:41:43.129089721Z","auid":4294967295,"parent_exec_id":"cWVtdS0xOjE6MA==","tid":2,"in_init_tree":false}},"node_name":"qemu-1","time":"2025-11-20T21:50:27.329867762Z"}

kkourt · 2025-11-20T12:57:41Z

api/v1/tetragon/tetragon.proto

  // tool like nsenter or kubectl exec.
  google.protobuf.BoolValue in_init_tree = 20;
+  // Environment variables passed to the binary at execution.
+  string environment_variables = 21;


Question: Should we parse this in user-space and offer an array to the user?
E.g.,

message EnvEntry { string Key = 1; string Value = 2; }

And return:

repeated EnvEntry environment_variables = 21

not sure.. as you already mentioned catching this impacts performance already,
also it's sort of similar to args string which we handle the same way together with the redaction support

Yeah, but splitting a string in userspace (I'm guessing based on a ) character shouldn't have a big impact on performance, no?

Yeah I'm wondering too if it's not a nice UX improvement that is pretty cheap considering everything that we already do in this gRPC conversion. No strong feelings.

ok, will add that

kkourt · 2025-11-20T12:59:53Z

pkg/option/flags.go

+	flags.Bool(KeyEnableProcessEnvironmentVariables, false, "Include environment variables in process_exec events. Disabled by default. Note that this option can significantly increase the size of the events and may impact performance")
+


Let's add a note about sensitive information in the flag.

E.g., "Include environment variables in process_exec events. Disabled by default. Note that this option can significantly increase the size of the events and may impact performance, as well as capture sensitive information such as passwords in the events"

"as well as capture sensitive information such as passwords in the events" might be super verbose but we can also mention the feature we have for redacting such content here.

mtardy

Thanks, here are a few comments, I think using size based parsing is cleaner indeed and easier to understand than searching for those NULL bytes.

bpf/process/bpf_execve_event.c

mtardy · 2025-11-20T19:05:20Z

api/v1/tetragon/tetragon.proto

  // tool like nsenter or kubectl exec.
  google.protobuf.BoolValue in_init_tree = 20;
+  // Environment variables passed to the binary at execution.
+  string environment_variables = 21;


Yeah I'm wondering too if it's not a nice UX improvement that is pretty cheap considering everything that we already do in this gRPC conversion. No strong feelings.

As the comment suggested the filename could be sent through data event, let's read the filename instead. Signed-off-by: Jiri Olsa <[email protected]>

We can return just proc object and error without the bool state. Making nopMsgProcess part of execParse init code so there's no longer need for the function. Signed-off-by: Jiri Olsa <[email protected]>

We are about to add another extra info to the exec event (environment variables), so instead of relying on separators lets add size for each data and allow the user space easier parsing. Signed-off-by: Jiri Olsa <[email protected]>

Let's share the functionality in single function. Signed-off-by: Jiri Olsa <[email protected]>

Adding test for execParse with api.EventErrorFilename and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

Add test for execParse with just filename and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

Adding support to read environment variables. It's guarded with tg_conf_map::env_vars_enabled variable and same as for arguments we either use space in the exec event or send it through separate data event. Signed-off-by: Mikita Iwanowski <[email protected]> Signed-off-by: Jiri Olsa <[email protected]>

It will store environment variables for the process. Signed-off-by: Jiri Olsa <[email protected]>

Signed-off-by: Jiri Olsa <[email protected]>

Adding parsing of the environment variables data from the exec event and the --enable-process-environment-variables option to enable this feature. Signed-off-by: Mikita Iwanowski <[email protected]> Signed-off-by: Jiri Olsa <[email protected]>

Add test for execParse with filename, args, cwd, envs and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

Add test for execParse with filename, args, cwd and empty envs and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

Add support to redact data from environment variables. Signed-off-by: Jiri Olsa <[email protected]>

Adding redact test for environment variables. Signed-off-by: Jiri Olsa <[email protected]>

Adding support to filter specific environment variables with --filter-environment-variables VAR1[,VAR2..] option. Only specified variables will be part of the event, like with: --filter-environment-variables "TEST_VAR1" only TEST_VAR1 will be stored in the event. Signed-off-by: Jiri Olsa <[email protected]>

Adding observer test with basic check for environment variables. Signed-off-by: Jiri Olsa <[email protected]>

Adding observer test with check for environment variables with applied filter. Signed-off-by: Jiri Olsa <[email protected]>

Adding observer test with check for environment variables with applied filter and redaction. Signed-off-by: Jiri Olsa <[email protected]>

Adding docs for environment variables redaction. Signed-off-by: Jiri Olsa <[email protected]>

mtardy

LGTM thanks!

mtardy · 2025-12-01T10:01:32Z

pkg/sensors/base/base.go

 		"size", strutils.SizeWithSuffix(entries*int(unsafe.Sizeof(execvemap.ExecveValue{}))))
+
+	if option.Config.EnableProcessEnvironmentVariables {
+		Execve.RewriteConstants["ENV_VARS_ENABLED"] = uint8(1)


nice thanks

mtardy · 2025-12-01T10:14:02Z

docs/content/en/docs/concepts/events.md

+It's also possible to store only requested environment variables with
+'--filter-environment-variables VAR1[,VAR2..]` option.


nit: feel free to ignore

Suggested change

It's also possible to store only requested environment variables with

'--filter-environment-variables VAR1[,VAR2..]` option.

It's also possible to store only requested environment variables with the

'--filter-environment-variables VAR1[,VAR2..]` option.

Also this could be documented in the part that describes this feature, but if we don't have any, that's cool

mtardy · 2025-12-01T10:15:04Z

bpf/process/bpf_execve_event.c

+	with_errmetrics(probe_read, &env_start, sizeof(env_start), _(&mm->env_start));
+	with_errmetrics(probe_read, &env_end, sizeof(env_end), _(&mm->env_end));


cool to use errmetrics here

olsajiri added the release-note/minor This PR introduces a minor user-visible change label Oct 13, 2025

olsajiri force-pushed the pr/olsajiri/exec branch 3 times, most recently from 0fbb328 to 9dd22a4 Compare October 13, 2025 22:08

olsajiri mentioned this pull request Oct 14, 2025

tetragon: assorted fixes #4131

Merged

olsajiri force-pushed the pr/olsajiri/exec branch 2 times, most recently from e35bac7 to 310e337 Compare October 17, 2025 11:42

olsajiri force-pushed the pr/olsajiri/exec branch 3 times, most recently from f906c12 to 6a4eec3 Compare November 11, 2025 20:59

olsajiri changed the title ~~Pr/olsajiri/exec~~ tetragon: Add support to retrieve environment variables Nov 11, 2025

olsajiri marked this pull request as ready for review November 11, 2025 21:06

olsajiri requested review from a team and mtardy as code owners November 11, 2025 21:06

olsajiri requested review from jrfastab and kkourt November 11, 2025 21:06

olsajiri force-pushed the pr/olsajiri/exec branch from 6a4eec3 to 8b4eaf6 Compare November 15, 2025 21:07

FedeDP reviewed Nov 18, 2025

View reviewed changes

bpf/process/bpf_execve_event.c Outdated Show resolved Hide resolved

bpf/process/bpf_execve_event.c Show resolved Hide resolved

olsajiri force-pushed the pr/olsajiri/exec branch 2 times, most recently from f4de6a0 to b546118 Compare November 18, 2025 22:16

FedeDP approved these changes Nov 19, 2025

View reviewed changes

kkourt reviewed Nov 20, 2025

View reviewed changes

mtardy reviewed Nov 20, 2025

View reviewed changes

mtardy added release-note/major This PR introduces major new functionality and removed release-note/minor This PR introduces a minor user-visible change labels Nov 20, 2025

olsajiri added 4 commits November 26, 2025 09:45

tetragon: Read 4.19 curr->bin.path from tracepoint filename

0b58655

As the comment suggested the filename could be sent through data event, let's read the filename instead. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Simplify execParse error value handling

dfab1a3

We can return just proc object and error without the bool state. Making nopMsgProcess part of execParse init code so there's no longer need for the function. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Factor data read from exec event into single function

5b7b8c2

Let's share the functionality in single function. Signed-off-by: Jiri Olsa <[email protected]>

olsajiri added 2 commits November 26, 2025 09:45

tetragon: Add test for execParse with api.EventErrorFilename

83f4120

Adding test for execParse with api.EventErrorFilename and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add test for execParse with just filename

8e86232

Add test for execParse with just filename and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the pr/olsajiri/exec branch from b546118 to 6f8def7 Compare November 26, 2025 12:02

olsajiri marked this pull request as draft November 26, 2025 12:02

slntopp and others added 3 commits November 26, 2025 14:29

tetragon: Add environment_variables string to Process message

5e5c3b8

It will store environment variables for the process. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add generated code for previous changes

bb1a9c3

Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the pr/olsajiri/exec branch 2 times, most recently from c69ad85 to 02c4cb2 Compare November 26, 2025 22:11

slntopp and others added 10 commits November 27, 2025 09:58

tetragon: Add test for execParse with environment variables

55caafa

Add test for execParse with filename, args, cwd, envs and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add test for execParse with empty environment variables

4bb6ac3

Add test for execParse with filename, args, cwd and empty envs and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Apply redaction filters also on environment variables

6fcae0b

Add support to redact data from environment variables. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add redact test for environment variables

646cc07

Adding redact test for environment variables. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add observer test for environment variables

318b205

Adding observer test with basic check for environment variables. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add observer test for environment variables with filter

61e5210

Adding observer test with check for environment variables with applied filter. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add observer test for environment variables with redaction

faead28

Adding observer test with check for environment variables with applied filter and redaction. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add docs for environment variables redaction

e0d7b24

Adding docs for environment variables redaction. Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the pr/olsajiri/exec branch from 02c4cb2 to e0d7b24 Compare November 27, 2025 11:49

olsajiri marked this pull request as ready for review November 27, 2025 13:27

olsajiri requested review from FedeDP, kkourt and mtardy November 28, 2025 16:28

mtardy approved these changes Dec 1, 2025

View reviewed changes

tpapagian approved these changes Dec 2, 2025

View reviewed changes

		flags.Bool(KeyEnableProcessEnvironmentVariables, false, "Include environment variables in process_exec events. Disabled by default. Note that this option can significantly increase the size of the events and may impact performance")

		It's also possible to store only requested environment variables with
		'--filter-environment-variables VAR1[,VAR2..]` option.

		with_errmetrics(probe_read, &env_start, sizeof(env_start), _(&mm->env_start));
		with_errmetrics(probe_read, &env_end, sizeof(env_end), _(&mm->env_end));

tetragon: Add support to retrieve environment variables #4184

Are you sure you want to change the base?

tetragon: Add support to retrieve environment variables #4184

Conversation

olsajiri commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for tetragon ready!

Uh oh!

FedeDP left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kkourt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtardy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtardy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

olsajiri commented Oct 13, 2025 •

edited

Loading

netlify bot commented Oct 13, 2025 •

edited

Loading