feat: collect environment variables in execve events #3797

slntopp · 2025-06-03T11:35:28Z

This patch adds:

Reads environment variables from the process memory during execve events.
Adds a new field envs to the ExecveEvent message in the Tetragon API.
Updates the BPF code to collect these environment variables.
Updates the Go API to include the new envs field in the ExecveEvent struct.

Partially resolves #2648

Description

We're trying to develop policies that would detect the malicious use of the LD_FLAGS and HTTP_PROXY environment variables, this PR brings in the patch that would enable it.

Example Event

Example event:

Click to expand

{
        "process_exec": {
            "process": {
                "exec_id": "bGltYS10ZXRyYWdvbjo2MTkzNDU5MzMwMTY0MzoxMTY5MDk=",
                "pid": 116909,
                "uid": 501,
                "cwd": "https://example.com /home/slnt_opp.linux",
                "binary": "/usr/bin/curl",
                "flags": "execve clone",
                "start_time": "2025-05-30T15:22:21.118200051Z",
                "auid": 501,
                "parent_exec_id": "bGltYS10ZXRyYWdvbjozMjY4MjMwMDAwMDAwOjU4NTgw",
                "tid": 116909,
                "in_init_tree": false,
                "environment_variables": "HTTP_PROXY=https://evil.com SHELL=/bin/bash COLORTERM=truecolor TERM_PROGRAM_VERSION=1.100.2 PWD=/home/slnt_opp.linux LOGNAME=slnt_opp XDG_SESSION_TYPE=tty VSCODE_GIT_ASKPASS_NODE=/home/slnt_opp.linux/.vscode-server/cli/servers/Stable-848b80aeb52026648a8ff9f7c45a9b0a80641e2e/server/node HOME=/home/slnt_opp.linux LANG=C.UTF-8 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90: SSL_CERT_DIR=/usr/lib/ssl/certs GIT_ASKPASS=/home/slnt_opp.linux/.vscode-server/cli/servers/Stable-848b80aeb52026648a8ff9f7c45a9b0a80641e2e/server/extensions/git/dist/askpass.sh SSH_CONNECTION=192.168.5.2 32771 192.168.5.15 22 VSCODE_GIT_ASKPASS_EXTRA_ARGS= LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=| /usr/bin/lesspipe %s USER=slnt_opp VSCODE_GIT_IPC_HANDLE=/run/user/501/vscode-git-9873813af7.sock SHLVL=2 XDG_SESSION_ID=2 XDG_RUNTIME_DIR=/run/user/501 SSL_CERT_FILE=/usr/lib/ssl/cert.pem SSH_CLIENT=192.168.5.2 32771 22 VSCODE_GIT_ASKPASS_MAIN=/home/slnt_opp.linux/.vscode-server/cli/servers/Stable-848b80aeb52026648a8ff9f7c45a9b0a80641e2e/server/extensions/git/dist/askpass-main.js XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop BROWSER=/home/slnt_opp.linux/.vscode-server/cli/servers/Stable-848b80aeb52026648a8ff9f7c45a9b0a80641e2e/server/bin/helpers/browser.sh PATH=/usr/local/go/bin:/home/slnt_opp.linux/.vscode-server/cli/servers/Stable-848b80aeb52026648a8ff9f7c45a9b0a80641e2e/server/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin:/sbin:/usr/sbin:/sbin:/usr/sbin:/sbin:/home/slnt_opp.linux/.vscode-server/data/User/globalStorage/github.copilot-chat/debugCommand:/usr/sbin:/sbin VERSION=1.54.0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/501/bus OLDPWD=/home/slnt_opp.linux/tetragon TERM_PROGRAM=vscode VSCODE_IPC_HOOK_CLI=/run/user/501/vscode-ipc-f6a9cd71-1155-437b-b0b1-e0a27e225b69.sock _=/usr/bin/curl"
            },
            "parent": {
                "exec_id": "bGltYS10ZXRyYWdvbjozMjY4MjMwMDAwMDAwOjU4NTgw",
                "pid": 58580,
                "uid": 501,
                "cwd": "/home/slnt_opp.linux",
                "binary": "/usr/bin/bash",
                "arguments": "--init-file /home/slnt_opp.linux/.vscode-server/cli/servers/Stable-848b80aeb52026648a8ff9f7c45a9b0a80641e2e/server/out/vs/workbench/contrib/terminal/common/scripts/shellIntegration-bash.sh",
                "flags": "procFS auid",
                "start_time": "2025-05-29T23:04:34.754897450Z",
                "auid": 501,
                "parent_exec_id": "bGltYS10ZXRyYWdvbjoyMTQ5MjAwMDAwMDA6MjgzNQ==",
                "tid": 58580,
                "in_init_tree": false
            }
        },
        "node_name": "lima-tetragon",
        "time": "2025-05-30T15:22:21.118199468Z"
    }

netlify · 2025-06-03T11:36:14Z

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	`1c7d28f`
🔍 Latest deploy log	https://app.netlify.com/projects/tetragon/deploys/68655cac8fd0520008ab6abf
😎 Deploy Preview	https://deploy-preview-3797--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

michi-covalent

thanks for the PR, i think this is super useful ❤️

could we do something similar to https://github.com/cilium/tetragon/pull/1296/files#diff-de32ec7437bcdafd9a6442b53e938fbf4a100cc12b014b28412e0e1a3146e510R264 and set the environment_variables field only for process_exec events to avoid increasing sizes of other events?
probably makes sense to put this behind a feature flag (--enable-environment-variables or something). some users might not want to expose environment variables as tetragon events, as they might contain sensitive information.
alternatively, make it configurable which environment variables get exported, like --export-environment-variables=LD_FLAGS,HTTP_PROXY. then default to an empty list, which effectively disables the feature, and maybe --export-environment-variables=* to export all the environment variables 💭

jrfastab · 2025-06-04T05:01:11Z

I also think exposing a feature flag for this is a good idea. Although BPF code is likely not too much overhead it also isn't free so if folks don't want to include it I think its easier to just have a big switch. Even with the flag having filters would be nice to have. But, if with the flag I think we can do filters as a follow up.

slntopp · 2025-06-05T15:16:01Z

@michi-covalent @jrfastab do you think it makes sense to parse them into a map as well then?
It would be easier to enable filtering later without incurring further changes to proto

olsajiri

thanks for the change, looks good

please split the generated changes in separate commits, I left some other comments

api/v1/tetragon/tetragon.proto

olsajiri · 2025-06-16T12:53:41Z

pkg/process/process.go

+	// EnvironmentVariables is a string map of unprocessed environment variables
+	// that were passed to the process. This is not stored into the process,
+	// but is used to construct the ProcessExec event.
+	environmentVariables string


I dont understand the command, what you mean by 'unprocessed environment variables' and 'this is not stored in the process' ?

Similar to binary properties above, it's stored as part of the ProcessExec, but not Process

bpf/process/bpf_execve_event.c

api/v1/tetragon/tetragon.proto

bpf/process/bpf_execve_event.c

olsajiri · 2025-06-16T13:13:11Z

bpf/process/bpf_execve_event.c

@@ -229,6 +302,7 @@ event_execve(struct exec_ctx_struct *ctx)
 	p->size += read_path(ctx, event, filename);
 	p->size += read_args(ctx, event);
 	p->size += read_cwd(ctx, p);
+	p->size += read_envs(ctx, event);


so we have the flag to enable this, but the bpf and parsing code is unconditional, right?
I think it should be behind some config check, and I wonder we could put it in tg_conf_map ? not sure..

We could move the feature flag handling completely into the bpf code, i like that, could do if you green-light patching tg_conf_map

olsajiri · 2025-06-18T11:22:59Z

@slntopp could you also please update the docs? and if possible it'd be great to have some filtering possibility, probably passed in as and options like --environment-variables="VAR1,VAR2,!VAR3,.." or any better way you could think of ;-) thanks

michi-covalent

protobuf & command line flag LGTM ❤️

api/v1/tetragon/tetragon.proto

olsajiri

thanks for the change, left some comments and some notes:

provide full commit subject and message for 2nd commit
keep just 1 Signed-off-by in 1st commit together with description of the changes
the change in execParse needs tests, please add
CI fails on existing execParse tests
you still read the environment unconditionally

bpf/process/bpf_execve_event.c

olsajiri · 2025-07-01T11:18:13Z

pkg/sensors/exec/exec_linux.go

+		// The content of 'args' is uncertain. For safety, assume no reliable args/envs.
+		// Alternatively, one might try to parse 'args' if BPF guarantees payload structure despite filename error.
+		// For now, if EventErrorFilename is set, we won't attempt to parse args/envs from this buffer.
+		args = nil
 	}


this seems like separate fix, please move it to separate commit

slntopp · 2025-07-03T12:18:31Z

pkg/api/confapi/confapi.go

@@ -10,6 +10,7 @@ type TetragonConf struct {
 	TgCgrpHierarchy uint32 `align:"tg_cgrp_hierarchy"`  // Tetragon Cgroup tracking hierarchy ID
 	TgCgrpSubsysIdx uint32 `align:"tg_cgrp_subsys_idx"` // Tracking Cgroup css idx at compile time
 	TgCgrpLevel     uint32 `align:"tg_cgrp_level"`      // Tetragon cgroup level
+	EnvVarsEnabled  uint64 `align:"env_vars_enabled"`   // Whether to read environment variables


Has to be uint64 due to the Golang's structs alignment

This patch adds: 1. Reads environment variables from the process memory during execve events. 2. Adds a new field `envs` to the `ExecveEvent` message in the Tetragon API. 3. Updates the BPF code to collect these environment variables. 4. Updates the Go API to include the new `envs` field in the `ExecveEvent` struct. 5. Adds `--enable-process-environment-variables` flag to conditionally read env vars with ProcessExec events Partially resolves cilium#2648 Signed-off-by: Mikita Iwanowski <[email protected]>

- protobuf - cli doc Signed-off-by: Mikita Iwanowski <[email protected]>

Handle potentially corrupted buffer data in execParse Signed-off-by: Mikita Iwanowski <[email protected]>

slntopp · 2025-07-17T12:10:22Z

Hey @olsajiri which docs or other patches are prio to complete this patch?
Also i'd prefer to do the env vars parsing & filtering with a followup PR, if that's okay

olsajiri

leaving some more comments, plus CI is failing, thanks

olsajiri · 2025-07-20T12:03:57Z

bpf/process/bpf_execve_event.c

+read_envs(void *ctx, struct msg_execve_event *event)
+{
+	__s32 key = 0;
+	struct tetragon_conf *conf = map_lookup_elem(&tg_conf_map, &key);


please add empty line between declarations and the code

olsajiri · 2025-07-20T12:04:47Z

bpf/process/bpf_execve_event.c

+		return 0;
+
+	struct task_struct *task = (struct task_struct *)get_current_task();
+	struct msg_process *p = &event->process;


stuck varibles declarations together at the top of the function

olsajiri · 2025-07-20T12:13:31Z

bpf/process/bpf_execve_event.c

+		return total_event_bytes_written;
+
+	if (envs_data_len < BUFFER && envs_data_len < remaining_space_in_event_buf) {
+		__u32 bytes_to_copy = envs_data_len & 0x3ff; /* BUFFER - 1 */


please add empty line between declarations and the code

olsajiri · 2025-07-20T12:14:27Z

bpf/process/bpf_execve_event.c

+	probe_read(&env_start_addr, sizeof(env_start_addr), _(&mm->env_start));
+	probe_read(&env_end_addr, sizeof(env_end_addr), _(&mm->env_end));
+
+	if (!env_start_addr || !env_end_addr || env_start_addr >= env_end_addr) {


please don't use {} for conditions with oneline code paths

olsajiri · 2025-07-20T12:19:26Z

pkg/sensors/exec/exec_linux.go

@@ -144,6 +144,9 @@ func execParse(reader *bytes.Reader) (processapi.MsgProcess, bool, error) {
 			proc.Filename = strutils.UTF8FromBPFBytes(args[:])
 			args = nil
 		}
+	} else if exec.Flags&api.EventErrorFilename != 0 {


this should be in commit together with the hunk below, no?
also I think this commit should go in before the env changes

diff --git a/pkg/sensors/exec/exec_linux.go b/pkg/sensors/exec/exec_linux.go index 47a6e3fe24ae..5e9d1912a129 100644 --- a/pkg/sensors/exec/exec_linux.go +++ b/pkg/sensors/exec/exec_linux.go @@ -139,41 +139,131 @@ func execParse(reader *bytes.Reader) (processapi.MsgProcess, bool, error) { if n != -1 { proc.Filename = strutils.UTF8FromBPFBytes(args[:n]) args = args[n+1:] + } else { + // Filename not null-terminated or buffer consumed + proc.Filename = strutils.UTF8FromBPFBytes(args[:]) + args = nil } }

olsajiri · 2025-07-20T12:27:14Z

pkg/sensors/exec/exec_linux.go

+	var argsPayload []byte
+	var envsPayload []byte
+
+	// --- Robust args/envs split logic ---


this change is too big and scary on very crucial code

if you need to change current args retrieval to fit in the environment stuff,
please do that in multiple incremental changes, so we can be sure we don't break things

olsajiri · 2025-07-20T12:28:50Z

bpf/process/bpf_execve_event.c

@@ -126,6 +127,84 @@ read_args(void *ctx, struct msg_execve_event *event)
 	return size;
 }

+FUNC_INLINE __u32


please split ebpf and golang change in separate commits (ebpf goes first)

olsajiri · 2025-07-20T12:34:32Z

Hey @olsajiri which docs or other patches are prio to complete this patch?

probably in here https://github.com/cilium/tetragon/blob/main/docs/content/en/docs/concepts/events.md

Also i'd prefer to do the env vars parsing & filtering with a followup PR, if that's okay

ok

slntopp marked this pull request as ready for review June 3, 2025 11:38

slntopp requested a review from a team as a code owner June 3, 2025 11:38

slntopp requested a review from jrfastab June 3, 2025 11:38

michi-covalent reviewed Jun 4, 2025

View reviewed changes

slntopp force-pushed the slntopp/collect-envs branch from 06b43e0 to 45b9c8c Compare June 5, 2025 15:37

slntopp force-pushed the slntopp/collect-envs branch 2 times, most recently from 029f7ef to 5bb052b Compare June 16, 2025 11:30

slntopp requested a review from mtardy as a code owner June 16, 2025 11:30

slntopp requested a review from michi-covalent June 16, 2025 11:30

olsajiri requested changes Jun 16, 2025

View reviewed changes

michi-covalent approved these changes Jun 18, 2025

View reviewed changes

api/v1/tetragon/tetragon.proto Show resolved Hide resolved

kkourt added the needs-rebase This PR needs to be rebased because it has merge conflicts. label Jun 19, 2025

slntopp force-pushed the slntopp/collect-envs branch 2 times, most recently from c5e2821 to 91fbbf3 Compare June 26, 2025 13:29

slntopp requested a review from olsajiri June 26, 2025 13:29

olsajiri requested changes Jul 1, 2025

View reviewed changes

slntopp force-pushed the slntopp/collect-envs branch 2 times, most recently from 1c7d28f to 0d067ad Compare July 3, 2025 12:17

slntopp commented Jul 3, 2025

View reviewed changes

slntopp force-pushed the slntopp/collect-envs branch from 0d067ad to 76c1cdb Compare July 3, 2025 12:35

slntopp added 3 commits July 3, 2025 14:42

codegen: generate Go protobuf code and CLI docs

eb4459c

- protobuf - cli doc Signed-off-by: Mikita Iwanowski <[email protected]>

fix: handle filename read errors in execParse function

538029d

Handle potentially corrupted buffer data in execParse Signed-off-by: Mikita Iwanowski <[email protected]>

slntopp force-pushed the slntopp/collect-envs branch from 76c1cdb to 538029d Compare July 3, 2025 12:43

slntopp requested a review from olsajiri July 3, 2025 12:48

olsajiri requested changes Jul 20, 2025

View reviewed changes

feat: collect environment variables in execve events #3797

Are you sure you want to change the base?

feat: collect environment variables in execve events #3797

Uh oh!

Conversation

slntopp commented Jun 3, 2025

Description

Example Event

Uh oh!

netlify bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for tetragon ready!

Uh oh!

michi-covalent left a comment

Choose a reason for hiding this comment

Uh oh!

jrfastab commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slntopp commented Jun 5, 2025

Uh oh!

olsajiri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olsajiri commented Jun 18, 2025

Uh oh!

michi-covalent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

olsajiri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slntopp commented Jul 17, 2025

Uh oh!

olsajiri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olsajiri commented Jul 20, 2025

Uh oh!

Uh oh!

netlify bot commented Jun 3, 2025 •

edited

Loading

jrfastab commented Jun 4, 2025 •

edited

Loading