Adding '--environ' to the service file run command leaks credentials

[dpantel]

The distributed .service files use ExecStart=/usr/bin/caddy run --environ... command to start caddyserver.

This means that all environment variables get logged to journald (or equivalent).

Many of us, following various guides and recommendations, use environment variables to store secrets, such as tls-dns API credentials, as one example, and would prefer to keep them out of the journal.

Is there a reason --environ option is used by default?

[carlwgeorge]

That flag was first added in 9188220 with a justification of making troubleshooting easier. If that's not what you want, you can trivially override parts of the unit file with overrides.

[dpantel]

I understand overrides. I am questioning this default, in light of so many places recommending ENVVARS for storing/passing secrets.

[carlwgeorge]

Personally I don't see a big difference in security posture between secrets being in environment variables versus logged to the journal. On most distros you have to be root or in an equivalent group (wheel or adm) to view system journal messages, and if you have that level of permissions you can just read /proc/<PID>/environ to see the environment variables for a process. Because of that I'm fine with the default as it currently stands, but I'm interested to see what the other Caddy maintainers think.

[mholt]

As discussed in Slack:

This is quite helpful for troubleshooting because it guarantees that the information is from the proper Caddy process. Sometimes people will run caddy environ or caddy run --environ and post that output, not realizing it's different when your service process prints the environment (because the environments are different!). Usually they just end up restarting their service process to surface the env info to the end of the logs.

So, other than a bit of a hacky trick suggested by @carlwgeorge to get the env vars from /proc, using --environ (or caddy environ) is the only way to get the actual process environment. And actually, we print a little more than just the literal environment, it includes version, data dir, config autosave path, working directory, and more.

I'd be OK with removing this from our service files by default if we implement an admin endpoint to output the same info. That way, it might actually solve the problem of people getting the environment of the wrong caddy process.

Another reason to have it on by default -- or have the endpoint -- is because you often need the environment AFTER an error/bug occurs, so if you have to go back and add a flag to the command, then re-start the process to print the environment, it could have changed from when the bug happened.

So, removing it is definitely not ideal.

Thus, if we agree that we can/should add an admin API endpoint for the environment, I think removing --environ can be more acceptable.

Any objections?

[dpantel]

As just an end-user, who stumbled upon this accidentally, I would appreciate at least a warning somewhere in the documentation. Perhaps the page about envvars.

[DemiMarie]

Personally I don't see a big difference in security posture between secrets being in environment variables versus logged to the journal. On most distros you have to be root or in an equivalent group (wheel or adm) to view system journal messages, and if you have that level of permissions you can just read /proc/<PID>/environ to see the environment variables for a process. Because of that I'm fine with the default as it currently stands, but I'm interested to see what the other Caddy maintainers think.

Logs are often shipped to other places, whereas secrets very rarely are. So this is a significant problem in many situations.