Feature request / RfC: Run devbox inside container to fully isolate dev environment?

[codethief]

Hi there! I've enjoyed using devbox and I think it's a really promising approach that could potentially make a lot of devs happy. That being said, I'm still a bit frustrated when it comes to isolating the dev box from the outside world, i.e. from my host machine and my $HOME:

1. The development environment inside the devbox will often not be independent of the outside environment. Instead, it might depend on it in subtle or accidental ways (e.g. if a script relies on a specific version of coreutils of the outside system, or on certain environment variables, or on binaries that were not installed in the devbox but are still available in the $PATH, et cetera).

2. The dev environment still has read/write access to the outside environment. Unfortunately, many tools – e.g. from the Python ecosystem – like write to $HOME without asking and, conversely, their behavior will depend on the dotfiles in $HOME. (I ran into this earlier: I had installed PDM through devbox/Nix and suddenly PDM was writing config settings to dotfiles in $XDG_CONFIG_HOME…)

So here comes a wild idea: What if, upon entering a devbox project, the devbox would auto-magically start a new pre-configured shell inside a devbox container that's fully isolated from the host machine? Obviously, this is by far not as easy as it sounds since the user would very likely still want to be able to use their usual shell, command line tools, shell aliases etc. inside the container. So one would have to think about how to provide the user with an easy way (probably through a config or something) to define what part of the ouside world of their host machine they would want to "leak" into the devbox on purpose, i.e. which dotfiles, binaries, etc.

@jetpack-io Do you think this could be a potential future feature for the devbox or is it out of scope?

[Lagoja]

Thank you @codethief for trying Devbox, and for the feedback!

I think we could explore providing more isolation than our current system provides. For example -- we could set $HOME or $XDG_* variables to point to directories within your Devbox project. A somewhat hacky way to test this would be to set $HOME and $XDG_* to a local folder in your project directory (e.g., ./.home/ or something similar).

As you indicate this might cause some of your expected settings/shell to break, so we'd need a way to either allow some settings to leak through, or configure some global state.

Can you share more details on the project you were working on so we can test it?

[codethief]

@Lagoja Thank you so much for your feedback and merry Christmas! :)

Can you share more details on the project you were working on so we can test it?

It wasn't anything complicated, I was merely setting up a new project and installing PDM through devbox/Nix. Then, PDM kept notifying me that a more up-to-date version was available through pip and asked me repeatedly to update it. As an alternative option it offered me to disable check_update – which I accepted. But, unfortunately, it wrote that config option to $XDG_CONFIG_HOME/pdm/, so it wouldn't get tracked by git and other contributors would be bothered by the same update-me message again – which in other projects/repos might obviously be very useful, so disabling updates globally in $XDG_CONFIG_HOME/pdm/ doesn't make much sense in the first place. Long story short, I think every devbox should also come, as you say, with separate $HOME and $XDG_CONFIG_HOME folders. But I wouldn't stop there and also provide a separate /usr/local/bin etc.

All this is not to say that the user shouldn't be able to re-use their global dotfiles & binaries but it should at least be a conscious decision.

[loreto]

Adding to what John is saying.

I think we can do two things:

1. Modify HOME/XDG to provide more isolation in devbox environments. This is still not perfect isolation, but it would increase it.

2. For cases where you need perfect isolation, we could provide an additional command that starts the shell inside a docker container. Something like devbox container shell or devbox shell --mode=container. This would give perfect isolation, but the downside is that it'll be slower and require docker. I can see giving users the option though, and they can decide which environment they need at a given time.

Is one of these two approaches better for your particular use case?

[ooraini]

I haven't started using devbox yet, but isolating my machine from the environment I working with is very important for me. In particular isolating the tools. configs and credentials.

What I'm doing right now is mounting some directory on the host as the home directory in the container:

start:
        podman run \
        --privileged \
        --name xyz \
        --userns=keep-id \
        --user=root \
        -v /home/ooraini/xyz:/home/ooraini \
        -it \
        -d \
        archlinux:base-devel /sbin/init || podman start xyz

replace xyz with your own environment. This way, cloud, Terraform, Kubernetes, etc... have their credentials in the subdirctory, which still accessible from the host in case I'm running some UI tool that needs to access them.

In addition you get an isolated network(relatively), and in the future you may even add a wireguard interface to the container, which makes some particular resources only available from inside the container.

[codethief]

@loreto Personally, I wouldn't make it (only) a command line option but rather a setting in the devbox.json. The reason being that the value proposition of devbox is that its behavior will be the same across developer machines.

As for performance, what bottlenecks do you have in mind specifically?

[loreto]

@loreto Personally, I wouldn't make it (only) a command line option but rather a setting in the devbox.json. The reason being that the value proposition of devbox is that its behavior will be the same across developer machines.

As for performance, what bottlenecks do you have in mind specifically?

Apologies for the delay – I went on holiday break and finally we're restarting all work.

When I talk about bottlenecks with containers: it's mainly for developers non-linux environments (windows or MacOS) usually. Docker introduces a virtualization layer to be able to run a container, and that layer usually has to map the file system inside the container to the filesystem in the host computer. For non-linux environments that tends to be a bit slow – so if you're compiling large projects, the time difference between working inside the container vs in your local machine can be large.

[codethief]

I see, thanks for elaborating! I wasn't even aware Nix runs on Windows. TIL.

In any case, I don't think there'd be anything wrong with a command line option to selectively turn off the containment feature. Or maybe one should think about introducing a devbox.user.json for such purposes which would not be version-controlled and which each developer could use to adapt the configuration to their needs, e.g. enable/disable containment, configure containment details (configure the shell, available system binaries, networking etc.), …

[dudicoco]

@loreto @codethief I believe using Nix Sandboxing would solve this problem.

[codethief]

@dudicoco I'm not a Nix expert by any means but as far as I understand this concerns (only?) the build process of Nix packages.

[dudicoco]

@codethief I'm barely a Nix beginner :)
I guess I was wrong about how that feature works.