Containers as a Development Environment

2022-12-21

I remember when I first started doing python development with Django and was introduced to the concept of a python virtual environment. I thought something along the lines of "oh yes! this is great! now I can easily share my project with other and they'll know the exact python packages and versions to install." I couldn't articulate it at the time, but notion of a reproducibility was indeed a revelation. Of course venv was limited to just python packages and even simple python projects have more dependencies than just the python packages they use.

It wouldn't be until years later that I was introduced to the concept of a Linux Container. It took me a moment to put things together but I realized that this was like venv but on steroids. Now, when sharing a project, I could reproduce the entire environment needed to run my application. So cool!

But what about my development environment?

The common use case for Containers is to package up an application. But can it be used for other things? Can we leverage the reproducibility of containers to help us solve other problems?

Over that last few years, and for various reasons, I've found myself jumping from one physical machine to the next (sometimes using multiple machines at the same time). For each switch, I would need to dedicate some time to getting the machine up and running. Managing my dotfiles was the first problem I tried to tackle. I tried a number of version control schemes, the benefit being two-fold: I got a history of my changes while being easily to able to sync them between machines. Recently, I've found chezmoi and I finally feel like I've got a dotfile management system that will last. Seriously, I can not say enough wonderful things about chezmoi.

But there's still many other things I need when setting up a machine. Namely, there's a lot of software I need install. I played around with various "setup" scripts. But these always had the issue of working differently on different systems. For example, installing software on Mac OS (brew) is markedly different from Linux (e.g. apt or pacman).

I've slowly come to the realization that containers might be the solution to my problem. I recently got a new laptop from System76 and decided to finally sit down and create my "Development Container". I've got some preliminary results and I'm quite happy with them. I've got an environment consisting of my primary text editor, chezmoi, and a few language servers. At this point adding new software should be trivial.

I should say I'm certainly not the first person to come up with this idea. There are a lot of solutions that already exist that are some variation on this theme. In fact, right about the same time I started thinking about all this, Uber (my current employer) also started playing with the idea. Uber calls them "Devpods" and the team building them has done a really great job. I feel obligated to say that I've learned a lot about the problem space from them and I've shamelessly cribbed some of their ideas for my solution.

Some notable highlights

My initial version of this Container was based on Debian since that's the Linux with which I'm most familiar. However, Debian is notorious for trading off up-to-date software in exchange for stability, and many of the tools I wanted to use (e.g. Neovim or Rust) are both new and have fast paced release cycles. After jumping through several hoops and building a small script to fetch and install binaries, I decided to give Arch a try instead. The result was a dramatically simplified Dockerfile.

I also wanted a clear separation between the software used for development and source code itself. The former being fully managed by my Dockerfile and essentially ephemeral (i.e. easily upgradeable) while the later is persistent. As a result, the system has the following properties:

The home directory is stored in a persistent volume that is mounted into the container at runtime. This allows me to keep my work while making changes to the conatiner image. I can even move the volume to be stored remotely and mounted over a network one day if I want to.
The root file system (except for /home/devuser) is essentially immutable during runtime. This has the nice property of making the things I want to be persistent independent from the things that will change over time. The result is something where I can easily install new software by changing the Dockerfile and then just pickup where I left off by mounting my persistent home directory into the new image. This is probably the most important thing I learned from my experience with Devpods, so I made sure to replicated it.

Why didn't you use Nix?

Nix is very much suited for what I'm trying to accomplish here. It's also more hermetic, ensuring that the exact versions of tools are installed when building an environment. I investigated this and I just couldn't get past the Nix DSL. For the time being, I much prefer the no-nonsense, easily readable Dockerfile.

Future work

There's a number of other language servers and binaries I want to add. But what I'm most looking forward is actually not having to open an interactive session into container. I've found a new Neovim interface, Neovide and it's pretty slick. Utilizing the --remote-tcp feature, I should be able to:

Run a headless nvim instance in my container (with all of my language server's and other dependencies installed).
Then connect to the Neovim instance with Neovide.

We'll see what happens!