Use bikeshare to get around? Make sure there are always docks where you’re going, and bikes when you’re leaving, with
My Adventure With LXC Containers
Over the last week or so, I’ve been playing around with LXC containers, low level virtualization containers in Linux. (Specifically, I’ve been using LXC version 2.0.9 with Ubuntu 16.04, on a cheap Digital Ocean box I setup for my experiment.)
Basic LXC Usage
First, in case you want to play along, installing LXC is incredibly easy:
$ sudo apt-get update && sudo apt-get install lxc
Create containers from a given template. For most of my playing around, I used a base
ubuntu template, but you can find far more with
apt-get install lxc-templates.
# lxc-create -n container_name -t template $ lxc-create -n webserver -t ubuntu
Each container lives in
/var/lib/lxc/$CONTAINER_NAME. Exploring this directory shows how simple a container really is:
$ ls /var/lib/lxc/webserver config rootfs/
config is created automatically, as is the root filesystem. By default, containers don’t start automatically, but you can change this easily by adding the following to your config:
lxc.start.auto = 1
Henceforth, you can start your container either by specifying which container to start, or by autostarting all of your containers based on their autostart settings:
$ lxc-start -n webserver $ lxc-stop -n webserver $ lxc-autostart
Connecting to your container is easy as pie. Either enter a console session from the host, or setup ssh as you would with any other Linux machine, except that your container is living in virtual private network, and thus only accessible from the host or with some port forwarding.
# lxc-console -n $CONTAINER_NAME $ lxc-console -n webserver
If you’re going to connect to your container remotely often, you may want to setup ssh forwarding. On your local machine, add the following to
Host my_webserver_host User myuser ProxyCommand ssh -q host_container nc -q0 private_local_ip 22
Here’s what my ssh config looked like:
Host host_container HostName 18.104.22.168 User root Host appname User deploy ProxyCommand ssh -q host_container nc -q0 10.0.3.96 22
Then I could ssh using
ssh appname or
ssh deploy@appname, which would first connect to my host container as a proxy, which would forward traffic to my app container. (Note: I added keys to both machines to avoid dealing with passwords.)
- Containers don’t autostart by default. Which is good! But easy to forget.
- Containers use DHCP on a virtual private network. You can make it a bit easier to connect by adding a line in
/etc/hosts, but either way, you’ll probably want to set the container’s IP to be static if you plan on connecting to it in any way other than
lxc-console(see update below to connect by hostname)
- Containers use DHCP by default, so if you plan to do this often, you’ll want to set your container’s IP to be static (in
- Containers, by default, use the same filesystem as the host. But you can also set them up to use different filesystems, such as
btrfs. The benefits are pretty mindblowing.
What I Learned
LXC Containers are pretty incredible.
As a result of my little experiment, I’m running three rack applications on a single host, using a container for each application (which contains only the code base,
rbenv, and unicorn, running on the only open port other than 22 for ssh), a container for nginx, which forwards requests to the appropriate container by hostname (bonus: I only need to setup pesky ssl stuff once!), and one container running postgres.
I can easily constrain resources on any container to ensure something going haywire doesn’t affect the others. I know each is secure and walled off, so there’s no risk that a process on one could, say, overwrite part of another’s filesystem, or kill off any processes.
At any given time, I can clone one of the application containers and add it to my nginx
upstream and I’ve got built in redundancy, or, if I’d rather move it to a different machine, I can do so easily by exporting the container to a new machine and merely changing my nginx config to look at that IP.
What I Really Learned
But realistically, this isn’t any different than running each of these five services on their own servers. What I really learned was that containerizing web applications has some really tremendous benefits.
Historically I’ve tended to put all my applications on their own server: each had source code, database, webserver (plus passenger/puma/unicorn for rack apps), a user for sudo stuff, a user for deploys and sometimes running the webserver. For really small websites, I’d setup several as virtual hosts on the same machine. This approach was easy and familiar, and pretty cost effective.
But over the years I’ve built so many small side projects. In fact, my exploration into LXC containers was because I built emoj.es (a stupid website that takes emoji and makes them into a big image, so when you share links in chat apps, they’d get a huge image as a “preview”, making your emoji nearly full screen), and couldn’t justify spending $7/mo for Heroku, or $5/mo for a new Digital Ocean droplet, and it was so stupid a site I didn’t want to risk affecting any of my other side projects.
I played with Docker for a few hours and learned a couple things, which ultimately resulted in my choosing LXC containers instead:
- Most of the base images everyone uses are marked as vulnerable. That doesn’t give me a warm and fuzzy feeling.
- Most of the base images are created and maintained by people who have little incentive to keep them around, which makes me very worried. Worse, they change all the time, so unless you – and they – are extremely careful about versioning, you risk small changes in your environment any time you run your deploy. (This is true for just about any package management (side note: a third party library we use at work removed and recreated a tag on github, so the same locked version had different behavior), but particularly scary when you’re talking about environments.)
- It took me forever to do basic things. Specifically, I wanted to start with a base Ubuntu image rather than the vulnerable ruby one, and it took me nearly three hours to get a new version of ruby installed. With
apt-geton ubuntu, the same task took me 15 minutes.
- Docker is really bad about permissions and security. If you’re using docker, I highly recommend you check whatever images you’re using that you didn’t write, and see how much is running as root.
But there’s one thing Docker does that my now-beloved LXC containers don’t: it automates provisioning and configuration. I tend to do that either as a set-it-and-forget-it because I’m lazy, which doesn’t do well over time, or with bash scripts that get grosser and grosser over time. This is better with containerized applications, though, because the steps to configurate a server are fewer and fewer. But it’s still a problem I wouldn’t consider solved (for me).
If you like Docker, awesome. If you like LXC Containers, that’s awesome(r). If you’re like I was, installing the full stack on a single box for your app, then, well, you’re livin’ in the past, man. I’ll never go back.
Moving the components of my application into different containers (whether servers, LXC containers, whatever) made me think more about how they talk to each other, what they need to run effectively, and how to isolate possibly vulnerabilities. For my more significant applications, like the API for Station to Station, I can have one machine with different containers, which I can easily scale, clone, whatever, as my needs change. Or for my smaller side projects, I can share resources by having a single webserver forwarding traffic to each application, a single database that I can backup or add slaves to, and I can keep costs way down by doing so on fewer machines.
Either way, if you host web applications, this is something worth looking into.
Also, I really love Linux.
Turns out you can connect to LXC containers by hostname pretty easily.
sudo apt-get install dnsmasq
- On Ubuntu 16.04, update
/etc/default/lxc-netto set the top level domain. The default is
lxc, which you can set by uncommenting the line
- Restart the LXC net service
sudo service lxc-net restart
- Add forwarding by editing
/etc/NetworkManager/dnsmasq.d/lxc.conffor Ubuntu 14.04)
Then you can connect using
ssh firstname.lastname@example.org, or verify using