My Adventure With LXC Containers
Over the last week or so, I’ve been playing around with LXC containers, low level virtualization containers in Linux. (Specifically, I’ve been using LXC version 2.0.9 with Ubuntu 16.04, on a cheap Digital Ocean box I setup for my experiment.)
Basic LXC Usage
First, in case you want to play along, installing LXC is incredibly easy:
$ sudo apt-get update && sudo apt-get install lxc
Create containers from a given template. For most of my playing around, I used a base
ubuntu template, but you can find far more with
apt-get install lxc-templates.
# lxc-create -n container_name -t template $ lxc-create -n webserver -t ubuntu
Each container lives in
/var/lib/lxc/$CONTAINER_NAME. Exploring this directory shows how simple a container really is:
$ ls /var/lib/lxc/webserver config rootfs/
config is created automatically, as is the root filesystem. By default, containers don’t start automatically, but you can change this easily by adding the following to your config:
lxc.start.auto = 1
Henceforth, you can start your container either by specifying which container to start, or by autostarting all of your containers based on their autostart settings:
$ lxc-start -n webserver $ lxc-stop -n webserver $ lxc-autostart
Connecting to your container is easy as pie. Either enter a console session from the host, or setup ssh as you would with any other Linux machine, except that your container is living in virtual private network, and thus only accessible from the host or with some port forwarding.
# lxc-console -n $CONTAINER_NAME $ lxc-console -n webserver
If you’re going to connect to your container remotely often, you may want to setup ssh forwarding. On your local machine, add the following to
Host my_webserver_host User myuser ProxyCommand ssh -q host_container nc -q0 private_local_ip 22
Here’s what my ssh config looked like:
Host host_container HostName 188.8.131.52 User root Host appname User deploy ProxyCommand ssh -q host_container nc -q0 10.0.3.96 22
Then I could ssh using
ssh appname or
ssh deploy@appname, which would first connect to my host container as a proxy, which would forward traffic to my app container. (Note: I added keys to both machines to avoid dealing with passwords.)
- Containers don’t autostart by default. Which is good! But easy to forget.
- Containers use DHCP on a virtual private network. You can make it a bit easier to connect by adding a line in
/etc/hosts, but either way, you’ll probably want to set the container’s IP to be static if you plan on connecting to it in any way other than
lxc-console(see update below to connect by hostname)
- Containers use DHCP by default, so if you plan to do this often, you’ll want to set your container’s IP to be static (in
- Containers, by default, use the same filesystem as the host. But you can also set them up to use different filesystems, such as
btrfs. The benefits are pretty mindblowing.
What I Learned
LXC Containers are pretty incredible.
As a result of my little experiment, I’m running three rack applications on a single host, using a container for each application (which contains only the code base,
rbenv, and unicorn, running on the only open port other than 22 for ssh), a container for nginx, which forwards requests to the appropriate container by hostname (bonus: I only need to setup pesky ssl stuff once!), and one container running postgres.
I can easily constrain resources on any container to ensure something going haywire doesn’t affect the others. I know each is secure and walled off, so there’s no risk that a process on one could, say, overwrite part of another’s filesystem, or kill off any processes.
At any given time, I can clone one of the application containers and add it to my nginx
upstream and I’ve got built in redundancy, or, if I’d rather move it to a different machine, I can do so easily by exporting the container to a new machine and merely changing my nginx config to look at that IP.
What I Really Learned
But realistically, this isn’t any different than running each of these five services on their own servers. What I really learned was that containerizing web applications has some really tremendous benefits.
Historically I’ve tended to put all my applications on their own server: each had source code, database, webserver (plus passenger/puma/unicorn for rack apps), a user for sudo stuff, a user for deploys and sometimes running the webserver. For really small websites, I’d setup several as virtual hosts on the same machine. This approach was easy and familiar, and pretty cost effective.
But over the years I’ve built so many small side projects. In fact, my exploration into LXC containers was because I built emoj.es (a stupid website that takes emoji and makes them into a big image, so when you share links in chat apps, they’d get a huge image as a “preview”, making your emoji nearly full screen), and couldn’t justify spending $7/mo for Heroku, or $5/mo for a new Digital Ocean droplet, and it was so stupid a site I didn’t want to risk affecting any of my other side projects.
I played with Docker for a few hours and learned a couple things, which ultimately resulted in my choosing LXC containers instead:
- Most of the base images everyone uses are marked as vulnerable. That doesn’t give me a warm and fuzzy feeling.
- Most of the base images are created and maintained by people who have little incentive to keep them around, which makes me very worried. Worse, they change all the time, so unless you – and they – are extremely careful about versioning, you risk small changes in your environment any time you run your deploy. (This is true for just about any package management (side note: a third party library we use at work removed and recreated a tag on github, so the same locked version had different behavior), but particularly scary when you’re talking about environments.)
- It took me forever to do basic things. Specifically, I wanted to start with a base Ubuntu image rather than the vulnerable ruby one, and it took me nearly three hours to get a new version of ruby installed. With
apt-geton ubuntu, the same task took me 15 minutes.
- Docker is really bad about permissions and security. If you’re using docker, I highly recommend you check whatever images you’re using that you didn’t write, and see how much is running as root.
But there’s one thing Docker does that my now-beloved LXC containers don’t: it automates provisioning and configuration. I tend to do that either as a set-it-and-forget-it because I’m lazy, which doesn’t do well over time, or with bash scripts that get grosser and grosser over time. This is better with containerized applications, though, because the steps to configurate a server are fewer and fewer. But it’s still a problem I wouldn’t consider solved (for me).
If you like Docker, awesome. If you like LXC Containers, that’s awesome(r). If you’re like I was, installing the full stack on a single box for your app, then, well, you’re livin’ in the past, man. I’ll never go back.
Moving the components of my application into different containers (whether servers, LXC containers, whatever) made me think more about how they talk to each other, what they need to run effectively, and how to isolate possibly vulnerabilities. For my more significant applications, like the API for Station to Station, I can have one machine with different containers, which I can easily scale, clone, whatever, as my needs change. Or for my smaller side projects, I can share resources by having a single webserver forwarding traffic to each application, a single database that I can backup or add slaves to, and I can keep costs way down by doing so on fewer machines.
Either way, if you host web applications, this is something worth looking into.
Also, I really love Linux.
Turns out you can connect to LXC containers by hostname pretty easily.
sudo apt-get install dnsmasq
- On Ubuntu 16.04, update
/etc/default/lxc-netto set the top level domain. The default is
lxc, which you can set by uncommenting the line
- Restart the LXC net service
sudo service lxc-net restart
- Add forwarding by editing
/etc/NetworkManager/dnsmasq.d/lxc.conffor Ubuntu 14.04)
Then you can connect using
ssh firstname.lastname@example.org, or verify using
It’s now 2017, almost 15 years since I met (and have since exceeded) my thousand mile goal. A very dear friend of mine recently has had the same urge for going. Like me, he grew up in Chicago, but he never drew a circle with a big radius around it. Instead, he’s had his sights on New York (Chicago’s older, grumpier brother). We’ve known each other for a few years, and I’ve always known about this dream of his, and, with my memory as bad as it is, I must’ve told him this story a dozen times.
He finally made the jump, and is leaving next week. We had a ceremonious last supper, when this story, again, came up. And the next day, he decided to look up the quote. (I have no idea why this never occurred to me. I could say it was too magical and important to me to ruin, but mostly it just never dawned on me that it was more than some World Market doodad the Hyatt installed in all their hotels.)
So, I love that quote you frequently recite: “Man travels the world in search of what he needs and returns home to find it.” So much so, that I looked up the guy who wrote it. And I couldn’t help but notice something…
I panicked when I received his message, worried it was Bruce Springstein. (Really.)
Time Traveling Novelist (Not Bruce)
Fortunately, it wasn’t a Bruce quote!
It was from a dude named George Moore, a novelist from the late 19th century, who, according to my friend, looks a bit like me. Then he asked if I was a time traveling novelist. With great enthusiasm, I decided to research the answer to this question, knowing full well that I don’t currently have the ability to travel through time.
I managed to find an old photo of me from around when this quote so changed my course in life. Here’s that photo, as well as George Moore’s portrait on Wikipedia:
I did a quick informal poll of people who know me pretty well, and on a scale of 1-10, the average response was a high 7 for visual similarlities.
Time Lord? (Still not Bruce)
It then dawned on me that for the last five or so years, I’ve been dressing up for Halloween as Matt Smith’s Doctor Who, a Time Lord. (Rarely known fact: I have worn and loved bowties ever since I borrowed one for that first Halloween.)
Let’s take a look at those alongside each other as well:
In conclusion, greetings from the future! I cannot tell you how I did it, but somehow the future-me was able to return to 1879 and 2011!
Further, I used what I learned in 2011 to say something famous in the late 19th century, that managed to influence me in 2011 to go back and say it before!
Even more, I probably knew that what I said in the 19th century would influence my friend in 2017 to tell me about my other selves so I would know to go back to 2011 and read that placard! Amazing.
Now I just need to find where I left the TARDIS…#permalink
One of the most important events in my life was, as they often are, an unexpected one.
It was early 2010. I’d been living in Hangzhou, China, and my world was beautifully small. It was maybe ten years since I required of myself that I put at least a thousand miles between me and Chicago: a thousand miles from the midwest, a thousand miles from the winters, a thousand miles from the same old same old. As Joni Mitchell said in what became my anthem, I got the urge for going, so I guess I had to go.
Since then, I took a 24-hour trip around Lake Michigan (1,100 miles in a car, listening to Bob Dylan, eating fudge from Mackinac Island). I spent four years in Colorado (1,100 miles away) for school. I traveled for a few weeks to Germany (4,382 miles), the Netherlands (4,110 miles), Belgium (4,114 miles), and France (4,137 miles). For a half a year, I studied Chinese Chess in Kunming, China (7,784 miles). And I took a last hoorah trip with an old friend to Thailand & Cambodia (8,567 miles).
And then I got a job in China, and my 1,000 mile minimum felt like a joke. Over the years I worked in China, I traveled regularly to Vietnam, Hong Kong, India, and Korea. I went to over a dozen cities in China regularly.
All this traveling meant I’d racked up some pretty sweet loyalty points, too. I was United Premier Executive (I still keep the card in my wallet even though it expired in 2012), and Hyatt Platinum which meant when I went to Japan to visit a friend, I got free airfare and stayed in the Park Hyatt Tokyo. One time my brother told me he had no plans for spring break, so the next day I used miles to get tickets to visit him for a week in Scotland.
I felt like I could go anywhere in the world, and I did.
My then-girlfriend (now-wife) and I had been dating, as of 2010, for about three years. While I was doing all this traveling, she stayed in Hangzhou. She didn’t have the urge for going, but I still did. Using miles, we booked tickets to San Francisco, to see some friends and family, to see the blue skies no longer familiar to either of us, and to drink some good wine. And I found a Hyatt promotion that was good for one free night at any Hyatt in the world for every two Hyatt stays, anywhere in the world. So in the weeks leading up to our trip, we took small trips to places we’d never been to stay at the cheapest Hyatts we could find.
The Park Hyatt Shanghai lobby is famously one of the highest in the world. To get to the elevators, you walk through a calm, dimly lit bamboo garden. We told the hotel that it was our honeymoon (it wasn’t), so when they escorted us to our free-upgrade suite, there was someone waiting for us with flowers and a bottle of champaign.
We went to our favorite restaurant in Shanghai that night, which brought us to that event that was so important to me. It wasn’t the dinner, or the swanky free hotel. It wasn’t our upcoming trip to San Francisco, or any of the trips I’d made leading up to it. It was what I saw, coming back from dinner, as we got off the elevator on the 91st floor. There was a small placard opposite the elevator that I’d missed upon arrival because luck had us on the other side.
Man travels the world over in search of something and returns home to find it.
Later that year, we got married. And except for an extended layover in Houston we try to forget, we’ve been living in Chicago ever since, and have no intention of leaving.
*Note: the actual quote is slightly different, which I learned only years later when telling this story, after I showed off a photo I took of the placard: “A man travels the world over in search of what he needs and returns home to find it.” Close enough.#permalink
I’ve long been an avid supporter of Betterment as a great, hands-off, effective investment vehicle. I’ve referred ten people, five of whom have funded their accounts, and two tell me they intend to. I’ve put nearly all of my eggs in this one basket because the way they invest your money is to spread it into as many carefully selected buckets as possible.
Earlier this morning, they announced a new pricing plan, effectively immediately for new customers, and June 1st, 2017, for existing customer.
Betterment’s Pricing Structure
Until now, their pricing structure was very simple, with benefits to putting in more money, as follows:
- An account balance less than $10,000 cost 0.35% annually (or $3/mo if they don’t have a recurring fee)
- Accounts with between $10,000 and $100,000 cost 0.25% annually
- Accounts over $100,000 cost only 0.15% annually
Their new pricing is much simpler, and charges more for human advice:
- Accounts of any balance costs 0.25% annually
- The privilege of receiving advice from their experts costs between 0.40% or 0.50%
This benefits users who were previously in the 0.35% bucket, and obviously doesn’t affect those in the 0.25% bucket. But for what was once the premium tier of customers, prices just went up by 66%.
What’s the big deal?
If you have $100,000 in your Betterment account at the end of May of this year, you’ll be paying around $150 per year in fees; the next month, when the new pricing goes into effect, this will jump to $250. That means you’re paying $100 extra as a customer once in their most valuable tier.
Further, they have a post about how important expense ratios are when choosing funds, comparing a 1% fee to Betterment’s 0.15% plus the underlying expense ratio of about 0.11%.
Assuming an average annual return of 7.1%, their new pricing structure will end up costing $1,823.06 more over ten years for accounts starting with $100k (source: begintoinvest.com). Over twenty years, the difference will be a whopping $7,105.78. And remember: this doesn’t include the expense ratio you’re already paying to the funds Betterment holds for you.
Why this doesn’t make sense
Enough of why this sucks. Betterment needs to make money just like the rest of us, right? And as prices of things go up, it’s only natural that they charge more money for their services. Right?
Well, not quite. For three reasons:
- As inflation goes up, their real prices go up. So there’s no need to periodically increase the percent cut they take
- They are indeed increasing the value of their service, with new (and awesome) features like tax loss harvesting, but that increases their earnings too
- This ironically punishes the customers currently most valuable to them
Let’s look at those one by one.
When I reached out to Betterment support earlier today (which replied almost immediately), the first reason for their change was: “Our pricing was last updated over four years ago.”
Lots of service cost more money over time due to inflation. This makes perfect sense! Except in this case, it doesn’t. Betterment is already charging a percentage of the account balance. So as inflation goes up, the account balance goes up. And as the account balance goes up, their real earnings go up. So if they’re currently making $150/yr on my $100,000, and inflation goes up 3% this year, next year my balance will be $103,000 (hopefully more), which means they’d be making $154.50. In contrast, with this new change, they’ll be making $257.50.
The value of their service
Further to their explanation that their pricing hasn’t changed in a while, Betterment support cited new features as justification for the price increase:
In addition to expanding our offerings to include joint accounts and trust accounts, below is a partial list of all the features we’ve added since then for all of our customers. Even though our fees have increased, we believe the value that we provide to our customers has increased even more through our features [such as RetireGuide, Tax Loss Harvesting+, and Tax-Coordinated Portfolio]
While this is absolutely true, and I completely agree that these new features have brought more value to their service, it’s important to point out that as I make money, they make money. Remember: they earn money as a percentage of my account balance. So as the value of my money increases, so does their cut. So even if you forget that fact that they should improve their value over time in order to stay competitive (you know, with investing in those same funds directly), their doing so is valuable to them as much as their customers.
Punishing those most invested
The final point for why this doesn’t make sense is the one that I take the hardest. I’ve had money in Betterment for years. It’s taken me quite a lot of aggressive saving, including several rollovers, to reach their top tier, reserved for those with $100k or more. In fact, I rolled over a beefy 401k earlier this month and put in a few thousand more to finally reach that long-awaited $100k benchmark. And not a month later, after having reached the top tier, they effectively bumped me back down.
This change will likely get them more customers in the <$10k range, because the pricing is lower, and won’t make a difference to those in the previously-middle tier. Further, it simplifies their pricing and focuses their offering on their human services in the form of advice. I have no idea what percent of their customer base was in that top tier before, but I will say without a doubt that it affected my relationship with Betterment deeply when they rewarded my loyalty with a price increase which they did not give to those with a lower investment.
Where to go from here
It’s realistically unlikely that I’ll change my investment strategy. Betterment still has the best returns, the best ease of use, and the most transparent practices. But this change isn’t insignificant: already, I had to convince myself (and others) that their fees were worth paying so much more than the expense ratios in the underlying funds, and I’ll soon need to re-evaluate. And I can’t help but feel burned, knowing that they punished those most invested in them, and rewarded those least.#permalink
I’ve always been somewhat of a coffee anti-snob, proud to say I was satisfied with whatever coffee was available. I always enjoyed good coffee, and could certainly taste the difference, but I reserved my snobbiness for whiskey, outright refusing to drink the cheap stuff.
That changed recently, after working alongside some of the snobbiest coffee snobs I’ve ever met. For two years, I had only fantastic coffee beans, ground fresh immediately before brewing, brewed using well maintained machines.
On a daily basis, I had easy access to:
- French press
- Cold brew
- Old fashioned coffee machine
with a heavy duty burr grinder and a wide selection of Big Shoulders beans, and occassionally others.
At times I grew frustrated by the need to make coffee, since all but one (or two) of those machines were single serving, so we couldn’t take turns making larger pots like at past offices.
But the coffee was fantastic.
(At one point, we had a blind tasting, in which we sampled the same beans brewed using each of the five machines. For me, the winner was overwhelmingly clear.)
Making Great Coffee
After two years of drinking the best of the best, I changed jobs and found myself with a single serving fancy automatic thingy machine which makes terrible over-caffeinated Starbucks coffee. The weak setting yields tasteless coffee yet leaves me shaking from the caffeine; the strong setting (occassionally mixed with water for a makeshift Americano) is no better. So I took the plunge and started bringing stuff to work.
Cuisinart DBM-8 Burr Grinder - $44.39
An old friend, who used to barista at Intelligentsia (best macchiato I’ve ever had), used to say that the best coffee with the best machine and the worst grinder would make worse coffee than the worst beans with the worst machine and the best grinder. I’m sure that’s a bit of an exaggeration, but he made his point clear: how you grind your coffee beans has a tremendous impact on the quality of your coffee.
However, as much as I love coffee, I do have my limits. I’ve used hand grinders and got frustrated with the effort and time needed. And there’s no way I’m going to spend over $200 for a grinder.
That said, I’m not going to grind my beans when I buy them and let them rot for weeks until I finish the bag.
I found a wonderful burr grinder for $45, and love it. The granularity is easily adjustable, and there’s an automatic shut-off so you can grind only what you need without waiting.
My only complaint is that the container the grounds go into seem to encourage grinding lots of beans at once, not a single serving. This makes it a bit of a pain to scoop out, especially when there’s coffee dust all over the place.
Aeropress - $29.95
It still amazes me that the best non-espresso coffee I’ve ever had is made using a thirty dollar plastic tube.
An Aeropress is pretty simple: you put grounds into a tube with a filter at the bottom, add hot water, stir it for a bit, and then plunge the water through the filter. The result is very similar to an espresso - strong flavor, slight crema, and very smooth. I like my Americanos, so I usually add water.
I cannot overstate how good this coffee tastes. Risking sounding dramatic, I’ll even say I didn’t know coffee was supposed to taste this good.
This can be a bit tough to find. But it’s worth the effort. I’ve found it at some (not all) Whole Foods around the city, as well as a handful of Marianos. I get a different roast every time, and my memory isn’t good enough to know which I enjoy the most. (This morning I had the Uganda Bugisu AA, and it was incredible.)
Anyway, I hope this was helpful for those of you looking for amazing coffee that doesn’t cost an arm, leg, or too much time.