abe hassan | blog

package management and the devops revolution

All this talk over the last few years about devops skirts around one particularly cynical take on traditional system administration: that it slows product development down. That error-proofing everything is unnecessary, that it's purely overhead and catastrophizing, at the expense of shipping new features. For many, certain failure modes aren't worth preventing, and there's a whole lot of these tradeoffs happening in the deployment space in order to ship new features, faster.

I think the core of this comes down to package management.

This is, by far, the biggest point of friction I've experienced. Certainly there are other places where traditional system administration is being improved or automated -- like better monitoring, better provisioning, etc. But I think that developers are looking for a more progressive approach to package management while systems administrators are a little more conservative.

In the broadest of strokes, packaging is the act of taking software and making the installation process repeatable. In turn this means that we're generating an artifact once, and installing it on a collection of servers; which means the state of the system is inspectable; which then means that we can ensure that every server is consistent. Tools like apt and yum are the usual things we think of -- and they're very mature package managers -- but even something like AMIs or tarballs could do the trick, depending on your needs.

It means that you're not re-compiling, re-building, re-executing your build steps on every single server. Putting aside the time it takes, you're ultimately multiplying the number of times you're doing an operation, increasing the likelihood of failure. (There's also a security angle here, in that some would argue that production hosts shouldn't have the "build" toolchain available: either installed or over the network. I personally agree with them, but I acknowledge there's tons of debate on this point.)

But it's slow and painful.

If you're relying on your upstream provider to give you packages, you're pretty much restricted to an out-of-date existence. Bugs are fixed in newer versions! Github is social coding! We add features to the libraries we use! Waiting on someone to give you packages is a losing proposition. When we optimize for multiple deploys a day, waiting a month or a year for your upstream to provide new functionality to you is a non-starter. If we want the latest and greatest -- and we often do -- you need to figure out how to package it.

Using pip or gem or cpanm is easy. Taking that output, packaging it, figuring out how to make it relocatable, etc, isn't very easy. Making sure that pip installs the exact same thing every time isn't easy. (You might specify exact versions of all your dependencies, but do those dependencies do the same?) Packaging each individual module takes forever. Packaging your app with its dependencies is bulky. Tools like fpm get us a lot closer, but it's still an extra step.

When "npm install" works most of the time, it's easy to start relying on it all of the time. When you're already deploying multiple times a day, it's easy to not want to make any particular installation repeatable. When you can replace any machine and just re-run your setup script, it's easy to not trap and handle failures in that process. But resiliency and convergence mean more than just "if I try this enough times, eventually it will work".

Over the last decade, I think the pendulum has swing back and forth from "package everything as an rpm and write an init script" to "just run it out of your homedir with screen". I don't like swinging between the extremes -- neither is great. Finding a healthy balance, somewhere in between, is devops is about. It's about finding a way to move fast while still maintaining a rapidly-deployable, operable system.

why we duplicate metrics

A few months back, we were getting the graphite and statsd stack set up in our world. The frontend functionality offered by Graphite and the easy-to-use interface offered by Statsd are incredibly powerful. But our use of statsd quickly presented a problem to us: we wanted to understand cluster health, but we also wanted to understand per-host health in a way where we could do blue/green deployment.

Statsd doesn't easily give you that ability, at least not out of the box:

1. If you run a statsd server on each machine, then your metrics stay separated even up until you get to Graphite, so you don't get an at-a-glance view of the cluster.
2. If you run a single, central statsd server, you lose per-host visibility. If your app emits a data point for "metricname", then all hosts' data points for that metric get combined together.

You might be thinking that the solution to (1) is simple: that you either use aggregation functions for statistics on the way in, or you use the functions in Graphite's frontend to combine metrics. Unfortunately the math doesn't work in your favor. You can get awfully close! But it won't be accurate.

The reason the math doesn't work is that the summarization is lossy.

For instance, if you're recording 99%ile latencies from each machine, you can't use that to determine the 99%ile across the cluster overall. There's only three cases where it's guaranteed to work: max, min, and sum (or count). But it doesn't work for averages, medians, percentiles, etc. You need more information, perhaps even the original data set, in order to derive that data from the constituent sets. It works when every machine's metrics are identical. It works less and less when when one machine is anomalous. And since detecting anomalous behavior is the original goal we had in mind, "close enough" wasn't good enough.

So we log our metrics twice, into a central statsd server: once with the hostname, once without.

Now, we didn't want our developers to have to call into statsd twice -- DRY and whatnot -- so we run a proxy on each host. This proxy intercepts statsd calls and multiplexes them¹. The proxy also munges the name into a standard naming scheme, based on the environment we're in. So, an application developer only needs to submit a data point called "appname.metricname" to statsd (really, the proxy), and the proxy emits it into statsd twice.

It's a little extra work at the head end of the processing chain, but it gives us a couple ways of bisecting data that we wouldn't otherwise get without either customizing statsd (or the clients) or asking our developers to duplicate all their calls. Hachi described how we've laid out our system in more detail, over at his blog.

For now, we've published our statsd proxies to Say Media's "devops-tools" repo on Github. If it's interesting enough to others, we'll put some effort into trying to get something like this moved upstream, either into the clients or into statsd itself. Let me know! Comment here, tweet me, or just find me at Velocity!

 

¹ Technically we multiplex everything other than "gauge". Because a gauge is an absolute number, recording a cluster-wide metric doesn't really work -- the last one to submit a value in a particular flushInterval wins. Not very interesting, and it's confusing, so we only log gauge values once.

using git in your /etc directory

About a year ago, I was trying to figure out how the right config mgmt recipes make a CentOS 4-based machine¹ properly auth against our internal LDAP servers. I was having a pretty hard time of it -- we had nailed it for Debian and for CentOS 5, but CentOS 4 changed some additional files when I ran `authconfig`.

I couldn't figure it out, and sort of paradoxically, I therefore only had one opportunity, per new machine, to figure it out. Otherwise I had to throw away the machine and re-instantiate, and even though it only took a couple minutes, it was still annoying. I knew that some file on disk was changing. I just couldn't figure out which file.

I took a page out of blueprint's book and used git to figure it out. First, I turned /etc into a git repository with a simple `cd /etc && git init`, and I committed the entire tree. Then I ran my authconfig command to set things up. Then, I could look at the changes with `git diff`, and I immediately found the culprit (which ended up being a line appended to one of the pam.d files).

I actually took it a step further and committed at three points in time: once after the machine first booted, another after cfengine ran, and then a third time after I ran `authconfig`. That allowed me to triangulate between those three states, to figure out which rules were missing and which rules were extraneous. It also allowed me to use `git checkout HEAD^^` to reset the machine to various states, to test my changes and my understanding of the system!

The best part is that when I was done, cleanup was a simple `rm -rf /etc/.git`, instead of cleaning up one directory within every single directory...

 

[1] Yes, CentOS 4. I know that it was super obsolete even a year ago. Don't ask, that's not the point. :)

google hangouts is almost what i want

Google's notion of bringing its Voice, Chat/Talk, Hangouts, and related products all under the umbrella of "Google Hangouts" is awesome. The way in which those tools were integrated into the rest of the Google ecosystem was pretty haphazard (why can I place a call from Gmail but not from Google Voice?). Seeing a concentrated effort to use the same messaging system is great.

Except.

I've switched to the new Hangouts on my personal account and I've found a few things that bug me:

1. There's no way to globally disable logging within Google. I can turn it off per-chat, but I can't disable it all the time always. I get there are some benefits to this -- device synchronization and such -- but man it's annoying. Of note, messages to/from people are recorded as separate emails with special headers that cause Gmail to display them a certain way. This conflicts tremendously with a filters hack I use, causing all my outgoing IMs to appear my Inbox. I had to create a filter for "in:chats from:<me>" to skip the Inbox; the filters UI says it'll never match anything but I know better.
2. The iOS app is nice! I wish I could tell it to not push notifications anytime anyone messages me. It's great if it's my primary SMS app, but it's not -- it's my IM app, and I don't need that from my phone. There's no way to permanently disable notifications; I can only do it for up to 72 hours at a time. Just give me an "unread" badge and stop it with the sounds and the banners. Nevermind! This one was just my dumb. But man, it wasn't easy; I had to disable notifications from the iOS settings instead of from the Hangouts app itself. No way to just disable notifications from the app if I wanted to do that.
3. Dropping XMPP support has been the biggest complaint I've heard. I don't honestly care, as a user; what I do care about is that I can continue to use a desktop app (in this case, Adium) to chat with people. Of course that brings its own mess of problems, particularly around message/state synchronization, but for me it's far more convenient to cmd-tab to Adium than it is to cmd-tab to Safari and then find the right tabs.
4. When I initiate an OTR chat session in Adium, any other screens syncronizing that chat (whether it's Gmail, G+, or the iOS app) all just show garbage. I get that's how the OTR chat looks on the wire, but man that's dumb. I'm guessing OTR support isn't really a priority, but I can totally see demand for an incognito mode for chat -- one that does more than just disable logging and enable SSL to Google.
5. The synchronization is a little bothersome because it's doesn't include things like which windows I've opened or closed. So if I respond to a message from Adium or Google+, and then close the window, Gmail still has the message boxes open. (Yes I know this is harder to do when supporting Adium, but even within the Google web apps it's annoying.)
6. Personally haven't verified this one way or the other, but I hope that Google Voice's SMS capability gets merged in with this, so that I can SMS with folks using the same systems. Maybe they could also fix the issue with Google Voice dropping incoming MMSes. These are a couple of the reasons I switched away from Google Voice.

Google, you are so close! A couple more tweaks and you will be perfect!

the future of digital media

Newspapers and magazines are dying, but blogs and news sites and Twitter and Facebook aren't enough to fill the space they're leaving. At Say Media, we're building the next generation of digital media. I thought I'd take a diversion from my usual tech blogging to share my perspective on what we're doing, and why I'm so excited for it.

We know that traditional media -- newspapers and magazines and television and radio -- aren't keeping up with the wants and needs of an increasingly online and off-the-grid society. We know that those companies are seeing this trend, and they're trying to build a digital media strategy: Conde Nast, New York Times, Hulu. They're all taking content surrounded by ads, and transposing it almost exactly to the Internet. High-quality content providers are still not entirely great at taking advantage of the rich features that the Internet can offer, which in turn impacts readers and marketers.

The current best-in-class online media offering is just a notch above scanning a magazine and putting it online. In a way, it's sort of a meta skeumorphism.

The entire CPM concept for advertising is based on those traditional models. Articles are split over multiple pages to crank up ad views. By having been addicted to that approach for so long, mobile is presenting one hell of a problem, where small displays and limited bandwidth make it near-impossible (or at least, uneconomical) to shove a bunch of ads on a page. What worked for media several decades ago doesn't work online. We can do much more, and we can know much more, and we need to put that to use.

The future of digital media is, well, digitally authentic.

Imagine a publication that lives and breathes online, that takes advantage of all the capabilities an online world has to offer. Personalized dashboards, frequent updates, two-way communication, community features, integrated advertising, direct commerce opportunities. Imagine interactive features -- not just photos, not just video, but truly interactive features (like the New York Times's election prediction charts). Imagine a publication that had bonus features for subscribers. Imagine a publication that had enough real-time data behind it that it could show you what's popular, that it could inform its editors about hot material and encourage them to engage with the community. Imagine a website that didn't feel like it was overrun by advertising, where a clean design ties the entire publication together, rather than just making room for an ad on the side.

The industry is still trying to just make newspapers online, and tweeting links to articles. That's how they stop their ships from sinking and adapt to an online world. But they're not building their publications any differently from how they have in the past. Let's be true to the medium. We're building the next generation of media, combining the best parts of the past and the capabilities of the future.

 

(And yes, before you ask: nobody asked me to write this, nobody edited this before I posted this. I'm actually sincerely excited about this. I see it so clearly in my mind, so I thought I would write it out and share it with others.)

does uptime matter?

Bank websites are down for something like 4-8 hours every Saturday evening for scheduled maintenance. Even non-technical people are familiar with Twitter's "fail whale". Tumblr has one of the worst uptime records of all blogging services (at least in 2011, though I'm not sure much has changed in 2012). The Apple Store falls over every time a new iPhone goes on sale (and it's down for "scheduled maintenance" on announcement days).

And yet that doesn't stop us from using those services.

I think a company's interest in uptime breaks down into two elements: monetary and reputational. Monetary implications are complex -- but generally measurable -- while reputational effects are harder to discern since they add up over the long term.

Google never goes down, to the point where it's what many of us test if our own connection is working. Yahoo's front page never goes down (and there's a dozen layers of redundancy and fail-safesbefore it actually does fail, all the way to rendering a static version of the page on disk). Amazon, Facebook. We notice when those guys go down. They've got more at stake. They know exactly how much each minute of downtime costs them, and whether that potential revenue is recoverable or not.  (For instance, retail customers might just defer some purchases until later, but they might talk themselves out of impulse purchases.

Reputational factors are harder to measure. Twitter and Tumblr have flipped traditional aspirations of availability on their head. They're poster children for successful services that didn't have great uptime track records. Certainly this would be different if we felt our data was unsafe with them. Based purely on availability, we seem to tolerate more. Refreshing a page or coming back later is annying, but it doesn't strike me as though it's a reason why users choose (or reject) a particular vendor.

(The implication is different in the B2B case, like Amazon Web Services, because your reputation is derived from the downtime implications your customers face.)

So maybe five-nines uptime isn't the most important thing. Maybe time to market or a robust feature set are worth having an hour of downtime here and there. We reach the point of diminishing returns perhaps closer to the 99% uptime point (to SWAG at a number) than we originally thought. And maybe then, after we've proven ourselves, should we worrry about adding more nines. 

ops, hard at work

About a month ago, we had a mini-hackathon at work wherein we ported alllll of our systems from talking to Ganglia to instead talking to Graphite. A coworker was amused and snapped a photo:

Nerdfest!

the road to software engineering

According to a lot of universities, software development is still a mix of computer science and electrical engineering. It was only in May 2012 that NCEES announced that they would be offering a Principles of Engineering exam in Software, starting this coming April.

The foundational work of Software Engineering has been happening for decades. The invention of calculators and computers and all the innovation on the hardware side of things; pieces of software that have become critical to the Internet (think about apache or memcache or even Linux); and revolutionary design patterns (think about map-reduce or the c10k problem). Over time, I think best practices emerge in a way that allows us to have high standards for software engineers; to be able to certify and license software engineers just like any other type of engineer.

Now, our field is fascinatingly different! And this difference is going to lend itself to some friction with other fields of engineering. Here's a couple ways we're unique:

1. You can learn this stuff on your own! You can practice this stuff on your own! You can't build a house without permits and inspections, but you can (usually) build a site or app with zero regulation.
2. And because of that: people who aren't licensed or certified, or who haven't gone through any sort of formal education program, can still change the way we do things. Ideas matter more than anything else, and anyone can have and promote them.
3. We aren't rooted in the physical sciences. This means that it's not a matter of truth or falsehood; not a matter of applying the laws of physics to the bridges we're building; etc. We weigh things against performance, productivity, consistency, and code volume. There's more than one way to do everything, and so to some extent, rigid implementation policies don't quite make sense as much as they would in other fields.
4. We generally believe in the Internet, in the free exchange of ideas, and so there isn't just one way to learn things. You aren't required to learn a certain amount about a certain set of subjects; you can pick and choose. And you can learn from anything: Google, books, classes, peers, you name it.

There are arguments that programming isn't engineering. I think the points in that article are actually spot-on ... for now. I generally think of software engineering as having a couple key requirements, some of which are exactly the distinctions called out in that article:

• deep, full-stack knowledge of software, hardware, computers, and networks;
• ability to translate customer requirements into a technical design and schematic;
• broad understanding of the field, to draw from our vast collective experience;
• practical understanding of an ever-expanding body of computer science theory;
• experience applying that knowledge to real-life situations and making tradeoffs.

I long for a world where the term "engineer" is only conferred on a person who has a proven understanding and ability at all of the above. I long for a world where continued education is a requirement for software engineers. I long for a world where we balance the science of engineering with the art of engineering -- balancing universal truths against situational realities. And, I long for a world where the ability to achieve these requirements is accessible to everyone, not just through a formal university program. (Something like Ops School is a great step forward here.)

To me, that's what differentiates an engineer from a "hacker" (someone who knows how to make a computer do a thing; but doesn't really have any of the requirements above) or a "programmer" (knows how to write code) or a "developer" (who may have portions of the first two above). Tom Limoncelli described the three phases of a system administrator in a similar way: you have a technician, an engineer, and a researcher. There are different roles, responsibilities, and requirements for each.

This seems to be a somewhat common pattern: technicians, engineers, scientists. In Software Engineering, we end up with programmers, engineers, and researchers. This is fantastic! For this field to grow, it must break apart into separate disciplines and career paths. And this is a sign that we're bursting past the "one size fits all" bucket. I'm looking forward to seeing and participating in that growth over the years to come.

(Thanks to Kyle Neath for reviewing this post, and for posting the tweet that got me motivated to finally write this up in the first place.)

off by one on purpose

Here are two problems that we ran into at LiveJournal: In a paginated system, how do you know whether to display the next/previous links? With password prompts being masked out, how can you tell a user that they kept typing but the prompt didn't record more? 

In the first case: if you display 20 items per page, load 21. If you have 21, throw out the last one and add the pagination links. If you get 20, then you know it's exactly right. Doing a database query with a "LIMIT 20" doesn't give you enough information to do this.

In the latter case, it's also easy: if your system limits passwords to (say) 15 characters, then make the change-password prompt take 16 characters. If the person typed in exactly 15 characters, then you're great. If you got 16, you can tell them that you didn't save their password the way they think you did.

(Edited to add: Yes, the thing here is to make your password input field accept more than the length of the password your service will store. If you actually limited the password prompt to 15 characters, in this example, the user doesn't have any visual indication that the password field stopped working since it's all just asterisked out anyway. So you have to take more than that: whether it's 16 or whether it's 200 doesn't matter.)

I love this idea. In order to gain certainty about something, you might just need one more piece of information. This trick is a bit clever and comes up every so often.

technical debt vs technical depreciation

Back when I worked at LiveJournal, Brad Fitzpatrick commented:

Our standards are continually being raised, so by definition any old code is ugly because new best practices have emerged in the meantime.

This has always stuck with me in the back of my mind, but it came up again in a conversation earlier this week. In trying to find the right way to manage a backlog of technical debt -- a whole topic in itself -- I pointed out that the discussion conflated two issues: one was technical debt, and I called the other "technical depreciation".

Your code may have been great when you wrote it two years ago. It epitomized every best practice of the time and is completely bug free. Unfortunately, two years later, and without doing anything to it at all, that's no longer the case. It might not compile against a newer version of libc, some security vulnerabilities have been discovered, a required dependency is no longer in the repository, its logic may not have kept up with changes to the business, and so on.

(Some of my favorites: a blog engine that worked when you had 1,000 entries posted to a blog, but four years later, it slows to a crawl when it's got 1,000,000 entries. Or, scrambling to find a bunch of old machines laying around because you require an ancient OS that doesn't have driver support for newer server components. Or, being stuck writing for an obsolete version of perl's Catalyst framework because you're so far behind that it's near-impossible to catch up.)

In the case of technical debt, you (usually) decided to take shortcuts, so you hopefully know where they are. But spending time managing your software's depreciation is equally important, but far more nebulous. I think this is the baseline cost of running any piece of software: managing dependency upgrades, managing security upgrades, maintaining build and release scripts, updating documentation, revising configuration for a changing environment, and so on.

People often think that Newton's Third Law ("an object in motion stays in motion") applies to software. But the world is in a constant state of change, and assumptions -- from hardware choice to business logic -- will eventually no longer hold. And just like technical debt, the longer you put off addressing depreciation, the more disproportionate the effort will be to address it.