abe hassan | blog

the future of digital media

Newspapers and magazines are dying, but blogs and news sites and Twitter and Facebook aren't enough to fill the space they're leaving. At Say Media, we're building the next generation of digital media. I thought I'd take a diversion from my usual tech blogging to share my perspective on what we're doing, and why I'm so excited for it.

We know that traditional media -- newspapers and magazines and television and radio -- aren't keeping up with the wants and needs of an increasingly online and off-the-grid society. We know that those companies are seeing this trend, and they're trying to build a digital media strategy: Conde Nast, New York Times, Hulu. They're all taking content surrounded by ads, and transposing it almost exactly to the Internet. High-quality content providers are still not entirely great at taking advantage of the rich features that the Internet can offer, which in turn impacts readers and marketers.

The current best-in-class online media offering is just a notch above scanning a magazine and putting it online. In a way, it's sort of a meta skeumorphism.

The entire CPM concept for advertising is based on those traditional models. Articles are split over multiple pages to crank up ad views. By having been addicted to that approach for so long, mobile is presenting one hell of a problem, where small displays and limited bandwidth make it near-impossible (or at least, uneconomical) to shove a bunch of ads on a page. What worked for media several decades ago doesn't work online. We can do much more, and we can know much more, and we need to put that to use.

The future of digital media is, well, digitally authentic.

Imagine a publication that lives and breathes online, that takes advantage of all the capabilities an online world has to offer. Personalized dashboards, frequent updates, two-way communication, community features, integrated advertising, direct commerce opportunities. Imagine interactive features -- not just photos, not just video, but truly interactive features (like the New York Times's election prediction charts). Imagine a publication that had bonus features for subscribers. Imagine a publication that had enough real-time data behind it that it could show you what's popular, that it could inform its editors about hot material and encourage them to engage with the community. Imagine a website that didn't feel like it was overrun by advertising, where a clean design ties the entire publication together, rather than just making room for an ad on the side.

The industry is still trying to just make newspapers online, and tweeting links to articles. That's how they stop their ships from sinking and adapt to an online world. But they're not building their publications any differently from how they have in the past. Let's be true to the medium. We're building the next generation of media, combining the best parts of the past and the capabilities of the future.

 

(And yes, before you ask: nobody asked me to write this, nobody edited this before I posted this. I'm actually sincerely excited about this. I see it so clearly in my mind, so I thought I would write it out and share it with others.)

does uptime matter?

Bank websites are down for something like 4-8 hours every Saturday evening for scheduled maintenance. Even non-technical people are familiar with Twitter's "fail whale". Tumblr has one of the worst uptime records of all blogging services (at least in 2011, though I'm not sure much has changed in 2012). The Apple Store falls over every time a new iPhone goes on sale (and it's down for "scheduled maintenance" on announcement days).

And yet that doesn't stop us from using those services.

I think a company's interest in uptime breaks down into two elements: monetary and reputational. Monetary implications are complex -- but generally measurable -- while reputational effects are harder to discern since they add up over the long term.

Google never goes down, to the point where it's what many of us test if our own connection is working. Yahoo's front page never goes down (and there's a dozen layers of redundancy and fail-safesbefore it actually does fail, all the way to rendering a static version of the page on disk). Amazon, Facebook. We notice when those guys go down. They've got more at stake. They know exactly how much each minute of downtime costs them, and whether that potential revenue is recoverable or not.  (For instance, retail customers might just defer some purchases until later, but they might talk themselves out of impulse purchases.

Reputational factors are harder to measure. Twitter and Tumblr have flipped traditional aspirations of availability on their head. They're poster children for successful services that didn't have great uptime track records. Certainly this would be different if we felt our data was unsafe with them. Based purely on availability, we seem to tolerate more. Refreshing a page or coming back later is annying, but it doesn't strike me as though it's a reason why users choose (or reject) a particular vendor.

(The implication is different in the B2B case, like Amazon Web Services, because your reputation is derived from the downtime implications your customers face.)

So maybe five-nines uptime isn't the most important thing. Maybe time to market or a robust feature set are worth having an hour of downtime here and there. We reach the point of diminishing returns perhaps closer to the 99% uptime point (to SWAG at a number) than we originally thought. And maybe then, after we've proven ourselves, should we worrry about adding more nines. 

ops, hard at work

About a month ago, we had a mini-hackathon at work wherein we ported alllll of our systems from talking to Ganglia to instead talking to Graphite. A coworker was amused and snapped a photo:

Nerdfest!

the road to software engineering

According to a lot of universities, software development is still a mix of computer science and electrical engineering. It was only in May 2012 that NCEES announced that they would be offering a Principles of Engineering exam in Software, starting this coming April.

The foundational work of Software Engineering has been happening for decades. The invention of calculators and computers and all the innovation on the hardware side of things; pieces of software that have become critical to the Internet (think about apache or memcache or even Linux); and revolutionary design patterns (think about map-reduce or the c10k problem). Over time, I think best practices emerge in a way that allows us to have high standards for software engineers; to be able to certify and license software engineers just like any other type of engineer.

Now, our field is fascinatingly different! And this difference is going to lend itself to some friction with other fields of engineering. Here's a couple ways we're unique:

1. You can learn this stuff on your own! You can practice this stuff on your own! You can't build a house without permits and inspections, but you can (usually) build a site or app with zero regulation.
2. And because of that: people who aren't licensed or certified, or who haven't gone through any sort of formal education program, can still change the way we do things. Ideas matter more than anything else, and anyone can have and promote them.
3. We aren't rooted in the physical sciences. This means that it's not a matter of truth or falsehood; not a matter of applying the laws of physics to the bridges we're building; etc. We weigh things against performance, productivity, consistency, and code volume. There's more than one way to do everything, and so to some extent, rigid implementation policies don't quite make sense as much as they would in other fields.
4. We generally believe in the Internet, in the free exchange of ideas, and so there isn't just one way to learn things. You aren't required to learn a certain amount about a certain set of subjects; you can pick and choose. And you can learn from anything: Google, books, classes, peers, you name it.

There are arguments that programming isn't engineering. I think the points in that article are actually spot-on ... for now. I generally think of software engineering as having a couple key requirements, some of which are exactly the distinctions called out in that article:

• deep, full-stack knowledge of software, hardware, computers, and networks;
• ability to translate customer requirements into a technical design and schematic;
• broad understanding of the field, to draw from our vast collective experience;
• practical understanding of an ever-expanding body of computer science theory;
• experience applying that knowledge to real-life situations and making tradeoffs.

I long for a world where the term "engineer" is only conferred on a person who has a proven understanding and ability at all of the above. I long for a world where continued education is a requirement for software engineers. I long for a world where we balance the science of engineering with the art of engineering -- balancing universal truths against situational realities. And, I long for a world where the ability to achieve these requirements is accessible to everyone, not just through a formal university program. (Something like Ops School is a great step forward here.)

To me, that's what differentiates an engineer from a "hacker" (someone who knows how to make a computer do a thing; but doesn't really have any of the requirements above) or a "programmer" (knows how to write code) or a "developer" (who may have portions of the first two above). Tom Limoncelli described the three phases of a system administrator in a similar way: you have a technician, an engineer, and a researcher. There are different roles, responsibilities, and requirements for each.

This seems to be a somewhat common pattern: technicians, engineers, scientists. In Software Engineering, we end up with programmers, engineers, and researchers. This is fantastic! For this field to grow, it must break apart into separate disciplines and career paths. And this is a sign that we're bursting past the "one size fits all" bucket. I'm looking forward to seeing and participating in that growth over the years to come.

(Thanks to Kyle Neath for reviewing this post, and for posting the tweet that got me motivated to finally write this up in the first place.)

off by one on purpose

Here are two problems that we ran into at LiveJournal: In a paginated system, how do you know whether to display the next/previous links? With password prompts being masked out, how can you tell a user that they kept typing but the prompt didn't record more? 

In the first case: if you display 20 items per page, load 21. If you have 21, throw out the last one and add the pagination links. If you get 20, then you know it's exactly right. Doing a database query with a "LIMIT 20" doesn't give you enough information to do this.

In the latter case, it's also easy: if your system limits passwords to (say) 15 characters, then make the change-password prompt take 16 characters. If the person typed in exactly 15 characters, then you're great. If you got 16, you can tell them that you didn't save their password the way they think you did.

(Edited to add: Yes, the thing here is to make your password input field accept more than the length of the password your service will store. If you actually limited the password prompt to 15 characters, in this example, the user doesn't have any visual indication that the password field stopped working since it's all just asterisked out anyway. So you have to take more than that: whether it's 16 or whether it's 200 doesn't matter.)

I love this idea. In order to gain certainty about something, you might just need one more piece of information. This trick is a bit clever and comes up every so often.

technical debt vs technical depreciation

Back when I worked at LiveJournal, Brad Fitzpatrick commented:

Our standards are continually being raised, so by definition any old code is ugly because new best practices have emerged in the meantime.

This has always stuck with me in the back of my mind, but it came up again in a conversation earlier this week. In trying to find the right way to manage a backlog of technical debt -- a whole topic in itself -- I pointed out that the discussion conflated two issues: one was technical debt, and I called the other "technical depreciation".

Your code may have been great when you wrote it two years ago. It epitomized every best practice of the time and is completely bug free. Unfortunately, two years later, and without doing anything to it at all, that's no longer the case. It might not compile against a newer version of libc, some security vulnerabilities have been discovered, a required dependency is no longer in the repository, its logic may not have kept up with changes to the business, and so on.

(Some of my favorites: a blog engine that worked when you had 1,000 entries posted to a blog, but four years later, it slows to a crawl when it's got 1,000,000 entries. Or, scrambling to find a bunch of old machines laying around because you require an ancient OS that doesn't have driver support for newer server components. Or, being stuck writing for an obsolete version of perl's Catalyst framework because you're so far behind that it's near-impossible to catch up.)

In the case of technical debt, you (usually) decided to take shortcuts, so you hopefully know where they are. But spending time managing your software's depreciation is equally important, but far more nebulous. I think this is the baseline cost of running any piece of software: managing dependency upgrades, managing security upgrades, maintaining build and release scripts, updating documentation, revising configuration for a changing environment, and so on.

People often think that Newton's Third Law ("an object in motion stays in motion") applies to software. But the world is in a constant state of change, and assumptions -- from hardware choice to business logic -- will eventually no longer hold. And just like technical debt, the longer you put off addressing depreciation, the more disproportionate the effort will be to address it.

graphite's derivative function lies

We had a day-long war room yesterday wherein we ported a lot of our scripts and systems from writing to Ganglia to instead write to Graphite. We got to simplify a lot of stuff, primarily around rate-of-change calculations. Instead of doing those ourselves, we were sending in total values, and letting Graphite handle the "per second" part of things.

So it was a little odd when Hachi called me over and pointed out something funny. He was plotting the function "derivative(metric.memcache_gets)", where memcache_gets is the total lifetime gets that it has served. Over the last hour, the graph hovers around 3k. When I zoom out to the last 3 hours, though, it hovers around 180k. Weird. Hachi did point out that the 1 hour mark is when we switch aggregation intervals and start storing coarser data. That didn't smell right, though.

A derivative is a rate of change of one thing with respect to another. It's a rate of change, and since Graphite graphs according to time, you get the units of the y-axis (memcache gets) divided by the units of the x-axis (sec). That's what a derivative is, basically. It's the slope of the graph. Rise over run. Change in y over change in x. Algebra.

So when I went code-diving, I was surprised to see that the "derivative" function in Graphite just plots the difference between successive data points. So as we moved to coarser intervals, the difference became greater. (I didn't notice at the time, but the delta was 60x). Apparently there's a "perSecond" function that takes the delta between two data points, divided by the delta in time between them. That's the one we want.

Jeez. "perSecond" is the real derivative. "derivative" really is just a delta. How confusing and misleading.

(Personally, I think everything should be normalized to "per <unit of time>", ideally seconds, in Graphite. The data here is completely dependent on the granularity of the backing file, which basically means that you don't know what units to use unless you know how the data is stored. Same with the "count" metric that statsd emits with counters. "rate" is normalized and is accurate no matter what, whereas "count" depends on the flushInterval, etc. At least that one, we've blacklisted so it doesn't get recorded in Graphite anymore.)

google plus is what i wanted livejournal to be

I spent many, many years caring about and caring for LiveJournal. One of our biggest challenges was figuring out how the site should evolve. Its age almost guaranteed that we had some sizable portion of the userbase using any given feature, which made it harder to retire features; our extraordinary ability to botch major announcements made us even more scared to do it. And the new stuff we were shipping didn't generally have a cohesive vision behind it. If it did, we'd be able to establish momentum behind that vision, and a better way of talking about it.

Anyway.

That chapter of my life has been over for a while now. Fast forward a bit.

Google+ isn't a great social network -- not many of my friends are on it -- but it's a fantastic service. But it's what I wished LiveJournal would have become. It's streamlined, but powerful. Think about it for a moment. Streams are like the friends page; circles are custom friend groups; photos are like a Scrapbook that worked; communities are like communities; +1s are memories; and so on. But it also eliminates a lot of the additional bells and whistles (like custom mood themes) that I think distracted us from our mission of connecting people. I'm not even counting the types of things that work because of Google's muscle -- better search, suggested friends, calendar integration, hangouts, etc.

Now, there is one or two things that Google+ is missing. Arguably, threaded comments is one of them, but I think the most important thing is personalization: owning your URL, customizing your theme/style, and selecting userpics for different posts or comments. Google+ is very clearly owned by Google, and the layout isn't very friendly to use. With that exception, though, Google+ provides exactly the set of features I think LiveJournal should. No more, no less.

The sense of community that LiveJournal created is what made it our home on the web, and Google+ is the first service I've seen that has the raw features to do the same thing. Tumblr doesn't do it, Twitter doesn't do it, Facebook comes close but it's not really emphasizing the written word as much, Wordpress doesn't do it, Typepad doesn't do it. Making the site easier to use would helped keep LiveJournal relevant instead of being relegated to being the great-grandfather of social media.

It makes me sad that we couldn't -- and that LiveJournal still can't -- evolve the site fast enough to keep up with the world. But that's why I'm hopeful for Google+, because I think something needs to pick up where LiveJournal left off.

the case of the disappearing metrics

statsd to carbon, flushInterval 10s, data is fine. carbon writes the value then overwrites with zeros. what the what?

— Abe Hassan (@burr86) January 10, 2013

I'm so mad at myself right now.

We're getting statsd and graphite deployed, and we've been feeding some counters into statsd. We've generally found this to be fairly straightforward, except that one of our counters has been a bit schizophrenic. Sometimes it reports correctly, and sometimes it reports zero (rather than null). In fact, watching the whisper files directly, I sometimes see the timestamp and the value written to the file, and then immediately overwritten with a zero in the next run.

Hachi, Franck, and I pored over every configuration file, we pored over the source code to carbon and whisper. We set up the manhole in carbon so we could ssh in and see what it thinks the world is. We ran tcpdumps every which way. We turned off various daemons under a hunch that it might be something else writing to carbon and we wanted to track it down. We twiddled frequencies, retention policies, everything. Nothing, no dice, everything looks like it should.

We've been digging into this for about two days now. Franck and I were banging our heads against it today when we saw something unusual:

$ echo counters | nc statsd 8126 | grep metric.name
metric.name 18342
metric.name. 0

Huh. Interesting. ... oh, no, those actually become the same metric in graphite. So even though statsd thinks it's two different metrics, carbon ends up coalescing them down into one. One of them wins, depending on how statsd flushes them and carbon sorts them, causing the flip-flopping. What's worst is that the second one is actually a bug in code that I had written. It's supposed to emit statistics for "metric.name.subcategory", but there's a subtle case where the subcategory could be empty.

(As an aside, we did find an issue with how we had statsd and graphite talking to each other. In particular, statsd's flushInterval needed to be configured to be the same as one of the archive intervals in graphite, or else some of the stats get discarded.)

For the record, I was pretty sure something else was overwriting the values.

— Franck Cuny (@franckcuny) January 11, 2013

controls in an outage

For Christmas two years ago, I bought a friend a flying lesson. In advance of the actual class, we were asked to read a couple chapters out of a flying manual. One section described how to hand off the controls to your co-pilot, called a three-way positive control exchange technique. It's explicit on both sides: the person handing off says "your controls", the person receiving says "my controls", and the person handing off says "your controls" again.

Within an hour of reading that, I had shared it with my team and asked everyone to adopt that practice for managing outages. We've had so many situations where either: two people unknowingly stomp on each other and make a situation worse; or, two people see that the other is working on the problem, and they both back off. A variation on that latter situation often occurs when someone more experienced shows up (in irc or in person), and everyone assumes that this person is taking care of the situation.

We don't generally have that problem anymore. The person who receives the page is responsible no matter what, until the controls are explicitly handed off to someone else. That someone else can be a manager, a domain expert, their on-call backup, whoever it is. Nothing happens automatically, there's no seniority-based triggers, nothing. And, the other person isn't responsible just because you said so; they have to accept it.

In fact, to help build the habit, I started using "your controls" in non-outage situations. When we're tinkering on a new system or working with a vendor or whatever the case may be, I generally handle all handoff this way. It's concise, it's explicit, and best of all, it makes me feel like I'm flying a plane. :)