step one: disable sshd?
lazyweb: ganglia vs graphite

baked servers vs fried servers

There's a classification of servers that I like: Phoenix Servers and Snowflake Servers. Phoenix servers can be rebuilt on demand, while Snowflake servers are uniquely hand-crafted. The ideal is to have a fleet of Phoenix servers, and there are two approaches to building those: having a gold image that you use as your baseline, or having a configuration system that ensures your systems match your spec. In general I believe the latter is more effective to making sure that your environment is running exactly the way you want it.

Baked servers are bootstrapped fully pre-configured. Often this means using a gold image or an AMI, configured and tuned exactly the way you want it. (A special type of baked server is one where you snapshot a VM's disk configuration as a backup.) You spin them up as you need them, and you roll any updates into the gold image. This is then paired with making those modifications to existing servers, or in the more extreme case, actually redeploying existing machines. Netflix does the latter, pretty aggressively, and that's a workflow that AWS makes super easy to do. 

Fried servers are bare-bones when they're first built, and additional updates are successively applied to your server to keep them in sync with your current standard. You install an operating system using kickstart (or whatever), and then your configuration management system takes care of the rest. This tends to take a little bit more time: the act of actually applying your changes against a base OS takes some time, but transcribing your changes so that Chef can apply them can sometimes be tricky too.

An interesting middle ground is to fry your gold image! That way you get the best of both worlds: speed of provisioning; a record of all the modifications you make; and continued compliance with your policy. But that's an optimization, and I think it's the only way that baking your servers can really be sustainable. Otherwise you end up with a bunch of Phoenix servers that are all based on a Snowflake, and that doesn't seem like a great position to be in.

comments powered by Disqus