« google plus is what i wanted livejournal to be | Main | technical debt vs technical depreciation »

Friday, January 25, 2013


Feed You can follow this conversation by subscribing to the comment feed for this post.


What is the bug # that you filed with the graphite devs?

Abe Hassan

I haven't, yet. It seems like it would break existing use to make this change, even if it is to fix it. (Maybe people have come to expect derivative to be a delta between data points.) But maybe I should let them make that decision?

Kevin G.

Have you looked into the way graphite calculates the 90th percentile? I suspect they're doing that wrong too. I've been thinking about that recently hard enough to strain a muscle, and I'm pretty sure you can't calculate percentiles from aggregated time periods. You need either the raw original data or a big enough random sampling of the original data where the confidence interval for the number of things in that top 10% is acceptable.

So if in the first time period you have the values 1..100, and in the second time period you have the values 1..30, the 90th percentile over the combined time period should be 87. But if you've aggregated away all the values and saved only something like the Five Number Summary, I don't think you're going to be able to come up with that.

This is much as I've been able to puzzle out from studying the Cartoon Guide to Statistics anyway.

> Algebra.

Calculus, actually ;-)

Abe Hassan

Yes, you're totally right. You need more information than the Five Number Summary. You can turn two averages into an average if you have the number of original data points; and you can turn two stddevs into one if you have the averages and number of data points. But percentiles are a whole 'nother story.

My suspicion is that Graphite's concept of percentiles is related to the data points it has stored. So it's not the 90th percentile *at that point*, but rather the 90th percentile of the data in the metric. To get 90th percentile at a given point in time, I would use statsd, which can calculate that and emit it to Graphite.

So there's a percentile at a point in time, and then a percentile across all time (or across the last X data points). I suspect Graphite is doing the latter. Technically valid, but super duper confusing.


If you want per second data, use scaleToSecond() rather than derivative().

If you want percentile you probably need data that Graphite do not store at all, smaller than one sample. Use Statsd in addition to Graphite and take its upper_90 metric.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

January 2014

Sun Mon Tue Wed Thu Fri Sat
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31  
Blog powered by Typepad
Member since 04/2007