In Matters of Measure

Sat 16 August 2014

tl;dr: regularly benchmark, don’t hang on to old assumptions regarding measurements

Oh, the times they are a-changing. Recently I was reviewing some code done by a coworker and saw they had .1 seconds for a timeout on one of their checks. I suggested a 1 second timeout, and in reply was told that I was bikeshedding and ‘thinking I could intuit socket timeouts’. "No! I’ve seen this behaviour before!" But to satisfy our (my) curiosity I ran a test that delivered an unrealistically extreme workload, saturating the CPUs, while still trying to get a socket timeout. And wouldn’t you know it, the timeout of .1 seconds was never reached.

So while eating humble pie, I tried thinking of the last time I had seen this particular behaviour where .1 seconds would be inadequate. And my mind went way, way back, to when I was a teenager, running a Pentium 3 desktop machine (I don’t recall how much RAM, but fairly sure not even a gig). At that time, 1 second was the limit of an adequate response, and the software in question was a good deal uglier and untuned. While over the years I’ve stuck to 1 second because it wasn’t necessarily wrong, performance has gotten so much better that frankly any response longer than a tenth of a second on modern equipment would be an indication of something being seriously off.

I’ve been reading Brendan Gregg‘s latest book on performance, "Systems Performance", and he stresses the importance of benchmarking the correct things, but also to run benchmarks regularly, as applications, kernels, and hardware changes and improves. It’s easy to agree that regular benchmarks are a good thing, actually doing them is another, especially when stale benchmarks still conveniently ‘fit’ the situation. We’re now in the age of sub-second results, and while in my lifetime I’ve had the good fortune of going from a TI-99/4A to a modern top-of-the-line MacBook Pro, and 28.8 kbps dial-up modems to fiber-optic, that luxury will not be afforded to the next generation. Features will be added to software systems, but I think it unlikely in the future for slow operations to be tolerated (not that we really tolerated the speed of floppy drives).

But before we settle comfortably into .1 seconds being ‘normal’, we need to take a look at the horizon of new hardware and software. Quantum computing is inching out of academia, flash drives are getting cheap enough for pretty much any application, and the Go language could be a very serious threat to C’s longstanding dominance of systems programming on the basis of comparable execution speed and order-of-magnitude build time speed increases. In the same way as what I thought was ‘normal’ ten years ago is to today, ‘normal’ today will be ten years forward from now.

Anyhow, back to reading "Systems Performance."