I built a counter that could only ever read zero

We had just moved Deflate compression off the recording writer thread and onto a worker pool. At 1 Msps a 256 kB block lands every ~29 ms, and compressing it inline was eating the writer. The pool fixed that — but only if the pool actually kept up. If it saturated, the writer would fall back to compressing inline and quietly perform exactly like the old code.

So the plan had a task whose entire purpose was making that visible: inlineCompressions, surfaced through the status API as recInlineCompressions. A recording that fell back for most of its blocks would say so, instead of leaving us to infer it from a box count.

I wired it to the obvious source:

/** Blocks the pool could not take, compressed on this thread instead. */
val inlineCompressions: Long get() = compressor.inlineFallbacks()

The pool counts its own refusals. The writer asks the pool for the number. Clean.

What the first hardware run said

{'recElapsedMs': 58701, 'recBoxes': 56718727,
 'recDroppedQueueBoxes': 1544313,
 'recInlineCompressions': 0, ...}

A million and a half dropped boxes, and zero inline compressions.

Those two numbers cannot both be innocent. The writer was visibly failing to keep up — that is what the dropped boxes are — while the counter I had added specifically to show the writer failing to keep up sat at zero.

Why it read zero

writeBlock has two ways to refuse the pool, and I had only instrumented one.

val accepted = if (inFlight < rawPool.size - 1) {
    compressor.submit(raw, rawLen) { ... }
} else {
    false     // no free raw buffer: compress inline rather than alias one
}

The submit path goes into the pool, and the pool counts its refusals there. The else branch never asks the pool anything. It is a raw-buffer gate — when inFlight is at cap there is no free buffer to hand over, so the writer short-circuits and compresses inline without the pool ever learning it happened.

Both branches land in the same fallback block. Only one of them was counted. The fix was to move the increment to where they converge:

compressor.drainAll()
inFlight = 0
inlineCompressions++
val comp = deflateInline(raw, rawLen)

Why my checks could not catch it

This is the part worth keeping.

The suite was green — 232 tests, including tests written specifically for pool saturation. They passed because they were correct: they drove blocks at the pool until it refused, and asserted the fallback happened and the file stayed byte-identical. Every one of them exercised submit. None of them could exercise the raw-buffer gate, because reaching that branch needs inFlight at cap — a condition about buffer ownership, not about pool capacity, and one my tests had no reason to construct.

So the tests covered “the pool refuses” thoroughly and “the writer refuses on the pool’s behalf” not at all. From the outside those look like the same scenario. They produce the same observable behavior — a block compressed inline. They differ only in which object learns about it, which is exactly the thing the counter reports.

And the failure mode is inverted from the usual one. A broken counter that reads high is obvious; someone chases the phantom. A broken counter that reads zero is indistinguishable from the system working perfectly. Two runs later, after the fix, runs 2 and 4 both read inline=0 — and this time I reported it as “pool never saturated, workers keep up with margin.” That claim was true. But it was only checkable because the counter had been fixed first. Before that, the instrument’s healthy reading and its broken reading were the same number, and no amount of staring at it would have separated them.

I should also be straight about the sequence: I quoted recInlineCompressions: 0 from that first run in the same breath as the 1.5 M dropped boxes before I worked out what it meant. The contradiction was sitting in one line of JSON output and I read past it once.

What caught it was not a test and not a review. It was running the thing on real hardware under real load, where a counter that should have moved didn’t.

The rule

An instrument that sits on one side of a branch measures one side of that branch. Before trusting a counter that reads zero, find every path that should increment it and confirm the counter is downstream of all of them — not downstream of the one you were thinking about when you wrote it.

The corollary is sharper: for any counter whose healthy value is zero, a green test suite tells you nothing about whether it can count. You need either a test that forces it non-zero through each path, or a hardware run where you know independently that it should have moved. We had the second one by luck — because the first run was bad enough that the zero was obviously a lie.