Estimating What You Cannot Measure

A traffic matrix inferred from link counters, the flow collector I decided not to build, and the honest limits of my own lab

22 August 2026 Michel Wijnberg

The previous post stopped one question short on purpose. Osprey can tell you that failing nrt1-cr1 ↔ sea1-cr1 changes 56 paths and isolates nobody. The question an operator asks next is the one that decides whether the change is safe:

Where does the traffic go, and does anything overflow?

Nobody signs off a change on path counts alone. The number that decides it is capacity: whether the links inheriting the traffic have room for it. This post is about how far you can honestly get toward that number when nothing in the network is measuring it directly, and about where that answer stops being one you should lean on.

Answering it requires knowing how much traffic flows from every source to every destination: a traffic matrix. Almost nobody has one.

The thing I decided not to build

The textbook way to get a traffic matrix is flow export: NetFlow, sFlow, IPFIX. Routers sample flows, ship records to a collector, and you aggregate them into origin-destination demands.

I planned that. There is a 1,139-line design document in this repository for a flow collector with LSDB-aware deduplication. It is marked, in its own first line:

Status: SHELVED (2026-03-12). Deferred indefinitely after cross-plan feasibility review.

with five stated reasons. The two that actually decided it:

Volume. 70–220 GB per week at 500 routers, roughly a hundred times the entire rest of the database. PostgreSQL is the only datastore in this product, on purpose; swallowing a flow pipeline means either a second datastore or a database that falls over.
Marginal value. About 80% of the practical traffic visibility was already available from SNMP interface counters, which were already being collected.

That is not a technical impossibility. It is a judgement that the last 20% costs a new subsystem, a new operational burden, and a category of storage risk, and I did not think it was worth it. Reasonable people disagree; some networks genuinely need flow. What matters here is that the decision is written down with its reasoning, so that anyone evaluating Osprey can check whether their situation matches mine.

So: no flow records. Which leaves the interesting problem.

Inferring the matrix from what the links admit

What Osprey does have is every link’s load, measured properly: 64-bit ifHCInOctets / ifHCOutOctets counters, polled on a cycle, differenced into bits per second, wrap-corrected, and divided by interface speed for a utilization percentage.

That is a set of link totals. What you want is a set of pair demands. Recovering the second from the first is an underdetermined inverse problem: many different traffic matrices produce exactly the same link loads. It has a well-known approach: Tomogravity (Zhang, Roughan et al., 2003).

It runs in two moves, and Osprey implements both:

// computeTrafficEstimate estimates traffic redistribution after topology
// mutations using the gravity model (Zhang/Roughan et al., "Tomogravity" 2003).
//
// Without a full traffic demand matrix, per-pair demands are estimated using
// the gravity model: demand(i,j) ∝ V(i)·V(j)/V_total, where V(i) is the
// total traffic volume at device i (derived from SNMP interface utilization).
// These gravity demands are then calibrated against observed per-link loads
// so that the model's predictions match reality.

The gravity step assumes traffic between two routers is proportional to the product of how much each one carries overall, the same shape as Newtonian gravity, or a trade model between two economies. It is a prior, not a measurement, and it is wrong in specific ways: it has no concept of a busy pair of neighbours or a quiet one.

The tomographic step is what makes it usable. The gravity guess is scaled so that, when every estimated demand is routed over the paths the IGP actually selects, the resulting per-link loads match the loads that were measured. The prior gets corrected by evidence, and the evidence is the one thing here that is not a guess.

Then, and only then, the calibrated demands are re-routed over the post-failure paths (ECMP-weighted, so a pair whose traffic splits four ways contributes a quarter to each link), and the difference is the projected redistribution.

Three deliberately unimpressive choices

The interesting engineering in an estimator is all in the places where you stop it from being confident.

Calibrate globally, not per link. The tempting version computes a scaling factor for every link independently. It fits the data better and it is much worse:

// Compute a GLOBAL calibration factor: total observed bps / total gravity sum.
// Per-link calibration can amplify noise when gravitySumPerLink is small
// relative to observed traffic, producing extreme scaling factors that
// inflate per-pair demands unrealistically. Global calibration distributes
// the error uniformly, producing more stable and realistic results.

A quiet link with a near-zero gravity sum and a little measured traffic produces an enormous scaling factor, which then inflates every pair that crosses it. The tighter fit buys you occasional nonsense. One global factor spreads the error evenly and never explodes.

Cap any single pair at 5% of total volume. A hard ceiling on how much of the network one estimated demand is allowed to be, because the failure mode of an inverse problem is one pair absorbing everything.

Refuse to report small changes. A projected shift smaller than 0.5% of link utilization is dropped, and 0.1% on the cost-change path, which also carries hypothetical links and nodes and spreads a new transit node’s effect thinly across many links. Below that, the number is indistinguishable from counter jitter, and printing it invites someone to act on noise.

Every one of those makes the output look less precise. Precision is not the goal; the goal is that a number shown on screen is one you can lean on.

And now the part where my lab cannot help me

Everything in the previous eight posts was validated against a lab that could contradict me. 4,032 router pairs checked against real routing tables. 360 fictional corridors found. Three protocols, three discovery paths, agreeing about the same single point of failure.

For traffic, the lab is nearly useless, and I want to be exact about why.

The routers in this lab forward almost nothing. They run protocols, they exchange LSAs and BGP updates, and that is essentially the whole of it. The measurements:

Measurement	Value
Interfaces with utilization data	1,279
…below 0.05% utilization	1,241
Highest utilization anywhere	0.431%, on a 10 Mbps port
Highest interface rate anywhere	~43 kbps
Peak across the 368 interfaces rated 10 Gbps	~24 kbps, which is 0.00024%

You can see the consequence in the simulation screenshot from the previous post, if you know where to look. The failed-links header reads “2 links, 6Kbps displaced”, six kilobits, and the panel shows no Traffic Redistribution section and no congestion risk at all. Not because the code did not run. Because every projected shift landed under that 0.5% noise floor and was correctly discarded.

The plumbing is verifiably right: counters are read, rates are differenced, gravity demands are formed, calibration runs, the redistribution is computed over the post-failure ECMP paths, and the noise floor rejects what it should reject. What the lab cannot tell me is whether the estimate is any good, because validating an estimator requires ground truth to compare against, and ground truth here means real traffic with a known origin-destination matrix.

A lab with no traffic cannot produce one. Adding synthetic load would not fix it either: I would be generating traffic from a matrix I chose, then congratulating the estimator for recovering the matrix I had just fed it. That is not validation, it is a very slow way of testing my own arithmetic.

So how much should you trust it?

Here is the honest ladder, in the order the numbers get less solid:

Link utilization, live and historical: measured. 64-bit counters, differenced, wrap-corrected, sanity-checked against interface speed. This is data.
Which paths change after a mutation: computed from the topology. Deterministic, and the topology underneath it is the one validated against 4,032 router pairs.
How much traffic moves onto each link: estimated, from a gravity prior corrected by measured link loads. Directionally useful, calibrated against real evidence, and not a measurement.

Rung 3 is genuinely valuable. It is the difference between “these twelve links absorb the failure” and “this one link absorbs most of it and is already at 70%”. It is also the only rung in this entire series that I cannot put a validation number next to, and it would be dishonest to present it in the same voice as “4,032 of 4,032”.

The product does draw that line where it can. When a BGP peer’s traffic has to be derived from prefix-count ratios instead of a matched SNMP interface, the row carries a tooltip saying exactly that, “Traffic estimated from prefix count ratios (no SNMP interface match)”, next to rows badged SNMP that came from counters. Same table, graded, which is the pattern this whole series keeps landing on, and this is the third time it has taken the same shape. A cross-domain path grades its segments resolved / inferred / opaque (post four); a port annotation grades its four verdicts from stated fact down to explicit ambiguity (post six); the three rungs above grade a traffic number. Three ladders, one habit: where the evidence underneath an answer is uneven, grade it in place rather than issuing a single confidence for the whole thing.

What the lab is for, restated

Post seven argued that the lab is not there to make Osprey look good, it is there to prove Osprey wrong. This post is the corollary nobody enjoys: a lab can only disprove the things it is capable of exercising.

Mine is exceptionally good at contradicting me about topology, protocol semantics and forwarding, because it has 192 real routers with real routing tables that I can read and compare against. It is incapable of contradicting me about traffic, because it carries none.

Knowing which of those two situations you are in, for each number your product prints, is most of what separates a tool that is trustworthy from one that is merely confident. I know exactly which claims in Osprey are backed by measurement, which by computation over a validated model, and which by an estimator I believe in but cannot prove here.

This post is the third category. There is exactly one of them, and now you know where it is.

Next, the last post in this series: the model is correct, and it is also a model of now. What did the network look like an hour before it broke? The Network Has a Memory.