A Network Engineer's Unit Tests

Testing a topology engine has an awkward property: for most of what it does, you cannot write down the expected answer.

You can unit-test a Dijkstra implementation against a graph you invented. That proves your Dijkstra is a Dijkstra. It says nothing about whether the graph you built from a real LSDB is the graph the routers are actually using, which is the only thing that matters and the only thing that can be wrong in an interesting way.

So the question becomes: what is the oracle?

---

The routers are the reference implementation

The answer I settled on splits it in two. RFC 2328 is the specification: it says what the protocol is supposed to do. The routers in the lab are the behavioural oracle: they say what an implementation actually does when you ask it. Osprey has to satisfy both, and where they diverge the divergence is itself the finding. The useful part is that the oracle is sitting right there, willing to be asked.

That turns validation into a pipeline:

64 real routers
      ↓  show ip route ospf  /  show ip ospf database summary
ground truth: every router's own installed table
      ↓
Osprey's reconstruction, same instant, same scope
      ↓
compare all 4,032 ordered pairs on four independent assertions

This is the reconstructed side of that comparison: one router’s own table, as Osprey holds it:

428 routes, split by type (C connected, O intra-area, O IA inter-area, E2 external), each with the metric, the advertising router, and, importantly for what follows, the full next-hop set, so 172.16.2.31, 172.16.2.32 appears as the two-way ECMP it actually is rather than as whichever one sorted first.

The four assertions matter more than the count:

Assertion	Why it is separate
Metric exact	Catches a wrong path that happens to have the right length
Path type exact (intra / inter / external)	Catches the right cost reached by the wrong rule
Full ECMP next-hop set identical	Catches picking one valid member and calling it the answer
Termination = delivered at the requested destination	Catches a walk that stops early and looks plausible

Any three of those can pass while the model is wrong. That is not hypothetical. The bug in the previous post produced paths with correct endpoints and cheaper-than-real cost, which is exactly what a lazier comparison would have waved through.

Today all four pass on 4,032 of 4,032 pairs. The number I care about is not 4,032. It is four: the number of independent ways I made it possible to fail.

---

When there is no oracle, fuzz

Ground truth works when you can ask the network. It does not work for the other half of the problem: what happens when a device sends you something malformed, truncated, or simply weird.

Osprey has one absolute rule for protocol parsers, and it is a product constraint rather than a code-quality aspiration: log the anomaly and continue; never crash. A monitoring system that dies on a malformed LSA takes your visibility away at exactly the moment something interesting is happening.

You cannot enumerate the ways a vendor will surprise you. So you fuzz:

make fuzz                          # every target, 10s each
make fuzz FUZZTIME=60s             # longer
make fuzz FUZZPKG=./internal/isis/ # one package

The current count in this repo:

Fuzz targets	110
Test files	419
Benchmarks	98

110 fuzz targets is not a vanity metric. It is roughly one per parse boundary where an attacker or a buggy agent controls the bytes: OSPFv2 and OSPFv3 packets and every LSA type, IS-IS PDUs and TLVs, BGP messages and path attributes, BMP headers, SNMP varbind decoding. Every one of them is a place where the honest answer to “what will this device send me?” is “I have no idea.”

The benchmarks exist for a related reason. The protocol parsers are on a hot path and are written for zero allocations per packet, with sync.Pool for buffers, and a benchmark with b.ReportAllocs() is the only thing that stops that quietly regressing the next time someone adds a convenient fmt.Sprintf.

---

The tests that need a real database

A third category resists both approaches. Osprey’s harder queries are CTEs with aggregates and multi-joins: “which devices are active but have never been polled”, “which areas are partitioned”, “reconstruct the graph as of T”. Mocking a database to test those proves only that your mock agrees with your expectations.

So those run against a real PostgreSQL via testcontainers, and the project rule is explicit: CTE / aggregate / multi-join queries require real DB tests, covering empty sets, NULLs, boundary conditions and the one that catches the most bugs: mutual exclusivity. If a query buckets devices into categories, the sum of the categories must equal the total. It is astonishing how often it does not, and how invisible that is on a dashboard where every number looks plausible on its own.

---

The vendors will not read the RFC to you

The last category is the one that no amount of internal testing finds, and it is where the lab earns its keep. Real equipment does not implement the standard you read. It implements something adjacent to it.

A few from this lab, all of which cost real time:

The MIB that isn’t there. The vEOS leaves in the lab’s EVPN services pod had to be discovered over OSPF rather than IS-IS, for the unglamorous reason that vEOS implements neither ISIS-MIB tree. Not partially. At all. No amount of correct SNMP code fixes that; the only correct response is to discover them by a protocol they do support, and to document why.

The pre-standard dialect. MPLS L2VPN pseudowires have a standard MIB, RFC 5601. Real IOS ships a pre-standard Cisco dialect instead. So the poller implements both and falls back automatically. And because “we support both” is exactly the kind of claim that rots, the two drivers are held to a parity gate so the standard-MIB path cannot silently diverge from the Cisco one.

The crypto that time forgot. The routers here run two IOL images (IOS 15.2(4)S7 on 124 of them, 15.4(2)T4 on the other 68), and every single one offers only SHA-1 key exchange and ssh-rsa host keys; the 15.2 fleet tops out at CBC ciphers on top of that. A stock modern OpenSSH client refuses the key exchange outright, so the connection dies before authentication is even attempted. Osprey’s SSH client carries the legacy fallbacks explicitly, because the alternative is a terminal feature that works on the vendor’s slide deck and not on the customer’s actual estate.

The server that was never started. An earlier 64-router build of this lab had ip ssh version 2 configured everywhere and no RSA key ever generated, so the SSH server was simply not running, on every node, invisibly. show ip ssh said Disabled and the config looked complete.

None of these are things you discover by reading a specification. They are things you discover by pointing your code at equipment and watching it fail.

---

What a topology engine really is

Somewhere in the middle of building the validation harness, the shape of the thing became clearer to me:

input:   messy, partial, contradictory network state
output:  a deterministic model
oracle:  the routers themselves

That is a compiler, near enough. And the useful consequence of thinking about it that way is that a compiler is judged by conformance, not by plausibility. Nobody accepts “the generated code looks about right”.

The reason I can write a series about a product that refuses to guess is not that I am unusually principled. It is that I have 64 machines that will contradict me on demand, and a harness that asks all of them at once.

Without that, “Osprey computes the path correctly” would be an opinion. With it, it is 4,032 out of 4,032, on four assertions, reproducible on a Tuesday afternoon.

That is the whole difference.

---

This is the last post in Series 2. Series 1, Three IGPs, One Map, Zero Footprint, Hop-by-Hop Truth and What Osprey Refuses to Guess, covers the principles this one puts under test.