What a Full Feed Actually Costs
I measured a full table into the production schema instead of estimating it, and the measurement killed one of my own defaults
Michel Wijnberg
“Can it handle a full table?” is the question every BGP product gets asked and almost none of them answer with a number. The honest reason is that the answer is embarrassing to produce: you have to load a real table into your real schema and look.
So that is what this post is. A full IPv4 table, 1,071,538 best paths from 78,538 origin ASes, loaded into the production schema on the same PostgreSQL major version the lab runs. Every figure below is a measurement of that, not a model of it.
It also contains the most useful thing measurement does, which is to contradict you. One of the numbers below was already shipped as a default in Osprey, and the measurement showed it sitting below ordinary traffic. It has since been raised by a factor of four.
Sessions are the unit, not routers
The single most common sizing mistake, and it is worth getting out of the way first.
A router with four full-table upstreams reports four peers over one BMP connection. It costs four times what a single-upstream router costs. Sizing by device count undercounts by exactly your multihoming factor, which is to say by the thing you built the network for.
What one full table costs, per unit
Sizes are PostgreSQL’s own reported relation sizes, which are 1024-based despite the MB and GB labels, and the per-row figures are derived from them. So 412 “MB” is 412 MiB: multiply 1,071,538 rows by 403 bytes and you get 411.8 MiB, not the 432 MB a decimal reading would suggest.
| Unit | Cost | Scope |
|---|---|---|
bgp_rib_entry including all 9 indexes | 412 MB (403 B/row) | per monitored peer session |
bgp_best_path including 4 indexes | 327 MB (319 B/row) | per AS and routing domain |
bgp_rib_entry_history including 3 indexes | 376 B per interval | per recorded transition |
| BMP per-peer prefix set, Go heap | 53 MiB (52 B/prefix) | per monitored peer session |
| Engine RIB debounce buffer at the default cap | 267 MiB (560 B/entry) | per engine, regardless of feeds |
Two things in that table are worth pulling out.
Indexes are slightly more than half the footprint. On bgp_rib_entry, per session:
primary key 82 MB, peer_prefix 42 MB, prefix_gist 32 MB, prefix 23 MB,
origin_as 8.9 MB, staleness 7.5 MB, next_hop 7.3 MB. Those are the seven worth
naming; the remaining two are smaller and do not move the point. The seven alone come
to 203 MB against a 412 MB total, and the primary key by itself is a fifth of
everything. If you have ever wondered whether one more index is free on a table this
size, the cheapest one here still costs 7.3 MB per session.
Measuring that also found a defect nobody had noticed. idx_bgp_rib_peer_lastseen
turned out to be byte-identical to the staleness index: 7.5 MB and one extra btree
write on every RIB row, buying nothing. It has been dropped. That kind of thing is
invisible until you put the sizes next to each other.
403 bytes a row is with communities. A realistic six-community string is included; routes without communities cost about 55 bytes less. Quote the cheaper number without saying which one it is and your sizing guide is 14% optimistic before anyone has deployed anything.
Worked deployments
| Deployment | Live database | BMP heap |
|---|---|---|
| 1 border router, 2 upstreams | 1.15 GB | 107 MiB |
| 2 border routers, 2 upstreams each | 1.98 GB | 213 MiB |
| 4 routers × 2 upstreams | 3.6 GB | 426 MiB |
| Route-reflector feed, 20 full-table sessions | 8.6 GB | 1.04 GiB |
These are live-state numbers. History is separate, and history is where it gets interesting.
Churn cannot be read off a snapshot
A table tells you what it costs to hold. It tells you nothing about what it costs to record, because that depends on how often the table changes, and a snapshot has no time axis.
So churn was measured from public MRT archives instead: RIPE RIS collector rrc00, 2026-08-10, four one-hour windows spread across the day, 6.8 million UPDATE messages across 98 distinct peer sessions. Osprey’s 30-second debounce is applied to the sample. Attribute-hash suppression is deliberately not modelled, so these are an upper bound on what would actually be recorded.
An interval here is one persisted state transition after the debounce. It is not a raw BGP UPDATE, which can carry many prefixes at once, and it is not a prefix event, which the debounce may collapse into an existing interval.
| Per session | Recorded intervals per day | History storage |
|---|---|---|
| Median | 413,000 | 155 MB/day → 14 GB per 90 days |
| p90 | 840,000 | 316 MB/day → 28 GB per 90 days |
| Heaviest observed | 8,300,000 | 3.1 GB/day → 281 GB per 90 days |
The debounce is doing real work: it collapses each 30-second window to one record per (session, prefix), which removes 45.6% of raw prefix events. Turning it off on a full-table target multiplies the history bill by about 1.8, and what you retain for that is finer-grained visibility into transitions occurring inside the window. On a DFZ feed that is almost entirely the kind of detail you were hoping not to store one row at a time.
Two caveats I would want if I were reading this. rrc00 is RIPE’s multihop collector and its peers are noisier than a typical IXP-local transit session, so treat these as pessimistic. And the spread matters far more than the median: the heaviest session is 20 times the median. Size on p90, alarm on the tail, and do not let anyone quote you the median as a capacity plan.
The number that was wrong
Osprey has a circuit breaker on history recording. If a target produces more than
storm_ceiling_per_hour intervals, recording is suppressed for the rest of that hour,
so a flapping session cannot fill a disk unattended.
The shipped default was 250,000 per hour.
Now put that next to the measurement. The heaviest single real session in the rrc00 sample runs at about 346,000 intervals per hour, which is the 8.3 million a day in the table above. Not a runaway. Not a flap loop. One real peer, on a real collector, on an ordinary Monday.
The default was tripped by normal operation, by itself. And the failure mode of a suppression breaker is the quiet kind: nothing crashes, nothing alarms loudly, history simply stops being complete and the gap is only visible if you go looking for it.
The default is now 1,000,000 per hour, derived from the same measurement rather than from intuition. p90 churn is roughly 35,000 intervals per hour per session, so the largest worked profile above, a route-reflector feed with 20 full-table sessions, runs around 700,000 per hour at p90. The new ceiling sits above that with headroom and above the heaviest single real session, while still being far below what a genuine runaway produces: a session replaying the table in a flap loop generates multiples of this, so the breaker still breaks.
Something the same measurement makes clear, which I want to state rather than leave implied: a target aggregating many noisy sessions can still trip the ceiling legitimately. The 98 rrc00 sessions together produce roughly 50 million intervals a day, about 2 million an hour, comfortably past the new ceiling. If you point Osprey at something like that, the breaker will fire, and it is right to.
Which is worth stating outright, because the two numbers look like they disagree: the ceiling is a runaway guard, not a capacity plan. It is set per target, above what normal operation produces on the largest profile we size for, and below what a session replaying its table in a loop produces. An aggregating target that legitimately exceeds it has outgrown one target, not the product.
That is why tripping it is recorded rather than merely logged. When the ceiling engages, history is incomplete for the remainder of that hour, and that gap is written into a blind-spot ledger you can query. A monitoring system that silently stops monitoring is worse than one that stops loudly, and the only thing that distinguishes them is whether the gap is a first-class record.
What this does not tell you
Everything above is bytes: what a full table occupies in PostgreSQL, what the BMP server holds in Go heap, and how fast history accumulates. That is one axis, and a sizing question has several.
Not measured here, and therefore not claimed:
- CPU, at ingest or at query time.
- Time to first useful answer after a session comes up, which is the number an operator actually feels.
- The message path between the BMP server and the database: what the bus and the live websocket state cost while a table is converging.
- Query latency against a table this size, as opposed to its footprint.
Those are separate measurements and I have not made them. A sizing guide that quietly implies otherwise, by giving you a confident storage number and letting you assume the rest scales the same way, is doing the thing this post exists to argue against.
Why publish a default you got wrong
Because the alternative is worse in a specific way.
A sizing guide built from estimates is unfalsifiable. Nobody can check it, so nobody finds the error, so the error ships. The reason this one was found is that the numbers were produced by loading a real table and replaying real updates, and a real update stream does not care what the configuration file thinks is reasonable.
The measurement cost about a day. It found a duplicate index, a default off by a factor of four, and it produced the first honest answer this product has to “can it handle a full table”. Which is: yes, at 412 MB and 53 MiB of heap per session for live state, and somewhere between 14 GB and 281 GB per 90 days for history depending entirely on how noisy your peers are. That last range is the real answer, not its midpoint.
The point
Every number in this post is a measurement of the shipped schema, and one of them contradicted the shipped code.
That is the argument for measuring your own product rather than reasoning about it. Not because reasoning is careless, but because a default that was chosen by reasoning has no way to tell you it was wrong. A real table does it in a day.
This is the last post in this series. The next one starts underneath all of it: how a recorder joins someone else’s protocol without becoming part of it, one front door at a time.