Context
Purpose-built networking silicon is fast, expensive and inflexible. General purpose servers are cheap and flexible, and — with the right software architecture — considerably faster than most people expect.
This reference design covers the software structure required to move packets at line rate on commodity x86, and just as importantly, the measurement discipline required to prove it.
Why the kernel path runs out
The standard Linux network stack is a general-purpose design making general-purpose trade-offs: per-packet interrupts, socket buffer copies, context switches, and a locking model built for correctness across many workloads rather than throughput for one.
At small frame sizes the budget is brutal. At 100 Gbps with 64-byte frames the arrival rate is roughly 148.8 million packets per second per direction — on the order of a few nanoseconds per packet per core. That is a handful of cache-line accesses. Any per-packet allocation, lock, or pointer chase into cold memory consumes the entire budget on its own.
Architecture
Poll-mode, not interrupt-driven. Dedicated cores spin on receive queues. Interrupt overhead disappears; the trade is a core that is always at 100% CPU by design. That looks alarming on a dashboard and is entirely correct — a monitoring note that belongs in the runbook.
Batching everywhere. Packets move through the pipeline in vectors, not individually. Amortising per-stage overhead across a batch is the single largest throughput lever, and it is what VPP’s graph-node architecture is built around: the instruction cache stays hot because one node processes many packets before control moves on.
NUMA-aware placement. Receive queues, memory pools and processing cores are pinned to the same NUMA node as the NIC. Crossing a socket boundary in the per-packet path is a silent throughput ceiling that no amount of code tuning recovers.
Cache-line discipline. Per-packet metadata is laid out so the hot fields sit in the first cache line. Structures shared between cores are padded to avoid false sharing. Prefetch is issued for the next packet’s headers while the current one is being processed.
Hugepages to reduce TLB pressure, and SR-IOV where workloads must be isolated without a software switch in the path.
Choosing the right tool
Not everything belongs in a userspace poll-mode dataplane:
- XDP/eBPF is the right answer for filtering, DDoS mitigation and load balancing that can act at the driver hook — very low overhead, and it keeps the rest of the stack intact.
- DPDK suits full custom pipelines where the application owns the NIC.
- VPP suits routing and switching functionality where a mature graph of standard nodes already exists and the work is extending rather than rebuilding.
Picking the heaviest option by default is a common and expensive mistake.
Measurement
Performance claims without methodology are marketing. The characterisation plan:
- RFC 2544 throughput, latency, frame-loss and back-to-back burst tests across the frame-size range — including 64-byte worst case, which is where designs actually fail.
- Realistic flow mixes, because a single-flow test measures one queue and one core, not the system. Flow entropy determines whether RSS distributes traffic evenly or collapses onto one core.
- Latency distributions to p99.9, since the tail is what the application experiences.
- Soak testing long enough to expose mbuf leaks and slow fragmentation that a five-minute run will never reveal.
What this demonstrates
Data-plane performance is an architectural property, not a tuning exercise applied at the end. This design shows the decisions that determine whether a software forwarder reaches line rate or plateaus at a third of it — and the test methodology that tells you honestly which one you have built.