When the conntrack table fills up

The symptom was maddeningly vague: a handful of connections per minute would hang, then recover on retry. Latency graphs looked fine. Error rates looked fine. The only hint was a thin line of kernel log messages nobody had bothered to alert on.

nf_conntrack: table full, dropping packet. Once you see it, the whole thing collapses into something obvious. The box was tracking far more flows than the default table size allowed, and every new connection past the limit was simply discarded.

Raising nf_conntrack_max is the quick fix, and it is usually the right one, but it is worth asking why the number is so high in the first place. In our case a health check was opening a fresh connection every second per target and never reusing it.

The lesson I keep relearning: put the boring kernel counters on a dashboard before you need them. A metric you only discover during an incident is a metric you discovered too late.