debugging a crash in haproxy
Horrors uncovered in kerberos
Last week we had several reports of disconnections to some of our haproxy fronted services after we upgraded to a longer/stronger SSL certificate. As the version of haproxy we are running is quite old, we tried an upgrade to the latest (dev21). This fixed the disconnections, but after a few packets, it reliably crashed on the production systems. However
we could not repeat the problem on any test or development systems, even through kickstart and puppet ensured they were identical to production.
This post details how we found the cause of the problem, so I have a record/teaching aid.