An old broker list that returned 504s

The symptom

On September 22 a screen of the finance system started to answer 504. The gateway in front of the app gives up after some seconds, so the user got a generic error page, and the application log had one line for each try:

ERROR: Local: Timed out

"Local" is the important word. In librdkafka, errors that start with Local: are produced by the client itself, before any broker says anything. The client was waiting for something and nobody answered.

The cause

The service's production config had the broker list hardcoded, like this (hostnames here, the real file had IPs):

defined('KAFKA_BROKER_PRODUCER') || define('KAFKA_BROKER_PRODUCER', ["kafka-old-1:9092", "kafka-old-2:9092"]);

That list came from the first commit of the project. In May the company moved to a new cluster with three brokers, and the services were repointed one by one; this one kept the old list, and the old brokers had become unreachable. The fix was one line, the same list that the other services already used in production:

defined('KAFKA_BROKER_PRODUCER') || define('KAFKA_BROKER_PRODUCER', ["kafka-1:9092", "kafka-2:9092", "kafka-3:9092"]);

Why nothing complained at startup

The library had a check that looked right:

$ok = $producer->addBrokers(implode(",", $brokers));
if ($ok == 0)
    throw new KafkaException("Could not connect to any broker");

addBrokers() returns how many brokers were added to the client's list. It parses the string and returns. The connection happens later, in the background, so with two dead brokers it returns 2 and the exception never fires.

Then each send did a poll(3000) and a flush(3000) with up to three retries. With no broker alive that is around 12 seconds before the error, for one message. Add a second send, or the flush that runs again in the destructor, and the request passes the gateway timeout. The user sees a 504 and the real error stays in a log nobody was reading.

Make it fail in one second

If you want to know at startup that a broker answers, ask for metadata. getMetadata() needs a real broker and throws when the timeout passes:

try {
    $producer->getMetadata(false, null, 1000);   // 1 s, no topic list
} catch (\RdKafka\Exception $e) {
    throw new KafkaException("No Kafka broker answered: " . $e->getMessage());
}

Running this on every request is too much, a cached health check is enough. And for anything a user waits for, keep the flush short and return a clear error, or write the message to an outbox table and send it outside the request, so a slow Kafka becomes a delayed message and the screen still works.

What I took from it

• A broker list per service, copied by hand, drifts. One shared place (an environment variable from the same source, or a DNS name per broker) moves with the cluster.
• addBrokers() counts strings. A metadata request is what proves a connection.
• Local: Timed out means the client never heard back. Look at the network and the config before the broker.
• A synchronous produce inside an HTTP request turns any Kafka problem into the gateway's problem, with a worse message.

Filed under the problem it solves

• kafka
• librdkafka
• php-rdkafka
• configuration drift
• timeouts
• reliability