Answers to the question you searched for, with the numbers attached.
-
Note
1.8 million messages, 200 million rows: checking Kafka against ClickHouse
Our players send one Kafka message per upload, about 112 rows each. How I counted a day of messages from offsets alone, compared them with the rows ClickHouse really inserted, and got 13,912,374 against 13,912,522.
measured ▸ 13,912,374 messages vs 13,912,522 rows in 24 h (0.001%) · 1,782,599 messages carrying ~199.3M rows
-
Note
Moving 14,000 screens to MQTT without losing one
Device presence and telemetry moved from HTTP to MQTT on a fleet of about 14,000 screens. A year-long project, four months of tests and a slow rollout, because the worst case was losing contact with screens in the field. Telemetry messages dropped 67%.
measured ▸ telemetry on Kafka: ~42M messages/day before, 13.9M after (−67%) · 13,835 screens reporting
-
Tip
An old broker list that returned 504s
A screen in a finance system answered 504. The cause was a Kafka broker list that still pointed to a cluster replaced months before. Why the client didn't complain at startup, and how to make it fail in one second.
measured ▸ 1 line changed · by the producer config, ~12 s per send (3 s poll + 3 flushes of 3 s) before Local: Timed out
-
Deep dive
Billing-grade telemetry from 15,000 devices
Every ad play is billing evidence, and the screens lose the network for hours. How the proof-of-play pipeline works: cache on the device, batch uploads, sort before insert, a weekly-partitioned MySQL with 2 billion rows, then Kafka and ClickHouse.
measured ▸ ~200 million play rows a day · importer runs of 155k to 643k rows in 7 to 20 s · MySQL era: 2B+ rows, weekly partitions