Moving 14,000 screens to MQTT without losing one

The fear

Every screen of the network talks with the platform in two ways. One is proof-of-play: the log of each ad, cached on the device and uploaded in batches, which is what we bill on. The other is presence and telemetry: "I'm alive, this is my state", a small message, sent all the time by every player over HTTP. Behind it, each message went to a Kafka topic, and from there to ClickHouse.

That second channel is also how we know a screen exists. If a change breaks it on a player, the screen keeps playing whatever it had, and we stop seeing it. We can't send it a fix, because the fix would go through the same channel that just broke. With about 14,000 screens in elevators and malls, a bad release there can turn into a truck roll to thousands of buildings, which for a company like ours is the worst case possible.

So when we decided to move presence and telemetry to MQTT, the technical part was the smaller one. The project was about not losing contact with one screen.

A year for a protocol change

It took the whole year. The first mockup test is from March. Then the change spread to the player, to the servers and to many processes that depended of the old way a screen announced itself, and each one had to move or be adapted.

Then about four months of tests before any real screen. After that, a pilot on a small part of the fleet, a week of it running without incident, and a controlled rollout from there. The last step was on September 30, 2026.

My role was to lead it. I was busy with other projects in the same months, so a senior engineer on the team did most of the hands-on work, and I set the direction and discussed each step with him before it went out. The question behind every step was the same one, what happens to a screen if this goes wrong and how do we reach it after.

How we knew it worked

The step is visible in the data we already trusted. The telemetry rows arriving in ClickHouse dropped from about 1.7 million per hour to about 0.6 million per hour around 4 pm on the day of the switch, and stayed there. This is the query I used, with generic names:

SELECT
    toStartOfHour(event_time) AS hour,
    sum(rows)                 AS rows_in
FROM system.part_log
WHERE event_type = 'NewPart'
  AND database = 'logs' AND table = 'device_presence'
  AND event_time >= '2026-09-29' AND event_time < '2026-10-02'
GROUP BY hour
ORDER BY hour;

A drop alone proves nothing. The same drop happens if half of the fleet goes silent, so the drop was the second thing I checked. The first one is the number of distinct screens reporting: on October 5, 13,835 devices sent telemetry in 24 hours, the same fleet we had before the switch. The message count on Kafka for those 24 hours was 13,912,374, against roughly 42 million a day from April to September, which is −67%. In conversation I round it to "almost 70%".

SELECT
    toDate(received_at)   AS day,
    uniqExact(device_id)  AS screens,
    count()               AS messages
FROM logs.device_presence
WHERE received_at >= today() - 3
GROUP BY day
ORDER BY day;

The raw telemetry stays only a few days in ClickHouse, so for the "before" side you need numbers saved earlier. part_log keeps the history of what was inserted, and that is why I like it for this kind of check (it has to be enabled in the server config, so check that system.part_log exists before you count on it).

What I'd repeat

• When the change touches the channel you use to fix things, plan the way back first. The rollback can't depend on the new channel working.
• Keep the billing path out of the migration. Proof-of-play didn't move one line, so a problem in presence could never become a problem in invoices.
• Measure with the instruments you already trust, and check the count of devices before the count of messages.
• Leading something you don't type is a full job. Reviewing each step with the person doing it caught more than any test plan I could write alone.

Filed under the problem it solves

• mqtt
• fleet management
• rollout
• telemetry
• kafka
• clickhouse
• leadership