Answers to the question you searched for, with the numbers attached.
-
Deep dive
How eventual is eventual? Measuring ReplacingMergeTree dedup
ReplacingMergeTree removes duplicates only when parts merge, at a time you can't plan. I measured how long it takes in production: 1.998 after 2 minutes, 1.11 after one hour, 1.0004 after 30 days. Plus the five layers that do the real dedup work.
measured ▸ duplicate factor 1.998 at 2 min · 1.11 at 1 h · 1.010 at 24 h · 1.0004 at 30 days
-
Deep dive
Three duplicates, three causes, one symptom
In one week I found duplicated processing in three systems: Kafka consumers outside their group, a summary that read ClickHouse before the merge, and two payment webhooks for the same order. Three different bugs, and the same piece missing in all of them: an idempotency key that every layer respects.
measured ▸ duplicate factor exactly 2.0 at insert · still 1.78 ten minutes later · control environment with one worker: 1.0000
-
Deep dive
Billing-grade telemetry from 15,000 devices
Every ad play is billing evidence, and the screens lose the network for hours. How the proof-of-play pipeline works: cache on the device, batch uploads, sort before insert, a weekly-partitioned MySQL with 2 billion rows, then Kafka and ClickHouse.
measured ▸ ~200 million play rows a day · importer runs of 155k to 643k rows in 7 to 20 s · MySQL era: 2B+ rows, weekly partitions