Answers to the question you searched for, with the numbers attached.
-
Deep dive
How eventual is eventual? Measuring ReplacingMergeTree dedup
ReplacingMergeTree removes duplicates only when parts merge, at a time you can't plan. I measured how long it takes in production: 1.998 after 2 minutes, 1.11 after one hour, 1.0004 after 30 days. Plus the five layers that do the real dedup work.
measured ▸ duplicate factor 1.998 at 2 min · 1.11 at 1 h · 1.010 at 24 h · 1.0004 at 30 days
-
Deep dive
Three duplicates, three causes, one symptom
In one week I found duplicated processing in three systems: Kafka consumers outside their group, a summary that read ClickHouse before the merge, and two payment webhooks for the same order. Three different bugs, and the same piece missing in all of them: an idempotency key that every layer respects.
measured ▸ duplicate factor exactly 2.0 at insert · still 1.78 ten minutes later · control environment with one worker: 1.0000