The $90/month CDN

The bill that didn't fit

In 2014 I was CIO at ATMO, a digital signage company in São Paulo. We had screens in stores and at events, and every player downloads its videos from somewhere. Video is heavy, the players ask for it all the time, and a commercial CDN charges by the gigabyte. For a company our size the math didn't close.

At the same time DigitalOcean was selling its smallest droplet for US$5 a month, and that droplet came with 1 TB of transfer included. Most people saw a cheap server there; what caught my eye was the terabyte, because transfer was exactly the thing we were paying too much for.

So the first version was born at ATMO, in 2014 and 2015. Some cheap droplets serving files with nginx, the bandwidth already paid in the price of the machine, my own sync scripts and a table in the database saying which file is where. It was small. But the whole idea was already there.

Version two, eighteen droplets

At the end of 2015 I co-founded helloo, a media company, and I built its technology from zero. The same problem came back, much bigger: players in elevators and malls, in a network that kept growing (today it has around 15,000 screens). I didn't take the old code. I wrote the CDN again from zero, with the same base idea, and this version went far past the first one.

Now the problem was the disk. A $5 droplet had a small SSD, so I couldn't put the whole library in every node.

Because of that I split the library by content. The setup I remember best had 18 droplets in 6 groups of 3. Each group held one slice of the library, and inside the group the three nodes had the same files. One node down, two still serve. A file lives in one group only, so adding a group adds disk, and adding a node to a group adds bandwidth and redundancy for that slice.

People like this part of the story, 18 × US$5 = US$90 a month, with 18 TB of transfer included, and that was the whole delivery layer for the helloo players.

The part that gives work

Nobody sells you the sync, and that is where the work was. I wrote scripts that push a new file to the three nodes of its group, check it arrived, and write that in a table in the database. The player only gets the URL of a file after the copies are confirmed, so it never asks a node for something that isn't there yet.

I don't have those scripts anymore. If I wrote the control table today it would look like this:

CREATE TABLE cdn_copy (
    content_id  INT UNSIGNED NOT NULL,
    node        VARCHAR(32)  NOT NULL,   -- e.g. g4-b
    synced_at   DATETIME     NULL,       -- NULL = still copying
    PRIMARY KEY (content_id, node)
);

-- a content is published only when its whole group has it
SELECT content_id
FROM cdn_copy
GROUP BY content_id
HAVING COUNT(*) = 3 AND COUNT(synced_at) = 3;

Then the boring jobs: a check that a node still answers and still has free disk, the move of a group when one slice grows too much, and the cleanup of files that no campaign uses anymore. The last one I underestimated for some time, disks fill up quietly and you discover it in the worst day.

When I'd do it again

My honest take is the same one from back then: for a project with a small budget, this kind of thing still pays. It only gives work. You become the CDN vendor, the on-call and the person who remembers why group 4 has a weird disk, and that cost is real even if it doesn't show in the invoice.

With money in the bank and a team busy with the product, I'd pay a commercial CDN without thinking twice. With a tight budget, a predictable library and someone who likes to operate things, eighteen small machines did the job for years.

Filed under the problem it solves

• cdn
• cost engineering
• digitalocean
• sharding
• replication
• digital signage