You have built an Erlang

This post will make more sense if you first read:

1. Dear Sir, You Have Built a Compiler and,
2. Dear friend, you have built a Kubernetes.

Dear friend…

I regret to inform you that, despite your best intentions, you have built an Erlang.

I know you never meant for this to happen. You just needed a simple way to notify services when data changed.

A message bus? Overkill. A background job queue? Same thing. You’d be done in an afternoon.

First, you wrote a small HTTP service. One endpoint to accept events, one to forward them to subscribers. A few dozen lines. Clean.

Then, another service needed to know when invoices were updated. So you added a second subscriber… a third subscriber wanted events filtered by type.

You added a routing table, just a map, nothing fancy…

Certain that this was enough, you deployed it.

Subscribers started timing out under load.

• You added retries.
• Retries piled up during deploys.
• You added a dead letter queue.
• The dead letter queue needed monitoring.
• You added a dashboard.

Somewhere around the fourth outage you realized that subscribers needed to be isolated from each other.

A crash in the billing subscriber shouldn’t take down the notification subscriber, so you gave each subscriber its own process.

Each process had its own mailbox. Each mailbox was a queue. Each queue had its own retry policy.

You had built lightweight processes with isolated heaps communicating through message passing. Done at last~

Then a subscriber got stuck in a retry loop and consumed all available memory. You added process monitoring.

• If a subscriber died? A monitor would restart it.
• If it died too many times? The monitor would give up and alert you.
• The monitors needed monitoring. You built a supervision hierarchy.

A colleague asked if the system handled network partitions.

It did not.

You added heartbeats between nodes.

You added leader election for the event router.

You added a distributed registry so subscribers could find each other across the cluster.

Eventually you found yourself implementing things that sounded suspiciously familiar.

• A hot code upgrade without dropping messages.
• Preemptive scheduling so one slow subscriber didn’t block the others.
• A shared-nothing architecture because shared state was where the bugs lived.
• An OTP-like library on GitHub that was suspiciously well-documented.
• Immutable messages copied between processes so nobody could mutate someone else’s state.

You sat back and looked at what you had built.

Lightweight processes. Isolated heaps. Message passing. Supervision trees. Preemptive scheduling. Hot code reloading. A distributed registry. Immutable data.

Dear friend, I am afraid to inform you that you have built an Erlang.

Addressed to,

Those who wanted to POST events between a few services…

Addendum

I’m very surprised this hit the front page of Hacker News ;w;

Y’all are a weird bunch. I threw this together inspired by the other two linked posts, and I didn’t expect it to resonate with so many people.

I guess a lot of us have been down this road before, and it’s nice to see that others have had similar experiences.