← writing

27 Oct 2026 7 min read

Writing a distributed rate limiter in Golang

What a rate limiter does, why an in-memory bucket stops working once two servers are involved, and how a Lua script makes the check atomic.

A rate limiter is the first line of defence for an API. It sits in front of your handler and answers one question for every request: should this one be allowed to continue? If yes, the request goes through. If not, it gets a 429 Too Many Requests and your database never sees it.

That sounds simple, and for one server it is. The interesting part starts when the same client can hit two servers, because now the limit has to live somewhere both of them can see.

Where the rate limiter sits in the request pathdeveloperadminsets the limitsedits limitsstored oncee.g. 100 req/minrate limiterrate limiterserviceredisget / update counter3userbackendservice1request2check4allow5if allowedprocessapplicationservice, dbcounter keyuser id → countip address → counttoken → countrate_limit:user:123 → 42
A developer sets the limit once. Every request then goes backend → limiter → Redis, keyed by user id, IP, or token, and only allowed requests reach the application.

What a rate limiter does

A limiter has two jobs:

  • stop one client from using more than its share
  • let a client that has been quiet send a short burst

The token bucket does both. Each client gets a bucket. The bucket holds up to max_tokens tokens, and every request takes one. Tokens come back at refill_rate per second, up to the maximum. If there is a token, the request is allowed. If the bucket is empty, the request is rejected.

A client that has been quiet has a full bucket, so it can send max_tokens requests at once. After that it is held to the refill rate. That is the shape most APIs want: it lets a short burst through, then holds the client to a steady rate.

Here is the bucket in Go:

go
type TokenBucket struct {
	mu         sync.Mutex
	tokens     int
	maxTokens  int
	refillRate int
	lastRefill time.Time
}

func (tb *TokenBucket) Allow() (bool, int, int) {
	tb.mu.Lock()
	defer tb.mu.Unlock()

	tb.refill()

	if tb.tokens > 0 {
		tb.tokens--
		return true, tb.tokens, tb.maxTokens
	}
	return false, tb.tokens, tb.maxTokens
}

There is no timer. With 10,000 clients, 10,000 goroutines adding a token every second would be 10,000 mostly idle timers. Instead the bucket refills when someone asks. It looks at the time since last_refill and adds only the tokens that time paid for. A bucket nobody calls costs nothing.

The detail that bites: do not set last_refill to now. Move it forward by exactly the time those tokens were worth. If you set it to now, every request throws away the leftover fraction of a token, and the client slowly gets less than the refill rate promised.

Here is the same rule in the browser, with 10 tokens and one token back per second:

Rate limiter request flowclientrequestsrate limiterrefill over timemax 10 tokensone request = one token1 tokenyour handlerdatabase, workdatabaseallowed requests only429bucket empty
The limiter runs before your handler. An allowed request spends one token; an empty bucket returns 429.

Inside one process

Inside one process, a map is enough: client id to bucket, with a mutex around the map. The decision is cheap. A bucket check is about 14 ns on the machine I measured. The whole HTTP request is milliseconds. The lock costs almost nothing at this scale.

This is the version that works until you run a second copy of the server.

Two servers, one client

Now put two copies behind a load balancer. Each copy has its own map. The client's limit is 10, but server A has its own 10 and server B has its own 10. The client can send 10 to each, 20 total. Add a third server and the limit becomes 30.

The limit is supposed to belong to the client. The bucket belongs to the process. Those are different scopes, and that is the bug.

Two processes do not share memory, so you cannot keep the map in one process and have the other read it. The state has to move to something both of them can talk to.

Two servers with separate in-memory bucketsclientlimit 10/sload balancerround robinserver Aown bucket: 10 tokensallows 10server Bown bucket: 10 tokensallows 10same client, two buckets: the limit is now 20add a third server and it becomes 30
Two servers, two buckets. The client's limit is enforced twice, once per process, so the real limit doubles.

The race in Redis

The usual fix is to move the bucket into a shared store. Redis is the common choice: every server reads and writes the same key.

A plain Redis implementation still has a race. It looks like this: read the token count, check it, then write the new count. Between the read and the write, another server can do the same thing.

Both servers read 1. Both decide the request is allowed. Both write 0. One token paid for two requests. At low traffic you may never see it. Under load it happens constantly.

Atomic means no other operation can run in the middle of it. The read, the check and the write have to be one step.

Naive read-check-write compared with a Lua scriptwithout Luaread, check, writeredistokens = 1one keyA GET → 1B GET → 1A allowB allowA SET 0B SET 0one token, two requestsboth requests saw the same 1the writes happen after the checkswith Luaone script, one stepserver Aserver Bredisone command at a timerefill + check + takeruns to completionone wins, the other gets 429one token, one requestno other command runs in the middlethe second request sees the updated count
Without Lua, two servers can read the same token and both allow the request. A Lua script closes the gap because Redis runs it as one command.

Lua makes it one step

Redis runs Lua scripts as a single unit. While a script is running, Redis does not run another command. So the whole bucket update can happen inside one script: read the hash, refill it, take a token, write it back, set an expiry. One round trip, no gap for another server to slip into.

This is a standard technique. Any time you need "read, decide, write" against Redis, a Lua script is usually the shortest correct version.

lua
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill', 'max_tokens', 'refill_rate')
-- work out how many tokens to add
-- if tokens > 0 then take one
redis.call('HSET', key, 'tokens', tokens, 'last_refill', last_refill, ...)
redis.call('EXPIRE', key, 3600)
return {allowed, tokens, max_tokens}

You could use INCR for a fixed-window counter, and that is already atomic. But a token bucket has to read the last refill time, work out how many tokens have accrued, and write both values back. That is more than one command, so it needs a script.

The catch is that the script blocks Redis while it runs. Keep it short. Ours is a few hash reads and writes.

Keys expire after an hour of silence, so Redis does not fill up with clients who left.

When one Redis is not enough

A single Redis is one process on one machine. When it runs out of room, there are three options.

Vertical scaling means a bigger instance: more CPU and RAM. It is the simplest change, and it has a ceiling.

Read replicas copy every write from the primary. They help reads. The limiter is write-heavy: each allowed request updates a counter, so the primary still takes all the write traffic.

Sharding splits the keyspace across instances. A router hashes the key and sends it to one shard, so rate_limit:user:123 always lands on the same instance. The Lua script still makes each shard's update atomic. The router becomes part of the correctness story too: if the same key can reach two shards, the limit doubles again.

Start with a bigger instance. Shard when one instance cannot keep up.

Scaling a Redis-backed rate limiterscaling a redis-backed rate limiterrate limiting is write-heavy: every allowed request updates a counter1. vertical scalingone larger redismore cpu / ramsimple, hits a hardware limit2. read replicasprimary redis (writes)replica(s) (reads)replicationdoes not fix write load3. shardingredis aa-hredis bi-predis cq-zsplit the keyspacebackendservicesrequest + keykey routersame key, same shardredis aa-hredis bi-predis cq-zroute by keythe same key must always reach the same shardscale up first; shard when one redis cannot keep up
Scale up first. Replicas add read capacity but writes still hit the primary. Sharding spreads the keyspace, as long as the router always sends a key to the same shard.

Who is the client?

None of this matters if the client can choose a new identity. If you read client_id from the request body, the caller sends a new id and gets a fresh bucket. The limiter is then optional.

The default here is the source IP. TRUST_CLIENT_ID is off unless a gateway you control sets the header. That way the id comes from something the client cannot rewrite.

When Redis is down

Any shared store can fail. The choice is fail open or fail closed.

This one fails open: if Redis is unreachable, requests are allowed. A limiter that takes the whole API down when its store breaks is worse than the traffic it was meant to stop. The cost is that while Redis is down, nothing is limited. Availability won. It might not for you.

The memory store has a cap for the same reason: at most 10,000 clients, idle ones dropped after 10 minutes. Without a cap, random client ids are a memory attack, and the limiter becomes the thing that kills you.

Proving it works

The correctness test sends 100 requests at one bucket at the same time. The bucket holds 20 tokens and does not refill. The result has to be exactly 20 allowed and 80 denied, every time.

js
thresholds: {
  allowed: ['count==20'],
  rate_limited: ['count==80'],
}

Exactly 20. If the update is not atomic, those numbers move, which is exactly how the read-check-write version would fail.

Measuring it

I ran everything on an M1 MacBook Air with 8 cores and 8 GB of RAM. hey, the server and Redis all ran on the same machine. 200 connections, one client, with a bucket big enough that it never runs out. Every response is a 200.

Backend Throughput p50 p99
In-memory 70K to 79K req/s 1.9 ms ~13 ms
Redis + Lua 33K to 35K req/s 5.4 ms ~10.5 ms

Redis is about half as fast as memory. That makes sense. Every check has to leave the process, cross to Redis and come back. That is the price of sharing state.

Read these as ranges. Your machine will give you different numbers. Everything shared one CPU. The HTTP tests only measured allowed requests. hey waits for each reply, so the tail latency is nicer than real traffic. Numbers moved 5 to 12% between runs.

What the benchmarks got wrong

My first numbers came from k6 and looked bad, around 18K req/s. Then I noticed k6 was using up to 600% CPU on the same machine. I was measuring the load generator. After switching to hey, the same server did about 30K on that run.

I also thought the mutex would be the bottleneck. The bucket decision takes about 14 ns. The whole HTTP request takes milliseconds. The lock had plenty of headroom. A profile showed the bucket math at around 2.5% of CPU time; the rest was Go's HTTP stack: reading sockets, parsing headers, encoding JSON.

The code

whysooharsh/rate-limiter

Rabbit holes

Things I read while building this, and would read again: