Case study · Distributed systems

Distributed Rate Limiter

Open source

Allow or deny for every tenant, in one atomic Redis round trip.

A multi-tenant rate-limiting service in Go. Services call POST /v1/check with an API key and a route and get back allow or deny. It is advisory, not a proxy: the caller enforces the decision.

Console GitHub

p95 at 2,000 req/s: 1.33 ms
Errors at 2,000 req/s: 0%
Redis round trip per check: 1
Monthly hosting cost: $0

Go
Redis + Lua
PostgreSQL
Prometheus
k6
Docker

The problem

Rate limiting one process is easy: keep a counter in memory. Rate limiting a fleet is the real problem. The moment two instances serve the same tenant, a read-then-write counter lets two requests both see the last free slot and both get through. As the design record puts it: multiple app instances must not race and over-allow traffic.

So the service has to answer allow or deny for many tenants, routes and callers, fast enough to sit on every request, and correctly when requests for the same key land on different instances at the same moment.

Constraints

Correct under concurrency across instances, not just inside one process.
On the hot path of every request, so latency matters more than features.
Rules change rarely; counters change on every request.
Advisory: return a decision and never proxy traffic.
Hosted for $0 a month on Render's free Docker tier, Neon Postgres and Aiven for Valkey.

Architecture

Callers
- Tenant service
- Gateway middleware
Rate limiter API (Go)
- Auth (SHA-256 key)
- Check handler
- Rule resolver
- Rule cache (30 s)
Data stores
- PostgreSQL config
- Redis Lua counters
Operations
- Prometheus /metrics
- k6 load tests

Callers POST a key and a route. The Go API authenticates against Postgres, resolves the rule from a 30-second cache, and makes one atomic EVALSHA call to Redis for the decision.

The key in X-API-Key is hashed with SHA-256 and looked up; an unknown or revoked key gets 401, a suspended tenant 403.
The tenant's rules come from an in-process cache, reloaded from Postgres when older than 30 seconds.
The resolver picks the most specific rule: exact route, then longest prefix, then the tenant default. No match means allow, with no Redis call.
One EVALSHA runs the token-bucket or sliding-window script against the counter key.
The response is 200 or 429, with X-RateLimit-Limit, Remaining, Reset, and Retry-After on a deny.

Key decisions

Counters live in Redis and the logic runs as Lua, one `EVALSHA` per check

Why: The increment-and-check has to be atomic, or two concurrent requests can both squeeze through the last slot. A Lua script runs server-side as one unit, so the read, the decision and the write cannot interleave with another request.
Trade-off: Redis becomes a hard dependency of the check path.
Rejected: In-memory counters per instance (limits multiply by replica count); Postgres row locks (higher latency, a poor fit for a per-request hot path); Redis WATCH/MULTI (more round trips, and harder to reason about than Lua).

When Redis is down, fail closed with 503

Why: Without Redis the service cannot safely decide allow or deny, and an accidental allow is the more expensive mistake.
Trade-off: A short Redis blip blocks checks instead of letting traffic bypass the limits.
Rejected: Fail open (unlimited traffic during an outage); fail closed with 429 (misleading, and a client may retry forever).

Deny with a real 429, not a 200 carrying `allowed: false`

Why: Proxies, SDKs and retry logic already understand 429 natively.
Trade-off: The caller still has to enforce the decision; the core is not a proxy.

A route with no matching rule is allowed

Why: Undefined routes should not be blocked by accident, and rules can be rolled out incrementally.
Trade-off: Callers cannot assume every route is limited.

Rules cached per instance for 30 seconds

Why: Rules change rarely, so the cache absorbs most reads and keeps Postgres off the hot path for rule lookups.
Trade-off: A rule change can take up to 30 seconds to apply on each instance, and there is no invalidation.

Only a hash of each key is stored, and check keys are separate from admin keys

Why: New keys are generated and hashed in the browser, so the real key never lands in logs or a backup. A key that leaks out of a deployed app cannot be used to raise that project's own limits.
Trade-off: Plain SHA-256 is fine for long random keys but would be weak for short, guessable ones.

Two algorithms, two shapes of traffic

	Token bucket	Sliding window
Behaviour	Allows bursts up to capacity	No burst at window boundaries
Used for	Payments	Orders and the tenant default
State in Redis	One hash: tokens and last refill time	Three counters: current, previous, window index
Time resolution	Milliseconds	Seconds
Writes on a deny	Yes	No

Failure modes

What goes wrong	What happens
Redis unavailable or a script error	503, covered by an integration test
Postgres unavailable	503
Unknown or revoked key	401
Suspended tenant	403
An admin key used for `/v1/check`	403
Reading another tenant's config	404
Invalid JSON, oversized route or cost	400, with the body capped at 1 MiB
Counters for idle keys	Expire through a TTL set in the script

Results

Measured with k6 at a constant arrival rate for 15 seconds per tier, against one API instance on a local Docker stack (Apple Silicon MacBook, 8 CPUs and 4 GB for Docker, with the API, Postgres, Redis and Prometheus sharing the machine). The latency is the full HTTP round trip including auth, Postgres and Redis, not Redis alone. Treat it as a lower bound, not a ceiling.

Target	Achieved	p50	p90	p95	Errors
500 req/s	500	0.99 ms	1.19 ms	1.30 ms	0%
1,000 req/s	1,000	0.75 ms	0.91 ms	1.09 ms	0%
2,000 req/s	2,000	0.82 ms	1.09 ms	1.33 ms	0%
5,000 req/s	4,755	12.31 ms	55.0 ms	70.6 ms	31.9%

On the deny path at 1,000 req/s: p50 0.77 ms, p95 1.50 ms, and 14,901 of 15,001 requests correctly denied.

Correctness is tested separately: 500 concurrent checks against a limit of 100 must allow between 99 and 101, run under Go's race detector in CI. Those tests use an in-memory Redis stand-in; only the load tests hit real Redis.

What I would change next

Cache key and tenant lookups as well as rules. Today every check still reads Postgres to authenticate.
Fall back from EVALSHA to EVAL on NOSCRIPT, so a Redis restart that drops the script cache does not turn into 503s until the app restarts.
Take time from Redis instead of each instance's clock, then benchmark more than one instance.
Commit the raw k6 summaries with p99, and rerun the deny path at 2,000 req/s.

Deep dive

One round trip: atomic rate limiting with Redis Lua

Why a rate limiter shared by many instances needs the read, the decision and the write to be one indivisible step, and how two short Lua scripts get there at 1.33 ms p95.

Read the post