Design a Rate Limiter System Design Interview Guide

← System Design Interview Preparation

This guide walks through Design a Rate Limiter the way you would in a backend or Java interview. The numbers are interview estimates. They help you show your thinking. They are not a production capacity plan.

1. Problem

A rate limiter controls how many requests a client can send in a time window.

Example:

• A public API allows 100 requests per minute per API key
• If the 101st request arrives in the same minute, the system rejects it with HTTP 429

Products use this to stop abuse, protect databases, keep costs fair, and give every customer a predictable share of capacity. You already need this in front of a URL shortener create API. Attackers will otherwise mint millions of links.

In this design we are building a service that:

1. Counts requests for a key (user, IP, or API key)
2. Allows the request if the count is under the limit
3. Blocks the request when the limit is crossed
4. Tells the client when they can try again

We are not building a full API gateway product. We are designing the core allow / deny decision.

2. Functional Requirements / FR

Out of scope for a 45-minute interview:

• Full billing and quota dashboards
• Bot detection with machine learning
• Per-endpoint WAF rules
• Global DDoS protection at the CDN layer

Confirm whether this sits inside the API gateway or as a shared library / sidecar. Both are valid. For the interview, a gateway + Redis counter is a clean story.

3. Non-Functional Requirements / NFR

A good interview sentence: the limiter must be faster than the API it protects, and it must work across many servers.

4. Back-of-the-Envelope Calculation

Say these assumptions out loud. Interviewers care more about the method than the exact number.

Traffic assumptions

• 100 million API requests per day through the gateway
• About 10% are writes, 90% are reads
• Read/write ratio ≈ 9 : 1
• We still rate-limit both reads and writes, so limiter QPS follows total traffic

QPS

Average QPS:

100,000,000 / 86,400 ≈ 1,160 QPS

If peak traffic is 5×, design for around 6,000 QPS.

Every request does one limiter check. So limiter QPS ≈ 6,000 at peak, not 1,160.

Redis operations

A simple token-bucket or sliding-window check is 1–3 Redis commands per request.

Peak 6,000 QPS × 2 commands ≈ 12,000 Redis ops/sec

One Redis instance can handle this easily. The hard part is many unique keys, not raw QPS.

Storage / memory estimate

Assume we track 5 million active keys in a 10-minute window (users + IPs + API keys).

Each key stores a count, a window start, and maybe a token remaining field ≈ 100 bytes.

5,000,000 × 100 bytes ≈ 500 MB

TTL on keys keeps memory bounded. Overnight idle keys disappear.

If we keep a 24-hour daily quota per API key for 2 million keys:

2,000,000 × 100 bytes ≈ 200 MB extra

Total cache memory ≈ 1 GB. Mention one Redis primary plus a replica.

Server estimate

The limiter can live in the API gateway or as a tiny Java sidecar.

A conservative 3,000 checks/sec per gateway instance:

Peak 6,000 QPS / 3,000 ≈ 2 gateway instances

Run 3–4 for deploys and failure. Redis is the shared brain, not the app servers.

These are interview estimates, not exact production numbers. The point is: shared Redis counters, tiny memory, check on every request.

5. APIs

Clients do not call the limiter directly in most designs. The gateway calls it. Still, show a clear interface.

Check limit

POST /internal/ratelimit/check

Request:

{
  "key": "api_key:ak_19f3",
  "ruleId": "search-read",
  "cost": 1
}

Success (200):

{
  "allowed": true,
  "limit": 100,
  "remaining": 72,
  "resetAt": 1770000060
}

Rejected (429 at the public edge, or 200 with allowed: false internally):

{
  "allowed": false,
  "limit": 100,
  "remaining": 0,
  "resetAt": 1770000060,
  "retryAfterSeconds": 18
}

Public response when blocked:

HTTP/1.1 429 Too Many Requests
Retry-After: 18
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1770000060

Body can be:

{
  "error": "rate_limited",
  "message": "Too many requests. Try again in 18 seconds."
}

Config (admin)

PUT /internal/ratelimit/rules/:ruleId

{
  "limit": 100,
  "windowSeconds": 60,
  "algorithm": "token_bucket"
}

Do not make every product team invent their own counters. One limiter, many rules.

6. Data Model

Redis is the hot store. A SQL table can hold rules, not live counts.

rate_limit_rules

Redis keys (live state)

Fixed window

• Key: rl:{ruleId}:{key}:{windowStart}
• Value: integer counter
• TTL: window length

Token bucket

• Key: rl:tb:{ruleId}:{key}
• Hash: tokens, updatedAt
• TTL: a few windows so idle users vanish

Sliding window log (only if volume is low)

• Key: rl:log:{ruleId}:{key}
• Sorted set of timestamps
• TTL: window length

Do not store one row per request in Postgres. That becomes the bottleneck you were trying to prevent.

7. High-Level Design

The limiter sits on the request path. Redis holds the counters. The backend never sees blocked traffic.

Components:

• Client: browser, mobile app, or partner integration.
• API Gateway: TLS, routing, auth. The limiter often runs here so every service is protected the same way.
• Rate Limiter: loads the rule, talks to Redis, returns allow or deny.
• Redis / Cache: shared counters across all gateway instances.
• Backend Service: only receives allowed requests.
• HTTP 429: returned immediately when the quota is gone.

Allow flow:

1. Client hits the gateway.
2. Gateway builds a key: user:42, ip:10.1.2.3, or api_key:ak_19f3.
3. Limiter runs one Redis script (INCR + EXPIRE, or token bucket Lua).
4. If allowed, request goes to the backend.
5. Response headers include remaining quota.

Block flow:

1. Same check.
2. Count is already at the limit.
3. Gateway does not call the backend.
4. Return 429 with Retry-After.

Why Redis? App servers come and go. A count in local memory would let a client send limit × number_of_servers requests. Redis is the single shared counter.

8. Deep Dives

Fixed window

Split time into buckets: 10:00–10:01, 10:01–10:02.

INCR rl:search:ak_19f3:1770000000
EXPIRE that key for 60 seconds
if count <= 100 → allow

Simple. Easy to explain. Problem: a client can send 100 requests at 10:00:50 and 100 more at 10:01:01. That is 200 requests in 11 seconds. Interviewers will ask this. Name it the burst at the window edge.

Sliding window

You want the last 60 seconds, not “this clock minute”.

Sliding log: store each request timestamp in a Redis sorted set, remove old ones, count the rest. Accurate, but memory grows with QPS per key.

Sliding window counter (good interview default): keep the previous window count and the current window count. Estimate:

previousCount * (overlap fraction) + currentCount

Almost as smooth as a log, much cheaper.

Token bucket

A bucket holds up to limit tokens. Tokens refill at limit / window per second. Each request spends one token.

This is the algorithm many teams actually ship. It allows a short burst (the bucket) and then a steady rate (the refill). In Java you can implement it with a Redis Lua script so read + compute + write is atomic.

// Interview sketch, not production code
boolean allow(String key) {
    BucketState state = redis.get(key);
    refill(state, now);
    if (state.tokens >= 1) {
        state.tokens -= 1;
        redis.set(key, state);
        return true;
    }
    return false;
}

Do the refill in Redis, not in each JVM, or two servers will both think tokens remain.

Leaky bucket

Requests enter a queue and leave at a fixed rate, like a leak. Smooth output. If the queue is full, drop (or 429). Good when you want to shape traffic to a downstream database. Slightly harder to explain than token bucket. Mention it, then pick token bucket unless the interviewer wants smoothing.

Redis-based counters

Rules for the interview:

• Use atomic commands or Lua. Never GET, add in Java, then SET. Two servers will over-allow.
• Set TTL so dead keys disappear.
• Prefer one round trip.
• If Redis is in another region, limiter latency becomes your API latency. Keep Redis next to the gateway.

Per-user / per-IP / per-API-key

You often apply more than one rule:

1. Per API key: 10,000 / hour (paid plan)
2. Per user: 100 / minute (fairness)
3. Per IP: 20 / second (abuse)

Deny if any rule fails. IP limits catch clients with no login. User limits catch a stolen key used by many IPs. API key limits match the customer contract.

Anonymous URL-shortener creates should use IP + a global daily cap. Logged-in creates can use user id.

What happens when the limit is exceeded

Do not hang. Do not wait until the window resets.

• Return 429
• Include Retry-After
• Include remaining = 0
• Log a metric rate_limited_total
• Optionally write a sample to an audit topic, not every blocked request

Some teams return 503 when the system is overloaded vs 429 when this client is over quota. Keep that distinction. 429 means “you sent too many”. 503 means “we are sick”.

9. Bottlenecks

The first bottleneck to name is a shared Redis key for a very hot client.

10. Tradeoffs

Fixed window vs sliding window vs token bucket

• Fixed window: simplest. Burst at edges.
• Sliding window: smoother, a bit more math.
• Token bucket: bursts + steady rate. Best default.
• Leaky bucket: smoothest outflow, can add queueing delay.

Say you would start with token bucket in Redis.

Accuracy vs speed

Counting every request exactly across three continents is slow. A 1–2% error at the edge is usually fine. Strong accuracy in one region, looser global cap, is a mature answer.

Fail open vs fail closed

• Fail open: Redis down → allow traffic. Product stays up. Abuse can slip through.
• Fail closed: Redis down → 429/503. Safer for payments. Worse user experience.

Interview choice: fail open for public read APIs, fail closed for login, OTP, and payment.

Gateway plugin vs library in each service

• Gateway: one place, cannot forget to protect a new service. See API Gateway.
• Library: each Java service has custom rules, but teams can forget it.

Use gateway for coarse limits, library for business quotas.

11. Failure Modes

Redis down
Decide fail open or closed by API type. Cache last-known remaining tokens in the gateway for a few seconds if you fail open.

Redis slow
Limiter latency becomes API latency. Time out the check (2–5 ms) and apply the fail policy. Never wait 1 second on Redis.

Clock jump
Windows reset early or late. Use Redis time.

Rule misconfig
A limit of 1 instead of 1000 outages your own app. Version rules, canary a new limit, keep an emergency “raise all limits” switch.

Thundering retry
Clients retry 429 immediately and make it worse. Teach Retry-After and jitter. Your own workers must respect 429 too.

NAT / shared IP
Many users behind one office IP hit the IP limit. Prefer user or API key once they are logged in. Keep IP limits only for anonymous traffic.

12. Interview Answer in 10 Minutes

Here is a version you can speak out loud.

"I would put a rate limiter in the API gateway so every backend is protected the same way. The limiter is a small check that runs before the service. It uses Redis so all gateway instances share one counter.

Assume 100 million requests a day. That is about 1,160 QPS average, and about 6,000 QPS at a 5× peak. Each check is one or two Redis commands, so Redis load is modest. Memory is also small: a few million keys with a TTL is well under a gigabyte.

I would store rules in a database: limit, window, algorithm, and whether the key is a user, IP, or API key. Live counts live in Redis. For the algorithm I prefer token bucket: it allows a short burst and then a steady rate. I would implement it with a Lua script so refill and consume are atomic. I would also mention fixed window as the simplest option and sliding window if the interviewer worries about the burst at minute boundaries.

If the client is over the limit, we do not call the backend. We return HTTP 429 with Retry-After and rate-limit headers. We can stack rules: per API key for the plan, per user for fairness, per IP for anonymous abuse.

If Redis is down, I fail open on public reads and fail closed on login or payment. If one key is extremely hot, that Redis key is the hotspot, so I isolate it.

That is the system: gateway check, Redis token bucket, 429 on deny, tiny memory, and a clear fail policy."

Practice this until it is under 10 minutes, then use leftover time for algorithms.

13. Interview Talking Points

• This check is on the hot path. Keep it in microseconds to low milliseconds.
• Redis exists so many servers share one count.
• Token bucket is a strong default. Know fixed window’s edge burst.
• 429, not a hang, not a silent drop without headers.
• Per-user, per-IP, per-API-key can all apply.
• Atomic updates. No get-then-set in Java.
• TTL on keys so memory does not grow forever.
• Fail open vs fail closed is a product decision. Say it out loud.
• Metrics: allowed, denied, Redis latency, hotspot keys.

14. Follow-up Questions

How would you handle 1 million QPS?
Shard Redis, keep the Lua script tiny, push coarse limits to the CDN/edge, and keep per-user limits only where needed. Local in-memory token caches with periodic Redis sync can cut Redis QPS.

Distributed vs local limiter?
Local is faster but wrong when you have many pods. Distributed Redis is the interview default. Hybrid: local bucket refilled from Redis.

How do you rate limit in Java / Spring?
Gateway filter or Spring filter. Redis via Lettuce. Lua for atomicity. Do not use a HashMap on the JVM as the source of truth.

What if limits differ by paid plan?
Rule table keyed by planId. When the user upgrades, update the rule. Existing Redis bucket can be resized on the next request.

How do you avoid punishing a NAT?
After login, key by user id. IP limits only for anonymous endpoints.

Other useful probes: sliding log memory cost, multi-region counters, and rate limiting GraphQL resolvers.

15. Internal Links

Related JavaThoughts reading:

• System Design Interview Preparation — top 20 questions
• Design a URL Shortener
• Design a Distributed Cache
• Java
• Spring Boot
• Caching
• API Gateway
• Rate limiting concept
• Microservices
• Distributed Systems
• 20 system design concepts

Next in this series: Design a Distributed Cache.