Latency vs Throughput

Definitions

• Latency — the time to perform some action or produce some result.
• Throughput — the number of such actions or results produced per unit of time.

The relationship

• These two are often traded against each other, but the goal is usually: maximal throughput with acceptable latency.

• A system can have low latency but low throughput (a single request is fast, but you can't handle many at once), or high throughput with high latency.

Analogy

Think of a factory assembly line:

• Latency = how long it takes one item to go through the line.
• Throughput = how many items come off the line per hour.

Adding more parallel lines (workers) increases throughput, but the time for any one item (latency) stays roughly the same.

Key takeaways

• Don't confuse the two: latency is per-operation time; throughput is operations per time unit.

• Many scaling decisions (e.g., batching, replication, async processing) deliberately accept higher latency to achieve higher throughput.

Rate Limiting Algorithms & Distributed Enforcement

Core algorithms

• Fixed window — count requests in the current time window; reset at each boundary. Simple, but allows up to twice the limit at window edges.
• Sliding window log — store a timestamp per request; reject if the number of timestamps in the last window exceeds the limit. Accurate but memory-heavy.
• Sliding window counter — approximate by combining the previous window's weighted count with the current window's count. Cheaper than the log approach.
• Token bucket — tokens refill at a fixed rate up to a burst capacity; each request consumes a token. Allows short bursts while preserving a long-term rate.
• Leaky bucket — requests enter a queue and are processed at a constant outflow rate. Smooths bursts but can add latency or drop requests.

Distributed enforcement

• Enforce at the edge/gateway or application tier; a distributed system needs shared counters in a store like Redis.
• Use Redis INCR + EXPIRE or a Lua script to perform atomic check-and-increment, avoiding race conditions between multiple API servers.
• Trade off single-node rate limiting (fast, local, but inconsistent across replicas) vs distributed rate limiting (consistent, but adds a Redis round-trip of latency).
• Return 429 Too Many Requests with Retry-After (and optionally X-RateLimit-* headers) to help clients back off.

Common Interview Questions

Design an API rate limiter

Why it's asked here: a rate limiter is a pure latency/throughput problem — you must cap throughput (requests/sec) while keeping latency acceptable.

Key points to discuss:

• Algorithms: token bucket, leaky bucket, fixed window, and sliding window (counter or log).

• Where to enforce: client, gateway, or service; single-node vs distributed.

• Trade-off: tight limits protect the backend (throughput) but may reject/queue requests (latency).

• Use a backing store (Redis) for distributed counters with atomic increments.

flowchart LR
    C[Client] --> RL[Rate Limiter]
    RL -->|allow| B[Backend API]
    RL -->|reject 429| C
    RL --> S[(Redis counters)]

Return the top-k requests during a time interval

Why it's asked here: a streaming top-k problem forces you to reason about how much work you can do per second (throughput) and how fresh results are (latency).

Key points to discuss:

• Count-min sketch / lossy counting for high-throughput approximate counts.
• Heavy-hitters via a heap over sliding windows.
• Batch vs streaming: batching raises throughput at the cost of result latency.
• Discuss read vs write throughput and how to bound memory.

flowchart LR
    E[Event stream] --> W[Sliding window]
    W --> K[Count-min sketch]
    K --> H[Top-K heap]
    H --> R[Result]

Further reading

• Understanding latency vs throughput