Built a small library to limit active concurrent streams (SSE / LLM tokens) in FastAPI #16506

Hi everyone,

Sharing a small library I built to solve an issue we had in production with streaming endpoints.

Standard rate limiters (like SlowAPI or token bucket middlewares) count requests per minute. But for LLM chat completions or SSE streams, a connection stays open for 30-60 seconds. A single user could open 5 or 10 parallel tabs and tie up worker sockets or run up API costs, while the rate limiter only saw 5 requests and allowed them all.

I built fastapi-stream-lease to limit active open connections right now rather than requests over time:
https://github.com/agustin18/fastapi-stream-lease

How it works:

• Uses Redis sorted sets and an atomic Lua script to acquire a lease when the stream starts.
• If user is at limit (e.g. max 2 streams), raises HTTP 429 before starting token generation.
• Slot is released immediately when stream finishes, or if client disconnects.
• Has local circuit breaker fallback so Redis downtime doesn't crash the API.
• Native Prometheus and OpenTelemetry metrics for active streams.

Just released v0.6.0 with 1-line exception handling (manager.register_exception_handlers(app)).

Feedback or critiques are very welcome!