Interactive
This is the AI side of the Service Assistant, wired up and actually running. Turn the traffic up past a million requests a day, starve the cache, kill a graphics card mid answer. Poke it however you like and watch what the design does about it. The interesting number is not how fast one card is. It is how few requests ever reach one.
where a request lands first, and the only part that scales by adding cheap machines
a repeat question, answered without touching anything downstream
finds the passage in that community's documents; arithmetic, not a model
nothing close enough, so it is refused here and no card is ever asked
what is left is batched; one card answers many questions per pass
drawn alongside, taken whenever the cards cannot
A million requests a day is about twelve a second on average, and three times that at the morning peak. Drag it past either.
Residents ask the same twenty questions. A repeat is answered from memory and never reaches retrieval, a queue, or a card.
A question the documents cannot answer is refused here, with no model called at all. That is honesty first, and capacity second.
A card does not answer one question at a time. It answers a batch each pass, so throughput rises with batch size and so does the wait. Autoscale is on: a burst buys a card and then gives it back.
Cheap machines. They do no thinking, so more of them costs almost nothing and buys headroom at the front.
A slow dependency is worse than a dead one. The breaker turns slow into fast and degraded, on purpose.
A model, not a measurement. Retrieval, the gate and the fallback behave as the running system does, and the refusal rate is the one measured on real questions. The card figures are modelled on an NVIDIA L4 running an 8B model with continuous batching, where a pass costs roughly 900ms plus 115ms per item in the batch. Nothing on this page connects to the live service.