Autoscaling

Every serving deployment scales within the min / max replica range you set — this page explains how those decisions are made. The rules below apply to LLM and embedding models. Image and video models use a simpler queue rule — see Image & video models.

Thresholds and timings are platform-managed and may be tuned over time; the shapes of the rules stay as described here.

What drives scaling

The scaler re-evaluates every 30 seconds, using a short rolling window rather than single readings — one spike never triggers anything by itself. It watches two signals:

Memory load — how much of each replica’s GPU memory is held by in-progress requests. This is measured in actual tokens, so a short chat (a few hundred tokens) and a 100k-token document analysis are weighted very differently, even though each is “one request”.
Engine queue — requests a replica has accepted but cannot start running yet. A growing queue means the fleet is genuinely out of capacity.

From the memory load it computes how many replicas the traffic actually needs:

Example. 3 replicas at 90% / 80% / 70% memory load carry ≈ 2.4 replicas’ worth of work. The target comfort level is ~60% per replica, so the scaler wants 2.4 / 0.6 → 4 replicas.

Replica capacity is counted in real terms: if one replica runs on a larger GPU than the others, its extra memory counts as extra capacity. Mixed-GPU deployments scale on what the fleet can actually hold, not on replica count.

When replicas are added

Path	Trigger	Time to decision
Sustained load	Needed replicas > current for 2 consecutive checks	~60 seconds
Burst	Engine queues jump sharply in a single check	Next check (≤30s)

A typical scale-up, end to end:

t=0 traffic doubles → t≈30s first check sees the higher load → t≈60s second check confirms it, a new replica is requested → GPU is assigned (cheapest fit under your price cap) → the replica cold-starts — instant if the model weights are already cached on that GPU node, a few minutes with a weight download → it joins the endpoint and starts taking new conversations.

Things to know:

At most 2 replicas are added per step, and never beyond your max.
A replica that is already being prepared counts as capacity on the way — the scaler won’t order a duplicate for the same load.
Scale-up can be blocked by your price cap, by temporary GPU shortage (banner reference), or by an exhausted credit balance. Fix the cause and it retries automatically.

When replicas are removed

Scale-down is deliberately cautious — removing a replica that was still needed means paying the cold-start again minutes later. All of the following must hold, continuously, for about 2.5 minutes:

No requests are queued anywhere.
Memory load is low across the window.
The remaining replicas could absorb the departing replica’s load and still stay comfortably below the scale-up point. This check uses the real capacity of the specific replica being removed.

Example. 2 replicas at ~25% load each: removing one puts the survivor at ~50% — under the limit → scales down to 1 (if your min allows). 2 replicas at ~40% each: the survivor would land at ~80% → stays at 2, even though both look half-idle.

Replicas are removed one at a time. Billing is per GPU-hour, so each removal lowers your cost immediately.

How requests are routed

Routing and scaling work together — balanced replicas mean the load signals above reflect reality:

Conversations stick to a replica. Follow-up turns of the same conversation return to the replica that served it, which keeps its response cache warm — that’s why follow-up turns start noticeably faster than first turns.
New conversations go to the replica with the most free capacity, weighted by GPU size. With replica capacities of 4 / 4 / 8 units, new conversations spread roughly 1 : 1 : 2.
Overloaded replicas shed new work. If a conversation’s home replica is near its memory limit, new requests detour to a freer replica for that request only — the conversation returns home once pressure drops. A detoured request may have a slower first token (its cache lives on the home replica); that’s the trade against queueing behind a saturated GPU.
Sticky routing heals itself. If a conversation’s home replica is removed (scale-down, replacement, failure), follow-up turns are reassigned to a live replica automatically — requests don’t fail because a replica went away, though the first turn after the move starts without the warm cache.

Image & video models

Diffusion models don’t have the token-level memory signal, so they scale on queued jobs per replica: image models scale up around ~4 concurrent jobs per replica, video around ~2 (jobs run minutes each), and scale down when fully idle. Everything else on this page — min/max range, blocked reasons, one-at-a-time removal — applies the same way.

What you control

Setting	Effect on scaling
Min replicas	Never scales below this — guaranteed floor, even with autoscale off. Failed replicas are replaced automatically.
Max replicas	Hard ceiling. Worst-case spend is `max replicas × price cap` per hour.
Autoscale	Off = hold at min; the rules above stop applying.
Price cap	Bounds which GPUs a scale-up may use — too low a cap is the most common reason scale-up blocks.

Signal thresholds (target load, windows, hysteresis) are platform-managed and not configurable per deployment. The same rules apply across /chat/completions, /completions, and /embeddings.

Troubleshooting

You might see	It means	Do
Load spiked but no new replica yet	Sustained-load path needs ~60s of confirmation	Wait one more check — bursts with real queueing react within 30s
Scale-up blocked (price cap / capacity) banner	No GPU under your cap, or temporary shortage	See banner reference
Scale-up stopped while credit is empty	Growth is paused to protect you from a stopped deployment	Top up credit — retries automatically
Stuck at max while queues grow	Ceiling reached	Raise max (and check the cap covers enough GPU types)
Won’t drop below N replicas	N is your min, or removing one would overload the rest	Lower min, or it’s working as designed
Fewer replicas added than expected during a burst	Replicas already provisioning count toward the target	None — they’ll come online
Occasional slow first token on a busy deployment	The request detoured off a saturated replica	None — protects overall latency