Parallel requests & scheduling

How Bedrova runs workloads in parallel, admits requests against the memory budget, and stays up under pressure.

Bedrova runs model types in parallel and schedules requests so one busy workload doesn’t stall the others or crash the app. You don’t configure most of this, but understanding it explains why a request sometimes queues or returns “overloaded” instead of failing.

Lanes per workload

The scheduler gives each workload its own lane — text, vision, embeddings/search, Core ML perception, speech, image — so a long image generation runs alongside a chat stream and an embedding batch rather than behind them. GGUF is a text runtime choice inside the text lane, not its own modality. This is what “every model type, in parallel, through one API” means in practice. See Concepts.

Admission against the budget

Before a request starts expensive work, an admission gate checks memory. If serving it would overcommit unified memory, Bedrova holds or rejects it rather than triggering an out-of-memory crash — the same budgeting discipline as the fit-check, applied at request time.

Concurrency limits

You set a maximum number of parallel requests in Settings → Server/API. Beyond that, requests queue. Some models are deliberately limited: frontier models like DeepSeek V4 run single-flight until concurrent use is proven safe, GGUF text concurrency depends on the selected llama.cpp/model proof, and smaller chat, embedding/search, perception, and document models scale up independently.

Overload behavior

When a lane is saturated, Bedrova returns a retryable response rather than dropping the request or crashing. Clients should back off and retry. If you see this often, lower your parallel request count or check whether a model is pinned that should be.

Cancellation

Cancelled requests release their resources and clean up — a cancelled generation doesn’t leave memory stuck allocated. This keeps long sessions and aborted streams from leaking the budget.