Fleet
The control plane. Tracks nodes, endpoints, models and health, pushes the routing table to Relay, applies per-node config live without restarts, and runs maintenance and decommission flows. It also reports version skew across your nodes.
FoundryOS: Control plane and router for self-hosted inferenceRun your GPU inference like a platform
FoundryOS routes to and monitors the self-hosted inference you already run across 1 to 20 GPU nodes. One endpoint for your tools, live visibility of every GPU and backend, and safe maintenance. Nothing leaves your network.
We run our own inference platform on FoundryOS in our rack at NextDC M1 Melbourne.
Release 2026.11
Four components that work together, and keep working when one is missing. Start them in any order; they degrade gracefully. FoundryOS does not deploy or place models. It routes to and monitors what you already run.
The control plane. Tracks nodes, endpoints, models and health, pushes the routing table to Relay, applies per-node config live without restarts, and runs maintenance and decommission flows. It also reports version skew across your nodes.
A lightweight agent on each GPU node. Reports NVIDIA GPU utilisation, VRAM, temperature, power, throttle state and ECC, plus CPU, memory, disk and network. It discovers models and reads health from vLLM, SGLang and llama.cpp.
The endpoint your tools talk to. One base URL speaks OpenAI chat, OpenAI Responses and Anthropic messages, and routes on live backend load, GPU pressure and measured time to first token. It keeps serving if Fleet goes down.
The web console embedded in Fleet. A wall-screen overview, a live flow view from relays to models, endpoints and nodes, a GPU wall, and the controls to drain, enable, re-probe and put nodes into maintenance.
Point Claude Code, Codex CLI, your SDKs or Open WebUI at a single base URL. Relay translates between API formats and picks the best backend for each request.
Clients on the left, Relay in the middle, your backends on the right. Relay translates the API format and picks the backend for each request.
Serve the same model from SGLang, vLLM and llama.cpp at once, behind one name and one URL. We use this ourselves at NextDC as we move between vLLM and SGLang when new models land.
Scout ties backend behaviour to the hardware underneath it, so you can tell whether a slow model is a utilisation problem, a thermal problem or a queue problem.
Deck is embedded in Fleet, so there is nothing extra to run. Glance at the overview, or follow a request from relay to model to endpoint to node.
Take a node out for driver updates or repairs and let traffic move to healthy endpoints. When something fails unexpectedly, Relay routes around it.
FoundryOS fits the tooling you already use for operations, and degrades gracefully instead of failing as a whole.
FoundryOS routes to and monitors the self-hosted inference you already run across 1 to 20 GPU nodes. It has four components: Fleet, the control plane that tracks nodes, endpoints, models and health; Scout, a per-node agent that reports GPU, system and backend telemetry; Relay, a single OpenAI and Anthropic compatible endpoint with load-aware routing and failover; and Deck, the web console embedded in Fleet. It does not deploy, schedule or place models for you, and it does not proxy to cloud APIs.
Teams running their own inference on a handful of NVIDIA GPU nodes who want one endpoint for their tools, live visibility of every GPU and backend, and safe maintenance without dropped requests. It suits organisations that must keep data on their own hardware, including private cloud and air-gapped environments.
FoundryOS ships monthly, calendar-versioned releases, with 2026.11 the first. Early access opens to select customers from 1 November 2026, with general availability in January 2027. Join the waitlist to be notified.
vLLM, SGLang and llama.cpp. Scout discovers models and reads health and cache metrics from each backend, and Relay translates between the OpenAI chat, OpenAI Responses and Anthropic messages APIs so clients can use whichever format they prefer, regardless of what the backend speaks.
Yes. Endpoints on different engines that serve the same model share one model name, so clients call one URL and never see which engine answered. You can drain one engine, bring up another and compare them live in Deck and Prometheus on per-endpoint time to first token, tokens per second, KV-cache, queue depth and error rate. Relay spreads requests across the endpoints with your chosen load-balancing strategy; it does not run controlled A/B splits.
No. You run your inference servers however you already do. FoundryOS discovers what is running, routes requests to it and shows you what is happening. It also does not send traffic to cloud APIs; Relay only routes to your own endpoints.
Olla is an open-source, single-binary LLM proxy suited to small and medium deployments. FoundryOS adds a control plane, per-node GPU telemetry and an operator console for multi-node GPU fleets. If you manage a single server or a small team, Olla is the right starting point. When you need fleet-wide visibility and maintenance workflows, FoundryOS is the next step.
Join the FoundryOS waitlist to register your interest in early access. Join the waitlist.
Early access opens to select customers from 1 November 2026, with general availability in January 2027.
Want independent evidence of what your GPUs deliver? See Validate.