Early Access

FoundryOS: Control plane and router for self-hosted inference

Run your GPU inference like a platform

FoundryOS routes to and monitors the self-hosted inference you already run across 1 to 20 GPU nodes. One endpoint for your tools, live visibility of every GPU and backend, and safe maintenance. Nothing leaves your network.

We run our own inference platform on FoundryOS in our rack at NextDC M1 Melbourne.

Release 2026.11

The Deck flow view on an RTX PRO 6000 Blackwell cluster.

Architecture Overview

Four components that work together, and keep working when one is missing. Start them in any order; they degrade gracefully. FoundryOS does not deploy or place models. It routes to and monitors what you already run.

FoundryOS architectureApplications send requests to Relay, which routes them to vLLM, SGLang and llama.cpp backends on GPU nodes. A Scout agent on each node reports telemetry to Fleet. Fleet pushes the routing table to Relay and serves the Deck console.Your tools and appsRelayOne base URL, routes requestsFleet + DeckControl plane and consolerouting tableGPU node 1vLLM endpointNVIDIA GPUsScout agentGPU node 2SGLang endpointNVIDIA GPUsScout agentGPU node 3llama.cpp endpointNVIDIA GPUsScout agenttelemetryFoundryOS architectureApplications send requests to Relay, which routes them to vLLM, SGLang and llama.cpp backends on GPU nodes. A Scout agent on each node reports telemetry to Fleet. Fleet pushes the routing table to Relay and serves the Deck console.Your tools and appsRelayOne base URL, routes requestsEach GPU nodevLLM, SGLang or llama.cppNVIDIA GPUsScout agenttelemetryFleet + DeckControl plane and consolerouting table

Fleet

The control plane. Tracks nodes, endpoints, models and health, pushes the routing table to Relay, applies per-node config live without restarts, and runs maintenance and decommission flows. It also reports version skew across your nodes.

Scout

A lightweight agent on each GPU node. Reports NVIDIA GPU utilisation, VRAM, temperature, power, throttle state and ECC, plus CPU, memory, disk and network. It discovers models and reads health from vLLM, SGLang and llama.cpp.

Relay

The endpoint your tools talk to. One base URL speaks OpenAI chat, OpenAI Responses and Anthropic messages, and routes on live backend load, GPU pressure and measured time to first token. It keeps serving if Fleet goes down.

Deck

The web console embedded in Fleet. A wall-screen overview, a live flow view from relays to models, endpoints and nodes, a GPU wall, and the controls to drain, enable, re-probe and put nodes into maintenance.

The Deck Relay page listing routing decisions across model endpoints on an RTX PRO 6000 Blackwell cluster

One endpoint for every tool

Point Claude Code, Codex CLI, your SDKs or Open WebUI at a single base URL. Relay translates between API formats and picks the best backend for each request.

  • Speaks OpenAI chat, OpenAI Responses and Anthropic messages, translating for whichever backend serves the model
  • Load-aware routing from live backend load, GPU pressure and measured time to first token
  • Least-connections, round-robin, sticky and weighted strategies when you want them
  • A routing-decision feed shows why each request went where it did
  • API-key guard on the endpoint

One request, end to end

Clients on the left, Relay in the middle, your backends on the right. Relay translates the API format and picks the backend for each request.

  1. Clients such as Claude Code, Codex CLI, Open WebUI or your own app call one base URL.
  2. Relay translates between OpenAI and Anthropic formats and routes on live load.
  3. Backends running vLLM, SGLang or llama.cpp serve the request, and failed ones are skipped.
The Deck Endpoints matrix, with one row per engine endpoint and a chart per metric on a shared time axis

Swap inference engines without users noticing

Serve the same model from SGLang, vLLM and llama.cpp at once, behind one name and one URL. We use this ourselves at NextDC as we move between vLLM and SGLang when new models land.

  • Endpoints on different engines serving the same model share one model name, so clients never see which engine answered
  • Drain one engine, bring up another, and clients keep the same URL and model; new endpoints join the routing table as soon as they are discovered
  • Compare engines live on real traffic: time to first token, tokens per second, KV-cache, queue depth and error rate per endpoint, in Deck and Prometheus
  • Adopt a new engine release or feature when a model lands, and roll back by draining the endpoint if it disappoints
The Deck GPU wall showing per-GPU activity across RTX PRO 6000 Blackwell nodes

See every GPU, not just the endpoint

Scout ties backend behaviour to the hardware underneath it, so you can tell whether a slow model is a utilisation problem, a thermal problem or a queue problem.

  • Per-GPU utilisation, VRAM, temperature, power, throttle state and ECC
  • Backend KV-cache and prefix-cache hit rate, queue depth and tokens per second
  • Which GPU each endpoint is running on
  • CPU, memory, disk and network per node
  • Model and health discovery for vLLM, SGLang and llama.cpp
Deck Overview for an RTX PRO 6000 Blackwell cluster

A console built for the wall screen

Deck is embedded in Fleet, so there is nothing extra to run. Glance at the overview, or follow a request from relay to model to endpoint to node.

  • Wall-screen Overview readable from across the room
  • Flow view from relays to models, endpoints and nodes
  • Endpoint table and matrix, logs and system status
  • Connect page generates ready-to-paste configs for Claude Code, Codex CLI, OpenCode, Open WebUI, the OpenAI and Anthropic SDKs, and curl
A node in maintenance in the Deck flow view, with traffic moving to the remaining RTX PRO 6000 Blackwell nodes

Maintain nodes without dropping requests

Take a node out for driver updates or repairs and let traffic move to healthy endpoints. When something fails unexpectedly, Relay routes around it.

  • Drain, enable and re-probe endpoints from Deck
  • Node maintenance and decommission flows driven from the UI
  • Retries, failover and circuit breakers keep requests flowing when a backend struggles
  • Relay keeps serving from its last routing table if Fleet is down
  • Version-skew reporting shows nodes that fell behind
The Deck Connect page, generating client configuration for an RTX PRO 6000 Blackwell cluster

Built to run, built to script

FoundryOS fits the tooling you already use for operations, and degrades gracefully instead of failing as a whole.

  • The foundry CLI has scriptable status exit codes for health checks and automation
  • Prometheus metrics on every component, ready for Grafana
  • Components start in any order and degrade gracefully
  • Separate operator and node-join tokens; Fleet refuses to start with placeholder tokens on a network bind
  • Self-hosted, with no cloud passthrough, so it suits air-gapped environments

Frequently Asked Questions

What is FoundryOS?

FoundryOS routes to and monitors the self-hosted inference you already run across 1 to 20 GPU nodes. It has four components: Fleet, the control plane that tracks nodes, endpoints, models and health; Scout, a per-node agent that reports GPU, system and backend telemetry; Relay, a single OpenAI and Anthropic compatible endpoint with load-aware routing and failover; and Deck, the web console embedded in Fleet. It does not deploy, schedule or place models for you, and it does not proxy to cloud APIs.

Who is FoundryOS designed for?

Teams running their own inference on a handful of NVIDIA GPU nodes who want one endpoint for their tools, live visibility of every GPU and backend, and safe maintenance without dropped requests. It suits organisations that must keep data on their own hardware, including private cloud and air-gapped environments.

When will FoundryOS be available?

FoundryOS ships monthly, calendar-versioned releases, with 2026.11 the first. Early access opens to select customers from 1 November 2026, with general availability in January 2027. Join the waitlist to be notified.

What inference backends does FoundryOS support?

vLLM, SGLang and llama.cpp. Scout discovers models and reads health and cache metrics from each backend, and Relay translates between the OpenAI chat, OpenAI Responses and Anthropic messages APIs so clients can use whichever format they prefer, regardless of what the backend speaks.

Can I run the same model on vLLM and SGLang at once?

Yes. Endpoints on different engines that serve the same model share one model name, so clients call one URL and never see which engine answered. You can drain one engine, bring up another and compare them live in Deck and Prometheus on per-endpoint time to first token, tokens per second, KV-cache, queue depth and error rate. Relay spreads requests across the endpoints with your chosen load-balancing strategy; it does not run controlled A/B splits.

Does FoundryOS deploy or schedule my models?

No. You run your inference servers however you already do. FoundryOS discovers what is running, routes requests to it and shows you what is happening. It also does not send traffic to cloud APIs; Relay only routes to your own endpoints.

How does FoundryOS differ from Olla?

Olla is an open-source, single-binary LLM proxy suited to small and medium deployments. FoundryOS adds a control plane, per-node GPU telemetry and an operator console for multi-node GPU fleets. If you manage a single server or a small team, Olla is the right starting point. When you need fleet-wide visibility and maintenance workflows, FoundryOS is the next step.

How do I get early access to FoundryOS?

Join the FoundryOS waitlist to register your interest in early access. Join the waitlist.

Join the FoundryOS Waitlist

Early access opens to select customers from 1 November 2026, with general availability in January 2027.

Want independent evidence of what your GPUs deliver? See Validate.