Architecture

hal0 is a single self-hosted service that turns one machine into an OpenAI-compatible inference platform. Everything runs behind one process — hal0-api — which owns configuration, model and slot lifecycle, request routing, and the web UI. Everything else — bundled agents, the memory brain, optional companion services — runs as its own sandboxed sibling unit that hal0-api provisions and talks to over HTTP or MCP, not in-process.

The pieces

hal0-api

A FastAPI application that binds 0.0.0.0:8080. It serves the REST control plane under /api/*, the OpenAI-compatible inference surface under /v1/*, the MCP servers under /mcp/*, and the built dashboard at /.

Slots

Each slot is one model served by one podman container, managed through a hal0-slot@<name>.service systemd unit. SlotManager owns their lifecycle and state; it never imports the dispatcher.

Stacks

A named, portable bundle of slot assignments plus the profiles and model metadata they reference — apply one to load, swap, and unload a whole slot lineup as a unit. See Stacks.

Dispatcher

The registry-aware router that resolves every incoming /v1 request to a slot or an external upstream, then forwards it — wrapping the slot in its serving state for the duration of the call. HTTP-only: it never starts or stops a slot itself.

Bundled agent

Hermes, a third-party agent runtime hal0 provisions and runs as its own sandboxed hal0-agent@<id>.service — the only bundled agent as of v1.0 (earlier speculative bundles, pi-coder and opencode, were removed) — plus hal0-brain, a first-class, Hermes-independent platform steward that drives the admin tool surface directly. See Agents.

Front ends

The React dashboard (served by hal0-api itself) for operating the platform, plus optional OpenWebUI as a chat front end pointed at the /v1 surface.

hal0-api: the control plane

The application is built once in create_app() and exposes a clearly grouped surface area:

• /v1/* — the OpenAI-compatible inference API. The model-listing probe (GET /v1/models) is intentionally separate from the inference writer surface, because OpenAI clients commonly list models before sending an Authorization header.
• /api/* — the operator control plane: slots, stacks, models, capabilities, profiles, providers, companion services, hardware, health, settings, activity, auth, and more.
• /mcp/admin and /mcp/memory — Model Context Protocol servers mounted as sub-applications so agents and tools can drive hal0 directly.
• / — the built dashboard SPA, mounted last so it never shadows an API route. Any unmatched non-API path falls back to the SPA’s index.html so client-side routing survives a reload.

When hal0-api starts, its lifespan wires the long-lived objects together: the model registry, the upstream registry, the SlotManager, the Dispatcher, the hardware probe, the capability orchestrator, and an event bus that streams state changes to the dashboard footer. These are shared across all requests — the API is a thin layer over them.

Note

hal0 ships open by default — no authentication is enforced unless you turn it on. A real, opt-in auth layer exists for when you need it. See Security for the auth tiers and the reverse-proxy pattern.

Slots: where models actually run

A slot is a named, long-lived place to serve one model. Behind each slot is a single podman container started by a hal0-slot@<name>.service systemd unit whose ExecStart runs podman run. SlotManager drives every transition through an explicit state machine and persists it so the dashboard reflects real lifecycle events, not systemd snapshots.

Slots are the unit hal0 reasons about everywhere: the dispatcher routes to them, the GPU arbiter coordinates exclusive GPU access between them, and profiles configure how they run. As of v1.0, the slot is also the sole owner of where a model runs and how much context it gets — a real split from profiles, covered below and in Slots.

Capabilities & profiles: what a slot does, and how

A slot’s behavior is shaped by two more ideas that sit on top of it:

• A capability groups related slots under one operator-facing idea (embed, voice, img, vision) so you configure a concept, not raw slot fields.
• A profile is a device-agnostic, reusable tune template — sampler settings, KV-cache policy, batch size, reasoning mode. As of the v1.0 hardware-ownership split, a profile carries no image and no hardware placement — that’s the slot’s job.

See Capabilities & profiles for the full model, including the write-boundary partition between what the slot owns and what the model owns.

Stacks: applying a slot set as one unit

Editing slots one at a time works, but swapping an entire lineup — the agent model, the utility model, their profiles, MTP settings — is tedious to do by hand. A stack is a single [stack.<slug>] entry in stacks.toml that bundles a set of slot assignments together with the profiles and model metadata they reference, so the whole bundle is portable to another machine.

Applying a stack is two phases: it’s first planned and committed as an atomic write through the same guarded slot-config pipeline every other slot edit goes through (guard_slot_write_payload — a stack can’t write anything a manual edit would be refused), then converged against the live slot set — loading, swapping, or skipping each slot and its capability children, and unloading anything running that the stack doesn’t mention. hal0 ships three seed stacks (saber, forge, pi) tuned to the reference Strix Halo roster; seeds are immutable, so customizing one means cloning it under a new name. See Stacks for the full model, and an honest read on how mature this feature is.

The dispatcher: routing a request

The Dispatcher reads the model registry and the list of upstreams to decide where each OpenAI-compatible request goes. It never starts or stops slots itself; it resolves a target, gates on readiness, and forwards. Resolution runs in a fixed order so behaviour is predictable:

1. Container-slot preemption — a loaded container slot is the authoritative server for the models it advertises, so it wins over any default registry binding.

2. Registry lookup — an exact model-to-upstream binding from the registry. If that upstream is online, forward there.

3. Passthrough — any upstream whose cached /v1/models already advertises the requested model id.

4. Cold-cache prefetch — fan out /v1/models against external upstreams with empty caches (coalesced so concurrent identical prefetches share one call), then re-check passthrough.

5. Capability / path routing — last-resort heuristics: /embeddings → the embed slot, /rerankings (and its /rerank alias) → the dedicated rerank slot, /audio/speech → the TTS slot, /images/generations → the image slot, an FLM name:tag model → the NPU slot, and explicit slot-name addressing. Falls back to the agent slot.

Every routing decision emits one structured log line, so you can always see why a request went where it did.

Data flow of a chat request

1. A client POSTs to /v1/chat/completions with a model field — that can be a model id, or a slot alias: the name of any enabled llm slot, such as agent or utility.

2. The route layer translates a slot alias to the slot’s configured model id, then asks the dispatcher to resolve the request to an UpstreamCall.

3. If the target is a local slot whose model isn’t loaded yet, the dispatcher kicks a load using the slot’s own hardware grid — device, binary, n_gpu_layers, threads — then gates on readiness. A slot that is still loading returns a structured slot.loading 503 with a Retry-After hint instead of a raw connection error.

4. Once the slot is ready, the dispatcher forwards the request, wrapping the slot in its serving state for the lifetime of the call. For a streamed response, that state is held open until the stream drains.

5. The upstream’s response (including streamed SSE chunks and any error envelope) is passed back to the client verbatim.

Tip

Addressing a slot by its alias — its slot name, such as agent or utility — is the stable way to pin a co-resident model: the alias doesn’t change when you swap the underlying model. See Slots.

The front ends

hal0-api serves the dashboard itself — a React single-page application. There is no separate web server to run. The dashboard is where you create and operate slots, pull models, pick capabilities, watch live activity, and configure the platform.

For end-user chat, point OpenWebUI (or any OpenAI-compatible client) at hal0’s /v1 surface. Because hal0 speaks the OpenAI API, anything that talks to OpenAI talks to hal0.

The hal0 dashboard — slot cards, live activity, and hardware metrics in one view.

Where to go next

SlotsThe lifecycle, single-flight dispatch, and the write-boundary partition.

Capabilities & profilesHow capabilities, profiles, and devices shape a slot.

StacksApplying a whole slot lineup as one declarative unit.

Strix HaloThe reference hardware and why it's the tuning target.

SecurityThe open-by-default posture and the opt-in auth layer.