One runtime, everywhere: inside TAOS
By Shirish Seetharam, Co-Founder and CVP, AI Software
Every new AI hardware company gets the same first question, and it is about the software stack. Teams want to know what happens to the models, kernels, and pipelines they have already built.
TAOS, the Tsavorite Agentic Operating Stack, is our answer. It serves the models you run today on the Omni Processing Unit (OPU), through the frameworks you already use, with no changes to the models or vendor lock-in. This post covers what TAOS supports, how it scales, and where it stands today.
Starting where developers already work
TAOS meets inference teams in the three places they already work:
- PyTorch: Models written for PyTorch run on the OPU as written. TAOS plugs in beneath the framework, so there is no model rewrite and no new modeling API to learn.
- vLLM: TAOS integrates with vLLM through its hardware plugin interface, so production serving features such as continuous batching and paged KV cache stay in place and your serving configuration carries over.
- llama.cpp: For edge and on-premises deployments that standardize on llama.cpp and GGUF models, TAOS provides an OPU backend.
No changes are required to serve a model, and that includes newly released open-weight models. When a new architecture lands in PyTorch or vLLM, it reaches the OPU through the same path rather than waiting on a hand port.
Scale up on one memory address space
Large models and long contexts outgrow a single device, and that is usually where serving software gets complicated. TAOS keeps the division of labor familiar. vLLM distributes the work across nodes, each running its own Linux instance, using the tensor and pipeline parallelism you already configure. TAOS provides the collective communications underneath, across a memory address space that spans multiple nodes.
Those collectives can use symmetric memory: each participating device allocates matching buffers that peers read and write directly, instead of staging data through intermediate copies. This is the primitive fast tensor-parallel and expert-parallel communication is built on, and the model recent PyTorch distributed work has standardized around. Your serving configuration does not change; the communication beneath it gets faster.
The MultiPlexus fabric is what makes this practical. It keeps memory coherent across devices and lets the compiler and runtime place and move data to make the most of available memory bandwidth, including delivering one load to many consumers instead of repeating it; this translates directly into more tokens per second.
A graph compiler and a runtime built to scale
Under the frameworks sit two pieces that do the heavy lifting. The TAOS graph compiler takes models from PyTorch and turns whole graphs into OPU code. Working at graph level lets it fuse operations, plan memory, and schedule data movement across the fabric, rather than optimizing one kernel at a time.
The TAOS runtime executes that code from a single OPU to multi-node systems. It manages memory placement, dispatch, and synchronization, so the same deployment scales from an edge box to a rack without a change in how you serve.
Bring your kernels, or let TAOS write them
Most inference teams carry some custom CUDA: an attention variant, a fused quantization step, a sampling kernel. TAOS includes a defined workflow for converting those CUDA kernels, so that investment carries over instead of it being rewritten from scratch.
We also generate kernels automatically. An agent-driven pipeline writes candidate kernels, compiles and runs them, checks numerical results against a reference, and keeps the best performer. That shortens the time from a new operator appearing in a model to an optimized OPU kernel serving it, and it is a large part of why new models do not need hand porting.
Running today
TAOS is running today, right now. The full stack, from framework integration through the compiler and runtime is running live today and early customers are using it to verify their own models. Contact us if you are interested in TAOS and evaluating our SDK.
In the next post, Chandra Joshi will introduce the Helix systems that bring the OPU and TAOS to the edge, the desk, and the datacenter.