Making scale a feature: inside the Omni Processing Unit
By Mike Uhler, Chief Technology Officer
Most AI systems get less efficient as they get bigger. Add accelerators to a cluster and the share of time spent computing usually falls, because more of the work turns into moving data and waiting for it.
We designed the Omni Processing Unit (OPU) to behave the other way, so that utilization rises as the system grows. This post explains how the two parts at its core, the MultiPlexus fabric and the AI OmniCore compute engine, make that possible.
Why utilization falls as clusters grow
In a conventional cluster, every accelerator has its own memory. Model weights, activations, and caches are split across devices, and any data that crosses a device boundary is copied over a link much slower than local memory.
Software has to schedule those copies, and compute often sits idle while it waits for them. As the cluster grows, a larger share of the traffic crosses the slowest links, so each added accelerator contributes less.
The underlying measure is arithmetic intensity: the number of operations a system performs for each byte it moves from memory. When intensity is low, memory bandwidth caps performance no matter how much compute the chip has.
MultiPlexus: one coherent fabric at every scale
MultiPlexus is the fabric that makes the OPU composable, and it is the same fabric at every level. On each die, it is a high-performance network on chip with distributed caches.
Over UCIe, it connects every chiplet in a package. Over SerDes, it connects every package in a system, and across systems and racks it forms unified memory pods.
Because the fabric is coherent end to end, cores share data through hardware rather than copying it between separate memory spaces under software control. Data stays closer to compute, and redundant transfers drop.
Why arithmetic intensity rises with scale
As an OPU system grows, the cache on the fabric grows with it, and fabric coherence lets any core reuse data held anywhere in that cache instead of fetching another copy from memory. More reuse means fewer bytes moved for the same work.
That is why arithmetic intensity rises as the fabric scales, by as much as 10x. Higher intensity moves a workload off the memory bandwidth limit and toward the compute limit, so utilization climbs with system size instead of falling.
AI OmniCore: keeping every core busy
AI OmniCore is the compute engine of the OPU: thousands of efficient cores, paired with Arm Neoverse, that run any AI model across training, fine-tuning, and inference.
Three features keep those cores busy. Flexible matrix shapes let the hardware fit the model instead of forcing the model to fit the hardware. Multiple contexts per core let a core run other work while tool calls happen, and hardware-managed resource arbitration schedules that work without relying on software.
Confidential computing and virtualization are built into the core itself. Several workloads or tenants can share an OPU with isolation enforced in hardware.
What it adds up to
Together, the fabric and the cores give the OPU higher performance density as systems grow. Multi-processor configurations scale from petaFLOP to exaFLOP performance with linear efficiency, and in the datacenter that reduces cost and power by up to 90%.
The same architecture spans a compact Helix-M at the edge to Helix-R racks in the datacenter, and software sees one programming model across all of it. In the next post, Shirish Seetharam explains how TAOS lets developers use the OPU from PyTorch without changing their code. If you would like a deeper technical briefing, contact our team.