Building an Autonomous Inference System

Simon Mo

Nov 2026

Company

X min read

Since we started Inferact, I have been getting a lot of questions about how it relates to vLLM: what stays in the open-source project, who maintains it, and what Inferact is for. All great questions; I want to lay out how we think about our technical north star.

The short version is that Inferact builds an autonomous inference system around vLLM, that vLLM stays open, and every engine optimization we and the system build improves vLLM for the whole community.

Inference is the problem

Inference is the only problem we are solving.

Visualization of the 3 frontiers (models, hardware, workloads)

Every week a lab ships a new model architecture, a chip vendor designs hardware with no tuned kernels, or a customer’s traffic shifts from short chat to long-context agents, and each novel combination needs its own tuning before it runs well. Models, hardware, and workloads each moves on their own pace, and the gap between them costs a lot: an unoptimized stack loses throughput and reliability.

vLLM’s job is to bridge this gap. But we realized, you can’t brute force a combinatorial explosion; a search space that’s 5000+ model, 1000+ chip architecture, and an infinitude of workloads.

Inferact is a center of gravity for best inference engineers in the world, many of them are the people who built vLLM. We are unified around one goal: build a system that improves the engine, and improves itself—full autonomy.

Our answer: An Autonomous Inference System around vLLM

At Inferact, we are building an autonomous inference system (AIS) around vLLM. Autonomous means it operates on its own: it takes a goal, works toward it, and seeks feedback, without anyone telling it exactly what to do next.

In practice, the system:

  • Sees the production workloads it operates or is authorized to observe, and learns from what the community shares, without needing access to their deployments.
  • Adapts to new models and new hardware, and to customer workloads as they shift.
  • Evolves its own code and its own brain, the accumulated knowledge and engineering taste behind every decision.
  • Drives itself toward its goals, working through roadblocks the way a strong engineering organization would.
  • Optimizes continuously against each workload’s own targets.

The point is extreme optimization for each point on the continuum of use cases our users and customers care about, on a reliable core that keeps getting better. We want the system’s evolution to be bound only by compute, up to the hardware’s speed of light.

What autonomous inference means in practice

Here is what it could look like on an ordinary day.

Flow diagram of a day in the life of an autonomous inference
  • 3am. A lab shares the preview of a new open model, and the system starts profiling it before anyone at Inferact is awake. Its novel attention variant has no tuned kernel, so it falls back to a generic path and misses the latency target.
  • 4am. Unprompted, the system starts a Hillclimb round: some agents write a megakernel for the variant, others fuse the new ops around existing kernels, and one rewrites an engine from scratch.
  • Overnight. Several candidates win in isolation and lose end to end. Only the combination that survives full serving benchmarks moves forward.
  • Morning. That combination is live at each customer’s workload requirements and chip architectures, and the winning changes land upstream in vLLM.
  • Evening. The system reviews its own day, notices that most of the failed candidates died at the same end-to-end gate, and rewrites its agents to run that check first.
  • Tomorrow. It keeps improving itself autonomously.

This flow depends on our production and open source flywheel:

Visualization for the data flywheel

Production. Writing a standalone engine and optimization is easy because inference tasks are generally verifiable. But this isolated gain typically falls short on production distribution and cannot be robust enough. vLLM runs production but it relies on the distributed inference stack to scale out. In order to verify the performance gain, only the engine that stands the test of prod traffic can be trusted.‍

Open source. vLLM is the land where optimization compounds. We carefully designed the core engine to be composable with many different optimizations. This allows every user to benefit from each release. The autonomous inference system benefits directly from the vast community and ecosystem of contributors, academic labs, and partners. I believe vLLM has access to the widest knowledge base of inference demand in the world; and we will enable the autonomous system to help us govern the project by triage issues and test pull requests faster.

Inferact exists to run this loop and to bring vLLM to production. That has three components.

  1. vLLM leads in the open, and we keep optimizing it. Every performance improvement we find goes upstream to vLLM.
  2. Inferact builds the autonomous system around vLLM. The goal is a system that maintains the inference engine and tailors it to each workload.
  3. Humans set the goals. Fundamentally, both of these require human taste to supply the quality bar and the ideas that improve the system.

Inferact only wins if vLLM keeps getting better for everyone.

Autonomous inference in production today

Parts of this loop already run in production at Inferact. Here are two examples.

MiniMax M3.1 Flash. M3.1 arrived with a new architecture: sparse attention with an indexer, a large MoE, vision input, and native DSpark speculative decoding. vLLM needed to run M3.1 on both NVIDIA and AMD at optimized performance from day 0, in production serving, and the workload kept changing shape underneath us, from long-context agentic coding to short-input, long-output traffic, and then to decode-bound serving once the final checkpoint doubled output length.

We ran open source and production on one branch, shadowed 100% of live M3 traffic onto M3.1 before cutover, and ran a production hillclimb against that traffic, promoting changes from local gates to online shadow evals.

Improvements to the production deployment, each upstreamed to vLLM as it landed:

  • p50 time to first token went from 1.5 seconds on the first non-PD endpoint to 0.8 seconds in production.
  • Patches to the expert-parallel prefill path cut TTFT by 60 to 70 percent.
  • Pipeline-parallel prefill delivered 19 to 46 percent more throughput than tensor parallel.
  • Output per AMD card improves 56%.

The missing kernels, including MSA attention, the indexer, and NVFP4 MoE, went upstream, and day-0 images shipped for NVIDIA and AMD.

Agent V on Kimi K3.1. Agent V is our shared agent infrastructure. Ask it in Slack and it completes the task in our sandbox with native support hillclimbing.

When we brought Kimi K3.1 up on our own serving stack, output quality degraded on certain batches. A 16-agent hillclimb on Agent V reproduced the problem, root-caused it to an attention routing switch that depended on batch composition, and shipped the fix.

  • K3.1 quality: the fix raised our DeepSWE score from 0.558 to 0.602, matching the model’s reference accuracy.
  • K3.1 prefill: a 24-hour hillclimb made time to first token 2.78x faster at 800k context on GB300.
  • K3.1 decode: a 30-hour hillclimb cut p99 time per output token by 35%.
  • K3 in production on GB300: throughput per GPU went from about 0.77M to 1.4M tokens per minute, roughly 1.8x, between mid-September and the end of October, through production hillclimbing.

Agent V runs mostly on Kimi K3 itself, served by vLLM: about 79 percent of its organic sessions and 53 percent of all its tokens come from our own open-weight deployment. More on Agent V later this week.

Where we go next

AIS is a north star. We haven’t got there yet; but we will keep building towards the full loop.

For now, compute is scarce; we can’t yet simulate the world well enough to test changes against traffic that never happened, which is also what Agent V would need to rewrite itself safely; and our feedback loop is still narrow, coming only from the deployments we operate and what customers and the community share.

However, we will aim to close that loop.

  • build one-click observability for vLLM so the system sees more of the world.
  • build an Agent V that works across partners and the vLLM community, so the whole ecosystem feeds it.
  • build this system that sees workloads, adapts to models chips, and improves itself without waiting on us.

The journey towards fully autonomous inference systems will be hard, we are fortunate to work with the best engineers and the entire inference community in the planet to achieve it. We will build the full loop end to end, crude as it is, expecting to fail and learn from it many times over.

Running at volume?

Tell us a model to serve. We’ll help optimize it.

Talk to an engineer