Summary
- Scality launched AI Inference Factory on 8 October, combining model serving, routing and shared cache storage.
- Company testing reports 166ms first-token response from stored context against about 2.3 seconds to recompute.
- The architecture shifts requirements towards fast storage and networking; operational benefits remain workload dependent.
Scality has launched an AI inference software stack that moves reusable model context into shared object storage, aiming to allow GPU servers to resume requests without repeating expensive calculations. AI Inference Factory, announced on 8 October, combines model serving, request routing and storage software for installations operated on customers’ own infrastructure.
The launch concerns the way existing computing resources are organised, rather than a newly commissioned data centre. Scality says the stack is available as a software licence or managed service, running validated open-weight models on standard servers. Its reported performance improvements come from company laboratory testing; no independent production deployment results were provided with the announcement.
Moving cached context beyond a GPU server
Inference systems retain intermediate information from processed prompts as a key-value, or KV, cache. When a request resumes, recovering that cache can save the GPU from processing the same context again. The cache grows with context length and concurrent sessions, competing with model weights for limited high-bandwidth GPU memory.
Scality proposes holding reusable context in its AI Data Infrastructure object storage, making it accessible to GPUs across a serving cluster. The design also separates prefill, which processes a prompt and generates its cache, from decode, which produces subsequent output tokens. Separate GPU pools can specialise in each stage, using the shared cache to transfer state between them.
That introduces storage and network performance into the critical path for delivering the first response token. A slow cache retrieval could eliminate the benefit of avoided recomputation, while contention across the storage fabric could affect multiple servers. Operators would therefore need to consider bandwidth, latency, concurrency and the availability of the shared tier when assessing any reduction in GPU requirements.
Scality’s detailed technical account describes remote direct memory access, or RDMA, connecting the storage servers to the GPU host. Its connector removes part of the conventional S3 request-processing path and transfers cached objects into GPU memory without routing the payload through the host CPU. The company says this approach achieved 97% of network line rate in one reported configuration.
Test results and their limits
In the launch account, Scality reports that restoring a 14,000-token context from its storage tier produced a 166-millisecond warm time to first token. Recomputing that context took approximately 2.3 seconds in its tests, giving a reported fourteenfold difference. For a much larger 438,764-token context, the company reports 7.3 seconds to restore versus 529 seconds to recompute.
Those measurements compare restoration with rebuilding context, not two identical end-to-end production services under all operating conditions. They depend on the selected models, serving software, network arrangement and whether the requested context has already been retained. A workload with little repeated context cannot be assumed to achieve the same result.
The test environment described by Scality included one eight-GPU server, five active storage servers and four 100Gbit/s RDMA links. A sixth storage server was unavailable because of a network-interface fault during the more detailed benchmark exercise. Scality also reports that its stored cache grew from 69GB to 8TB without a comparable increase in first-token latency in the measured configuration.
The company’s separate benchmark account reports 42.47GB/s of sustained object-read throughput against a measured fabric ceiling of 48.8GB/s. It attributes the limiting throughput to the network rather than the storage servers. The figures are useful design evidence, but they are still vendor measurements rather than audited service-level guarantees for customers.
Scality says it contributed changes to open-source components used in the stack, including an adjustment to a vLLM cache connector that previously fetched more blocks than each restoration needed, and support for object-storage model loading. These changes address data movement overhead rather than introducing new computing hardware.
Operating consequences for data centres
The infrastructure proposition is to increase useful work from already installed accelerators while permitting cache capacity to grow independently. That could alter the balance between GPUs, network ports, NVMe storage and associated power and cooling equipment. It does not demonstrate a reduction in total facility electricity demand; additional storage and network hardware also consume power.
Customers would also have to establish how the shared cache behaves during component failures, upgrades and congestion. Cache information may be reproducible, but losing it can increase recomputation and change response times. Those effects need testing against the required availability and latency of each hosted service.
Scality lists vLLM, NVIDIA Dynamo, its connector and AI Data Infrastructure among the software components, with deployment options spanning established server manufacturers. Its announcement identifies the architecture and initial benchmarks, while leaving open how different organisations will size, operate and verify the system under sustained production traffic.
