Local inference contested between hardware cost and model control
Sam Altman advocates cloud for local inference, while Eclypsium shows a model can be altered in memory without touching its file. Two visions of security and control collide.
The debate on AI model execution is changing tone: on one side, Sam Altman believes it is better to concentrate resources in the cloud; on the other, Eclypsium shows that a compromised infrastructure can modify a model’s behavior during inference without altering its original file. Between component shortages, memory pressure, and infrastructure security, the question is no longer just where to run AI, but who really controls what is executed.
Cloud as a response to shortage, according to Sam Altman
In a recent interview on the Cheeky Pint podcast, the OpenAI CEO argued for maximum cloud use, on the grounds that global silicon capacity remains constrained. According to him, multiplying personal machines capable of running models at home wastes a scarce resource, especially as memory and storage demand is already rising sharply. His reasoning mainly targets industrial chip allocation, not the comfort of users who like to keep their models on hand. It’s less romantic than a gamer PC under the desk, but the calculation is clear.
Altman also claimed that many people overestimate the capacity of local inference, i.e., running a model on one’s own machine rather than on a remote server. For him, hyperscalers offer better infrastructure utilization. However, this position fuels an underlying tension: the more AI depends on the cloud, the more access to models, data, and operating costs concentrates among a few players.
When infrastructure modifies the model without touching the file
Eclypsium offers a very different answer, but on another ground: security. In research published on October 29, the company explains it demonstrated that a compromised infrastructure can modify a model’s behavior during inference, even though its stored files remain intact. In other words, verifying a signature or hash is not enough to prove that the model in memory is the one that was approved.
The demonstration, conducted with NVIDIA’s support, relied on a data processing unit, or DPU, a programmable processor responsible for network, storage, or security functions. By combining this compromised component with permissive memory isolation, researchers modified model weights at runtime. The produced responses changed without affecting the inference service or the original file. The experiment even impacted a geopolitical question, reminding us that the issue goes far beyond a simple lab test.
The scenario, however, depended on specific conditions: an NVIDIA Grace Hopper environment, a DMA access path, and settings such as IOMMU passthrough and permissive PCIe isolation. Researchers specify these were not the default Linux settings. With stricter isolation configuration, the same attack path was no longer available. The alert is therefore less about a universal flaw than a stark reminder: protection capability exists only if it is actually enabled.
AI security does not stop at the model
The core point, for Eclypsium as for NVIDIA, is simple: AI trust boundaries also include hardware, firmware, and device configuration. NVIDIA highlights mechanisms on its BlueField for secure boot authentication, hardware root of trust, attestation, and update protection. But the availability of a feature alone says nothing about its proper configuration in a given environment.
This nuance becomes critical when models access tools, enterprise data, or launch actions. NVIDIA presents its Open Agent Safety Platform as a layered response: OpenShell for execution governance, and Sentry on BlueField to monitor from an isolated zone of the system. The idea is coherent: move control outside the application to reduce risk. However, this new control point must also be trustworthy. Otherwise, the problem is simply shifted from one layer to another.
What operators must verify in practice
Eclypsium describes what it calls the AI Infrastructure Assurance Gap, the gap between model-centered assurance and actual infrastructure integrity. The company proposes inventorying components, assessing firmware and configuration, comparing equipment status to approved references, and tracking changes over time. For operators, this means treating GPUs, DPUs, management controllers, and firmware as full assets, with identified owners, tracked versions, and known update processes.
This monitoring must also survive lifecycle changes: deployment, update, incident recovery, reassignment. In shared environments, hardware may serve multiple successive clients; a clean system on the surface therefore guarantees nothing at lower layers. Model signatures remain useful, but they only say one thing: the loaded file is the expected one. Actual execution depends on a broader chain. And that chain does not forgive blind spots.
Key points
- Sam Altman advocates cloud use to maximize silicon utilization.
- Eclypsium published its research on October 29.
- The demonstration used an NVIDIA Grace Hopper environment.
- The compromised DPU modified model weights in memory.
- NVIDIA highlights BlueField, attestation, and root of trust.
- Eclypsium speaks of an “AI Infrastructure Assurance Gap.”
By the numbers
- October 29 — date of Eclypsium’s research publication.
- 1 model — behavior modified during inference without changing the stored file.
- 0 modification of the original file — the model’s disk hash remained consistent according to the demonstration.
- 2 configurations — a permissive setting tested, then a stricter isolation blocking access.