You Don’t Need a $2,000 GPU to Run a 70B Model. This Free Tool Just Proved It, And Nobody’s Talking About It
For three years the unspoken law of local AI was brutal and simple: the bigger the model, the bigger the GPU, the deeper your pocket. Buy a 70-billion-parameter model and the hardware store expects a down payment on a data center. A single open-source library called AirLLM just rewrote that law in roughly forty lines of Python, and the fallout is genuinely ugly for anyone still trying to sell you a $2,000 graphics card as a prerequisite for serious local inference.
The pitch is almost too clean to be true, which is exactly why you should read this before your next hardware purchase. AirLLM doesn’t make your model smaller. It doesn’t make your GPU faster. It does something far sneakier: it refuses to load the whole thing at once.
The Rule That Ruled Local AI — And Why It Was Always a Racket
Here’s the thing nobody likes to say out loud about running big models locally. VRAM is the gatekeeper. Every single parameter of a model has to live in your GPU’s memory at the same time, plus the KV cache that keeps track of context. A 70B model at normal precision wants roughly 140GB of that memory. A 405B model wants nearly five times that. On consumer hardware, the answer to “can I run this?” has routinely been “no, and neither can almost anyone else.”
So the industry shipped us a comfortable lie: *if you want quality, buy the card.* The 24GB RTX 4090 became the aspirational finish line. The 48GB and 96GB workstation cards became the “grow up” upgrade. Anyone who wanted to run a genuinely huge model on a single machine was told to either spend a fortune or rent cloud time by the minute.
The hidden assumption behind all of it was that a model is a single, monolithic block that must be fully resident to function. That assumption is the entire business. Break it, and the whole pricing ladder underneath it starts to slide.
AirLLM breaks it. And it does so without asking you to sacrifice a single drop of quality.
The Trick Nobody Wants to Admit Is That Simple
The core idea of AirLLM is so embarrassingly basic that most people scroll past it, which is precisely why it works. Instead of loading the entire model into GPU memory all at once, it loads one transformer layer at a time, computes that layer, shoves it back out, and streams in the next one.
Think of it like reading a book you’re not allowed to keep. You don’t swallow the whole page. You read one line, process it, move on. The book never fully exists in your working memory, but the meaning comes out the same.
The reason this matters is that a model’s memory requirement is dictated by its *total* parameter count, while a single layer’s requirement is dictated by its *per-layer* size. A 70B model has 70 billion parameters spread across, say, 80 layers. That means each individual layer is only about 875M parameters — tiny. If you only ever keep one layer in VRAM at once, your peak memory demand collapses from “140GB” to “what one layer takes,” which a modest card handles without breaking a sweat.
This is why AirLLM can make claims that would get a startup laughed out of a funding pitch:
- A 70B model on a 4GB GPU — full precision, no quantization.
- Llama 3.1 405B on 8GB — the 405-billion-parameter giant, on a card most people use for spreadsheets.
- DeepSeek-V3 (671B) on roughly 12GB — a nearly 700B-parameter model, on a genuinely consumer-priced card.
- Qwen3-235B on about 3GB — a quarter-trillion parameters, compressed down to a coffee-mug footprint in VRAM terms.
- Kimi K3 (2.8 trillion parameters) on 3.72GB — yes, that’s a 2.8 *trillion* parameter model, running on less memory than a mid-range card from three years ago.
None of that requires quantization, distillation, or pruning. The weights stay in full fidelity. That last part is the one that should make hardware vendors nervous.
Why “No Quantization” Is the Real Story Here
Let me explain why this detail matters more than the VRAM numbers, because most people get distracted by the gimmicky “70B on 4GB” headline and miss the actual revolution.
The usual way to shrink a big model for small hardware is to quantize it — round those beautiful floating-point numbers down to coarser, smaller representations. Q4 means 4-bit weights. Q2 means 2-bit. The lower you go, the smaller the footprint, but the more the model’s brain gets sanded down like a rock in a tumbler. You can still ask it questions. You just get slightly dumber answers, and the degradation accelerates the further you push.
AirLLM sidesteps the entire trade. Because it’s solving a *memory* problem with a *scheduling* trick rather than a *compression* trick, the weights never have to be mangled. You get the model exactly as the authors intended it. Full-precision reasoning, full-precision output.
That distinction is the difference between “running the model” and “running a ghost of the model that happens to look like it from across the room.” For a lot of work — careful coding, precise reasoning, nuanced writing — that gap is the entire product. AirLLM hands you the real thing on hardware that was never supposed to host it.
Now, there’s a nuance worth being honest about: AirLLM *does* offer optional block-wise 4-bit or 8-bit quantization, but it’s a lever you pull only when you need speed, not a requirement you’re forced into. With bitsandbytes backing it, that optional quant can roughly triple inference speed with minimal accuracy loss. So you get a choice: run in pristine full precision and be patient, or shave a little quality for a little more throughput. The freedom to choose is itself the win.
The Honest Part: It’s Slow, And That’s the Whole Game
Here’s where the story stops being a hype piece and starts being useful. AirLLM trades throughput for the ability to run something that otherwise simply wouldn’t run. Every time you stream a layer in and out, you’re paying a tax on the PCIe bus and memory bandwidth. The result is that AirLLM is *slow*. We’re “1 to 3 tokens per second” slow for the really big models.
And it gets cheeky. The community review of AirLLM’s flagship demo put it best: the 2.8-trillion-parameter Kimi K3 runs at around 292 seconds per token on a 48GB card. Two hundred and ninety-two seconds for a single token. That’s not a language model. That’s a very expensive typewriter that takes longer to write a word than a human does to read a sentence. It’s comically slow, and the people using it know it.
So let’s be brutally clear about what AirLLM is and isn’t.
It is good for:
- One-off inference on a model you own but can’t otherwise afford the hardware for — running a 70B model to answer a few carefully crafted questions without renting a cloud GPU.
- Development and debugging, where you need to *see* a giant model behave, not serve it to a thousand users.
- Experimental workhorses and research play, especially with the newest models that drop faster than your hardware can keep up with.
- Edge and constrained deployments where “it runs” beats “it’s fast.”
It is not good for:
- Real-time chat. If you want snappy back-and-forth, you want a smaller model on a proper card, or a cloud API.
- Serving many users. AirLLM is a solo-instrument hobby, not an orchestra.
- Anything where latency actually matters to the outcome.
The community sentiment tracks this reality. Of the discussion around AirLLM’s big demo, roughly 20% positive, 30% mixed, and 50% skeptical. The skeptics aren’t wrong — they’re just measuring AirLLM against a standard it never promised to meet. Judge a submarine by how well it flies and you’ll conclude it’s useless. Judge it by how well it dives, and it’s a masterpiece.
The MoE Loophole That Makes the Crazy Numbers Possible
There’s a second trick inside AirLLM that explains the most jaw-dropping claims, and understanding it will change how you think about model sizing forever.
Modern giant models are mostly Mixture-of-Experts (MoE) architectures. A 70B “dense” model uses all 70 billion parameters on every single token. A MoE model of similar total size, though, activates only a small slice per token — maybe 3B or so — and routes each token to just a handful of “expert” sub-networks. Total parameters tell you how much memory you’ll *need*. Active parameters tell you how much compute you’ll *use*.
AirLLM leans into this. For MoE models it doesn’t even stream whole layers — it streams only the experts a token actually routes to. That’s how the 2.8-trillion-parameter Kimi K3 fits into 3.72GB: the library never loads the whole MoE monolith, only the tiny sliver of experts relevant to the current token.
This is also why the numbers scale the way they do. A sparse MoE model needs dramatically less memory than a dense model of similar scale, precisely because its per-layer, per-expert footprint is so small. It’s the reason a 671B DeepSeek can live on 12GB while a dense 70B already asks for a fairly serious card. Total parameters are a marketing number. Active parameters and per-layer size are the physics.
What This Actually Means For You (And For the People Selling You Hardware)
Strip away the specs and there are three real-world implications here, and they’re uncomfortable for certain businesses.
First, the ceiling on “what runs locally” just got knocked out of the park. The question used to be “how much VRAM can I afford?” Now the question is “how big a model can I tolerate being slow on?” For anyone who’s ever drooled over a 70B or 405B model and then checked their bank balance and closed the tab, AirLLM quietly unlocks that dream at zero dollar cost. The hardware gatekeeper has been disarmed.
Second, quality is no longer for sale as a function of your GPU. For years, “you get what you pay for in hardware” meant “if you want the un-compromised model, buy the expensive card.” That equation is broken. You can now run the real, full-precision model on a card that costs less than a decent gaming chair. The people who made their money selling you the compromise are going to notice.
Third, and this is the quiet one: the fastest path to running something isn’t always to make the model fit your hardware. It’s to change how the model executes. AirLLM didn’t wait for GPU makers to ship more VRAM at a price ordinary people could pay. It changed the execution. That reframing — fit the workload to the machine instead of forcing the machine to the workload — is a toolbelt move you’ll want to keep in your pocket for every constrained-hardware problem ahead.
For the hardware vendors, the message is less comforting. If a free library can run a 70B model on a $300 card in full quality, the premium on “AI-ready” RAM shrinks fast. It doesn’t kill the market — people still want speed, and AirLLM is no substitute for a real inference card when latency matters. But it absolutely caps the ceiling of what a scarcity-of-VRAM story can extract from you.
The Bottom Line
The rule that said “bigger model equals bigger GPU” was never a law of physics. It was a convention, enforced by a particular way of loading models, and AirLLM just proved the convention was optional. A 70B model on a 4GB card, a 671B model on 12GB, a 2.8-trillion-parameter behemoth on under 4GB — all in full precision, all free, all without cutting a single quality corner.
The catch is real: it’s slow, and it’s a solo act, not a serving platform. But “slow but possible” will knock a lot of people who’ve been told “impossible” off their stools.
So here’s the question I’d leave you with: if running the best open model in full quality no longer requires a second mortgage on your graphics card, what are you still paying extra for?