Hybrid LLMs: Cloud for Code, Local for Home

The rough draft of this post was written by a 27 billion parameter model. It runs on a small box in my house. It used a coding agent to read my notes, outline the post, and produce a first pass. I did the editing. I’m telling you this up front because it’s the point.

I’m a heavy Claude Code and Antigravity user. I still am. My work harness runs on frontier cloud models, and for real software I don’t want to compromise. But recently I started running a second stack at home. A hybrid approach: cloud where quality is the bottleneck, local where privacy, cost, and experimentation matter.

The debate is the wrong question

You can feel the local versus cloud debate in every AI feed. One side wants all inference on your own hardware. The other side says frontier models are so far ahead that local is a hobby. The reality is more nuanced. I run both, and I’m not running them at the same job.

Cloud models do my work. When I’m shipping software, the cost of a wrong edit is real. I pay for the model with the giant context window incorporated into a harness that can manage hundreds of tools.

Local models do everything else. They answer my daughter’s questions. They remember what we talked about last week. They cost zero dollars per million tokens after the box is paid for and the electricity bill auto-deducts. And they keep working when the internet doesn’t, which matters more than you’d think.

I wrote about how frontier models were always going to be a commodity and how cheap tokens change your ergonomics. The hybrid is the practical conclusion of both: the models are a commodity, so buy where it’s cheap and run where it matters. If you’ve built workbenches for local inference before, my GKE post is the other half of this story.

The box: Infplane Hilbert

I wanted room. Not room for one chat model. Room for a stack.

I ended up with an Infplane Hilbert. It’s a home AI computer built around an AMD Ryzen AI Max+ 395 with 128GB of RAM (up to 96GB can be exposed as VRAM). The product page claims it can run a 235B MoE model locally, or eight 32B models at once. It has server-class power and an OS developed by Infplane that dynamically reallocates CPU, GPU, and memory in real time.

I’ll be honest. The marketing is a little loud. But the core claim is the one I was after. This is a box that holds a small model fleet, not a single model. And it sits in the house, on, quiet, and private. The liquid cooling and low power consumption were the primary reasons I chose this over an Nvidia device.

The roster

Five models. Each one has a job.

ID	Name	Role
qwen3.6-35b-a3b	Qwen3.6 35B-A3B (Q8_0)	chat
qwen3.8-27b	Qwen3.8 27B (Q4_K_M)	hard reasoning
gemma-4-e4b (alias gemma-4-e2b)	Gemma 4 E4B (QAT q4_0)	task
qwen3-vl-embedding	Qwen3-VL Embedding 2B (Q8_0)	embeddings
qwen3-reranker	Qwen3 Reranker 0.6B (Q8_0)	reranker

A few notes on how this breaks down:

• Chat is the daily driver. The 35B is an MoE with about 3B active parameters, so it’s fast enough to feel snappy for family conversation.
• Hard reasoning is the “think harder” model. Math, planning, and the gnarly questions go here.
• Task is the small, fast one. Summaries, classification, quick jobs.
• Embeddings is the memory. It turns what we say into vectors. The “VL” means it handles images too, which is nice for a family that photographs everything.
• Reranker is the quality layer on top of memory. When the agent looks for a past conversation, the embedding model finds the candidates and the reranker sorts them by relevance.

This is the part that surprises people. It’s not one brain with a big file system. It’s an orchestration problem. A small crew of specialists, each doing what it’s good at. The embedding and reranker pair is what makes the family agent feel like it remembers you. That’s most of the RAG, and none of it leaves the house.

Sadie

The models feed an OWUI (Open WebUI) based, ever-growing family agent. We call her Sadie.

The family uses her. My daughter asks questions. My wife asks questions. I ask questions I don’t want in a cloud vendor’s logs. I want to be hyper-personal. Sadie is the front door for all of it.

There’s a privacy reason to do this, and it’s the one I care about most. A family generates a lot of text, and a lot of it is personal. Family finances and medical records shouldn’t be used for training data.

There’s also a lab reason. Sadie is my test bed for agent workflows. When I want to try a new orchestration idea, Sadie gets it first. The family is my test group. They’re a patient one. Mostly.

The experiment: local coding

This is where it gets fun. I’m starting to investigate local coding, and this post is the experiment.

The setup: qwen3.8-27b as the brain, pi as the harness. pi is a coding agent. It gives the model the same tools I use every day: read files, run bash, edit code. The model does the thinking, the harness does the plumbing. I gave it the brief, it produced the rough draft of this post, and I rewrote the parts that needed it.

Here’s the honest report:

• What worked: structure and coverage. It read the material, laid out the sections, and got to a first draft in one pass. For a writing task, the friction was low. But the quality gaps were also easy to find.
• What didn’t work: really? not much. Some of the skills from Claude or Antigravity didn’t work. I also did 0 prep. The model went down a bit more of a security rabbit hole than I would have. You can see it above, even after I toned it down a bit. On this system I have room for a 128,000 token context window. This task fit easily inside that window.
• The verdict: a 27B local model won’t replace a frontier model for serious coding yet. I wouldn’t trust it with a migration or a security-sensitive change. But for a blog draft, a personal script, a first pass on a small task, it’s good enough. And it’s free once the box is paid for.

The part I want to emphasize: the harness matters as much as the model. pi gives the local model the same tools the cloud model gets (mostly). The gap between the two is the model’s capability, not the tooling. That gap is closing fast, and it’s the reason I’m experimenting now instead of in a year.

So what

You don’t have to pick a side. That’s the whole point.

• Cloud frontier for work. When quality is the bottleneck and the cost of a mistake is real, pay for the best model.
• Local for home and the lab. When privacy, offline, cost, and experimentation matter, run your own.
• Start with a stack, not a model. One chat model, an embedder, a reranker. That’s a family agent. Add the specialists when you feel the gap.

The real story is the hardware. A few years ago, a home box ran one model and you tuned around it. Now a 96GB machine runs a fleet. It’s not a single brain. It’s a small data center that fits on a shelf.

I’m not leaving the cloud. My serious work still runs on the best models money can buy. I’m just keeping a second brain in the house. And this post, rough draft and all, is proof that it’s starting to write its own words.