My Local AI Model Stack

i’ve been really getting into local AI — localmaxxing if you will. a few weeks ago i got a DGX Spark, and a few weeks before that i got a new Macbook Pro M5 Pro with 48GB of unified memory. with these, i’ve really been able to test out what’s happening in the world of Local LLMs & inferencing.

because it’s been a few weeks, and because i’ve landed on somewhat of a stable set of tools (even though the space moves really fucking fast), i figured it would be a good time to go through all tools i’m using for local inferencing.

Hardware

i’ll keep this section short, since i’ve already outlined the hardware i’m running on above.

DGX Spark

  • 128GB unified memory
  • Nvidia GB10 GPU
  • ARM CPU
  • 4TB hard drive

i have this connected to the internet and on my personal tailnet. this gives me access from wherever, whenever. even my phone (which i do way more than i thought i would be, surprisingly).

Macbook Pro

  • M5 Pro
  • 48GB unified memory
  • 1TB hard drive

also connected to my tailnet. this is my main computing device. i’m on it all the time.

Software

this is the fun part, and i want to break it up into model serving and interaction.

Model Serving

i’ve tried out a lot of software for model serving. i got my start with ollama, moved to LM Studio, but then landed on two main pieces of software: Llama.cpp & oMLX

Llama.cpp

I decided to land on Llama.cpp because I get to use llama-swap which allows me to keep several models on device and easily swap between them through API requests. Sure, i lose some time to first token, and it maybe isn’t as performant as vLLM or SGLang, but it’s robust and fast enough for my work.

this runs on the DGX Spark.

oMLX

oMLX is a macOS-native MLX server with smart caching. the thing that drew me to oMLX was that it persists every cache block to SSD. so when the agent circles back to a previous prefix, it’s restored from disk in milliseconds, not recomputed from scratch.

so, not only is it doing cool things with the KV cache, it’s also taking advantage of apple’s mlx framework for improved inference speeds.

Model Interaction

similar to model serving, i’ve messed around with and used a lot of apps, tools, harnesses, cli’s, etc. but through all that, i feel like i’ve landed on a really good set of tools.

my coding harness of choice is pi. when i’m on my macbook using pi, i route prompts to a few different places: my DGX Spark, locally on the macbook, openrouter, AND anthropic models via my claude subscription. when i’m ssh’d into my spark and using pi, i’m routing prompts through models running on the spark itself, or through openrouter.

pi is great. it’s minimal, extensible, and can be customized. i’ve been a big fan.

for chatting with the models through a chat app, i’ve been co-developing a macos app with a friend, called Modelo. it runs locally, has a menubar icon & quick chat, as well as a small agent that runs on my spark to report device utilization with nvidia-smi.

i also have a hermes agent running in two places: on my macbook pro and on an old dell optiplex i have on my tailnet as well. both are, 95% of the time, routed through openrouter. the version that runs on my macbook (of which i use the hermes desktop client) i’ll sometimes run local models with.

Models

ok so hardware and software out of the way, which models am i running and what am i using them for? here’s the full list:

DGX Spark

  • Gemma 4 31B QAT + MTP
  • Qwen3.6-35B-A3B
  • North-Mini-Code-1.0
  • Step-3.7-Flash

it’s quite the spread. but they each kind of have their place. the only two that are really competing in my mind are the Gemma 4 and Qwen3.6 models. North Mini is my main agentic coding model. Step-3.7 is the big boy, the one i go to if i need a bit more horsepower.

for coding models, my main plugin that i use is ponytail. for the daily driver models, i have firecrawl. as i use local models more, i’ll definitely be extending the skills and plugins i use for each of them.

Macbook Pro

  • Gemma-4-26B-A4B-it-QAT-MLX-4bit
  • Qwen3.6-27B-MLX-4bit

similarly to the DGX spark, there’s a bit of overlap. but primarily the gemma 4 model is used as my “daily driver” the one i use by default for daily driver type stuff, and qwen3.6-27b is what i use to code locally.

Conclusion

local ai is changing, what seems like, hourly. and, boy, is it exciting. i’m very happy with my current setup, and the good thing is that (at least i think) for awhile i’ll be good on hardware while the models keep getting better and better.

i’ll try to keep this blog updated with my stack information as i change things, but because it’s changing and moving so fast, i’m guessing it’ll be hard.

if you want to chat about this with me, or if you have questions, the best place to reach me is on x/twitter or via email. let’s yap localmaxxing.