Gemma 26B fine-tuned on a 24 GB MacBook: offline assistant gets 703 of 766 right

Six months of work, one MacBook with 24 GB of unified memory, and a hard constraint: no cloud, no subscriptions. The project's author built a voice assistant for macOS that doesn't chat but acts: it opens folders, searches for files, launches movies and reads email. To do this, he fine-tuned Gemma 4 26B A4B (a MoE with ~4 billion active parameters) using QLoRA through mlx-lm. The rules that used to take up about 5 000 tokens of system prompt moved into the weights of a small LoRA adapter. On a frozen test bench of 766 conversations, version v4b with a short prompt solved 703, and the median response time was 1.64 seconds.

The brain picks a tool instead of writing text

The app is built on Electron, with separate processes running around the window. Python handles hearing, with a speech detector and Whisper built on MLX. The brain is llama-server (llama.cpp) running Gemma in 4-bit quantization with a LoRA in GGUF format. Qwen-TTS synthesizes the voice from a reference recording. A Swift helper covers macOS system functions (sound, microphone, echo). A watchdog restarts any process that crashes, and the voice comes back up in about 7 seconds.

For each utterance, the model returns a single step in a fixed JSON format with the fields think, tool, args, say and next. The window runs the tool and, if needed, passes the result back to the model. Any change or deletion of files requires voice confirmation. The same format goes to ChatGPT, Claude and Gemini if the user connects them: tools are passed to them as native function calls. In that case the local llama-server shuts down and frees about 13 GB.

A 766-conversation test bench and a repair pass

Before training the model, the author built a frozen test bench: phrases with reference steps that he never trains on. It covers folder commands, movie search, email, questions about documents, multi-turn chains, refusals and speech-recognition mishearings. Two lessons were expensive. Checking utterance by utterance threw out more than 400 chains and painted too rosy a picture, so you have to compare whole conversations. Examples built from the bench's own errors inflate the score on that same bench, so an honest check needs a fresh set.

How the versions progressed (the bench grew from 533 to 766 conversations, so direct comparison is valid only within the same size):

• No adapter, full prompt: 475 of 533 (89%), median 1.94 s
• v3, full prompt: 487 of 533 (91%), median 1.63 s
• v4 with new sections (documents, drawings, English, chains): 684 of 766
• v4b, short prompt: 703 of 766, median 1.64 s

In v4 the model improved on the new sections but regressed on the old ones. "Play the movie" started launching music, "find the movie" lost its search type, and "my own folder" got confused with the notes folder. A repair pass fixed it: 416 examples covering these cases plus a random sample of old and new data, about 1100 in total. Training continued for another ~1000 steps at a learning rate of 1.5e-5 instead of 3e-5 and without validation. It took about 70 minutes and produced the best result of the whole project. Training from scratch, by contrast, made the model forget old behavior.

What broke on the laptop

The Metal limit dictates example length: no more than ~1400 tokens, or it crashes with Insufficient Memory. Training with the full prompt on a Mac is physically impossible, which is one more argument for moving the rules into the weights. The script discards examples that don't fit in advance. Speed is 0.25–0.4 steps per second, and a pass takes about three hours, so the model trains overnight.

Other pitfalls:

• The Mac went to sleep. After one training run, the bench check took six hours. Now the whole chain (training, conversion to GGUF, bench) runs under a single caffeinate -i -s.
• Memory failures. The browser and Spotlight indexing push swap up to 6–7 GB, and the failures coincided with validation every 200–400 steps. The fix: save checkpoints, resume automatically with --resume-adapter-file, and turn off validation in later stages.
• Editing a bash script on the fly crashed the chain with a syntax error, because bash reads the file as it executes.
• Gemma 4 needed two small patches to mlx-lm, and converting the adapter to GGUF requires a separate environment with torch.

There is plenty of engineering around the model, too. The movie is ducked based on the start of speech, not on the assistant's name: if the voice is 6 dB above the "floating floor" and it isn't an echo, the sound is lowered, and 2.5 seconds later it returns if the name wasn't spoken.

A synthesis defect, where "сэр" ("sir") turns into "сёр" in roughly every third take, is caught acoustically. The vowel is measured with 12th-order LPC analysis, and the second formant is compared (about 2000–2300 Hz for "э", 750–900 Hz for "ё"). The check takes about 3 ms, and the detector catches 39 of 46 failures; the phrase is then re-synthesized with a different take.

Spotlight handles questions about documents: search in 60–700 ms, the two or three best files, windows of ~600 characters, and an answer based only on them. DWG drawings are parsed by LibreDWG in WebAssembly in a separate thread, and the result is converted to SVG and cached.

A recipe for your own project

If you're building a local agent, start with a test bench, not with training. Freeze a set of conversations and compare versions only on it, and for the final evaluation keep a fresh sample that neither the model nor you has seen.

Rules that drag thousands of prompt tokens into every request are worth moving into a LoRA: accuracy and latency improve at the same time. Add new skills on top of the existing adapter using a mix of old and new data, with a lower learning rate. 24 GB is enough for a 26B MoE model if you cut example length, save checkpoints and don't let the machine fall asleep. The author's next step is personal memory and self-training on phrases the assistant failed to understand.