Original title: Gemma 4 26B-A4B inference in about 2 GB of RAM
Article
TurboFieldfare is a Swift and Metal runtime for Apple Silicon that runs Gemma 4 26B-A4B from a pinned checkpoint with about 2 GB of RAM in use by keeping shared weights and FP16 KV cache resident while paging only required experts from SSD each token. The project targets macOS 26+ on arm64 Macs with at least 8 GB and ships as a native app, CLI, decode service, loopback OpenAI-compatible server, and installer utility, all built from a single package, with the app handling chat formatting and defaults for sampling. The installer repacks Hugging Face ranges directly into a compact .gturbo layout without materializing full checkpoint files in memory, then validates manifest and hashes before generation; users are instructed to run one model-owning process at a time and verify installs when needed. Architecture details include 4-bit quantized weights, 8-bit routing, 16-slot LFU expert cache, bounded pread reads, chunked prefill, and token-by-token decode, with memory tradeoffs driven by prompt length, cache state, and hardware. Reported throughput spans 5.1–6.3 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on a 24 GB M5 Pro, presented as reference measurements rather than hard limits. Benchmarks and experiments are documented in a 103-result registry, and the project is positioned as text-only inference only, with no native tool execution except client-authorized calls via the server. The author frames it as an independent project, explicitly not affiliated with Google, with model rights handled through external terms and project licensing under Apache 2.0. Community discussion adds scrutiny and suggestions for broader hardware and model support, including extension ideas for other MoE models and longer-running workloads.
Commenters reported successful builds on older macOS versions by conditionalizing language settings, but with reduced prefill speed benefits, indicating compatibility work remains practical but costly. Several users compared the approach to mmap-based llama.cpp usage and asked for quantitative evidence such as per-token I/O, hit rates, and the exact tuning that drives the major speed gains. The 8 GB versus 16/24 GB performance gap drew repeated questions, with explanations generally pointing to memory capacity, cache behavior, and hardware limits rather than simple SSD bandwidth assumptions. Others asked whether the method could scale beyond Gemma and into models like Qwen or Kimi, and whether the same streaming architecture can generalize across model families and bigger parameter counts. Thermal and storage endurance concerns appeared, including 8-hour usage, SSD wear, and sustained throughput on fanless machines. Additional feedback included calls for local-model strategy guidance, interest in future hardware generations, and comparisons with related projects such as DwarfStar4, while some users explicitly praised the expert-caching design, others highlighted that Ollama and similar tools may not implement this style of expert paging. A few commenters validated throughput claims on newer Macs, and several requested Windows or broader platform support, which is currently outside scope.