Given the improvements in models as of late, I’ve been toying with running local models to avoid token usage and costs with the big cloud players.
Setup
M1 Pro with 16GB. I’ve got llama.cpp installed and running models, then using OpenCode to interact with the models.
┌─────────────┐ ┌────────────────┐
│ OpenCode │───▶│ llama.cpp │
│ (Harness) │ │ (Server) │
└─────────────┘ └────────────────┘
│
▼
┌──────────────┐
│ GGUF Models │
│ (~4-12GB) │
└──────────────┘I wrote a simple Python script that handles model switching and server management.
The models I’m experimenting with:
- Qwen 2.5 Coder 7B: Code tasks (~4GB)
- Qwen 2.5 7B Instruct: General chat (~4GB)
- GPT-OSS 20B: When I need more power (~12GB)
Each model runs on its own port so I can switch between them in OpenCode.
Results
For basic text interaction and simple code tasks, it’s quick and outputs are good. Complex coding tasks take longer.
llama.cpp shows some interesting stats:
3782.84 ms/559 tokens (6.77 ms per token, 147.77 tokens per second)
26201.59 ms/482 tokens (54.36 ms per token, 18.40 tokens per second)That’s ~18 tokens/second generation. Makes you think about every token.
The quantized models (Q4_K_M) are a decent balance - small enough to run smoothly, quality good enough to be useful.
The other big gap - llama.cpp comes with a UI out of the box, but it has no tools, which makes it pretty useless. So, that’s what was driving the usage of the OpenCode harness. Qwen 2.5B just does not like using the tools correctly. It was very difficult to get it to actually write a markdown file.
Anywho, cool setup, and more experiments to come.