What I Learned Testing AI Models and Harnesses for Months
A few months of trial-and-error across AI platforms, models, and harnesses. The model is not enough — harness quality, open options, and cost efficiency decide what actually ships.
For the past few months I regularly tried different AI platforms, models, and harnesses. I looked at different pieces at different times — used them, compared them, broke things. Not everything is perfect, but a few observations keep showing up.
1. A good model is not enough — the harness matters equally
No matter how strong the model is, if tool calling, routing, context handling, or agent execution is weak, you burn tokens and still miss the expected output. In my experience, model + harness together decide real performance.
2. Betting only on overhyped closed models is risky
I expect companies to lean harder into open-source models for data sovereignty, security, and internal control. Running models trained or adapted on private data inside secure environments will keep growing. RAG is a major part of that shift.
3. Open-source model + good harness = practical value
Open models and their tooling are improving quickly. Getting strong output at lower cost matters more every month. Kimi, DeepSeek, and other open options are useful examples. On many use cases I am seeing roughly 3x to 15x cost differences.
4. Free AI usage is already useful
Not every free model is enough for heavy work, but there are usable options without paying:
- OpenCode
- CommandCode
- Freebuff (ads included, but high free usage)
- Cerebras, Groq
- Devin AI via credits for higher volume
- Paid tools like Codex and Claude Code with BYOK options
5. My current cost-efficient preferences
DeepSeek’s Flash-family options are among the best value I have found when you look at input, output, and cache together. Free tiers of smaller models remain useful for basic-to-medium work.
If you have an international card, Xiaomi and Alibaba platforms are worth trying. Kimi and related coding-oriented options can perform well at low price. You will not get Claude-level output on every task — but if a $5 model can do ~90% of the work for ~10% of the cost, that is a huge practical advantage.
6. Harness is now its own priority for me
- OpenCode — flexible, multi-provider, strong tool integration
- JCode — agent swarm, parallel work, auto-poke
- Prime Agent — long-running work, token efficiency, programmatic tool calling
- OhMyPi — Pi-based and customizable
- CommandCode — solid pricing
Among these, OpenCode, JCode, and Prime Agent felt most promising. Among paid tools, Claude Code still feels the most balanced.
Bigger picture
Another large observation: the AI race is not only a model race. Compute, memory, chip supply, export restrictions, and local alternatives are making infrastructure more important fast.
This summary comes from about three months of continuous trial-and-error. Not everyone can follow AI news every day — hopefully a few of these notes save someone else some time and money.