News
Local LLMs need RAG for accurate technical answers
Developer benchmarked local models (Gemma, Qwen, Apple Intelligence) on 7,648 technical QA questions from popular documentation. Standalone models scored poorly; adding RAG with document retrieval dramatically improved accuracy. Thinking/reasoning added minimal gains but heavy compute cost.
A developer tested the accuracy of local LLMs—including Qwen, Gemma QAT, and Apple's AFM 2 3b—using 7,648 multiple-choice questions. These questions were generated by deepseek-v4-flash based on markdown documentation for Node, TypeScript, Vue, Langchain.js, and transformers.js.
The results showed that standalone models performed poorly. However, accuracy improved significantly when the models were integrated with a Retrieval-Augmented Generation (RAG) system. Apple Intelligence (AFM 2 3b) achieved a strong 86% score, despite a limited 4k context window compared to the 32k provided to other models.
Experiments with "thinking" or reasoning modes yielded only a marginal improvement of about 1% while significantly increasing compute time and latency. The developer is applying these findings to "Chatwise," a learning app available on the App Store for iPhone and macOS.
Readers’ Forum
No contributions yet — open the debate.
Open the discussion
No account or password needed — just enter your e-mail and we’ll send you a one-time sign-in link. First time here? You’re set up automatically.
Your rating will be applied automatically after you sign in.
Check your inbox
We’ve sent a sign-in link to …. Open it on this device — this tab will sign you in automatically.
Waiting for your click …
·