A developer tested the accuracy of local LLMs—including Qwen, Gemma QAT, and Apple's AFM 2 3b—using 7,648 multiple-choice questions. These questions were generated by deepseek-v4-flash based on markdown documentation for Node, TypeScript, Vue, Langchain.js, and transformers.js.

The results showed that standalone models performed poorly. However, accuracy improved significantly when the models were integrated with a Retrieval-Augmented Generation (RAG) system. Apple Intelligence (AFM 2 3b) achieved a strong 86% score, despite a limited 4k context window compared to the 32k provided to other models.

Experiments with "thinking" or reasoning modes yielded only a marginal improvement of about 1% while significantly increasing compute time and latency. The developer is applying these findings to "Chatwise," a learning app available on the App Store for iPhone and macOS.