Developer builds a RAG system on local models, boosting search accuracy from 20% to 53%

Developer builds a RAG system on local models, boosting search

accuracy from 20% to 53%

A developer going by tzar26 on GitHub has published a breakdown of their news-analysis system — fully self-contained, with zero calls to cloud APIs like OpenAI or Anthropic. All the models run locally through Ollama: Qwen 2.5 14B handles reasoning and classification, while BGE-M3 generates text embeddings. The article states the motivation plainly: resilience against sanctions and independence from external providers. The resulting stack — Python, FastAPI, PostgreSQL with the pgvector extension, Celery for background jobs, LangGraph for the agent, and Langfuse for monitoring — was packaged in Docker containers and covered by tests through GitHub Actions.

Why plain vector search failed on homonyms

The system collected more than 12,000 articles and initially searched them using only cosine distance between BGE-M3 embeddings. Simple queries worked fine, but more complex topics like "the central bank's key rate" returned garbage. The clearest example was the query "gold": instead of news about metal prices, the search dutifully returned articles about a figure skater who won gold at a competition — a classic case of a word having multiple meanings that a vector representation fails to disambiguate by context.

The fix was hybrid search: vector comparison was paired with precise BM25 text search built on PostgreSQL's native tsvector, and the results of both methods were merged using Reciprocal Rank Fusion (RRF), which takes each document's rank across both result sets and computes a combined score. BM25 excels at catching exact terms like "OPEC" or "rate," while vector search captures synonyms and semantic relationships. Together, they raised the share of relevant documents in the top five search results (the Recall@5 metric) from 0.2 to 0.53 — more than double.

Breaking the habit of dumping everything into "other"

A separate issue showed up during automatic topic classification. In the first version of the prompt, the model routed 74% of all articles into the "other" category — meaning nearly three-quarters of the database ended up without a useful label. The cause turned out to be simple: the instructions included the phrase "if in doubt, choose other," and the model, afraid of getting things wrong, leaned on that escape hatch constantly.

The developer rewrote the prompt, adding clear descriptions for each category — including retail, macroeconomics (macro), and incidents (incident) — and removed the "if in doubt" language entirely. The share of unclassified articles then dropped to 0.03%. To confirm the numbers held up, the author also manually reviewed a random sample of labeled articles, finding 85% classification accuracy.

An agent that calculates, not just searches

Plain database search can answer questions like "what's being written about oil," but that's not enough for real analytics. So the author built a LangGraph agent with three tools to choose from:

• `search_articles` — standard hybrid text search
• `get_metrics` — SQL aggregation, for example counting energy-related articles over the last three days
• `search_by_entity` — search by extracted company names with sentiment attached (the output of separate NER and sentiment models)

When asked something like "is there negative news about the central bank," the agent decides on its own which tool to invoke — there's no need to explicitly tell it to compute sentiment rather than just retrieve text. To debug this decision-making, the team deployed a self-hosted instance of Langfuse, which traces every step of the chain — planner → tool call → final answer — along with token usage at each step and the exact arguments the model passed into each function.

Technical headaches of running on local hardware

Running models locally brought its own set of problems, ones that simply don't come up with cloud APIs:

• Ollama was unloading models from GPU memory when switching between them, occasionally stretching a single request out to 60 seconds — fixed by setting `OLLAMA_MAX_LOADED_MODELS=2`
• LangChain's internal streaming-response aggregation was causing a ninefold slowdown — it was ripped out of the pipeline in favor of calling `ollama.AsyncClient` directly
• SQLAlchemy ties its connection pool to the event loop, so when Celery interrupted a task, connections were left stranded as "zombies" — solved by calling `await engine.dispose()` after every task
• Ollama processes requests sequentially by default; classifying a single article was taking around 10 seconds, and raising `OLLAMA_NUM_PARALLEL` from 1 to 2 sped the process up by at least 1.5x

The project is currently a public MVP, available openly on GitHub. The next step, according to the author, is already underway behind closed doors: linking mentioned companies to Moscow Exchange tickers (like SBER and GAZP) and building forecasting dashboards for specific stocks. For engineers weighing a local stack against cloud APIs, the practical takeaway is straightforward: the savings on API calls are real, but you pay for them in time spent tuning infrastructure — from dialing in Ollama's parallelism to wrestling with connection management in async code.