16 models tested on 7 search tools: Exa averaged 10.5 out of 14, SearXNG about 3

A Habr author ran 16 models through 7 search tools, all with the same question about Bitrix24. The agent had to find the legal entity and its registration details, revenue for the last known year, the CEO and CMO, products and pricing, 2026 updates, two past and two upcoming webinars, and media mentions over the previous six months. Each of the seven items was scored 0, 1 or 2 points, for a maximum of 14. The author compiled the reference answer by hand. Besides completeness, he counted E, the number of items with a verifiable error, and "confident fabrications": wrong facts the model marked as "confident." The ranking metric is the net score, meaning completeness minus E.

The question contained traps. OOO "BITRIX24" in Kaliningrad is a shell holding company with no employees, not the company in question. Revenue data has been published only for 2023. The chief marketing officer is known mostly from comments in trade media.

What the models know without search

First, eight models answered without internet access. Gemini 3.8 Flash got 5 correct points out of 14; the rest got 1–3. Sonnet 5.5, Qwen 3.8 Max and GLM-5.3 made up an INN (taxpayer ID). Nemotron Super named an INN that belongs to Sberbank. Mistral Large 3 got confused about the legal entity, while MiMo v2.6 Pro honestly wrote "I don't know" on revenue.

gpt-oss:120b behaved worst: six confident fabrications, including "2026 releases," four nonexistent webinars and RIA Novosti quotes dated 22.10.2026. Six models named the CEO; none named the current CMO.

Seven search tools: what each one returns

The author described the tools and tested them on two free models with the fewest fabrications, dots-3-note and Nemotron 3 Super, with one run per combination.

• Exa is a semantic search over its own index. It returns full text and highlights, and offers $10 in free credits per month. Average score: 10.5.
• DuckDuckGo (ddgs) is a keyless library that returns a title, URL and a 200–300-character snippet. It is fragile because of IP blocking. Average score: 10.0.
• Keenable has its own index with "point-in-time" search. Result: 7–8 points.
• Tavily is an aggregator of up to 20 sources that slices content into chunks of 500–800 characters; 1 000 credits per month. Result: 5–7.
• Firecrawl is a scraper, not a search engine: it renders a page and returns clean markdown. Paired with Parallel, it scored 5–9 points in the matrix.
• SearXNG is a self-hosted metasearch engine covering up to 244 sources. Out of the box it returns only snippets, and the result was about 3. The author allows that it could do better with tuning.
• Brave Search API has an independent index of 30+ billion pages and a /llm/context endpoint. The 2 000 free requests were not enough, and the author had to top up the account with $5, so it is not in the matrix.

One telling case involved dots: the agent cited helpdesk.bitrixrix24.ru, a typo domain that returned HTTP 200. The fact turned out to be wrong, and the domain itself may be a phishing site. Of roughly 66 URLs checked with curl, only 2 were dead. The main error was not an invented link but a misreading of a live page. The author advises adding domain validation to the agent harness.

Sixteen models: reading pages matters more than "intelligence"

On Exa, eight paid models averaged 11.1 out of 14. The author then ran each combination three times. A single model's spread was 3–6 points: Sol scored from 7 to 13 on the same prompt. A ranking based on one run is partly a lottery. Final average completeness scores:

• MiMo v2.6 Pro — 12.0, 0 fabrications, ≈$0.06 per run;
• Nemotron 3 Ultra — about 12.0, 0 fabrications (2 of 3 runs completed);
• GLM-5.3 — 11.7, 1 fabrication;
• DeepSeek v4.1 Flash — 11.7, 4 fabrications: the high score was bought with guesswork;
• Sonnet 5.5 — 11.3, 0 fabrications, ≈$0.16;
• Gemini 3.8 Flash — 11.3, 2 fabrications, ≈$0.05;
• Qwen 3.8 Max — 11.0, 0 fabrications, ≈$0.14;
• dots-3-note — 11.0 and 5 fabrications;
• Sol — 10.7, 0 fabrications, ≈$0.15;
• Qwen 3.8 27B — 10.7, 2 fabrications;
• Luna — 10.3, 1 fabrication, ≈$0.01;
• Gemma 4 31B — 8.7, 3 fabrications;
• Nemotron Super — 8.3, 3 fabrications;
• Inkling — 8.3, 2 fabrications;
• Mistral Large 3 — 7.3, 3 fabrications;
• gpt-oss 120B — 6.0, 6 fabrications.

An analysis of tool_calls.json explained the pattern. The agent uses web_search (to find) and web_extract (to read a page). On Brave, Qwen Max made about 23 searches and 6 extractions in 5–6 minutes. Inkling and Mistral made 7 searches each and no extractions, scoring 3.7 and 4.0. Sonnet and Gemini made 7–11 searches and 0–1 extractions in 40–50 seconds: fast, but without enough budget left for webinars and media. Completeness grows with the number of web_extract calls, not with model power. Prompt hints are unstable: for Luna, a tip about webinars added one meetup, but completeness fell from 10.3 to 7.7.

Source snapshots and the Brave failure

When the code itself downloaded sources into snapshot.json and the model merely extracted the fields, the ranking got shuffled. Gemini 3.8 Flash scored 12 points with no errors on the DDG snapshot, but 9.3 on live Brave. Mistral and Gemma rose to 10.7 on the snapshot, while Nemotron 3 Super, by contrast, fell from 9.0 on live Brave to 3.3–3.7. DeepSeek: 11.7 on Exa, 6.7 on the DDG snapshot, 10.7 on the Brave snapshot. Average completeness by mode: Exa — 10.1, Brave — 8.9, DDG snapshot — 9.2, Brave snapshot — 8.6. The Brave snapshot barely covered media: in 34 of 45 runs that item was left blank.

Typical errors included revenue of "4.6 billion for 2024" instead of 4.57 billion for 2023, Bitrix24 confused with the "1C-Bitrix" CMS, and the Kaliningrad OOO trap, which caught Inkling, Mistral, Gemma, Nemotron Super and gpt-oss. In a separate incident, after the limit was exhausted, Brave returned HTTP 402 while the detector caught only 429, so 9 of 42 runs went through without any search.

The limitations are obvious: a single test case, a small number of runs, and manual scoring done with the help of Grok Bot. The practical takeaway for anyone building agentic search: don't trust a single run, count confident fabrications separately, and leave the agent a budget for reading pages.