Saiga Llama 3 8B on 10 GB of VRAM: How Habr Reached 93% Accuracy on War and Peace

Habr AI has published a practical walkthrough of running Saiga Llama 3 8B on a home RTX 3080 with 10 GB of video memory to summarize the first two volumes of War and Peace. The experiment showed that the main problem for a local LLM on this task is not just limited memory, but also hallucinations in facts, names and chronology.

Running on 10 GB

The author built a pipeline around IlyaGusev/saiga_llama3_8b with 4-bit quantization and ran the model on a home RTX 3080 with 10 GB of VRAM. The full text of the two volumes did not fit in memory, so the novel had to be split by chapter and the size of each chunk capped. After a series of runs, the workable compromise was about 7,500 characters per chunk: smaller chunks lost too much context, while larger ones raised the risk of crashes and VRAM overflow. The stack used transformers and bitsandbytes, and the author checked the accuracy of the summaries with Gemini.

Unexpected side effects turned up along the way: Qwen2.5-7B-Instruct once produced a long chunk of Python code with library tips instead of a summary. The "sliding window" idea, in which the model retells a summary it has already produced, was also dropped quickly: quality degraded like a game of telephone, and processing time ended up noticeably longer.

Where the hallucinations came from

The naive prompt looked fine at first: the model produced short summaries of 3–5 sentences, but it soon began mixing up surnames, family relationships and chronology. Pierre Bezukhov could suddenly become a son of the Rostovs, and Prince Vasily Kuragin his father. When a character database with strict rules was added to the system prompt, the errors did not disappear but shifted: the network became more confident in stating factually wrong conclusions about individual chapters.

The most telling failure involved Nikolai Rostov. In the episode after the Battle of Schöngrabern, the model decided the hero had died, although in the text he is only wounded and continues in the story later. The author explains this as a skew in probabilities: Tolstoy describes pain, blood and the sense of approaching death at length, while the brief confirmation that Rostov is alive comes later and weighs less for the model. A logit check showed that a prompt can indeed shift the choice of the next token radically.

"Killing characters is forbidden!

Nikolai Rostov survives at Schöngrabern."

What actually helped

In the working version of the pipeline, the rules became extremely direct: check surnames against the character database, do not invent romantic storylines, remember that the action takes place in 1805, and say so honestly if a passage breaks off before the resolution. In parallel, the author tightened the generation parameters: temperature 0.1, top_p 0.85 and repetition_penalty 1.15. The idea was simple: less creativity, less temptation to finish Tolstoy on one's own. And the more stable the output.

• 4-bit quantization instead of full-size loading
• Splitting the text by chapter with a limit of about 7,500 characters
• A strict system prompt with a character database
• Low temperature and a restricted top_p
• Post-processing of rare errors in surnames

This set of measures did not make the system error-free, but it sharply reduced the number of critical hallucinations. The final evaluation with Gemini 3 Flash gave an average factual accuracy of about 93%, and most chapters stayed within a range of 90–98%. The most striking misses remained at the level of tokens and morphemes: in one place "Pierre Bezdarovsky" appeared, a hybrid of Bezukhov's surname and the Russian word for "talentless." The author believes such rare failures are easier to catch with post-processing than by complicating the prompt further.

What this means

This case shows something important for local LLMs: even on a consumer GPU, you can build a useful pipeline for long texts, but success depends on more than the model and the amount of VRAM. What often decides it is strict instructions, control over generation and post-processing, that is, engineering around the LLM, not a single magic "read the book for me" button.