New Computer Raised Agent Memory Search Accuracy by 50% with LangSmith

Startup New Computer achieved a 50% increase in recall accuracy of its AI agent's memory system by using the LangSmith observability tool. The success is based not on new architecture, but on disciplined prompt testing methodology.

Why AI Agents Need Long-Term Memory

New Computer's product is a personal AI assistant that truly remembers the user: their preferences, past conversations, habits, and life context. This approach fundamentally distinguishes the assistant from a regular chatbot, which starts with a clean slate each time.

Implementing long-term memory is technically non-trivial. You cannot simply save everything said and send it to the model context each time: the data volume grows, and the cost of requests makes this approach unviable. Instead, memory systems are built on a search principle: from the accumulated database, only relevant fragments are extracted. This is where the recall metric emerges (search completeness): how often the system finds exactly the memories needed right now. Low recall means the agent "forgets" important details even when they exist in the database. For a personal assistant, this is a critical problem.

What LangSmith Provided

LangSmith is an observability tool from LangChain: it logs each step of an LLM application's work, allows you to create test datasets, and compare system versions in a visual interface. The New Computer team built an iterative process around several functions:

• Comparison View — visual matching of two runs: see which scenarios improved and which regressed
• Regression tracking — automatic detection of cases where the new system version performs worse than the previous one
• Fast iteration cycle — changed prompt → ran test → compared results → accepted or reverted
• Conversation logging — complete query history helped identify patterns where memory search failed

The key insight: without structured comparison, regressions remained invisible. When a prompt improved some scenarios, it silently broke others. LangSmith made these breakages visible immediately — before changes reached production.

From Intuition to Measurable Results

Before implementing LangSmith, the team operated on the principle "tried it — seems better." Subjective assessments did not allow confident version comparison: too many variables, too little measurable data. After switching to comparative testing, each prompt change went through a set of test scenarios. Results were recorded, compared to the previous version — and only after confirmed improvement were changes accepted.

Regression became a diagnostic, manageable problem instead of random "something broke." The outcome — a 50% improvement in recall: the agent became 1.5 times more likely to find relevant memories from past conversations. For a personal assistant whose value is measured by memory quality — this is a fundamental product improvement.

What This Means

The New Computer case demonstrates: AI product quality today is determined not only by architecture or choice of base model, but by the observability system. Teams that see their regressions in real time iterate faster — and this directly converts to metrics. Tools like LangSmith move LLM application development from the mode "seems to work" to the mode of measurable, reproducible improvements.