Kolab: 115 Hours of Recordings Show That AI Agents Don't Understand Their Human Colleagues

Researchers at Minnesota NLP have published Kolab, a dataset and benchmark for evaluating how AI agents work alongside people rather than in their place. It is built on 115+ hours of screen recordings: 50 professionals from 40 occupations worked with agents on realistic, long-horizon tasks. The main finding is an uncomfortable one for the industry. Frontier models, both agentic and non-agentic, overestimate the quality of collaboration and struggle to tell good collaboration from bad.

What Happens to an Agent's Output After the Chat

Most popular agent benchmarks measure automation: did the agent complete the task in a single pass? The authors of Kolab, including Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruiqi Wang, Zhouyi Li, Han Chi Le, Yong-Jun Lee, and Dongyeop Kang, start from a different observation. On long tasks, people value ongoing interaction with the agent, such as revising the plan, more than the ability to finish the job on the first attempt.

Professional workflows suit this kind of test well. Their steps depend on one another: a decision at stage two changes everything that follows at stage five. Kolab records how experts talk to the agent, what they take from its answers, and what they edit. Participants also rated the quality of the collaboration and of the final result themselves. The result is something rare: data on what people do with an agent's contribution once it has left the chat window.

The analysis revealed distinct interaction patterns. Much depends on when and how the specialist brings the agent into the work.

Two Tests That Automation Benchmarks Don't Have

The authors argue that a collaborative agent must be able to do two things, and automation tests check neither. This gives the benchmark two tasks:

• Predicting how the agent's contribution will be used. The model must predict what the expert will do with the agent's suggestion: accept, modify, or reject it. The agent's next steps depend on this.
• Verifying collaboration ratings. Given an unfolding workflow, and without continuous feedback from the expert, the model must determine how satisfied the participant is with the collaboration and how good the result is.

The dataset covers 50 participants, 40 occupations, and more than 115 hours of recordings. It includes the specialists' own ratings, so the ground truth is a human's opinion, not an automatic metric.

Where the Models Go Wrong

The results were consistent. Frontier models in both categories, agentic and non-agentic, systematically overestimate the quality of collaboration with experts. They also struggle to distinguish high-quality interaction from low-quality interaction. In effect, a model tends to assume everything went well, even when the expert was dissatisfied or rewrote half of what was proposed.

The least intuitive result concerns context. A longer workflow history does not consistently improve predictions. The common fix of "give the model more information about how the work is going" does not work here. The problem, it seems, is not a shortage of tokens but an inability to read signals: how a person reacts, what they quietly edit, when they stop trusting the agent.

What Agent Developers Should Do

For teams building agents for professional use, Kolab offers two practical takeaways. First, add contribution-acceptance metrics to task-success metrics. An agent that solves 80% of tasks in a single pass, but whose output experts rewrite, is less useful in practice than the leaderboard suggests. Second, an agent's self-assessment is unreliable. If a model overestimates the quality of joint work, you cannot rely on its own "everything is going well" signals when automatically deciding whether to ask a human for confirmation.

For researchers, the dataset provides material for training models that predict an expert's reaction. The code and data are openly available on the Minnesota NLP project page and in a GitHub repository. The paper is on arXiv under number 2610.21109 and appeared in the daily Hugging Face Papers selection.

One key limitation is worth keeping in mind: 50 participants is a small sample, although the coverage of 40 occupations is broad. But the direction is clear. Until agents can predict what an expert will do with their output, they remain an automation tool, not a partner.