Veai shows how it tests an AI agent inside JetBrains IDEs without depending on the model

Veai has described how it built UI tests for its AI agent inside JetBrains IDEs so that checks do not depend on the whims of the language model. The team separated testing of the interface, the business logic and the LLM itself, then assembled a stable pipeline for PRs and nightly runs.

Why this is hard

The main problem with such products is that the same LLM request can return different wording, response length and even a different output format. If you test an AI plugin like an ordinary chatbot and wait for specific text, UI tests quickly turn into a lottery.

At the same time, a significant part of the product is entirely deterministic: user settings, transitions between chat states, buttons, skill badges, dialog history and the plugin's internal business logic. Veai decided to move exactly this part into a separate automation layer.

The team works from a simple principle: the quality of the model's answer and the functioning of the interface are different tasks, and they should not be mixed in a single test. The project already had checks at other levels, including IDE model tests, scenarios for different LLM providers and a separate agent benchmark. UI automation therefore became the top of the test pyramid rather than an attempt to replace everything else.

This matters for an IDE plugin: the agent responds both in the chat window and through the terminal, so the number of places where a test can accidentally turn flaky is noticeably higher than in an ordinary web interface.

How the tests are set up

To keep goals from getting mixed up, Veai split runs into several modes. Smoke tests run in every pull request and check the basic functioning of the interface without a real LLM call. Full runs start at night, work with a real server and go through the whole chain from the plugin to the model's response. Separately, there is a benchmark: it is not a set of UI tests but an evaluation of user scenarios and agent quality using the LLM-as-a-Judge principle.

As a result, the team catches UI regressions faster during the day and does not lose end-to-end verification of the product at night.

• Smoke in every PR: checking the UI and basic logic without load on the LLM
• Full at night: working with a real server and waiting for a response in the interface
• Benchmark separately: evaluating agent scenarios and result quality
• Parallel runs across different IDEs: broader coverage without more manual routine

In the full run, developers use a very short request: 2+2=? Answer briefly. But the goal of the test is not to see the number 4 specifically. The point is different: after the necessary preconditions, the agent must reach the Ready state, get a response from the server, correctly finish streaming the tokens and show the result in the interface. This scenario checks not the model's creativity but that the combination of the IDE plugin, the license server, the LLM server and internal libraries has not fallen apart after the latest change.

"We are not trying to get the answer 4."

What helped in CI

On the technical side, Veai relies on the JetBrains Starter and Driver libraries. Starter prepares the IDE, configures the test project and collects run artifacts, while Driver works with the real interface and lets the team describe elements through Page Object and a DSL approach. When locators are not enough, the team adds accessibleName to the product code or adjusts obfuscation rules so that elements remain discoverable.

Part of the state is prepared in advance through the plugin's XML settings, so tests do not have to go through the Welcome screen and onboarding every time.

Another important element is access to the IDE's internal state through JMX. This makes it possible not only to click through the interface but also to check what was actually written to the agent's chat, which provider is selected and what the JSON state looks like from the plugin's point of view.

For CI, the team maintains a matrix of different IntelliJ Platform versions, several JetBrains IDEs, feature flags and obfuscated builds. Heavy checks run at night, Xvfb is used on Linux, and limited retries are enabled to reduce noise: no more than one restart of a test and no more than three failures per entire run.

The practical effect is already visible: several dozen UI tests cover plugin settings, transitions between chat states, skill calls, dialog history and the sign-in of a new user. In the first months, these checks found quite practical bugs: after drag and drop of files was added, copy and paste broke; the context progress bar did not account for all LLM providers; agent selection in the UI also did not work for all configurations; and the error-recovery scenario started to visibly flicker along with the tests. Nightly runs even highlighted instability in one of the test configurations of the LLM server.

What this means

For products with AI features, this is a good reference point: do not make UI tests judge model quality where it is enough to check the stability of the interface and the integration chain. Separating the deterministic and non-deterministic parts of a system makes tests faster, more useful and noticeably more honest, for both developers and QA.