Why AI Assistants Deliberately Hide Systems Thinking from Users

AI assistant developers deliberately limit their systems' thinking: according to researchers' observations, users turn away from AI that speaks too intelligently and complexly, so models by default adjust to the low level of the interlocutor.

Where This Dilemma Came From

A few years ago, the author of material on Habr told about systems thinking at the Analyst Days conference and received feedback asking: "This is interesting, but why so complicated? Why not just take c4-model and build a model on its basis — isn't that much simpler?" Today, a similar question arises regarding AI assistants: do they need to apply a systems approach when a user is engaging them in designing complex projects.

For model developers, the answer was obvious from the start, and it was negative. An AI aware of systems thinking would communicate "too intelligently," and research shows that people don't like that — they turn away from such an interlocutor. Therefore, the model by default orients itself to the statistically low level of the interlocutor, so as not to scare away the mass user.

Why the Phrase "Use a Systems Approach" Doesn't Work

Simply asking AI to "use a systems approach" is not enough, because many different meanings are hidden behind these words. In the simplest case, it is the ability to apply some sequence of reasoning instead of chaotic mental wandering. But only a small part of these meanings relates to real systems thinking: the ability to identify systems in the surrounding world, build correct relationships between them, distinguish emergent properties inherent to the system as a whole from the properties of its individual parts, and not confuse the functional division of the system with the modular.

• Task — teach the model to distinguish between emergent properties of the system and properties of its individual parts
• Assistants by default orient themselves toward the low level of the interlocutor, because that is statistically more comfortable for the majority of users
• A simple command "think systemically" does not reveal the content of systems thinking and can lead the model to error
• The author of the material claims that there is now an accessible and free solution for revealing this content

Revealing this content — that is, explicitly describing what patterns constitute a systems approach — is no easy task. Without explicit description, the model easily confuses different levels of reasoning: applies a sequence of steps where it needs to identify systems and connections between them, or mixes functional division with modular.

What This Means

The story with systems thinking demonstrates a more general problem in AI assistant design: the balance between accessibility for the mass user and expert depth by default is resolved not in favor of depth. For complex tasks — whether software architecture or systems analysis — the user will have to explicitly formulate what exactly systems approach they need from the assistant, rather than relying on the model to understand the request literally.