On November 16, Microsoft Research Asia published a blog post introducing a new component called UI-Evol, designed to address the accuracy and reliability issues faced by computer-use AI agents due to frequent changes in software interfaces.
According to IT Home, which cited the blog post, computer-use agents are an emerging type of AI system capable of autonomously operating various software applications through graphical user interfaces (GUIs), just like humans. They can complete complex tasks such as filling out forms and managing workflows.
Despite their promising potential, these agents often perform poorly in real-world applications. They typically rely on external knowledge obtained from the internet to interpret screen content and perform actions, but usually fail to translate this knowledge into effective behavior—a challenge known as the “knowledge–action gap.”
Microsoft referenced a study highlighting the severity of this issue: even when AI agents receive up to 90% correct instructions, their eventual task success rate is only 41%.
Moreover, the behavior of these AI agents is difficult to predict; they may use different methods each time they perform the same task, displaying high instability. This severely limits their applicability in practical scenarios.
Microsoft showcases a breakthrough in intelligent office automation: solving the ‘knowing but not doing’ problem so AI can handle the work—and your boss praises your efficiency.
Figure 1: The top image shows how correct external knowledge still fails to work in practice. The bottom photo shows how UI-Evol narrows this gap by integrating knowledge with the software environment, enabling more reliable performance.
To tackle this core challenge, Microsoft Research Asia developed UI-Evol, a plug-and-play component that integrates seamlessly into an agent’s workflow. Instead of relying solely on static external knowledge, UI-Evol guides agents directly from the real software interface.
UI-Evol continuously updates and refines its understanding of the interface. By dynamically aligning knowledge with the actual software environment, agents can complete tasks more accurately and reliably, effectively bridging the gap between theoretical knowledge and real-world execution. This research has been accepted for presentation at the ICML 2025 Workshop on Computer-Use AI Agents.
UI-Evol’s mechanism consists of two key stages.
The first stage is “retrace,” where the system precisely records every step the agent takes to complete a task—including all clicks, keystrokes, and other actions—capturing a complete and verifiable action trajectory.
Microsoft showcases a breakthrough in intelligent office automation: solving the ‘knowing but not doing’ problem so AI can handle the work—and your boss praises your efficiency.
Figure 2: UI-Evol’s two stages refine external instructions using the agent’s real behavior, producing guidance that works in practice.
The second stage is “critique,” where the system compares this actual action trajectory with the external instructions. Whenever discrepancies are found, UI-Evol adjusts the knowledge base to reflect the steps that truly work within the software. Through this iterative cycle, general-purpose instructions gradually evolve into highly reliable, practice-validated guidance for agents.
The research team evaluated UI-Evol using the OSWorld benchmark on Agent S2, a state-of-the-art computer-use agent. OSWorld is designed to assess multimodal agents on open-ended tasks in genuine software and workflows.
The results show that UI-Evol significantly improves task success rates and solves a long-standing problem—“high behavioral variance,” i.e., inconsistent behavior when agents perform the same task multiple times. With UI-Evol integrated, agents powered by advanced large language models, such as GPT-4, exhibit greater stability and predictability.
The above content is compiled by ModeZone, a fashion and entertainment magazine.
