Microsoft logs 1M desktop screenshots to teach AI agents when to probe
Microsoft researchers Michael Solodko and Justin Wagle have detailed ScreenSearch, a distributed exploration system that logged 1,007,406 screenshots across 11 desktop applications. The goal is teaching computer-using agents when to keep probing a screen and when to commit to an action. The paper first appeared in May 2026, and the pair posted a revision on August 17.
The problem is that desktop agents work under partial observability. Screens that look nearly identical can hide different workflow states, which means the same click can produce sharply different outcomes. An agent that trusts its eyes doesn’t always know what it’s looking at.
ScreenSearch reads each screen through UI Automation, the Windows accessibility framework that exposes programmatic information about interface elements. It converts those element trees into structural features, deduplicates near-identical screens, and shares one state graph across isolated virtual machine workers. On top of that graph, a one-step PUCT selector picks a single action at a time, with no simulated rollouts. It balances plausible actions, expansion of the frontier, and movement toward states that respond more consistently.
The core idea is a structural ambiguity signal, and the balance matters. Novelty alone can over-reward superficial visual change, but ambiguity alone can keep a policy circling a small uncertain region. So when similar screens produce different next states under the same action signature, the system treats the screen as unresolved and probes it further instead of moving on. That’s the test deciding when an agent should investigate and when it should act.
| Reported figure | Value |
|---|---|
| Screenshots recorded | 1,007,406 |
| Unique observations after post-processing | 39,902 |
| Merged graph states | 31,146 |
| Recorded VM hours | 5,524 |
| Desktop applications explored | 11 |
| Extracted three-step traces | 55,770 |
The search rule earns its keep in a matched test with 50 replay starts in each of Notepad, File Explorer and LibreOffice Calc. There, PUCT lifted macro-average new-state AUC from 4.820 to 50.487 over greedy selection using the same prior. New-state AUC is a curve-derived score of how quickly a policy accumulates fresh states, so that gap separates wandering from covering ground.
Even so, coverage wasn’t uniform. State discovery rates ranged from 1.03 percent in Notepad to 7.83 percent in LibreOffice Calc, and some policies cut ambiguity quickly while finding little new frontier. The authors are blunt about what that implies.
Ambiguity reduction alone is therefore not a sufficient exploration objective.
Michael Solodko and Justin Wagle, in the ScreenSearch paper
The backdrop is a field that still fails routine computer work. When the OSWorld benchmark launched in 2024, humans completed more than 72 percent of its 369 tasks spanning real web and desktop apps. The best model managed 12.24 percent, struggling most with GUI grounding and operational knowledge. That gap matches the 2026 research on long-horizon agents, which points at context handling rather than raw capability.
The uncertainty theme runs wider than one system. A survey posted on August 17 by Abderrahmene Boudiaf, Irfan Hussain and Sajid Javed argues that a multimodal model’s uncertainty shouldn’t be judged as a confidence number alone. In their framing, calibrated uncertainty should determine what the system does next: answer selectively, abstain, ask for clarification, retrieve more evidence, or escalate. ScreenSearch makes the same argument at the operating system level, with a graph instead of a confidence score.
Where this goes next is data. The corpus was built to feed downstream policy training, and the paper’s appendix reports that stronger proposal priors can materially improve unique-state discovery. Microsoft isn’t the only lab that wants records of real desktop use either; OpenAI’s Computer History logs Mac clicks and keystrokes on the consumer side. The thing to watch is whether exploration corpora at this scale start moving the benchmark scores that agents keep failing.
Get the daily rundown
One email each weekday with the AI news that matters, every claim linked to its primary source.
Free, one email each weekday, unsubscribe in one click. We never sell or share your address.
