Skip to content
vivek pal

wiki / tiny llms

gui grounding

How small models turn an instruction and a screenshot into a point on screen, and how they are measured.

updated · 4 min read

the task

GUI grounding takes an instruction, such as “the search icon in the toolbar”, plus a screenshot, and returns a location: a point to click or a bounding box. The SeeClick paper (Cheng et al., 2024) called it the key challenge in building visual GUI agents, the ones that work from screenshots instead of HTML or other structured data.

Output formats differ by model. OS-Atlas returns relative coordinates in a 0 to 1000 range, either a center point or a box given by its top-left and bottom-right corners, so you scale them back to the real image size. TinyClick returns the screen coordinates of the element that matches the command.

why it splits from planning

An agent does two different jobs. Planning decides what to do next across many steps and needs reasoning and context. Grounding is one perception step that needs pixel precision. Splitting them lets a large model pick the target in words and a small model resolve the words to pixels.

ScreenSpot-Pro shows the effect. Grounding models scored at most 18.9% on its high-resolution professional screenshots. The authors found that shrinking the search area helps. ScreenSeekeR, which has a strong planner guide a cascaded search before grounding, reached 48.1% with no extra training.

UI-TARS goes the other way: one end-to-end model takes screenshots and emits keyboard and mouse actions, with planning and grounding in the same weights.

how the models are built

modelwhat it is
SeeClick (2024)Screenshot-only agent with GUI grounding pre-training and automated curation of grounding data. Introduced ScreenSpot.
OS-Atlas (2024)Grounding and action models trained on a synthesized cross-platform corpus of over 13 million GUI elements from Windows, Linux, macOS, Android, and the web. OS-Atlas-Base-4B is fine-tuned from InternVL2-4B and OS-Atlas-Base-7B from Qwen2-VL-7B-Instruct.
ShowUI (2024)A 2B vision-language-action model trained on 256K examples. It treats a screenshot as a connected graph of UI regions and drops redundant visual tokens, 33% fewer during training and 1.4x faster. Reports 75.1% zero-shot screenshot grounding.
TinyClick (2024)A 0.27B single-turn agent built on Florence-2-Base. Multi-task vision training and data augmentation with a larger multimodal model. Training took 56 GPU-hours, about 40 USD.
UI-TARS (2025)Native GUI agent. Large-scale screenshot data for perception, a unified action space across platforms, and iterative training on reflective traces collected on hundreds of virtual machines.

Most of the effort in these papers goes into data. SeeClick automates curation, OS-Atlas built a synthesis toolkit, ShowUI curates a small high-quality set and resamples to balance it, and TinyClick augments its training data with a larger model. TinyClick in particular shows that a sub-billion-parameter model can be a useful grounder on a small compute budget.

how they are measured

benchmarkintroduced bywhat it covers
ScreenSpotSeeClick, 2024The first realistic grounding benchmark, across mobile, desktop, and web.
ScreenSpot-v2OS-Atlas, 2024ScreenSpot with annotations fixed. The OS-Atlas authors found about 11.32% of the original annotations were wrong.
ScreenSpot-ProLi et al., 20251,581 instructions, each on its own high-resolution screenshot, from 23 professional applications across five industries and three operating systems.

The score is click accuracy: the share of instructions where the predicted point falls inside the target element’s bounding box. ScreenSpot and ScreenSpot-v2 have low-resolution interfaces and large targets. ScreenSpot-Pro has small targets on big screens, which is where small grounders struggle most.

accessibility tree or pixels

There are two ways to find a button. The structured route reads the accessibility tree or the DOM, picks the element, and clicks its frame. When the element is labeled, this is exact and cheap. SeeClick’s authors point out its limits: structured data can be very long (HTML), and it is sometimes unavailable (desktop apps). Canvas-based apps, custom-drawn controls, and icons without labels never show up in it.

The pixel route works on anything that renders, at the cost of a vision model call per lookup and some error on small targets.

These approaches combine well. Use the tree when it names the target, and fall back to a grounding model when it doesn’t.

open questions

  • How much accuracy does a grounder lose when a large screenshot is downscaled to fit its input?
  • Can a planner that crops the screen first, as ScreenSeekeR does, make a sub-billion-parameter grounder good enough for professional apps?

sources

← all notes