Skip to content
vivek pal

wiki / self-improving models

learning from computer-use traces

Notes on improving a small computer-use model from its own action logs, kept on the user's machine.

updated · 5 min read

A computer-use agent leaves a record of everything it did. That record becomes training data once you can tell which runs worked. These notes collect the relevant pieces from the literature, plus some ideas for doing this on one person’s machine. The ideas are design sketches, not a description of a shipped system.

what a trace holds

Each step needs enough to replay the decision and judge it later:

{
  "task": "export the Q3 chart as a PNG to Desktop",
  "steps": [
    {
      "t": 3.42,
      "app": "Numbers",
      "observation": { "screenshot": "step-007.png", "ax_tree": "step-007.json" },
      "intent": "open the File menu",
      "action": { "type": "click", "x": 412, "y": 18, "target": "File menu" },
      "result": "menu opened"
    }
  ],
  "end": "success | user_stopped | timeout | error",
  "user_signal": "accepted | undone | took_over | re-asked | none"
}

Episode-level fields matter as much as the steps: the task in the user’s own words, how the run ended, and what the user did afterwards.

labeling success

Labeling success is the hard part, and the label quality sets the ceiling for everything trained on it (see the self-improvement loop). Three sources, from most to least reliable:

  • Execution-based checks. OSWorld (Xie et al., 2024) gives each of its 369 tasks a custom evaluation script that inspects the final state of the machine. This is the most reliable kind of label, but someone has to write a checker per task type.
  • A model evaluator. DigiRL (Bai et al., 2024) trained Android device-control agents with autonomous RL, using a VLM-based evaluator to judge whether each episode succeeded. This scales to open-ended tasks, at the cost of the evaluator’s own errors.
  • User signals. The user accepted the result, undid it, took over, or asked again. These are cheap and noisy, but they are the most direct evidence of what the person wanted.

hindsight relabeling

Hindsight Experience Replay (Andrychowicz et al., 2017) handles sparse, binary rewards by replaying failed episodes as if the goal had been whatever the agent actually reached. The computer-use version: a run asked to export a chart as PNG that ended by saving an SVG is a failure for the original task and a correct example for “export the chart as SVG”. Relabeling turns some failures into positive examples for the instructions they did satisfy.

The catch is that describing the reached state as an instruction is itself a model call, and it can be wrong. A relabeled example should pass the same outcome check as any other example before it enters the training set.

skill libraries before weight updates

Voyager (Wang et al., 2023) is an LLM agent in Minecraft that stores behaviours as executable code in a growing skill library. It retrieves skills for new tasks and refines them using environment feedback, execution errors, and self-verification. The authors report that this compounds the agent’s abilities and alleviates catastrophic forgetting, because skills are stored outside the weights.

The desktop analogue: when a run succeeds, save its action sequence as a parameterized routine keyed by the task description. Next time, retrieve and replay it, and fall back to step-by-step planning only when the replay fails its check. A routine is easy to inspect and delete, and it improves the agent without training anything.

verbal self-reflection

Reflexion (Shinn et al., 2023) also skips weight updates. After a failed trial the agent writes a reflection on the feedback it got and keeps it in an episodic memory buffer for the next attempt. The paper reports 91% pass@1 on HumanEval against 80% for GPT-4 at the time. It works with a frozen model and is cheap per user. The costs are that the memory grows and needs pruning, and that a reflection is only as useful as the feedback behind it.

keeping it on device

Traces contain screenshots of whatever the person was doing: messages, documents, sometimes credentials. By default they should not leave the machine. Some ideas:

  • Store traces locally and encrypted, with a retention window.
  • Redact before anything is kept for training. Drop frames from password fields and password managers, and mask personal data in text.
  • Train small per-user LoRA adapters (Hu et al., 2021) on device instead of changing shared weights. The base model stays the same for everyone, and an adapter can be inspected, switched off, or deleted. Biderman et al. (2024) found LoRA forgets less of the base model than full fine-tuning, which matters when the training set is one person’s narrow habits.
  • Train shared base-model updates only on data people explicitly choose to share.

open questions

  • How many verified traces does a small grounding or planning model need before a per-user adapter beats the base model on that person’s apps?
  • Can undo and takeover signals be trusted as labels on their own, or do they always need a second check?
  • How should a skill library notice that an app update broke a stored routine?
  • If many users’ adapters improve the same skill, is there a private way to merge them, such as federated averaging with differential privacy, and how much quality does it cost?
  • How do you evaluate a per-user model when every user’s eval set is different?

sources

← all notes