Browser tools

Participants can look at a web page and operate it, in a browser AutoDev keeps for the run. It is off until you turn it on, it uses a profile of its own, and some actions always ask for a gate.

What they are #

Browser tools let a participant see and operate a web page that has no API, no CLI and no MCP server to use instead. There are five, and a participant reaches them through the tools AutoDev serves it:

• browser_screen opens an address, or shows the page the last call left, as a screenshot.
• browser_click clicks, or double-clicks, a point in the last screenshot.
• browser_type types text, into the field at a point or where the focus is.
• browser_key presses one key: Enter, Tab, Escape, Backspace, Space, an arrow, PageUp or PageDown.
• browser_scroll scrolls the page by a number of pixels.

Every call answers with the page as it is afterwards: a 1280 by 800 screenshot, its title and its address. The participant acts by pixel on that image. Only http and https addresses open.

The participant looks and decides with its own model, on its own subscription. The screenshots are sent to that model, like the code it reads.

One browser lasts the whole run and closes when the run ends, so a login, an open tab and a scroll position carry from one call to the next.

Turn them on #

They are off until you turn them on, because a capability that operates interfaces is not a default. Open the Config destination, and in the Computer use section set capability.computerUse.enabled, or run this in the project:

autodev config set capability.computerUse.enabled true

While it is off, a participant that calls a browser tool is told that computer use is off for the project, and to ask you for that step instead. Every config key lists the keys.

The tools need Chrome or Edge on the machine. When neither is found, set AUTODEV_BROWSER to the path of one.

Its own profile #

The browser runs on a profile of AutoDev's own, kept on the machine in the browser-profile/ folder of the project's machine-side state. It is not your everyday browser: nothing in it reads your cookies, your history or your saved passwords, and your browser is not opened or driven. Where state lives says where that folder is. autodev new-run leaves it where it is, so the logins outlive a run.

The configuration file can name a folder of your own in capability.computerUse.browserProfile. autodev config set refuses that key, because a path there is your own browser profile, and it carries every cookie you have into an automated session.

Log in to a site #

A login is yours to make, once per site, and it stays in the profile for every later run. A participant never types a password: it refuses a password field, and a field labelled as a code or a key.

In the panel, open the Config destination and press Log in to a site… in the Computer use section. The panel asks for the address. From a terminal:

autodev browser login <url>

The address must be a full http:// or https:// URL, and it opens in AutoDev's own browser, visible, on that page. Log in there, then close the window. A run started while it is open uses it, and closes it when the run ends.

If a run already has that browser open and visible, the page opens in a new tab of it, and you close that tab after the login, because the next call of the run continues on the newest tab. If the run has it hidden, there is no window to log into: log in after the run ends, or set capability.computerUse.headless to false. autodev browser login has the details.

Hidden by default #

The browser runs without a window. capability.computerUse.headless set to false shows the window while the participant works, in the Computer use section or with autodev config set. Changing it during a run swaps the browser at the next call. Leave the window alone while a call is running.

What a participant can and cannot do #

A participant can look at a page, click, type, press the keys above, and scroll. It works only in this browser: nothing else on the desktop is within reach.

What it does not do on its own is decided from the page, not from what the participant says it is doing. AutoDev reads the role and the label of the element under the pointer, or of the field that has the focus, and sorts the action into a class. Payments, account changes, destructive actions, publishing, sending a message and administrator actions are the blocked classes. A label that looks like one of them is enough: the patterns are broad on purpose, because a blocked click costs a gate, and a wrong one costs something nobody approved. Scrolling is never blocked.

A blocked action comes back to the participant as a blocker that names the control. The way through is a gate that names it, and once the gate is approved the participant calls again with its id. An approval is used once, and it clears only the control it names. A gate follows your gate settings like any other, and by default AutoDev approves the gates it is shown, so set gateMode to gated when you want each of these to wait for you. Your first objective covers gates.

A secret field is never typed into, whatever is approved: a password field, or one labelled as a password, a code, a key or a token. The participant asks you to type it, or you log in first, as above.

Each action a participant takes, and each action refused, is written to the event log. A notice a participant leaves can also hold the browser tools until you answer it, and The shared thread says how a hold is released.