DOM manipulation and element targeting

How we find and use page elements: selectors, forms, inputs, interactive widgets, visibility, polling, and timeouts on slow pages.

The DOM manipulation and element targeting capabilities are implemented across:

• Background service worker: orchestrates tab-level actions, executes injected scripts, and manages timeouts.
• Content script: performs lightweight actions directly on the active page and recognizes common interactive patterns.
• Utilities: capture DOM context, build interactive element lists, and route actions to the active tab.
• Python service: formats DOM context for LLM-driven action planning.

graph TB
subgraph "Extension"
BG["Background Script<br/>background.ts"]
CS["Content Script<br/>content.ts"]
UA["Utils<br/>executeAgent.ts / executeActions.ts"]
end
subgraph "Python Backend"
Svc["Agent Service<br/>browser_use_service.py"]
end
User["User Action"] --> BG
BG --> UA
UA --> BG
BG --> CS
BG --> Svc
Svc --> BG

• Selector-based element targeting: CSS selectors are used to locate elements for click, type, and visibility checks.
• Form and input detection: heuristics identify inputs, textareas, selects, and contenteditable regions; appropriate events are dispatched to simulate user input.
• Interactive component recognition: a curated set of selectors targets buttons, links, and role-based controls.
• Visibility and state detection: computed styles and offset checks determine visibility; wait conditions support dynamic content.
• Dynamic content handling: polling loops with timeouts ensure reliability on slow-loading pages.
• Accessibility-aware attributes: aria-label and other attributes are captured to improve targeting accuracy.

The system uses a message-driven architecture:

• Background script receives high-level commands and injects page scripts to perform DOM operations.
• Content script handles simple actions directly on the page.
• Utilities capture DOM context and route actions to the active tab.
• Python service formats DOM context for LLM-driven action planning.

sequenceDiagram
participant User as "User"
participant BG as "Background Script"
participant CS as "Content Script"
participant Page as "Web Page"
User->>BG : "Click/Type/Wait/Scroll"
BG->>BG : "Validate action and parameters"
BG->>Page : "Inject script to execute DOM operation"
Page-->>BG : "Result or error"
BG-->>User : "Success or failure"
Note over BG,Page : "Visibility checks and timeouts are performed inside the page context"

Selector-Based element targeting#

• CSS selectors are passed to the page context for querying elements.
• Actions include clicking elements identified by a selector and typing into input/select/textarea/contenteditable elements.
• Visibility filtering supports returning only visible elements when listing candidates.

flowchart TD
Start(["Action Request"]) --> BuildSel["Build CSS Selector"]
BuildSel --> Inject["Inject Script into Page"]
Inject --> Query["document.querySelector(selector)"]
Query --> Found{"Element Found?"}
Found -- "No" --> Err["Throw Error: Element not found"]
Found -- "Yes" --> Operate["Perform Operation (click/type/scroll)"]
Operate --> Dispatch["Dispatch Events (input/change/key*)"]
Dispatch --> Done(["Return Success"])
Err --> Done

Form element detection and input identification#

• Heuristics identify interactive elements: anchors, buttons, inputs, selects, textareas, and role-based buttons.
• For contenteditable areas, innerText/textContent is set and input/change events are dispatched.
• For standard inputs/textareas, value is set and input/change/keyboard events are dispatched.
• A convenience action fills multiple fields and optionally submits the form.

flowchart TD
Detect["Detect Interactive Elements"] --> FilterVis["Filter Visible Elements"]
FilterVis --> Attrs["Extract Attributes (id/class/type/name/aria)"]
Attrs --> ListOut["Return Interactive List"]
ListOut --> Fill["Fill Form Fields"]
Fill --> Submit{"Submit Selector Provided?"}
Submit -- "Yes" --> ClickSubmit["Click Submit Button"]
Submit -- "No" --> Done(["Done"])
ClickSubmit --> Done

Interactive component recognition#

• Buttons and links are targeted broadly using selectors for anchor tags, buttons, and elements with role=“button”.
• Content script includes a simple parser to match user intent to visible buttons by text matching.

sequenceDiagram
participant User as "User"
participant CS as "Content Script"
participant Page as "Web Page"
User->>CS : "Click [verb] button"
CS->>Page : "Query buttons/links"
Page-->>CS : "Candidate elements"
CS->>CS : "Match by text"
CS->>Page : "Click matched element"
Page-->>CS : "Click result"
CS-->>User : "Success"

Element state detection and visibility checks#

• Visibility is determined by computed styles (display, visibility) and offsetParent presence.
• A wait loop polls until an element exists or meets a visibility condition within a timeout.

flowchart TD
Enter(["Wait for Element"]) --> Loop["Poll Every 100ms"]
Loop --> Query["Query Element"]
Query --> Exists{"Condition Met?"}
Exists -- "Yes" --> Success(["Return Success"])
Exists -- "No" --> Timeout{"Timeout Reached?"}
Timeout -- "No" --> Loop
Timeout -- "Yes" --> Fail(["Return Timeout Error"])

Dynamic content handling and timeout strategies#

• Wait actions use a configurable timeout and polling interval to accommodate slow-rendering pages.
• Navigation and tab operations include completion waits and fallback timeouts.

sequenceDiagram
participant BG as "Background Script"
participant Page as "Web Page"
BG->>Page : "Wait for selector (condition, timeout)"
loop Poll Until Condition or Timeout
Page-->>BG : "Exists/Visible/Hidden?"
end
alt Condition Met
BG-->>BG : "Return success"
else Timeout
BG-->>BG : "Return error"
end

Accessibility considerations and attribute capture#

• Captured attributes include id, class, type, placeholder, name, aria-label, and visible text to improve reliability and accessibility.
• The interactive list is limited to avoid excessive payloads.

flowchart TD
Scan["Scan DOM for Interactive Elements"] --> Extract["Extract Attributes"]
Extract --> Limit["Limit Count"]
Limit --> Payload["Return DOM Context"]

• Background script depends on page injection to perform DOM operations safely.
• Content script provides lightweight actions and pattern-based button matching.
• Utilities coordinate DOM capture and action routing to the active tab.
• Python service consumes DOM context to generate structured action plans.

graph LR
BG["background.ts"] --> CS["content.ts"]
BG --> UA1["executeAgent.ts"]
BG --> UA2["executeActions.ts"]
UA1 --> BG
UA2 --> BG
BG --> PY["browser_use_service.py"]
PY --> BG

• Prefer specific CSS selectors to reduce query overhead.
• Limit the number of returned interactive elements to avoid large payloads.
• Use visibility filters when listing candidates to minimize false positives.
• Batch operations and introduce small delays between actions to prevent overwhelming the page.

Common issues and resolutions:

• Element not found
• Verify the selector specificity and scope.
• Ensure the element exists before targeting; use wait conditions for dynamic content.
• Confirm the page is fully loaded or use navigation waits.
• Visibility errors
• Use visibility checks or filter visible elements when listing candidates.
• Adjust selectors to target visible subtrees.
• Dynamic content timing
• Increase timeout values for wait actions.
• Use polling intervals suited to the page’s rendering speed.
• Contenteditable vs. inputs
• For contenteditable, set innerText/textContent and dispatch input/change events.
• For inputs/textareas, set value and dispatch input/change/keyboard events.
• Accessibility attributes
• Include aria-label/id/class/name in selectors to improve reliability.
• Mixed environments
• For SPA or framework-heavy pages, ensure selectors target stable attributes and avoid brittle text-based targeting.

Prefer stable selectors, capture useful attributes (id, name, aria-label, visible text), and poll with timeouts instead of single-shot queries on dynamic pages.