Browser automation tools

Browser automation tool and AgentService: goal in, DOM-aware JSON plan out, sanitizer, and execution through the extension.

The browser automation system spans backend services, FastAPI routes, LangChain prompts, a browser extension, and React-based UI. The key layers are:

• Backend API and service layer for generating action plans
• Prompt templates and sanitization utilities
• Extension background script, content script, and side panel executor
• Agent orchestration and tool integration

graph TB
subgraph "Extension UI"
UI["Side Panel Executor<br/>AgentExecutor.tsx"]
AgentMap["Agent Map<br/>agent-map.ts"]
end
subgraph "Extension Runtime"
BG["Background Script<br/>background.ts"]
CS["Content Script<br/>content.ts"]
ExecUtil["Action Execution Utils<br/>executeActions.ts"]
ExecAgent["Agent Runner<br/>executeAgent.ts"]
end
subgraph "Backend"
API["FastAPI Router<br/>routers/browser_use.py"]
SVC["AgentService<br/>services/browser_use_service.py"]
PROMPT["Prompt Template<br/>prompts/browser_use.py"]
SAN["Sanitizer<br/>utils/agent_sanitizer.py"]
MODELS["Pydantic Models<br/>models/requests/agent.py<br/>models/response/agent.py"]
end
subgraph "Agent System"
RA["React Agent Graph<br/>agents/react_agent.py"]
RT["Agent Tools<br/>agents/react_tools.py"]
end
UI --> BG
AgentMap --> UI
BG --> API
API --> SVC
SVC --> PROMPT
SVC --> SAN
SVC --> MODELS
BG --> ExecAgent
BG --> ExecUtil
BG --> CS
RA --> RT
RT --> SVC

• AgentService: Generates a JSON action plan from a user goal, optional target URL, DOM structure, and constraints. It formats the DOM info, constructs a user prompt, invokes the LLM, and validates/sanitizes the resulting JSON.
• FastAPI Router: Exposes a POST endpoint to generate scripts, validating inputs and returning structured responses.
• BrowserActionInput Schema: Defines the input contract for the browser action tool, including goal, target_url, dom_structure, and constraints.
• Prompt Template: Provides a system prompt and examples for DOM manipulation and tab/window control actions, with strict JSON output requirements and critical rules.
• Sanitizer: Validates the generated JSON plan: required fields, valid action types, and safe custom script patterns.
• Extension Integration: The side panel executor triggers generation, captures DOM context, and executes actions via background and content scripts.

The system orchestrates a user intent into executable browser actions:

• The side panel collects the goal and optional URL, optionally resolves a tab context, and captures the DOM.
• The backend receives the request, builds a prompt enriched with DOM info, and asks the LLM to produce a JSON action plan.
• The sanitizer validates the plan and returns it to the UI.
• The UI executes the plan via the extension’s background script, which performs tab/window control or injects DOM actions into the page.

sequenceDiagram
participant User as "User"
participant UI as "Side Panel<br/>AgentExecutor.tsx"
participant BG as "Background Script<br/>background.ts"
participant API as "FastAPI Router<br/>routers/browser_use.py"
participant SVC as "AgentService<br/>services/browser_use_service.py"
participant PROMPT as "Prompt Template<br/>prompts/browser_use.py"
participant SAN as "Sanitizer<br/>utils/agent_sanitizer.py"
User->>UI : Enter goal and optional URL
UI->>BG : Capture DOM and prepare payload
UI->>API : POST /api/agent/generate-script
API->>SVC : generate_script(goal, target_url, dom_structure, constraints)
SVC->>PROMPT : Build user prompt with DOM info
SVC->>SVC : Call LLM and parse response
SVC->>SAN : Validate JSON and actions
SAN-->>SVC : Problems or sanitized plan
SVC-->>API : {ok, action_plan} or error
API-->>UI : Response
UI->>BG : Execute actions (tab/window or injected DOM)
BG-->>UI : Execution results

AgentService: intelligent script generation#

AgentService orchestrates the generation of a JSON action plan:

• Formats DOM info for the prompt, including URL, title, and interactive elements.
• Constructs a user prompt with explicit rules for DOM vs tab actions, search URL construction, and constraints.
• Invokes the LLM pipeline and parses the response.
• Sanitizes and validates the JSON plan, reporting problems and returning raw response context on failure.

flowchart TD
Start(["Entry: generate_script"]) --> FormatDOM["Format DOM Info"]
FormatDOM --> BuildPrompt["Build User Prompt"]
BuildPrompt --> InvokeLLM["Invoke LLM Chain"]
InvokeLLM --> ParseResp["Parse Response"]
ParseResp --> Sanitize["Sanitize and Validate JSON"]
Sanitize --> Valid{"Valid Plan?"}
Valid --> |Yes| ReturnOK["Return {ok: true, action_plan}"]
Valid --> |No| ReturnErr["Return {ok: false, error, problems, raw_response}"]
ReturnOK --> End(["Exit"])
ReturnErr --> End

FastAPI router: endpoint contract and validation#

The router enforces:

• Required goal field.
• Delegates to AgentService and handles validation errors vs general errors.
• Returns a standardized response model including ok flag, action_plan, error, problems, and raw_response.

sequenceDiagram
participant Client as "Client"
participant Router as "Router<br/>routers/browser_use.py"
participant Service as "AgentService"
participant ModelR as "GenerateScriptRequest"
participant ModelS as "GenerateScriptResponse"
Client->>Router : POST /generate-script
Router->>ModelR : Validate request
alt Missing goal
Router-->>Client : 400 Bad Request
else Valid
Router->>Service : generate_script(...)
Service-->>Router : {ok, action_plan} or {ok : false, error/problems}
Router->>ModelS : Serialize response
Router-->>Client : 200 OK with response
end

BrowserActionInput schema and tool integration#

The schema defines:

• goal: The user’s instruction.
• target_url: Optional URL to navigate to.
• dom_structure: Optional DOM snapshot for context.
• constraints: Optional constraints for the action.

The tool wraps AgentService and exposes a structured tool for agent workflows.

classDiagram
class BrowserActionInput {
+string goal
+string target_url
+dict dom_structure
+dict constraints
}
class AgentService {
+generate_script(goal, target_url, dom_structure, constraints) dict
}
BrowserActionInput --> AgentService : "used by tool"

DOM structure analysis and interactive elements#

The extension captures DOM context for the current tab:

• Executes a content script function to collect interactive elements (links, buttons, inputs, selects, textareas, ARIA roles).
• Filters visible elements and extracts attributes (tag, id, class, type, placeholder, name, aria-label, innerText/textContent).
• Limits payload size and returns URL, title, and raw HTML alongside interactive elements.

flowchart TD
Start(["Capture DOM"]) --> QueryTabs["Query Active Tab"]
QueryTabs --> ExecScript["Execute Content Script to Collect DOM"]
ExecScript --> FilterVisible["Filter Visible Elements"]
FilterVisible --> ExtractAttrs["Extract Attributes and Text"]
ExtractAttrs --> LimitPayload["Limit Interactive List"]
LimitPayload --> ReturnDOM["Return {url, title, interactive, raw_html}"]
ReturnDOM --> End(["Done"])

Action planning and execution patterns#

The extension supports two execution modes:

• Side panel executor: Parses slash commands, resolves agent endpoints, captures DOM context, and executes the returned action plan.
• Background runner: Receives action plans and executes them with tab/window control and DOM injection.

sequenceDiagram
participant UI as "Side Panel"
participant BG as "Background Script"
participant CS as "Content Script"
participant Tab as "Browser Tabs"
UI->>BG : Execute actions
loop For each action
alt Tab/Window Control
BG->>Tab : Create/Close/Switch/Navigate/Reload/Duplicate
Tab-->>BG : Status
else DOM Manipulation
BG->>CS : Inject script to CLICK/TYPE/SCROLL/WAIT
CS-->>BG : Result
end
end
BG-->>UI : Aggregate results

Agent orchestration and tool integration#

The React agent integrates browser actions with other tools:

• Builds a LangGraph workflow with tools including the browser action tool.
• Routes tool calls to the appropriate tool and returns results to the agent.
• Supports dynamic tool composition based on context (e.g., Google tokens, JIIT session).

graph LR
RA["React Agent<br/>react_agent.py"] --> RT["Agent Tools<br/>react_tools.py"]
RT --> BA["Browser Action Tool<br/>tools/browser_use/tool.py"]
BA --> SVC["AgentService<br/>services/browser_use_service.py"]

• Service-to-Prompt: AgentService depends on the prompt template for constructing the user prompt.
• Service-to-Sanitizer: AgentService relies on the sanitizer to validate JSON and detect unsafe patterns.
• Router-to-Service: FastAPI router delegates to AgentService and returns standardized responses.
• Extension-to-Service: The side panel executor calls the backend; the background script executes actions locally when needed.
• Agent System: React agent composes tools, including the browser action tool, enabling multi-modal automation.

graph TB
SVC["AgentService"] --> PROMPT["Prompt Template"]
SVC --> SAN["Sanitizer"]
API["Router"] --> SVC
UI["Side Panel"] --> API
BG["Background Script"] --> BGExec["Action Execution"]
BG --> CS["Content Script"]
RA["React Agent"] --> RT["Agent Tools"]
RT --> BA["Browser Action Tool"]
BA --> SVC

• DOM capture limits: Interactive element lists are truncated to avoid large payloads and excessive token usage.
• Action throttling: A small delay is introduced between actions to prevent rapid-fire operations that may overwhelm the page.
• LLM token budget: DOM info is limited to a subset of interactive elements and concise text to fit within context windows.
• Tab operations: Navigation and reload operations wait for completion to avoid race conditions.
• Sanitization overhead: JSON validation and safety checks occur synchronously; batching or caching could reduce repeated work.

Common issues and resolutions:

• Missing goal: The router returns a 400 error when goal is empty.
• Validation failures: If the sanitizer detects missing fields or invalid action types, the service returns problems and a raw response preview.
• DOM action on chrome:// pages: The prompt explicitly forbids DOM actions on chrome-specific URLs; use tab control actions instead.
• Search URL construction: For search intents, use OPEN_TAB with a fully constructed search URL rather than opening blank tabs and typing.
• Element not found: DOM actions throw errors when selectors do not match; verify selectors and ensure the page has loaded.
• Tab switching: SWITCH_TAB requires either tabId or direction; ensure one is provided.
• Execution timeouts: Tab operations include timeout fallbacks; adjust expectations for slow-loading pages.

Sanitize every plan. Reject dangerous EXECUTE_SCRIPT patterns. DOM writes stay in content scripts with trusted selectors.

Browser action types and constraints#

• DOM Manipulation Actions: CLICK, TYPE, SCROLL, WAIT, SELECT, EXECUTE_SCRIPT
• Tab/Window Control Actions: OPEN_TAB, CLOSE_TAB, SWITCH_TAB, NAVIGATE, RELOAD_TAB, DUPLICATE_TAB
• Constraints: Provide target_url for navigation, prefer specific selectors, and avoid DOM actions on chrome:// pages.

Example scenarios#

• Open a new tab and search: Use OPEN_TAB with a constructed search URL; avoid typing into chrome:// pages.
• Fill a login form: Capture DOM, select precise selectors for input fields, and use TYPE with WAIT for page readiness.
• Switch tabs and click a button: Use SWITCH_TAB with tabId or direction, then CLICK on the target element.
• Scroll and extract links: Use SCROLL to reveal content, then use EXECUTE_SCRIPT to gather link data safely.

Security considerations#

• Custom script validation: The sanitizer rejects potentially dangerous patterns in EXECUTE_SCRIPT actions.
• DOM injection safety: Content scripts and background scripts restrict DOM manipulation to trusted selectors and validated contexts.
• CORS and permissions: Ensure the extension has scripting permissions for target origins and respects site policies.