Component interactions and data flow

Request paths across side panel, background/content scripts, WebSocket, FastAPI, and the LangGraph agent, including HTTP fallback.

Pieces:

• Extension (React UI, background script, content script, WebSocket client, utilities)
• Backend API (FastAPI app, routers, services, models)
• Agents and tools (LangGraph-based React agent, tool registry)
• MCP server (external service integration)

graph TB
subgraph "Extension"
UI["AgentExecutor.tsx<br/>Side Panel UI"]
WS["websocket-client.ts<br/>WebSocket Client"]
BG["background.ts<br/>Background Script"]
CT["content.ts<br/>Content Script"]
EXA["executeAgent.ts<br/>HTTP Execution"]
EXB["executeActions.ts<br/>Browser Actions"]
PARSE["parseAgentCommand.ts<br/>Command Parser"]
end
subgraph "Backend API"
API["main.py<br/>FastAPI App"]
RUN["main.py<br/>Uvicorn Runner"]
end
subgraph "Agents & Tools"
REACT["react_agent.py<br/>LangGraph Agent"]
end
UI --> WS
UI --> EXA
UI --> PARSE
WS --> API
EXA --> API
BG --> CT
EXB --> BG
API --> REACT

• WebSocket client encapsulates connection lifecycle, event handling, and agent execution over WebSocket with automatic reconnection.
• Side panel UI manages user input, command parsing, progress updates, and browser storage-backed sessions.
• Background script handles cross-tab automation, content script injection, and runtime messaging.
• Content script provides page context and executes DOM-level actions.
• HTTP execution utility builds payloads, captures page context, and invokes backend endpoints.
• Backend API exposes routers for agents, tools, and services; runs on Uvicorn.
• React agent orchestrates LLM reasoning and tool execution via LangGraph.

The system supports two primary execution modes:

• Real-time via WebSocket: UI emits commands; backend streams progress and returns results.
• HTTP fallback: UI sends commands via HTTP; backend responds with completion.

sequenceDiagram
participant User as "User"
participant UI as "AgentExecutor.tsx"
participant WS as "websocket-client.ts"
participant BG as "background.ts"
participant CT as "content.ts"
participant API as "main.py"
participant REACT as "react_agent.py"
User->>UI : Enter command
UI->>UI : parseAgentCommand()
alt WebSocket connected
UI->>WS : executeAgent(command, onProgress)
WS->>API : emit "execute_agent" {command}
API->>REACT : run_react_agent(messages)
REACT-->>API : streaming tool calls
API-->>WS : "generation_progress" events
WS-->>UI : onProgress updates
API-->>WS : "agent_result" or "agent_error"
WS-->>UI : resolve/reject
else HTTP fallback
UI->>BG : fetch skills / upload / execute agent
BG->>CT : inject and send PERFORM_ACTION
UI->>API : POST /api/... with payload
API->>REACT : run_react_agent(messages)
REACT-->>API : final answer
API-->>UI : JSON response
end

WebSocket client and real-time execution#

• Establishes persistent connections with automatic reconnection and transport fallback.
• Emits connection status and generation progress events.
• Provides executeAgent, stopAgent, and stats APIs with timeouts and cleanup.

classDiagram
class WebSocketClient {
-socket
-listeners
-autoConnect
+connect()
+on(event, callback)
+off(event, callback)
+isSocketConnected() bool
+executeAgent(command, onProgress) Promise
+stopAgent() Promise
+getStats() Promise
+disconnect()
+connectSocket()
+enableAutoConnect()
+disableAutoConnect()
}

Side panel UI and command parsing#

• Parses slash commands into agent/action endpoints and validates availability.
• Manages sessions, voice input, file attachments, and mention menus.
• Streams progress updates and renders Markdown responses.

flowchart TD
Start(["User enters command"]) --> Parse["parseAgentCommand()"]
Parse --> Stage{"Stage"}
Stage --> |agent_select| SuggestAgents["Show agent suggestions"]
Stage --> |action_select| SuggestActions["Show action suggestions"]
Stage --> |complete| BuildPayload["Build payload and execute"]
BuildPayload --> WSMode{"WebSocket connected?"}
WSMode --> |Yes| WSExec["wsClient.executeAgent()"]
WSMode --> |No| HTTPExec["executeAgent() via HTTP"]
WSExec --> Progress["Update progress"]
HTTPExec --> Result["Render result"]
Progress --> Result

HTTP execution pipeline#

• Resolves active or mentioned tab context, captures client HTML, normalizes URLs, and constructs payloads.
• Supports special endpoints (React agent, YouTube, website, GitHub, PyJIIT, skills).
• Uses GET/POST based on endpoint and returns structured responses.

sequenceDiagram
participant UI as "AgentExecutor.tsx"
participant EXA as "executeAgent.ts"
participant BG as "background.ts"
participant API as "main.py"
participant REACT as "react_agent.py"
UI->>EXA : executeAgent(fullCommand, prompt, chatHistory, attachedFile)
EXA->>EXA : Resolve tab context + capture HTML
EXA->>EXA : Build payload per endpoint
alt WebSocket mode
EXA->>API : POST /api/... (payload)
else HTTP mode
EXA->>BG : Runtime message (if needed)
BG->>API : HTTP request
end
API->>REACT : run_react_agent(messages)
REACT-->>API : final answer
API-->>EXA : JSON response
EXA-->>UI : result

Browser automation and content scripts#

• Background script injects content scripts and routes actions to active tabs.
• Content script performs DOM-level actions (click, type, scroll) and page info extraction.
• Action executor translates agent action plans into tab messages.

sequenceDiagram
participant UI as "AgentExecutor.tsx"
participant BG as "background.ts"
participant CT as "content.ts"
participant TAB as "Target Tab"
UI->>BG : EXECUTE_ACTION {action}
BG->>TAB : scripting.executeScript(files)
BG->>TAB : tabs.sendMessage {type : "PERFORM_ACTION", action}
TAB->>CT : performAction(action)
CT-->>TAB : result
TAB-->>BG : response
BG-->>UI : result

Backend API and agent orchestration#

• FastAPI app aggregates routers for agents, tools, and services.
• Uvicorn runner starts the server on a configurable host/port.
• React agent compiles a LangGraph workflow and executes tool calls asynchronously.

graph TB
API["main.py"] --> Routers["Routers: agent, react, website, youtube, upload, skills"]
API --> App["FastAPI App"]
App --> Uvicorn["main.py: uvicorn.run()"]
App --> REACT["react_agent.py: run_react_agent()"]

• UI depends on WebSocket client and HTTP execution utility.
• Background script depends on content script and browser APIs.
• HTTP execution utility depends on browser storage, tabs, and scripting APIs.
• Backend API depends on routers and the React agent.
• React agent depends on LLM and tool registry.

graph LR
UI["AgentExecutor.tsx"] --> WS["websocket-client.ts"]
UI --> EXA["executeAgent.ts"]
UI --> PARSE["parseAgentCommand.ts"]
EXA --> BG["background.ts"]
BG --> CT["content.ts"]
API["main.py"] --> REACT["react_agent.py"]
WS --> API
EXA --> API

• WebSocket streaming reduces latency for long-running agent executions; HTTP fallback ensures reliability when WebSocket is unavailable.
• Payload construction captures minimal client HTML and limits DOM introspection to reduce overhead.
• Action executor introduces small delays between actions to prevent race conditions and improve stability.
• Caching: React agent graph is cached via LRU to avoid recompilation costs.
• Recommendations:
• Prefer WebSocket for interactive sessions; degrade to HTTP.
• Limit DOM capture size and scope; avoid unnecessary reflows.
• Batch browser actions and debounce UI updates.

• WebSocket connectivity:
• Monitor connection_status events and fallback to HTTP when disconnected.
• Use getStats with timeout to detect backend responsiveness.
• Command parsing:
• Ensure slash commands are complete; partial suggestions guide users.
• Browser automation:
• Verify content script injection and tab permissions.
• Handle unknown action types and timeouts during navigation/reload.
• HTTP errors:
• Normalize error messages for rate limits, gateway errors, and service unavailability.
• Storage and sessions:
• Persist sessions in browser storage; migrate legacy chat history if needed.

UI → transport → agent → tools → browser. WebSocket streams when it can; HTTP covers the rest. State lives where the component that needs it can see it.