Agentic Browser

AI agent for web automation and scraping, designed to plan, execute, and verify actions across any website

Source 14 Live site

Tool system

Introduction#

Tool framework: StructuredTool interface, argument schemas, dynamic registration from runtime context, and the domains tools cover.

Project structure#

The Tool System spans several layers:

  • Tool definitions and registration live in the agent module
  • Tool implementations reside under tools/
  • Services orchestrate tool logic and external integrations
  • Routers expose endpoints for direct API access
  • Prompts and utilities support tool-specific workflows

Core components#

  • Tool interface standard: Each tool is a StructuredTool with a typed Pydantic args_schema and a coroutine executor. Inputs are validated automatically; outputs are normalized to text or structured JSON.
  • Registration mechanism: The agent builds a dynamic toolset from context (e.g., Google access tokens, PyJIIT session payloads). Tools are conditionally included to avoid unnecessary dependencies.
  • Execution patterns: Tools run asynchronously; long-running or blocking operations are offloaded to threads to prevent blocking the event loop.
  • Validation and sanitization: Tools normalize outputs and apply sanitization for generated JSON plans; routers validate inputs and enforce constraints.

Key implementation anchors:

  • Tool registry and builders: react_tools.py
  • Tool schemas and coroutines: react_tools.py
  • Browser automation tool: browser_use/tool.py
  • GitHub ingestion: github_crawler/convertor.py
  • Web search pipeline: google_search/seach_agent.py
  • Website context fetchers: website_context/__init__.py
  • YouTube utilities: youtube_utils/__init__.py
  • Gmail service: services/gmail_service.py
  • Calendar router: routers/calendar.py
  • YouTube router: routers/youtube.py

Architecture overview#

Tool stack:

  • Agent layer defines tools and builds the toolset from context
  • Tool layer implements domain-specific logic and integrates with services/utilities
  • Service layer orchestrates external integrations and validations
  • Router layer exposes endpoints for direct API access

Detailed component analysis#

Tool interface standards#

  • StructuredTool: Each tool is a StructuredTool with a name, description, coroutine executor, and args_schema. Inputs are validated automatically by Pydantic.
  • Args schemas: Define required fields, constraints (min/max values), and descriptions. Examples include GitHubToolInput, WebsiteToolInput, YouTubeToolInput, GmailToolInput, CalendarToolInput, and PyjiitAttendanceInput.
  • Output normalization: Tools return either plain text or structured JSON. A helper ensures consistent stringification of outputs.

Implementation anchors:

  • StructuredTool definitions and schemas: react_tools.py
  • Output normalization helpers: react_tools.py

Registration mechanisms#

  • Static toolset: AGENT_TOOLS provides a baseline set of tools (GitHub, web search, website, YouTube, browser automation).
  • Dynamic toolset builder: build_agent_tools(context) adds Google and PyJIIT tools when credentials/payloads are present in context. It uses partial to inject default tokens/payloads into tool coroutines.

Implementation anchors:

  • Static toolset and builder: react_tools.py

Execution patterns#

  • Async-first design: Tools are coroutines. Long-running or blocking operations are executed in threads using asyncio.to_thread to avoid blocking the event loop.
  • Example patterns:
  • Web search tool: bounded results and thread-offloaded pipeline invocation
  • Gmail tools: token validation and thread-offloaded operations
  • Calendar tools: ISO 8601 validation and thread-offloaded creation
  • Browser automation: service-based generation of JSON action plans with sanitization

Implementation anchors:

  • Tool coroutines and thread offloading: react_tools.py
  • Browser automation service: browser_use_service.py

Browser automation tools#

  • Purpose: Generate a JSON action plan for browser tasks given a goal, target URL, DOM structure, and constraints.
  • Implementation:
  • Tool schema defines goal, target_url, dom_structure, and constraints
  • Coroutine invokes AgentService.generate_script
  • Service composes a prompt, invokes an LLM, sanitizes the result, and returns structured action plan

GitHub crawler tools#

  • Purpose: Convert a GitHub repository to markdown (tree, summary, content), then answer questions using a retrieval-augmented chain.
  • Implementation:
  • URL normalization and ingestion (async/sync fallback)
  • Optional file attachment processing via Google AI SDK
  • LLM-based answer generation with chat history support

Gmail integration tools#

  • Purpose: Fetch latest emails, list unread messages, mark messages as read, and send emails using OAuth access tokens.
  • Implementation:
  • Service methods wrap tool functions and centralize error logging
  • Routers validate presence of access_token and enforce max result bounds

Calendar management tools#

  • Purpose: Retrieve upcoming events and create new events using OAuth access tokens.
  • Implementation:
  • Routers validate ISO 8601 timestamps and enforce max result bounds
  • Tools delegate to service-layer logic for event operations

YouTube processing utilities#

  • Purpose: Extract video IDs, fetch subtitles, and retrieve video info for downstream tooling.
  • Implementation:
  • Utility functions exposed via init.py
  • Router answers questions about videos using YouTube service

Website context extraction tools#

  • Purpose: Convert HTML to markdown and fetch markdown for a given URL to enable question answering.
  • Implementation:
  • Exposed via website_context/init.py
  • Used by website_agent tool to answer questions about a page

Google search tool#

  • Purpose: Perform web search using Tavily and return summarized results.
  • Implementation:
  • Pipeline sets max_results and maps Tavily response to expected format
  • Tool caps results and summarizes snippets for downstream use

PyJIIT attendance tool#

  • Purpose: Fetch attendance data from the JIIT web portal using a session payload.
  • Implementation:
  • Validates session payload and adapts to nested structures
  • Hardcoded semester mapping and regex parsing for subject codes
  • Runs blocking IO in a thread

Dependency analysis#

  • Tool-to-service coupling:
  • Browser tool depends on AgentService for action plan generation
  • GitHub tool depends on GitHubService and ingestion utilities
  • Gmail tool depends on GmailService and tool functions
  • Router-to-service coupling:
  • GitHub router depends on GitHubService
  • Gmail router depends on GmailService
  • Calendar router validates inputs and delegates to service logic
  • YouTube router depends on YouTube service
  • External dependencies:
  • LangChain StructuredTool and TavilySearch
  • Pydantic for input validation
  • Optional Google AI SDK for file attachments

Performance considerations#

  • Async execution: Use asyncio.to_thread for blocking operations to maintain responsiveness.
  • Input bounding: Tools cap max_results and truncate content to fit model context windows.
  • Token limits: Repository content is truncated to a safe upper bound to prevent context overflow.
  • Prompt composition: Browser automation service limits interactive element listings to reduce token usage.
  • External API throttling: Respect rate limits for external services (e.g., Tavily, Gmail, Calendar).

Troubleshooting guide#

Common issues and resolutions:

  • Missing credentials:
  • Gmail/Calendar tools require access tokens; errors instruct providing tokens or include them in the tool call.
  • Invalid input formats:
  • Calendar tools enforce ISO 8601 timestamps; GitHub router validates URL and question presence.
  • Web search tool bounds max_results; website tool trims summaries.
  • External service failures:
  • GitHub ingestion handles invalid URLs and inaccessible repositories with plain messages.
  • Gmail service logs exceptions and re-raises for router handling.
  • Action plan validation:
  • Browser automation service returns validation problems and raw response for debugging.

Conclusion#

Same interface for every tool, validate inputs, run async. Registration can depend on context so you do not expose Gmail tools without a token.

Appendices#

Guidelines for creating custom tools#

  • Define a Pydantic args_schema with clear field descriptions and constraints
  • Implement a coroutine executor that:
  • Validates inputs
  • Handles errors and returns plain messages
  • Offloads blocking operations to threads when needed
  • Wrap tool logic in a service if it interacts with external APIs
  • Register the tool in the agent toolset and conditionally include it via build_agent_tools when appropriate
  • Add a router endpoint if exposing the tool via API

Tool validation and error handling#

  • Use Pydantic validators to enforce input constraints
  • Normalize outputs consistently (strings or JSON)
  • Log errors early and propagate meaningful messages
  • For external APIs, handle common failure modes (invalid tokens, rate limits, timeouts)

Security considerations#

  • Never embed secrets in tool code; pass tokens via context or request parameters
  • Validate and sanitize inputs to prevent injection attacks
  • Limit tool capabilities to least privilege scopes
  • Avoid printing sensitive data in logs

Resource management and debugging#

  • Use bounded max_results and content truncation to manage memory and compute
  • Enable structured logging and return minimal diagnostic details in responses
  • For browser automation, sanitize and validate generated JSON plans before returning