Websurferagent

If you need an agent that can browse, extract, or interact with the web, WebSurferAgent is a good choice. The agent actions the request(s) given to it by determining what to do on the web and browsing and crawling it, returning the details of what it finds.

The WebSurferAgent has two in-built web tools to choose from: 1. browser-use - uses an actual browser instance (visible or headless), interacting with the web pages in realtime 2. Crawl4AI - crawls without a visual browser instance

Tip

If you want to add browsing capabilities to your existing agents, see this notebook for browser-use and this notebook for Crawl4AI.

Warning

Browser Use requires Python 3.11 or higher.

To get started with WebSurferAgent, install AG2 with the browser-use and/or crawl4ai extras. 

pip install ag2[openai,browser-use]
and/or 
pip install ag2[openai,crawl4ai]

Tip

If you have been using autogen or pyautogen, all you need to do is upgrade it using: 

pip install -U autogen[openai,browser-use]
or 
pip install -U pyautogen[openai,browser-use]
as pyautogen, autogen, and ag2 are aliases for the same PyPI package.

And then setup Playwright: 

# Installs Playwright and browsers for all OS
playwright install
# Additional command, mandatory for Linux only
playwright install-deps

Now, you can create an agent, nominating the web tool:

Tip

Browser Use supports the following models: Supported Models

We had great experience with OpenAI, Anthropic, and Gemini. However, DeepSeek and Ollama haven't performed as well.

Tip

Crawl4AI is built on top of LiteLLM and supports the same models as LiteLLM.

We had great experience with OpenAI, Anthropic, Gemini and Ollama. However, as of this writing, DeepSeek is encountering some issues.

from autogen.agents.experimental import WebSurferAgent

# Put your key in the OPENAI_API_KEY environment variable
llm_config = {"api_type": "openai", "model": "gpt-4o-mini"}

# Create our agent
websurfer = WebSurferAgent(
    name="WebSurfer",
    llm_config=llm_config,
    web_tool="browser_use",
)
# or
websurfer = WebSurferAgent(
    name="WebSurfer",
    llm_config=llm_config,
    web_tool="crawl4ai",
)

Tip

Crawl4AI doesn't always require an LLM configuration, see this notebook for examples with and without one.

Let's browse the web for news on AG2.

import Example from "/snippets/python-examples/websurferagent.mdx";

Let's break it down:

  1. Import WebSurferAgent and create an LLM configuration for the browser-use tool to use.

  2. We create a configuration dictionary turning off the headless mode (so we can see what's happening) and saving an animated GIF of the process (shown below).

  3. Create the agent, nominating the web tool and passing in the LLM and tool configurations.

  4. Run the agent, ensuring we pass the agent's tools through to the run method so it can add them to the internal executor agent to execute.

WebSurferAgent in action