Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/md_v2/core/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,54 @@ This page provides a comprehensive list of example scripts that demonstrate vari
| Chainlit Integration | Shows how to integrate Crawl4AI with Chainlit. | [View Guide](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/chainlit.md) |
| Crawl4AI vs FireCrawl | Compares Crawl4AI with the FireCrawl library. | [View Code](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/crawlai_vs_firecrawl.py) |

## Community Integrations

Crawl4AI works well with AG2 (formerly AutoGen) for agentic browsing. The example below uses `Crawl4AITool` with a Pydantic schema to return structured data.

```python
# Install: `pip install "ag2[openai,crawl4ai]"`

import os
from autogen import AssistantAgent, UserProxyAgent, LLMConfig
from autogen.tools.experimental import Crawl4AITool
from pydantic import BaseModel, Field


class NewsArticle(BaseModel):
title: str = Field(description="The headline of the article")
summary: str = Field(description="A brief summary of the content")


llm_config = LLMConfig({
"api_type": "openai",
"model": "gpt-4o",
"api_key": os.environ["OPENAI_API_KEY"],
}
)

researcher = AssistantAgent(
name="WebResearcher",
system_message="You are a precise web researcher. Use the crawl tool to get data.",
llm_config=llm_config,
)

user_proxy = UserProxyAgent(
name="Admin",
human_input_mode="NEVER",
max_consecutive_auto_reply=2,
)

crawl_tool = Crawl4AITool(llm_config=llm_config, extraction_model=NewsArticle)
crawl_tool.register_for_execution(user_proxy)
crawl_tool.register_for_llm(researcher)

user_proxy.run(
recipient=researcher,
message="Extract the top 3 headlines and summaries from https://techcrunch.com using the news schema.",
max_turns=2,
).process()
```

## Content Generation & Markdown

| Example | Description | Link |
Expand Down
3 changes: 2 additions & 1 deletion docs/md_v2/core/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -461,5 +461,6 @@ If you’re ready for more, check out:
- **Hooks & Auth**: Learn how to run custom JavaScript or handle logins with cookies, local storage, etc.
- **Deployment**: Explore ephemeral testing in Docker or plan for the upcoming stable Docker release.
- **Browser Management**: Delve into user simulation, stealth modes, and concurrency best practices.
- **Code Examples**: Browse practical examples, including a community AG2 (formerly AutoGen) integration snippet.

Crawl4AI is a powerful, flexible tool. Enjoy building out your scrapers, data pipelines, or AI-driven extraction flows. Happy crawling!
Crawl4AI is a powerful, flexible tool. Enjoy building out your scrapers, data pipelines, or AI-driven extraction flows. Happy crawling!