This example demonstrates how to use Scrapegraph-ai for deep web crawling and content exploration.
- Deep web crawling
- Content discovery
- Link analysis
- Recursive search
- Install required dependencies
- Copy
.env.example
to.env
- Configure your API keys in the
.env
file
from scrapegraphai.graphs import DepthSearchGraph
graph = DepthSearchGraph()
results = graph.search("https://example.com", depth=3)
Required environment variables:
OPENAI_API_KEY
: Your OpenAI API key