tavily-crawl
tavily crawl
What it does
- You need content from many pages on a site (e.g., all /docs/)
- You want to download documentation for offline use
- Step 4 in the workflow: search → extract → map → crawl → research
- Start conservative — --max-depth 1, --limit 20 — and scale up.
- Use --select-paths to focus on the section you need.
- Use map first to understand site structure before a full crawl.
- Always set --limit to prevent runaway crawls.
Requirements & configuration
- You need content from many pages on a site (e.g., all /docs/)
- | --chunks-per-source | Chunks per page (1-5, requires --instructions) |
- Use --select-paths to focus on the section you need.
Derived from the skill's own SKILL.md documentation · extracted 2026-07-23
tavily-crawl FAQ
What does the tavily-crawl skill do?
| Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", "bulk extract", or needs content from many pages on the same domain. Supports depth/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file. You need content from many pages on a site (e.g., all /docs/) You want to download documentation for offline use
What does tavily-crawl require?
You need content from many pages on a site (e.g., all /docs/) | --chunks-per-source | Chunks per page (1-5, requires --instructions) | Use --select-paths to focus on the section you need.
How do I install tavily-crawl?
Run: npx -y skills add https://github.com/tavily-ai/skills --skill tavily-crawl --agent claude-code — the source lives at github.com/tavily-ai/skills.
Maintain tavily-crawl?
Let people know it's listed here — add the badge (live metrics, light/dark aware) or a plain link to your README or docs.
[](https://getagentictools.com/skills/tavily-ai-skills-tavily-crawl?ref=badge) npx agentictools info skills/tavily-ai-skills-tavily-crawl The second line is the CLI lookup for this page — handy in READMEs and docs.
Related skills
flutter-animating-apps Visual feedback, transitions, and physics-based motion for Flutter apps using the Animation framework.
flutter-managing-state Manage ephemeral and app-level state in Flutter using StatefulWidget, MVVM, and Provider.
tavily-map tavily map
angular-new-app angular new app
flutter-architecting-apps Layered architecture framework for scalable Flutter apps with strict separation of concerns.
tavily-extract tavily extract