# Install from crates.io
cargo install docrawl

# Or build from source
git clone https://github.com/neur0map/docrawl
cd docrawl
cargo build --release

docrawl "https://docs.rust-lang.org"          # crawl with default depth
docrawl "https://docs.python.org" --all       # full site crawl
docrawl "https://react.dev" --depth 2         # shallow crawl
docrawl "https://nextjs.org/docs" --fast      # quick scan without assets
docrawl "https://example.com/docs" --silence  # suppress progress/status output
docrawl --update                              # update to latest version

• Documentation-optimized extraction - Built-in selectors for Docusaurus, MkDocs, Sphinx, Next.js docs
• Clean Markdown output - Preserves code blocks, tables, and formatting with YAML frontmatter metadata
• Path-mirroring structure - Maintains original URL hierarchy as folders with index.md files
• Polite crawling - Respects robots.txt, rate limits, and sitemap hints
• Security-first - Sanitizes content, detects prompt injections, quarantines suspicious pages
• Self-updating - Built-in update mechanism via docrawl --update

Unlike general-purpose crawlers, docrawl is purpose-built for documentation:

docrawl combines crawling, extraction, and conversion in a single tool optimized for technical documentation.

Add to your Cargo.toml:

[dependencies]
docrawl = "0.1"
tokio = { version = "1", features = ["full"] }

Minimal example:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cfg = docrawl::CrawlConfig {
        base_url: url::Url::parse("https://example.com/docs")?,
        output_dir: std::path::PathBuf::from("./out"),
        max_depth: Some(3),
        ..Default::default()
    };
    let stats = docrawl::crawl(cfg).await?;
    println!("Crawled {} pages", stats.pages);
    Ok(())
}

Optional docrawl.config.json:

{
  "selectors": [".content", "article"],
  "exclude_patterns": ["\\.pdf$", "/api/"],
  "max_pages": 1000,
  "host_only": true
}

output/
└── example.com/
    ├── index.md
    ├── guide/
    │   └── index.md
    ├── assets/
    │   └── images/
    └── manifest.json

Each Markdown file includes frontmatter:

---
title: Page Title
source_url: https://example.com/page
fetched_at: 2025-01-18T12:00:00Z
---

docrawl is optimized for speed and efficiency:

• Fast HTML to Markdown conversion using fast_html2md
• Concurrent processing with configurable worker pools
• Intelligent rate limiting to respect server resources
• Persistent caching to avoid duplicate work
• Memory-efficient streaming for large sites

docrawl includes built-in security features:

• Content sanitization removes potentially harmful HTML
• Prompt injection detection identifies and quarantines suspicious content
• URL validation prevents malicious redirects
• File system sandboxing restricts output to specified directories
• Rate limiting prevents overwhelming target servers

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

MIT