Skip to content
@scrapy

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Pinned Loading

  1. scrapy scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python 58.9k 11.1k

  2. scrapyd scrapyd Public

    A service daemon to run Scrapy spiders

    Python 3.1k 576

  3. parsel parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    Python 1.3k 154

  4. w3lib w3lib Public

    Python library of web-related functions

    Python 411 106

  5. protego protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    DIGITAL Command Language 71 28

  6. itemadapter itemadapter Public

    Common interface for data container classes

    Python 68 12

Repositories

Showing 10 of 29 repositories
  • protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    scrapy/protego’s past year of commit activity
    DIGITAL Command Language 71 BSD-3-Clause 28 7 (3 issues need help) 1 Updated Nov 8, 2025
  • queuelib Public

    Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

    scrapy/queuelib’s past year of commit activity
    Python 285 BSD-3-Clause 55 3 3 Updated Nov 8, 2025
  • itemloaders Public

    Library to populate items using XPath and CSS with a convenient API

    scrapy/itemloaders’s past year of commit activity
    Python 47 BSD-3-Clause 16 18 (1 issue needs help) 5 Updated Nov 8, 2025
  • itemadapter Public

    Common interface for data container classes

    scrapy/itemadapter’s past year of commit activity
    Python 68 BSD-3-Clause 12 10 3 Updated Nov 7, 2025
  • cssselect Public

    CSS Selectors for Python

    scrapy/cssselect’s past year of commit activity
    Python 302 61 18 5 Updated Nov 7, 2025
  • w3lib Public

    Python library of web-related functions

    scrapy/w3lib’s past year of commit activity
    Python 411 BSD-3-Clause 106 11 (1 issue needs help) 6 Updated Nov 7, 2025
  • parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    scrapy/parsel’s past year of commit activity
    Python 1,287 BSD-3-Clause 154 33 14 Updated Nov 6, 2025
  • scrapyd Public

    A service daemon to run Scrapy spiders

    scrapy/scrapyd’s past year of commit activity
    Python 3,074 BSD-3-Clause 576 7 0 Updated Nov 6, 2025
  • scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    scrapy/scrapy’s past year of commit activity
    Python 58,923 BSD-3-Clause 11,143 465 (21 issues need help) 220 Updated Oct 27, 2025
  • sphinx-scrapy Public

    Sphinx extension for documentation in the Scrapy ecosystem

    scrapy/sphinx-scrapy’s past year of commit activity
    Python 1 BSD-3-Clause 1 0 0 Updated Sep 16, 2025
View all repositories

Top languages

Loading…

Most used topics

Loading…