Download
A libp2p DHT crawler and monitor that tracks the liveness of peers. The crawler connects to DHT bootstrap peers and then recursively follows all entries in their k-buckets until all peers have been visited. The crawler supports the IPFS, Filecoin, Polkadot, Kusama, Rococo, Westend networks and more. The crawler can store its results as JSON documents or in a postgres database - the --dry-run flag prevents it from doing either. Nebula will print a summary of the crawl at the end instead. A crawl takes ~5-10 min depending on your internet connection. You can also specify the network you want to crawl by appending, e.g., --network FILECOIN and limit the number of peers to crawl by providing the --limit flag.
Features
- The crawl sub-command starts by connecting to a set of bootstrap nodes and constructing the routing tables (kademlia k-buckets) of these peers based on their PeerIDs
- If Nebula is configured to store its results in a database, every peer that was visited is persisted in it
- The monitor sub-command polls every 10 seconds all sessions from the database
- The resolve sub-command goes through all multi addresses that are present in the database and resolves them to their respective IP-addresses
- You need a running postgres instance to persist and/or read the crawl results
- Migrations are applied automatically when nebula starts and successfully establishes a database connection
Project Samples
