Installation:
npm update
Usage:
node ./src/main.js
Note: Please be courteous when mapping.
- Check robots.txt first
- What you need is probably already in the sitemap
- Keep a generous interval between requests (default is 1000ms)
-
Notifications
You must be signed in to change notification settings - Fork 0
License
AaronDabney/domain-crawler
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
No description, website, or topics provided.
Resources
License
Stars
Watchers
Forks
Releases
No releases published