Skip to content

AaronDabney/domain-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Installation: npm update

Usage: node ./src/main.js

Note: Please be courteous when mapping.

  • Check robots.txt first
  • What you need is probably already in the sitemap
  • Keep a generous interval between requests (default is 1000ms)

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors