crawl rate

What is robots.txt ? A robots.txt file instructs search engine crawlers on which URLs on your site they may access....