Apache Lounge



Forum Index Downloads Search Register Log in  RSS Apache Lounge

Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.



A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Apache Lounge is not sponsored.

Your donations will help to keep this site alive and well, and continuing building binaries.

Crawl-delay to stop crawler overload the server.

Post new topic   Reply to topic    Apache Forum Index -> How-to's & Documentation & Tips

View previous topic :: View next topic  
Author Message

Joined: 15 Oct 2005
Posts: 2720
Location: Hilversum, NL, EU

PostPosted: Fri 28 Jun '13 10:35    Post subject: Crawl-delay to stop crawler overload the server. Reply with quote

When you want that Google, Bing, Yandex etc. index your site(s) and it is overloading the server then you can use in robot.txt the Crawl-delay directive.

See for expample for an explanation: www.bing.com/blogs/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx


From the Yandex help:
If the server is overloaded and does not have enough time to process downloading requests, use the Crawl-delay directive. It enables you to specify the minimum interval (in seconds) for a search robot to wait after downloading one page, before starting to download another. To achieve compatibility with robots that somewhat deviate from standard behaviour when processing robots.txt, the Crawl-delay directive must be added to the group that starts from the 'User-Agent' entry, right after the 'Disallow'('Allow') directive(s).

Yandex search robot supports fractional values for Crawl-Delay, e.g. 0.5. It does not mean that the search robot will access your site every half a second, but it gives the robot more freedom and may speed up the site processing.
Back to top

Post new topic   Reply to topic    Apache Forum Index -> How-to's & Documentation & Tips
Page 1 of 1