logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> How-to's & Documentation & Tips View previous topic :: View next topic
Reply to topic   Topic: Crawl-delay to stop crawler overload the server.
Author
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3049
Location: Hilversum, NL, EU

PostPosted: Fri 28 Jun '13 10:35    Post subject: Crawl-delay to stop crawler overload the server. Reply with quote

When you want that Google, Bing, Yandex etc. index your site(s) and it is overloading the server then you can use in robot.txt the Crawl-delay directive.

See for expample for an explanation: www.bing.com/blogs/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx

Steffen


From the Yandex help:
If the server is overloaded and does not have enough time to process downloading requests, use the Crawl-delay directive. It enables you to specify the minimum interval (in seconds) for a search robot to wait after downloading one page, before starting to download another. To achieve compatibility with robots that somewhat deviate from standard behaviour when processing robots.txt, the Crawl-delay directive must be added to the group that starts from the 'User-Agent' entry, right after the 'Disallow'('Allow') directive(s).

Yandex search robot supports fractional values for Crawl-Delay, e.g. 0.5. It does not mean that the search robot will access your site every half a second, but it gives the robot more freedom and may speed up the site processing.
Back to top


Reply to topic   Topic: Crawl-delay to stop crawler overload the server. View previous topic :: View next topic
Post new topic   Forum Index -> How-to's & Documentation & Tips