3

Old school knowledge needed:

The IP-based method of exempting GoogleBot from throttling (as detailed in the MIA EE wiki) no longer works - GoogleBot doesn't have a fixed IP range.

I need Google to be able to constantly and reliably index a news site, but would like to keep throttling active, to help deal with weak DOS nonsense.

Ideas?

Improve this question

1 Answer 1

Reset to default
1

You might look at changing this in Google's configs directly. If you set up a Webmaster Tools account and connect the domain, you can adjust the Crawl Rate to something that will work better for your site.

enter image description here

Improve this answer
5
  • A great tip, but this site has been assigned special crawl rates because of the amount and frequency of new content. Google won't allow changing the settings in Webmaster Tools.
    – Dylan Smith
    Mar 19, 2013 at 16:24
  • Assigned how? So this setting is disabled on your end?
    – Anna_MediaGirl ♦
    Mar 19, 2013 at 17:51
  • No, on Google's. They do that for news sites with heaps of new content.
    – Dylan Smith
    Mar 20, 2013 at 1:09
  • To be clear, the setting in Webmaster Tools doesn't affect the frequency of the bot visits, instead it allows you to change the speed at which the bot consumes pages on the site while it's there visiting. You can slow the speed down to fit within your throttling limits. Apparently, if you set this you need to readd it every 90 days.
    – Anna_MediaGirl ♦
    Mar 20, 2013 at 14:59
  • And if you have a site with frequent enough updates, Google apparently disables any ability to modify any crawl settings. "Crawl rate Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate."
    – Dylan Smith
    Mar 24, 2013 at 23:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.