Advertisement
Help Keep Boards Alive. Support us by going ad free today. See here: https://subscriptions.boards.ie/.
If we do not hit our goal we will be forced to close the site.

Current status: https://keepboardsalive.com/

Annual subs are best for most impact. If you are still undecided on going Ad Free - you can also donate using the Paypal Donate option. All contribution helps. Thank you.
https://www.boards.ie/group/1878-subscribers-forum

Private Group for paid up members of Boards.ie. Join the club.

Googlebots hogging bandwidth

  • 25-04-2008 04:56PM
    #1
    bazzjazz
    Registered Users, Registered Users 2 Posts: 82 ✭✭


    Not sure if this is the correct board for this...

    I host a clients site on a shared server with 50gb bandwidth per month. At the moment Googlebots are using the entire 50gb crawling the site. Total bandwidth for my shared server this month is about 130gb and most of this is robots crawling various sites.

    I have a robots.txt file which is only allowing the major search engines and it is restricting access to all directories on the site.

    I have also used Googles webmaster tools to slow the crawling of the googlebots but nothing seems to be working.

    Does anyone have any suggestions on how to dramatically slow the crawling of googlebots?

    Thanks!

    Barry


Comments

  • #2
    Bob the Builder
    Registered Users, Registered Users 2 Posts: 2,919 ✭✭✭Bob the Builder


    i've never known google to do that to be honest. Are you sure that it's google's IP address though?

    I had a problem when google used 400mb of bandwith, but webmaster tools sorted out the problem...


  • #3
    bazzjazz
    Registered Users, Registered Users 2 Posts: 82 ✭✭bazzjazz


    I'm sure that Google is the main culprit ok, Yahoo is also doing a lot of crawling.

    One of the googlebots is 'crawl-66-2549-65-130.googlebot.com'. There are others with the same naming structure.

    What did you do in Webmaster tools to sort your problem?

    regards,

    Barry


Advertisement