Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Googlebots hogging bandwidth

Options
  • 25-04-2008 4:56pm
    #1
    bazzjazz
    Registered Users Posts: 80 ✭✭


    Not sure if this is the correct board for this...

    I host a clients site on a shared server with 50gb bandwidth per month. At the moment Googlebots are using the entire 50gb crawling the site. Total bandwidth for my shared server this month is about 130gb and most of this is robots crawling various sites.

    I have a robots.txt file which is only allowing the major search engines and it is restricting access to all directories on the site.

    I have also used Googles webmaster tools to slow the crawling of the googlebots but nothing seems to be working.

    Does anyone have any suggestions on how to dramatically slow the crawling of googlebots?

    Thanks!

    Barry


Comments

  • Options
    #2
    Bob the Builder
    Registered Users Posts: 2,919 ✭✭✭Bob the Builder


    i've never known google to do that to be honest. Are you sure that it's google's IP address though?

    I had a problem when google used 400mb of bandwith, but webmaster tools sorted out the problem...


  • Options
    #3
    bazzjazz
    Registered Users Posts: 80 ✭✭bazzjazz


    I'm sure that Google is the main culprit ok, Yahoo is also doing a lot of crawling.

    One of the googlebots is 'crawl-66-2549-65-130.googlebot.com'. There are others with the same naming structure.

    What did you do in Webmaster tools to sort your problem?

    regards,

    Barry


Advertisement