Unhappy with service on Starter + VM

philboy

Hi,
I'm having an issue for the past few months with "Googlebot can't access your site". It occurs every 2-3 weeks. It has an error rate of 10-90% of reading robots.txt and it stops Google from crawling my site which is majorly affecting our SEO.

I've been on to Google and they recommend going to the Google support forums. They don't provide any more information on any errors in Webmaster Tools (WMT) *which is what blacknight support keep telling me to do. 

I've created a thread on Google Support forums and they seem to think it may be due to 1) googlebot being blocked 2) Site overloading.

I don't have access to your firewall logs so can't tell whether it's one and can't give you the IP of the affected Googlebot or the HTTP error codes as Google won't provide them. I've analysed my access logs and still nothing untoward show ups.

In relation to the Site overloading, I looked at the graphs on my VM and the cpu load is fine on the days when these errors occur. I also have pingdom setup and don't have any alerts for those days

I offered to upgrade to the next package (Business) to try and see if this solves the issue.

All I'm looking for and would appreciate is a proper investigation into this issue. I've analyzed everything I can on my end and can't get anymore information from Google. The issue is really frustrating. I don't mind if I have to pay for support, I'd just like to get the issue sorted.

Thanks.

mneylon

Hi

If you've opened tickets with us in the past can you please provide the ticket ID(s)?

Thanks

Michele

philboy

Hi Michele,
Thanks for your quick reply. The ticket number is #101271. I also used the chat support aswell quite a bit in relation to this issue. I always found the support, both chat & email very good up until this point.

I can't get any more information from Google on this issue, I've emailed them, was on their product forums and was on their live chat and they can't provide anymore information. I've all transcripts saved if you need anymore information. It's just very frustrating as I don't want it to affect our SEO and I can't seem to see what's wrong. I looked at all of the logs and spent most of the afternoon today going through them and still nothing. No 404,  503 or 504 etc... error codes on the days when this issue occurs. I also checked the load on our server at those times and there's nothing there indicating the site was overloaded at the time and couldn't server the robots.txt file. I checked our DNS settings and they're fine also.

If there's anything you guys can do I would appreciate it. I can upgrade to the Business package if you think it will solve it.

Thanks a lot,
Philip

mneylon

I asked our technical team to followup with you on this

Michele

philboy

Thanks Michele, no joy yet unfortunately with it. I'll keep researching on it and looking into it.

Digital Society

Try Cloudflare and see if it gives you a more detailed answer when the problem occurs.

Surprised GoogleBot deosnt give you something like a 522 error if the site is overloading.

So Google literally gives no number with the error?