Search engine tester

Da Man

I have a site with dynamic pages and as per usual, the problem is to get search engines to pick those up. This is using ASP.NET. I'm quite happy to limit myself to search engines that can handle query strings. My plan is to generate a static page with links to all the relevant pages with query strings and then have a link from the home page to the static page that only appears if the browser identifies itself as a crawler.

That's no big deal to do, but I would like to test it somehow in my development environment. Are there crawlers out there that would simulate a search engine crawl to give me some assurance that this might actually work?

mneylon

If you poke around on google and look for search engine optimisation tools, you should find a few that will give you some idea of how a spider would view the site.

I'm not sure about the possibilities using .NET and IIS6, but with some systems (such as vBulletin) the search engine issues have been overcome by building an alternate, search engine friendly version of the pages.

tomED

You could server a dynamic page to the bot once it visits, but the only way you would be able to decide whether the bot is a crawler or a regular visitor is by finding out what the agent type is.

Set up your script to serve the "search engine friendly" content once a bot from your list is found. Testing purposes - try it using mozilla, IE, and netscape.

Google will follow most dynamic page links, but not all, thats why I would recommend you start building your site to account for search engine friendky urls. So for example - www.mysite.com/test.asp?id=34 will become - www.mysite.com/test.asp/id/34 or something simliar.

Theres loads of info out ther for figuring out how to do this on the web. Search for search engine friendly urls.

Da Man

ASP.NET claims to be able to make the distinction between a bot and an ordinary user through an API call (HttpBrowserCapabilities.Crawler to be precise). I presume there's something in the request somewhere that provides this information to the server?

How does one do this otherwise? A hidden link from the home page to the static page with all the links to the dynamic pages?

tomED

Be careful with hidden anything! Google will penalise you if it notices that you are using hidden links - but in saying that, it does work..

yellum

You could build a sitemap with the links on that.