If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)

Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Computer Literate Bible Bashers

ballooba · 2003-05-02 15:53:01

Does anyone else have these wankers crawling their site or is it just that we represent crude drunken debauchery of its lowest form? ChristCRAWLER 2.0 (http://www.christcrawler.com), I'm tempted to crawl their website with 'F*ck off and die' as my user agent.

Options

02-05-2003 4:53pm

#1

ballooba

Registered Users Posts: 6,315 ✭✭✭

Join Date: June 2002

Posts: 5777

Does anyone else have these **** crawling their site or is it just that we represent crude drunken debauchery of its lowest form?

ChristCRAWLER 2.0 (http://www.christcrawler.com),

I'm tempted to crawl their website with 'F*ck off and die' as my user agent.

0

Comments

Options
#2 03-05-2003 2:24am
mneylon

Registered Users Posts: 7,739 ✭✭✭

Join Date: April 2001

Posts: 7231

Yep - got a few of them in there

If the robot complies with the 'standard' you can easily block it using a robots.txt file - and they say that it does
Have a look here:
http://www.robotstxt.org/wc/robots.html
http://www.searchengineworld.com/robots/robots_tutorial.htm

If you're having problems with creating the robots.txt there is some software available to do it:
http://www.rietta.com/robogen/index.shtml -PAY
http://www.webtoolcentral.com/webmaster/tools/robots_txt_file_generator/ - FREE Online Tool

Blog | Tweets

0