Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Robots and Spiders - Keep clear!

Options
  • 19-04-2004 4:10pm
    #1
    Philbert
    Registered Users Posts: 358 ✭✭


    A quick question!

    What do I need to put at the top of a webpage to stop all search engine robots and spiders from searching and indexing the webpage?

    Is there code you can put at the top of the homepage (mydomain.com, mydomain.com/index.htm) that will stop all search engine robots and spiders from searching and indexing the entire website?

    TIA..


Comments

  • Options
    #2
    dahamsta
    Banned (with Prison Access) Posts: 16,659 ✭✭✭✭dahamsta


    http://www.robotstxt.org/wc/exclusion.html#meta


  • Options
    #3
    Philbert
    Registered Users Posts: 358 ✭✭Philbert


    Thank you very much ;)


  • Options
    #4
    buddy
    Registered Users Posts: 1,031 ✭✭✭buddy


    When do people want to do this?

    Frame page's etc or under what other circumstances?


  • Options
    #5
    mneylon
    Registered Users Posts: 7,739 ✭✭✭mneylon


    Originally posted by buddy
    When do people want to do this?

    Frame page's etc or under what other circumstances?
    When you don't want your pages to be indexed.

    Of course there is a downside - "rogue" spiders do not follow the robots directives and will be more than happy to index anyway.
    If you need to block them you may have to look into .htaccess exclusion rules.

    Blog | Tweets



  • Options
    #6
    buddy
    Registered Users Posts: 1,031 ✭✭✭buddy


    I understand its to prevent pages being indexed, but I don't see a reason that I'd want that prevented.

    What types of pages do people have that they don't want indexed?


  • Advertisement
  • Options
    #7
    dahamsta
    Banned (with Prison Access) Posts: 16,659 ✭✭✭✭dahamsta


    Originally posted by buddy
    What types of pages do people have that they don't want indexed?
    Well, Google doesn't want anything indexed, for obvious reasons. People with personal websites mightn't want their personal details easily findable by every TD&H on the Internet. Businesses might want certain sections of their website (prices for example) restricted to genuine prospects (i.e. active visitors). There's lots of other reasons.

    adam


Advertisement