Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Best way to seach google from desktop

Options
  • 12-12-2009 8:04am
    #1
    techguy
    Registered Users Posts: 2,234 ✭✭✭


    Hi,

    I'm currently working on a web crawler and need to search google to get start links.

    At the moment i'm screen scraping the result from a simple query but its getting very complicated to just get a clean set of results.

    I've been reading up on Google search API but it seems they have discontinued what I am looking for.

    Maybe i'm overlooking some new service they have introduced. It would be nice to be able to have something that would return the results in xml.


Comments

  • Options
    #2
    Páid
    Registered Users Posts: 916 ✭✭✭Páid


    DMOZ provide their database of urls free of charge - http://rdf.dmoz.org

    Might be easier and faster than querying Google for links.


  • Options
    #3
    techguy
    Registered Users Posts: 2,234 ✭✭✭techguy


    How detailed is this DB? I'm going to be searching for pages that are pretty new so would this do the job?

    I've been working with Yahoo Boss over the last few days and it seems pretty good.

    Thanks.


  • Options
    #4
    Páid
    Registered Users Posts: 916 ✭✭✭Páid


    It contains a lot more than just urls e.g. hierarchical categories, etc. No idea how up to date it is.

    Yahoo BOSS looks good. I noticed that there are unlimited queries presently but that they intend to introduce a fee structure.


Advertisement