Advertisement
Help Keep Boards Alive. Support us by going ad free today. See here: https://subscriptions.boards.ie/.
If we do not hit our goal we will be forced to close the site.

Current status: https://keepboardsalive.com/

Annual subs are best for most impact. If you are still undecided on going Ad Free - you can also donate using the Paypal Donate option. All contribution helps. Thank you.
https://www.boards.ie/group/1878-subscribers-forum

Private Group for paid up members of Boards.ie. Join the club.

Best way to seach google from desktop

  • 12-12-2009 08:04AM
    #1
    techguy
    Registered Users, Registered Users 2 Posts: 2,238 ✭✭✭


    Hi,

    I'm currently working on a web crawler and need to search google to get start links.

    At the moment i'm screen scraping the result from a simple query but its getting very complicated to just get a clean set of results.

    I've been reading up on Google search API but it seems they have discontinued what I am looking for.

    Maybe i'm overlooking some new service they have introduced. It would be nice to be able to have something that would return the results in xml.


Comments

  • #2
    Páid
    Registered Users, Registered Users 2 Posts: 917 ✭✭✭Páid


    DMOZ provide their database of urls free of charge - http://rdf.dmoz.org

    Might be easier and faster than querying Google for links.


  • #3
    techguy
    Registered Users, Registered Users 2 Posts: 2,238 ✭✭✭techguy


    How detailed is this DB? I'm going to be searching for pages that are pretty new so would this do the job?

    I've been working with Yahoo Boss over the last few days and it seems pretty good.

    Thanks.


  • #4
    Páid
    Registered Users, Registered Users 2 Posts: 917 ✭✭✭Páid


    It contains a lot more than just urls e.g. hierarchical categories, etc. No idea how up to date it is.

    Yahoo BOSS looks good. I noticed that there are unlimited queries presently but that they intend to introduce a fee structure.


Advertisement