Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Web scraping to database

Options
  • 07-06-2007 3:41pm
    #1
    jellie
    Registered Users Posts: 15,503 ✭✭✭✭


    Hi,

    Im wondering if anyone can recommend a programming language or push me in the right direction with a project.
    I have to scrape data from a webpage & store this data in a database. The data will have to be updated regularly, every 10-15mins probably, and deleted when its no longer relevent.

    Ive done a lot of java, a bit of c & c++, and have done some basic basic php (just connecting to a mysql database & adding/retrieving data). There just seems to be so many different options i dont really know where to start.

    If anyone has any suggestions/recommendations id be really grateful!
    Thanks! :)


Comments

  • Options
    #2
    Eoin
    Registered Users Posts: 21,257 ✭✭✭✭Eoin


    If you've done a bit of PHP before, then this should do the trick?

    http://www.tgreer.com/class_http_php.html


  • Options
    #3
    NutJob
    Closed Accounts Posts: 884 ✭✭✭NutJob


    Anything with regular expressions and a decent networking liburies.

    Have done this before in Java and also in perl.


  • Options
    #4
    daymobrew
    Registered Users Posts: 6,509 ✭✭✭daymobrew


    I use some perl scripts with cron to download a weather data page, extract the required data (regular expressions) and push the data to a database (via a perl CGI on a web site). Then PHP reads the data to produce a nice graph.


  • Options
    #5
    Trampas
    Registered Users Posts: 7,678 ✭✭✭Trampas


    I have used VB.NET for doing this.

    Are you scraping big chunks of text or word here and a word there?

    Regular Expressions can do it


  • Options
    #6
    jellie
    Registered Users Posts: 15,503 ✭✭✭✭jellie


    Thanks for the help.

    Its data from a table that im scraping, thered be quite a lot of it


  • Advertisement
Advertisement