Need help with an idea

cmssjone

Hi all,

I've been toying with an idea for a while now and I think I'm going to take the plunge and see if it will work.

The idea is pretty simple and essentially is a database of products/information that would be updated daily/weekly and be queried as needed.

The website does not need to be overly dynamic (lots of flash etc) but the information will have to be taken from 3rd party websites and then broadcast on my own.

I don't have the programming skills to build the website and am not sure how much it would cost to maintain or how to obtain the information from 3rd party websites. The majority of these would be foreign but would not need to be translated. If it were to be successful, then the database would be increased and would obviously have to be modified at regular intervals.

Would someone be able to offer a ball park cost for the type of website I am looking to develop?

The Corinthian

cmssjone wrote: »

The website does not need to be overly dynamic (lots of flash etc) but the information will have to be taken from 3rd party websites and then broadcast on my own.

I don't have the programming skills to build the website and am not sure how much it would cost to maintain or how to obtain the information from 3rd party websites. The majority of these would be foreign but would not need to be translated. If it were to be successful, then the database would be increased and would obviously have to be modified at regular intervals.

By the sounds of things, unless you can access data feeds from these sites that specifically share such information, you're looking at a site that screen-scrapes these third party sites, so you can update your own data. Basically, there's two applications, with a database between them.

Screen-scraping is where an automated process is set up on your server to access the HTML of a third party site like a normal person would and from that parse or 'pick out' the relevant information. It's notoriously difficult to maintain, as sites regularly change format (thus breaking your parsing logic) or even will block attempts at such harvesting.

An initial site and screen-scraping solution done together would probably set you back a few thousand (I'd guess anything between €2.5k to €10k), depending on overall functionality and design. Maintenance of the screen-scraping will be your biggest long-term cost, as you may require no work done for six months and then suddenly a large chunk of it rewritten (so this might mean an ongoing monthly fee of anything between €100 to €600 p.m.).

I'd recommend either taking on a partner to do this or splitting the project into the constituent parts, with the site done by a Web developer and hiring someone to do the screen-scraping solution and maintenance.

Please note, the above figures are totally off the top of my head, so don't treat them as anything other than a set of very ballpark figures. However, from a technical architecture point of view - without any further details - that's what would need to be done, given your description.

cmssjone

The Corinthian wrote: »

By the sounds of things, unless you can access data feeds from these sites that specifically share such information, you're looking at a site that screen-scrapes these third party sites, so you can update your own data. Basically, there's two applications, with a database between them.

Screen-scraping is where an automated process is set up on your server to access the HTML of a third party site like a normal person would and from that parse or 'pick out' the relevant information. It's notoriously difficult to maintain, as sites regularly change format (thus breaking your parsing logic) or even will block attempts at such harvesting.

An initial site and screen-scraping solution done together would probably set you back a few thousand (I'd guess anything between €2.5k to €10k), depending on overall functionality and design. Maintenance of the screen-scraping will be your biggest long-term cost, as you may require no work done for six months and then suddenly a large chunk of it rewritten (so this might mean an ongoing monthly fee of anything between €100 to €600 p.m.).

I'd recommend either taking on a partner to do this or splitting the project into the constituent parts, with the site done by a Web developer and hiring someone to do the screen-scraping solution and maintenance.

Please note, the above figures are totally off the top of my head, so don't treat them as anything other than a set of very ballpark figures. However, from a technical architecture point of view - without any further details - that's what would need to be done, given your description.

Thanks for the very detailed reply. I wasn't aware how difficult this is to maintain and naively thought it would just be a one-off start up cost with tinkering here and there. Looks like it could be back to the drawing board so...

The Corinthian

cmssjone wrote: »

I wasn't aware how difficult this is to maintain and naively thought it would just be a one-off start up cost with tinkering here and there.

Information has a value; as such, unless you have an agreement with the third party site, whereby they allow you to take the data, then essentially you're taking it without either their permission or cooperation.

How you may arrive at such an agreement is really up to what you can work out with or is offered by them. If you're reselling their goods, then they may be amenable to such an arrangement. Or you could offer to pay a monthly fee for the data. Or some other arrangement.

If you're taking it without either their permission or cooperation, then at best automating this will be flaky as any change to how they present it will likely essentially 'brake' how you suck it down to your own site, thus needing ongoing maintenance and development to fix such occurrences. They may also sue you, in a worst case scenario, as Ryanair has done, for such screen-scraping.

cmssjone

The Corinthian wrote: »

Information has a value; as such, unless you have an agreement with the third party site, whereby they allow you to take the data, then essentially you're taking it without either their permission or cooperation.

How you may arrive at such an agreement is really up to what you can work out with or is offered by them. If you're reselling their goods, then they may be amenable to such an arrangement. Or you could offer to pay a monthly fee for the data. Or some other arrangement.

If you're taking it without either their permission or cooperation, then at best automating this will be flaky as any change to how they present it will likely essentially 'brake' how you suck it down to your own site, thus needing ongoing maintenance and development to fix such occurrences. They may also sue you, in a worst case scenario, as Ryanair has done, for such screen-scraping.

Again thanks for the informative reply. I would not be looking to resell any products and would not be a competitor in any way to the businesses. The idea would be to present the data for purely informational purposes for my users. I could visit the websites myself and then manually input the prices into my own website and update this (manually) on a weekly/monthly basis. Perhaps I will try this method, as even I can make such a basic website as this, before trying to automate the process. This will also indicate the demand, if any, that exists for the service.

The Corinthian

cmssjone wrote: »

I could visit the websites myself and then manually input the prices into my own website and update this (manually) on a weekly/monthly basis.

Depends on how much source data there is and how often it changes. You don't want inaccurate data sitting on your site for any length of time or in any great volume, as you'll lose any marketing value to the information - people will just not trust it and thus won't visit your site to check it.

Perhaps I will try this method, as even I can make such a basic website as this, before trying to automate the process. This will also indicate the demand, if any, that exists for the service.

There are probably better, and cheaper, ways of doing this. Indeed, market research is something you realistically want to do before launch.

Peterdalkey

Is this intended to be a hobby or a revenue generating commercial website? If it is for profit, it is very hard to see a business model for a directory from scaped data, that whole market has long come and gone. If it is a hobby site then this thread is probably in the wrong forum.

The Corinthian

Peterdalkey wrote: »

Is this intended to be a hobby or a revenue generating commercial website? If it is for profit, it is very hard to see a business model for a directory from scaped data, that whole market has long come and gone.

I'd imagine that the scraped data is a marketing tool, something to attract people to the site and to return to it, as it does not appear to be directly connected to whatever he/she is selling, with the actual service or product being something (as yet revealed) else.

There is merit to such a marketing strategy, but it depends on what that information is, if there are other places one can go for it and, frankly, how much cost and hassle it takes to maintain it.

cmssjone

The Corinthian wrote: »

Depends on how much source data there is and how often it changes. You don't want inaccurate data sitting on your site for any length of time or in any great volume, as you'll lose any marketing value to the information - people will just not trust it and thus won't visit your site to check it.

There are probably better, and cheaper, ways of doing this. Indeed, market research is something you realistically want to do before launch.

Everyone who I have mentioned the idea to has remarked that it seems like a good idea. (I'm sure many people have said that!!) However, I agree that formal market research would be needed. The data would not need to be live and up to the minute as the data would lead to a general picture, rather than be important as single entities.

Peterdalkey wrote: »

Is this intended to be a hobby or a revenue generating commercial website? If it is for profit, it is very hard to see a business model for a directory from scaped data, that whole market has long come and gone. If it is a hobby site then this thread is probably in the wrong forum.

I would like to think that it would earn revenue but this would only be through advertising so would need pretty serious traffic. I do believe that it would be a useful website and the target market would be large enough to gain the traffic needed to earn some decent returns.