Data Transfer Conundrum

Slaanesh

Lets get straight to it ...

Background:

Here in work we have developed a background music system that plays music (wow how ingenius!). It runs on Debian Linux and on a MIPS based processor.

Now the problem:

Customers normally receive new music every month on CD to the tune of ~450MB. We now have to possibility of a customer putting 8 of these boxes in their outlets around the country. This customer does not want updates via CD so we are left to do the updates "down-the-line". Due to backup/general internet usage amongst the stores there is only a one night window where these updates can be done.

Each store has a 3Mbit down/ 1Mbit up internet connection. So if the customer is able to receive the update at the max of 3Mbit the update could be done in about 25-30 minutes. So 8 boxes at 25-30 mins a piece is about 4 hours ... grand. The problem lies with the upload capacity here in work, we have 4Mbit/384Kbps so we can only transfer to each individual box at probably a max speed of 384Kbps. This obviously sucks.

I then thought of putting a box in the HQ of the company so that that box could update each site at a speed of 1Mbit but it's still a bit too slow and security concerns become apparent for the company.

The fastest connection out there that's within budget is the metro 8Mbit/2Mbit option which isn't too bad but that 2Mbit isn't guaranteed. Eircom will have a synchronous 10Mbit product coming out in the next couple of months but that will probably cost a small fortune.

We've also thought of daisy chaining the updates from box to box but that still has it's limitations.

Anyone else ever had to do something similar? Is there a hosting company that we could use to quickly transfer the data to each box and then just pay for the traffic?

Hope this all makes sense!

Stephen

That's how I'd do it anyway - upload the files at your leisure to someplace external that's got a fast connection, and then push the updates from there to the customer.

SwampThing

This won't be much help, but does the 120GB limit on Metro 8Mb/2Mb service not limit you? I've no idea how many customers you have, but for instance, this one customer receiving 8 X 450MB will use ~3.6GB a month.

To be honest, I don't know where you're going to get the bandwidth you need without paying for it. Unfortunately, broadband in this country sucks.
no doubt, someone will correct me, but maybe you're thinking the right way - get a decent hosting company, with generous limits and a wide pipe and upload to it, but I wouldn't even know where you point you, sorry.

irlrobins

Moving to broadband, I think its better suited to that forum.

hshortt

Couldn't you use a simple file hosting like rapidshare.de or something similar. You upload once and the customers get a single url to go to for downloading, you'll have to take the hit on the first upload.

Alternatively, what format are the tracks in, raw audio cda files? Why not compress to MP3 or WAV and have the recieving end burn as audio cd?

Cheerio
Howard

Wookie

What about using some sort of secure pier to pier file sharing software?

Bundle the software and set off your base site as a seed and connect up the other sites. In theory it should reduce the amount of data that the base site needs to upload by a significant amount. You could also try copying the file(s) to one other site and then kicking off both of them seeding the updated software etc…

bambam

I'd agree with the others, purchase some secure online storage. Then upload up to it at your leisure. Later, all clients can pull from the 3rdparty site at full speed.

Google for 'secure online storage', you should find plenty of options.

Alternatively, you could use some form of peer to peer protocol over VPN or https for getting the files distributed.

TheHig

Irish Broadbands' lines offer 3Mb Up/Down if memory serves. Something to look into, although I'd never recommend IBB to an end user, it seems as though it'd solve your problem

irlrobins

Yes I have the IBB 2Mb package. Works fine for me but I'm among the lucky few. Interestingly for whatever reason, my upload is not 2Mbps but 5Mbps. No idea why, it just was set to that after a fault. But even before that, had a perfect 2Mbps upload link.

Thomas_S_Hunterson

Torrents maybe?

Just email your clients the .torrent file every month

Slaanesh

Thanks for the replies lads, I don't think I was specific enough.

The customer does not interact with the box, it has 3 buttons and an LCD and plays music constantly. I need to be able to FTP files to the box, this will then trigger the update. The files are in MP3 format already.

Is there some sort of hosting company that I could store the update on and use a remote session to batch ftp the files to all the boxes?

Cheers.

irlrobins

mmm would this work:

Solo Linux account with Blacknight.

500Mb space and 5000Gb monthly bandwidth. Should be enough for you to upload the update to it. Then run a script on the remote boxes to ftp to the account and download the update.

Only cost you 35e a year for the account. That's the smallest one they have, you can go bigger if you need to.

Slaanesh

That looks like a brilliant solution, I'll look into it. Thanks irlrobins!

irlrobins wrote:

mmm would this work:

Solo Linux account with Blacknight.

500Mb space and 5000Gb monthly bandwidth. Should be enough for you to upload the update to it. Then run a script on the remote boxes to ftp to the account and download the update.

Only cost you 35e a year for the account. That's the smallest one they have, you can go bigger if you need to.

irlrobins

No worries. I've just switched my hosting to them from a provider in States. Every efficient, was in contact with them before the switch to ask a bunch of questions. Responses were precise and very prompt. They even set up a test account to confirm compatability with some of our equipment.

So I'm sure they'll be happy to answer any queries you might have.

And no I'm not shilling. I'm jsut stating my personal experience.

steve-hosting36

Just to clarify - that plan has 5GB not 5000

http://www.hosting365.com/hosting-win-personal.php has 10GB for example for the same price. Also remember, you'll need lots of storage - bandwidth allowance isnt much use if you cant store the content.

Also, using an Irish provider will ensure the fastest possible downloads for your end users.

If the end sites are located on or near any of the new MAN's (Metropolitan Area Networks) or Fiber in the main cities, you could also fiber them up relatively cost effectively.


HTH, STephen

irlrobins

steve-hosting36 wrote:

Just to clarify - that plan has 5GB not 5000

Sorry I was thinking in MB