Huge installers/downloads

cornbb

I'm currently working on an installer for a very large application (c. 1.7GB) which my company wants to sell via downloads as well as on DVD. We want an installation solution which will allow users to reliably download a small installation package which will then connect to our servers and download the rest of the necessary files - something akin to Windows Update or what have you. This would let users resume downloads if they fail - my superiors basically reckon that this option, cumbersome as it may seem, would be better than requiring users to download one huge file.

To complicate matters, we are developing for Mac OS X and are currently using Apple's PackageMaker installer solution, so I think that might restrict our options a bit

Hope I'm making sense here - I'm not sure where to start, but if anyone else here has had experience with similar installer scenarios I'd love to hear them. Thanks!

Martyr

i don't have any experience doing something like this, but perhaps you should use existing protocols to get the job done, like HTTP.

its simple enough if you know how to program tcp/ip sockets, assuming you do, i'd say use the content-range: http header.

what i did for a resume, get the file size on disk, get the total size of file on the http server from Content-Length: header after HEAD request.

if the file is incomplete,send a GET request with the Content-Range: set.

HEAD /big_installer.exe HTTP/1.1
Host: www.yourwebsiteaddress.com
Connection: close
User-Agent: Cornbb HTTP client

total_size = Content-Length: (from HEAD request)
range = (size_on_disk+1)-total_size

for example, if the total file size is 1KB, and the amount already downloaded is 500 Bytes, the request would be something like..

GET /big_installer.exe HTTP/1.1
Host: www.yourwebsiteaddress.com
Connection: Keep-Alive
Content-Range: 501-1000

...any other relevant information.it might be worth a look, rather than writing a whole new protocol.

Neurojazz

most times people use the torrent system (which is open source) - so you could stream data out to clients that way...

The Corinthian

Neurojazz wrote:

most times people use the torrent system (which is open source) - so you could stream data out to clients that way...

Unless their application is likely to have thousands of downloads per day any P2P solution will, given the nature of P2P, be pretty useless.

phil

And specifically talking to the person who said to try and implement part of the HTTP specification, don't bother unless there really is no library on OSX to do this. And I'd be really really surprised if there wasn't. Maybe there's not, but try not to reinvent the wheel too much!

cornbb

Thanks for all the suggestions! Average Joe's method would work quite well in principle I think (bear in mind that I'm not very familiar with this area of programming/development). However, the thing is we need a method like this (i.e. splitting downloads) which will work seamlessly with some sort of installer/installation solution, which would involve the user manually downloading a small package, which would in turn take care of grabbing the huge files from our servers, reassembling them and launching the installer.

We are currently using Apple's PackageMaker/Installer which has no facilities to do this. ViseX is the other major installer for OS X and they don't seem to do this either. The only software vendors I've seen do things like this are Apple, Microsoft and Adobe. There don't appear to be any off-the-shelf solutions, certainly not for Mac, and I'm not looking forward to the prospect of developing one myself.

Thanks again for the suggestions, please keep em coming

Ginger

If you ever used the MSDN download site, they use a custom downloader. HTTP has limits on the file sizes it can download, so MS use their downloader to get the files from the website (the MSDN libs are a download similar in size to what you want)

Just get the clients to download the installer and use that to copy the files down. It also acts as a download manager so that they dont have to do it in one go.. I am fairly sure this is something similar to Average Joes suggestion

amen

of interest what are you developing that has 1.7GB application?
thats a lot of code

The Corinthian

amen wrote:

thats a lot of code

I very much doubt if it's code. It's probably bloated by dependencies such as media files or data.

bushy...

Could the small installer use wget ( if its still in mac os ) , since it has resume and all the rest ?

ressem

I might be wrong but I think Microsoft use a rebranded version of
http://www.akamai.com/html/technology/products/http_downloads.html

for their download manager.

decob

440Hz wrote:

Are you ceiling cat???

Are you watching us code in here! Big brother style... im scared... im off home now... STOP WATCHING ME

no, i think i could be beta-testing his companies software...

gah.. stupid boards doing funky stuff again...

cornbb

amen wrote:

of interest what are you developing that has 1.7GB application?
thats a lot of code

Thats true The app itself is only 15mb, the rest consists of audio samples, WAV/AIFF files which have been compressed into 3 huge propitiatory container files. The fact that these container files are so huge is part of the problem, but unfortunately we can't split them up

ressem wrote:

I might be wrong but I think Microsoft use a rebranded version of
http://www.akamai.com/html/technolog...downloads.html

for their download manager.

Perfect! Something like that is exactly what I seem to be looking for. Looks like it might be costly, although could be a breeze to implement.

bushy... wrote:

Could the small installer use wget ( if its still in mac os ) , since it has resume and all the rest ?

Also perfect!! Well not quite perfect, as I'd love an off-the-shelf solution, but that tool looks like just what I would need to implement a homemade solution. There's no man entry for it in OSX but some quick googling indicates that someone has made a widget/extension available.

I'll be back with an update. Thanks everyone!

decob

cornbb wrote:

Thats true The app itself is only 15mb, the rest consists of audio samples, WAV/AIFF files which have been compressed into 3 huge propitiatory container files. The fact that these container files are so huge is part of the problem, but unfortunately we can't split them up

slightly off topic, but would i be right in assuming you work for a company being with 'a' ending in 't'.

440Hz

decob wrote:

slightly off topic, but would i be right in assuming you work for a company being with 'a' ending in 't'.

Are you ceiling cat???

Are you watching us code in here! Big brother style... im scared... im off home now... STOP WATCHING ME

cornbb

decob wrote:

slightly off topic, but would i be right in assuming you work for a company being with 'a' ending in 't'.

Ha, indeed I do, as does 440Hz as you might gather ^^

440Hz

Oooohhh a Beta Tester Nice one! Small world wide web eh!

cornbb you win the prize again!

decob

440Hz wrote:

Oooohhh a Beta Tester Nice one! Small world wide web eh!

it is indeed. ..

440Hz

Just FYI, for anyone looking at this in the future... was looking yesterday and found this http://atterer.net/jigdo/ quite a nifty little tool. for any platform!

cornbb

Also http://aria2.sourceforge.net/ or http://www.gnu.org/software/wget/ for anyone wishing to use a CLI or homemade solution. Solution, God, I hate that word...