What do I need to learn to do this?

Supercell

I want to import data from a csv file into a database and output that information into an application that reads it and can generate graphs, summary's etc on the fly as the user chooses variables.

For example, if I input a csv file of today's weather data from my automated weather station, I would like to be able to launch the program and choose to see a graph of temperatures in the last 6 hours/days, maybe see what the top 10 temperatures were in the last year. Can this be done on a web page too without needing a separate application?

I have a MSDN account so can use any Microsoft product for this.

I would also import data automatically from the web via ftp (synoptic weather station data is available for free over ftp in csv format also).

Thanks.

fergalr

There's loads of ways of doing this, with varying complexity and flexibility.


One way you could go about it is have a simple script running on a server somewhere, for example written in Perl, that would periodically connect to the FTP server, parse the CSV and shove it into the database. This would be quite easy to knock together in Perl, or indeed most other languages.

Then, the other part of your app that displays the graphed data to the user. It sounds like you want it on a web page, so you could write a simple piece of j2ee that runs in tomcat, connects to the backend database, and renders the appropriate data on the web page, maybe using a library like JFreeChart to render the graphs.

That's one way to go about it, that'd give a lot of flexibility and control over the content. You could probably also find a set of canned tools that'd do most of the displaying of the data for you, as it's a really common enterprise task to graph data that's in a database...
Are you looking for more detail than that?

Webmonkey

Currently working with BIRT: http://www.eclipse.org/birt/phoenix/

I must say I'm impressed with it. Its so damn easy to use!

You can run a it on a Tomcat server then as a java app to view the reports.

Just another idea if you want to make life simple.

Supercell

Thanks guys,

Certainly stuff to read up on there.
Any non Java suggestions?

sobriquet

That you mention MSDN implies you'd prefer to use MS tech.

If so, my first thoughts would be a program running on your windows box to do what fergalr says - I'd make it a Service so that it could be started and restarted automatically using standard Windows mechanisms. It'd periodically grab the csv file, parse it, and mung the data into your DB (SQL Server most likely, the Express editions are free and functional). You might have a config file specifying where the file is, and how often to grab it, every 15 minutes say.

As the lads say with regard to reporting, you'd do the same thing but with an ASP.Net web application that reads the data from the same DB. Have a look around for graph generating tools in C#/ASP.Net, there'll likely be loads out there. A benefit of this approach is you could do both in C#.

Cantab.

Perl and gnuplot

Tom Dunne

sobriquet wrote: »

If so, my first thoughts would be a program running on your windows box to do what fergalr says - I'd make it a Service so that it could be started and restarted automatically using standard Windows mechanisms. It'd periodically grab the csv file, parse it, and mung the data into your DB (SQL Server most likely, the Express editions are free and functional). You might have a config file specifying where the file is, and how often to grab it, every 15 minutes say.

That sounds like a bit of an overkill.

A simple ftp scheduled task would do the trick. Drop the csv file into a directory and have SQL server load it.

No need for Windows services or anything fancy.

sobriquet

Tom Dunne wrote: »

No need for Windows services or anything fancy.

Writing a Service isn't actually hard at all, I don't recall it being much harder than writing the actual working code to perform a task. The main benefit is that the whole thing is more self-contained and you can use the management console to stop/start it like other services. But yeah, it's little different in practice from using scheduled tasks.

FWIW, I'm a linux head, I'd more likely go with python and cron if it were up to me.

Gurgle

Supercell wrote: »

I want to import data from a csv file into a database and output that information into an application that reads it and can generate graphs, summary's etc on the fly as the user chooses variables.

That sounds like a bog-standard report generating database.
MS Access will do it for you, all with wizards so practically no knowledge is needed.

Start to finish development would be under 2 hours for what you describe.

Or you could go off hacking at Java, Perl, Ftp servers, BIRT and gnuplot until christmas.

BigglesMcGee

Do it with c# or vb.net.

I reckon abotu 2 hours too including nice charts etc

Gurgle

BigglesMcGee wrote: »

Do it with c# or vb.net.

I reckon abotu 2 hours too including nice charts etc

I included learning to use Access in the two hours.