Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Easiest way to make graph from text files?

Options
  • 18-08-2010 4:41pm
    #1
    Moderators, Science, Health & Environment Moderators, Social & Fun Moderators, Society & Culture Moderators Posts: 60,098 Mod ✭✭✭✭


    Hello, I want to make graphs from various text files generated by simulations. An example would be these two files. 12 nodes each with a radio total value. So say graphing motes on the x axis and radio total on the y axis to generate a line for each file or perhaps taking the value after mote 1 radio total in each file and graphing that in a line with same axis values? What would be the quickest way for me to do this, excel, some script, any tools about, not made much graphs before.

    Thanks!
    Efficiency table loaded, size=102
    Recovery table loaded, size=11
    trace file used: /opt/tinyos-2.1.1/apps/EasyDissemination/DIP/Simulations/Energy.txt
    energy model used: micaZ_energy_model.txt
    maxseen 11
    mote battery starting energy: 21600000 mJ
    Mote 0, cpu total: 0.9
    Mote 0, radio total: 59094.2
    Mote 0, adc total: 0.0
    Mote 0, leds total: 0.0
    Mote 0, sensor total: 0.0
    Mote 0, eeprom total: 0.0
    Mote 0, cpu_cycle total: 0.0
    Mote 0, Total energy used: 59095
    Mote 0, Battery energy remaining (linear): 21540905
    Mote 0, Battery energy remaining: 21538572
    Mote 0, Battery mAh remaining: 1994.3


    Mote 1, cpu total: 0.9
    Mote 1, radio total: 34609.3
    Mote 1, adc total: 0.0
    Mote 1, leds total: 6561.3
    Mote 1, sensor total: 0.0
    Mote 1, eeprom total: 0.0
    Mote 1, cpu_cycle total: 0.0
    Mote 1, Total energy used: 41172
    Mote 1, Battery energy remaining (linear): 21558828
    Mote 1, Battery energy remaining: 21558918
    Mote 1, Battery mAh remaining: 1996.2


    Mote 2, cpu total: 0.9
    Mote 2, radio total: 59094.2
    Mote 2, adc total: 0.0
    Mote 2, leds total: 0.0
    Mote 2, sensor total: 0.0
    Mote 2, eeprom total: 0.0
    Mote 2, cpu_cycle total: 0.0
    Mote 2, Total energy used: 59095
    Mote 2, Battery energy remaining (linear): 21540905
    Mote 2, Battery energy remaining: 21538572
    Mote 2, Battery mAh remaining: 1994.3


    Mote 3, cpu total: 0.9
    Mote 3, radio total: 59094.1
    Mote 3, adc total: 0.0
    Mote 3, leds total: 0.0
    Mote 3, sensor total: 0.0
    Mote 3, eeprom total: 0.0
    Mote 3, cpu_cycle total: 0.0
    Mote 3, Total energy used: 59095
    Mote 3, Battery energy remaining (linear): 21540905
    Mote 3, Battery energy remaining: 21538572
    Mote 3, Battery mAh remaining: 1994.3


    Mote 4, cpu total: 0.9
    Mote 4, radio total: 59094.1
    Mote 4, adc total: 0.0
    Mote 4, leds total: 0.0
    Mote 4, sensor total: 0.0
    Mote 4, eeprom total: 0.0
    Mote 4, cpu_cycle total: 0.0
    Mote 4, Total energy used: 59095
    Mote 4, Battery energy remaining (linear): 21540905
    Mote 4, Battery energy remaining: 21538572
    Mote 4, Battery mAh remaining: 1994.3


    Mote 5, cpu total: 0.9
    Mote 5, radio total: 59094.1
    Mote 5, adc total: 0.0
    Mote 5, leds total: 0.0
    Mote 5, sensor total: 0.0
    Mote 5, eeprom total: 0.0
    Mote 5, cpu_cycle total: 0.0
    Mote 5, Total energy used: 59095
    Mote 5, Battery energy remaining (linear): 21540905
    Mote 5, Battery energy remaining: 21538572
    Mote 5, Battery mAh remaining: 1994.3


    Mote 6, cpu total: 0.9
    Mote 6, radio total: 59094.1
    Mote 6, adc total: 0.0
    Mote 6, leds total: 0.0
    Mote 6, sensor total: 0.0
    Mote 6, eeprom total: 0.0
    Mote 6, cpu_cycle total: 0.0
    Mote 6, Total energy used: 59095
    Mote 6, Battery energy remaining (linear): 21540905
    Mote 6, Battery energy remaining: 21538572
    Mote 6, Battery mAh remaining: 1994.3


    Mote 7, cpu total: 0.9
    Mote 7, radio total: 59094.1
    Mote 7, adc total: 0.0
    Mote 7, leds total: 0.0
    Mote 7, sensor total: 0.0
    Mote 7, eeprom total: 0.0
    Mote 7, cpu_cycle total: 0.0
    Mote 7, Total energy used: 59095
    Mote 7, Battery energy remaining (linear): 21540905
    Mote 7, Battery energy remaining: 21538572
    Mote 7, Battery mAh remaining: 1994.3


    Mote 8, cpu total: 0.9
    Mote 8, radio total: 59094.1
    Mote 8, adc total: 0.0
    Mote 8, leds total: 0.0
    Mote 8, sensor total: 0.0
    Mote 8, eeprom total: 0.0
    Mote 8, cpu_cycle total: 0.0
    Mote 8, Total energy used: 59095
    Mote 8, Battery energy remaining (linear): 21540905
    Mote 8, Battery energy remaining: 21538572
    Mote 8, Battery mAh remaining: 1994.3


    Mote 9, cpu total: 0.9
    Mote 9, radio total: 59094.1
    Mote 9, adc total: 0.0
    Mote 9, leds total: 0.0
    Mote 9, sensor total: 0.0
    Mote 9, eeprom total: 0.0
    Mote 9, cpu_cycle total: 0.0
    Mote 9, Total energy used: 59095
    Mote 9, Battery energy remaining (linear): 21540905
    Mote 9, Battery energy remaining: 21538572
    Mote 9, Battery mAh remaining: 1994.3


    Mote 10, cpu total: 0.9
    Mote 10, radio total: 59094.1
    Mote 10, adc total: 0.0
    Mote 10, leds total: 0.0
    Mote 10, sensor total: 0.0
    Mote 10, eeprom total: 0.0
    Mote 10, cpu_cycle total: 0.0
    Mote 10, Total energy used: 59095
    Mote 10, Battery energy remaining (linear): 21540905
    Mote 10, Battery energy remaining: 21538572
    Mote 10, Battery mAh remaining: 1994.3


    Mote 11, cpu total: 0.9
    Mote 11, radio total: 59094.0
    Mote 11, adc total: 0.0
    Mote 11, leds total: 0.0
    Mote 11, sensor total: 0.0
    Mote 11, eeprom total: 0.0
    Mote 11, cpu_cycle total: 0.0
    Mote 11, Total energy used: 59095
    Mote 11, Battery energy remaining (linear): 21540905
    Mote 11, Battery energy remaining: 21538572
    Mote 11, Battery mAh remaining: 1994.3


    Simulated seconds: 999.9
    Real seconds: 14.7
    Efficiency table loaded, size=102
    Recovery table loaded, size=11
    trace file used: /opt/tinyos-2.1.1/apps/EasyDissemination/DIP/Simulations/Energy.txt
    energy model used: micaZ_energy_model.txt
    maxseen 11
    mote battery starting energy: 21600000 mJ
    Mote 0, cpu total: 0.9
    Mote 0, radio total: 59088.4
    Mote 0, adc total: 0.0
    Mote 0, leds total: 0.0
    Mote 0, sensor total: 0.0
    Mote 0, eeprom total: 0.0
    Mote 0, cpu_cycle total: 0.0
    Mote 0, Total energy used: 59089
    Mote 0, Battery energy remaining (linear): 21540911
    Mote 0, Battery energy remaining: 21538578
    Mote 0, Battery mAh remaining: 1994.3


    Mote 1, cpu total: 0.9
    Mote 1, radio total: 34057.7
    Mote 1, adc total: 0.0
    Mote 1, leds total: 6597.4
    Mote 1, sensor total: 0.0
    Mote 1, eeprom total: 0.0
    Mote 1, cpu_cycle total: 0.0
    Mote 1, Total energy used: 40656
    Mote 1, Battery energy remaining (linear): 21559344
    Mote 1, Battery energy remaining: 21559505
    Mote 1, Battery mAh remaining: 1996.3


    Mote 2, cpu total: 0.9
    Mote 2, radio total: 59088.4
    Mote 2, adc total: 0.0
    Mote 2, leds total: 0.0
    Mote 2, sensor total: 0.0
    Mote 2, eeprom total: 0.0
    Mote 2, cpu_cycle total: 0.0
    Mote 2, Total energy used: 59089
    Mote 2, Battery energy remaining (linear): 21540911
    Mote 2, Battery energy remaining: 21538578
    Mote 2, Battery mAh remaining: 1994.3


    Mote 3, cpu total: 0.9
    Mote 3, radio total: 59088.3
    Mote 3, adc total: 0.0
    Mote 3, leds total: 0.0
    Mote 3, sensor total: 0.0
    Mote 3, eeprom total: 0.0
    Mote 3, cpu_cycle total: 0.0
    Mote 3, Total energy used: 59089
    Mote 3, Battery energy remaining (linear): 21540911
    Mote 3, Battery energy remaining: 21538578
    Mote 3, Battery mAh remaining: 1994.3


    Mote 4, cpu total: 0.9
    Mote 4, radio total: 59088.3
    Mote 4, adc total: 0.0
    Mote 4, leds total: 0.0
    Mote 4, sensor total: 0.0
    Mote 4, eeprom total: 0.0
    Mote 4, cpu_cycle total: 0.0
    Mote 4, Total energy used: 59089
    Mote 4, Battery energy remaining (linear): 21540911
    Mote 4, Battery energy remaining: 21538578
    Mote 4, Battery mAh remaining: 1994.3


    Mote 5, cpu total: 0.9
    Mote 5, radio total: 59088.3
    Mote 5, adc total: 0.0
    Mote 5, leds total: 0.0
    Mote 5, sensor total: 0.0
    Mote 5, eeprom total: 0.0
    Mote 5, cpu_cycle total: 0.0
    Mote 5, Total energy used: 59089
    Mote 5, Battery energy remaining (linear): 21540911
    Mote 5, Battery energy remaining: 21538578
    Mote 5, Battery mAh remaining: 1994.3


    Mote 6, cpu total: 0.9
    Mote 6, radio total: 59088.3
    Mote 6, adc total: 0.0
    Mote 6, leds total: 0.0
    Mote 6, sensor total: 0.0
    Mote 6, eeprom total: 0.0
    Mote 6, cpu_cycle total: 0.0
    Mote 6, Total energy used: 59089
    Mote 6, Battery energy remaining (linear): 21540911
    Mote 6, Battery energy remaining: 21538578
    Mote 6, Battery mAh remaining: 1994.3


    Mote 7, cpu total: 0.9
    Mote 7, radio total: 59088.3
    Mote 7, adc total: 0.0
    Mote 7, leds total: 0.0
    Mote 7, sensor total: 0.0
    Mote 7, eeprom total: 0.0
    Mote 7, cpu_cycle total: 0.0
    Mote 7, Total energy used: 59089
    Mote 7, Battery energy remaining (linear): 21540911
    Mote 7, Battery energy remaining: 21538578
    Mote 7, Battery mAh remaining: 1994.3


    Mote 8, cpu total: 0.9
    Mote 8, radio total: 59088.3
    Mote 8, adc total: 0.0
    Mote 8, leds total: 0.0
    Mote 8, sensor total: 0.0
    Mote 8, eeprom total: 0.0
    Mote 8, cpu_cycle total: 0.0
    Mote 8, Total energy used: 59089
    Mote 8, Battery energy remaining (linear): 21540911
    Mote 8, Battery energy remaining: 21538578
    Mote 8, Battery mAh remaining: 1994.3


    Mote 9, cpu total: 0.9
    Mote 9, radio total: 59088.3
    Mote 9, adc total: 0.0
    Mote 9, leds total: 0.0
    Mote 9, sensor total: 0.0
    Mote 9, eeprom total: 0.0
    Mote 9, cpu_cycle total: 0.0
    Mote 9, Total energy used: 59089
    Mote 9, Battery energy remaining (linear): 21540911
    Mote 9, Battery energy remaining: 21538578
    Mote 9, Battery mAh remaining: 1994.3


    Mote 10, cpu total: 0.9
    Mote 10, radio total: 59088.3
    Mote 10, adc total: 0.0
    Mote 10, leds total: 0.0
    Mote 10, sensor total: 0.0
    Mote 10, eeprom total: 0.0
    Mote 10, cpu_cycle total: 0.0
    Mote 10, Total energy used: 59089
    Mote 10, Battery energy remaining (linear): 21540911
    Mote 10, Battery energy remaining: 21538578
    Mote 10, Battery mAh remaining: 1994.3


    Mote 11, cpu total: 0.9
    Mote 11, radio total: 59088.2
    Mote 11, adc total: 0.0
    Mote 11, leds total: 0.0
    Mote 11, sensor total: 0.0
    Mote 11, eeprom total: 0.0
    Mote 11, cpu_cycle total: 0.0
    Mote 11, Total energy used: 59089
    Mote 11, Battery energy remaining (linear): 21540911
    Mote 11, Battery energy remaining: 21538578
    Mote 11, Battery mAh remaining: 1994.3


    Simulated seconds: 999.8
    Real seconds: 14.4


Comments

  • Registered Users Posts: 7,468 ✭✭✭Evil Phil


    I would say Excel, you might have to write a Macro to extract the data though (should be easy enough really). Alternatively, if you wanted a web based solution you could use Google Charts or something similar, again you'll need to write something to extract the data.


  • Registered Users Posts: 40,038 ✭✭✭✭Sparks


    Probably something in a scripting language like python hooked up with a graphing library like reportlab (like so), since the file format's not as simple as a list of X and Y coordinates.


  • Closed Accounts Posts: 285 ✭✭Plebs


    Personally, I'd use Perl and gnuplot here.

    20 minutes and a couple of tweaks, you'll have lovely graphs.


  • Moderators, Science, Health & Environment Moderators, Social & Fun Moderators, Society & Culture Moderators Posts: 60,098 Mod ✭✭✭✭Tar.Aldarion


    Thanks a lot guys, I'll have a look at these. Mmmm pretty graphs.
    Got excel and bothered to boot into windows, so number one on the list, may the prettiest method win.


  • Registered Users Posts: 1,109 ✭✭✭Skrynesaver


    The following is a simple graph using Perl's GD::Graph, you can then play around with colour (spelled correctly for once ;) ), format (line, chart, bar, etc...), fonts, logos etc.. until you are delighted with the result ;)
    #!/usr/bin/perl
    
    use strict;
    use warnings;
    use GD::Graph;
    use GD::Graph::area;
    
    for my $data_file (@ARGV){
            my (@motes, @radio);
            my $x_max=0;
            my $y_max=0;
            open (my $data_fh, "<" , $data_file);
            while (<$data_fh>){
                    if (/^Mote (\d+), radio total: (\d+\.?\d?)$/){
                            push @motes, $1;
                            push @radio, $2;
                            $y_max=$2 if ($2 > $y_max);
                    }
            }
            my @data = (\@motes, \@radio, );
            my $graph = GD::Graph::area->new(1024, 768);
    
    
            $graph->set(
                 x_label           => 'Mote',
                 y_label           => 'Radio total',
                 title             => 'Radio total v\'s Mote',
                 y_max_value       => $y_max,
                 y_tick_number     => $y_max / 1000,
                 y_label_skip      => 2,
                 transparent       => 0,
            ) or die $graph->error;
    
    
            my $gd = $graph->plot(\@data) or die $graph->error;
            my $image_file = $data_file;
            $image_file=~ s/(.+)\.[^\.]+/$1.png/;
            open(my $img_fh, '>', $image_file) or die $!;
            binmode $img_fh;
            print $img_fh $gd->png;
            close $img_fh;
    }
    

    link to documentation of module


  • Advertisement
  • Moderators, Science, Health & Environment Moderators, Social & Fun Moderators, Society & Culture Moderators Posts: 60,098 Mod ✭✭✭✭Tar.Aldarion


    Not used perl before but great, thanks very much! Still haven't started making them, will be days yet, will have a look at yours!


  • Closed Accounts Posts: 1,397 ✭✭✭Herbal Deity


    I've found this perl script, which makes use of GNUPlot, to be very useful for this:
    http://burningcutlery.com/derek/bargraph/


  • Moderators, Science, Health & Environment Moderators, Social & Fun Moderators, Society & Culture Moderators Posts: 60,098 Mod ✭✭✭✭Tar.Aldarion


    Thanks a lot! Keep em coming guys! :D
    /simulates more data


  • Registered Users Posts: 413 ✭✭noxqs


    You might want to check out Python and Matplotlib

    http://matplotlib.sourceforge.net/

    Examples and code at:

    http://matplotlib.sourceforge.net/examples/index.html

    Its a package that can do virtually anything when it comes to plotting. And will serve you well if it is something you expect to do alot. Excel is fine, but not nearly as flexible to work with as Python when you have data which requires a bit of parsing to extract values from.


  • Registered Users Posts: 599 ✭✭✭ambasite


    kst - plots real time data from txt files: http://kst.kde.org/


  • Advertisement
  • Registered Users Posts: 297 ✭✭stesh


    noxqs wrote: »
    You might want to check out Python and Matplotlib

    http://matplotlib.sourceforge.net/

    Examples and code at:

    http://matplotlib.sourceforge.net/examples/index.html

    Its a package that can do virtually anything when it comes to plotting. And will serve you well if it is something you expect to do alot. Excel is fine, but not nearly as flexible to work with as Python when you have data which requires a bit of parsing to extract values from.

    This.

    Alternatively, take a look at R. It has decent graphing functions, and reads CSV with ease.


  • Moderators, Science, Health & Environment Moderators, Social & Fun Moderators, Society & Culture Moderators Posts: 60,098 Mod ✭✭✭✭Tar.Aldarion


    Thanks guys, will look. :)


  • Moderators, Science, Health & Environment Moderators, Social & Fun Moderators, Society & Culture Moderators Posts: 60,098 Mod ✭✭✭✭Tar.Aldarion


    The following is a simple graph using Perl's GD::Graph, you can then play around with colour (spelled correctly for once ;) ), format (line, chart, bar, etc...), fonts, logos etc.. until you are delighted with the result ;)
    #!/usr/bin/perl
    
    use strict;
    use warnings;
    use GD::Graph;
    use GD::Graph::area;
    
    for my $data_file (@ARGV){
            my (@motes, @radio);
            my $x_max=0;
            my $y_max=0;
            open (my $data_fh, "<" , $data_file);
            while (<$data_fh>){
                    if (/^Mote (\d+), radio total: (\d+\.?\d?)$/){
                            push @motes, $1;
                            push @radio, $2;
                            $y_max=$2 if ($2 > $y_max);
                    }
            }
            my @data = (\@motes, \@radio, );
            my $graph = GD::Graph::area->new(1024, 768);
    
    
            $graph->set(
                 x_label           => 'Mote',
                 y_label           => 'Radio total',
                 title             => 'Radio total v\'s Mote',
                 y_max_value       => $y_max,
                 y_tick_number     => $y_max / 1000,
                 y_label_skip      => 2,
                 transparent       => 0,
            ) or die $graph->error;
    
    
            my $gd = $graph->plot(\@data) or die $graph->error;
            my $image_file = $data_file;
            $image_file=~ s/(.+)\.[^\.]+/$1.png/;
            open(my $img_fh, '>', $image_file) or die $!;
            binmode $img_fh;
            print $img_fh $gd->png;
            close $img_fh;
    }
    

    link to documentation of module

    Hi was messing around with this and got

    DHV1000test1.png

    Is it easy just to change that so it is a line rather than have all that red under the line?

    Also messing up when i try to edit it and take in data from multiple files to make multiple lines. /back to editing


  • Registered Users Posts: 515 ✭✭✭NeverSayDie


    Hi was messing around with this and got



    Is it easy just to change that so it is a line rather than have all that red under the line?

    At a guess, I'd say this line
    "my $graph = GD::Graph::area->new(1024, 768);"
    is telling it to use an area graph and not the line graph you want it to. The docs for GD::Graph should have the rest.


  • Moderators, Science, Health & Environment Moderators, Social & Fun Moderators, Society & Culture Moderators Posts: 60,098 Mod ✭✭✭✭Tar.Aldarion


    Cool thanks I'll have a look, thought that just setting area of graph.


  • Registered Users Posts: 1,109 ✭✭✭Skrynesaver


    At a guess, I'd say this line
    "my $graph = GD::Graph::area->new(1024, 768);"
    is telling it to use an area graph and not the line graph you want it to. The docs for GD::Graph should have the rest.

    Correct.
    my $graph = GD::Graph::lines->new(1024, 768);
    
    Would define a line chart.
    gdgraph.com shows a range of charts and the code to generate them

    The original code was to graph one file specified on the command line, you wish to have a data structure such that there are an array of data points associated with each Mote ID?

    To do this try the following (WARNING: Haven't had any coffee yet and work box doesn't have GD::Graph installed :( , this may fail badly, if it does "use Data::Dumper; print Dumper \@data;" before writing graph)
    #!/usr/bin/perl
    
    use strict;
    use warnings;
    use GD::Graph;
    use GD::Graph::area;
    
    my (@motes, %radio, $count);
    my $x_max=0;
    my $y_max=0;
    $count=0;
    for my $data_file (@ARGV){
            open (my $data_fh, "<" , $data_file);
            while (<$data_fh>){
                    if (/^Mote (\d+), radio total: (\d+\.?\d?)$/){
                            push @motes, $1 if $count == 0; #Assumes all files have same Mote count
                            push @{$radio[$count]}, $2;
                            $y_max=$2 if ($2 > $y_max);
                    }
            }
    }
            my @data = (\@motes);
            push @data, values %radio;
            my $graph = GD::Graph::area->new(1024, 768);
    
    
            $graph->set(
                 x_label           => 'Mote',
                 y_label           => 'Radio total',
                 title             => 'Radio total v\'s Mote',
                 y_max_value       => $y_max,
                 y_tick_number     => $y_max / 1000,
                 y_label_skip      => 2,
                 transparent       => 0,
            ) or die $graph->error;
    
    
            my $gd = $graph->plot(\@data) or die $graph->error;
            my $image_file = $data_file;
            $image_file=~ s/(.+)\.[^\.]+/$1.png/;
            open(my $img_fh, '>', $image_file) or die $!;
            binmode $img_fh;
            print $img_fh $gd->png;
            close $img_fh;
    }
    

    ps: Welcome to Perl, it gets in on you quite quickly


  • Registered Users Posts: 1,922 ✭✭✭fergalr


    If you have to do much plotting or analysis, I'd really recommend taking a look at R.
    http://www.r-project.org/
    http://math.illinoisstate.edu/dhkim/rstuff/rtutor.html

    Its very very good at this sort of thing, there are a lot of libraries for different types of plotting and statistical analysis; its also free and widely available.
    The language itself is a bit of a mess, but the library support means you dont spend much time in the language.

    You could use Perl to do the initial extraction and tabulation of the data.
    But Perl is a bit manky. Python is usually better if you've to do anything complex... although the perl regex support is hard to beat.

    Speaking of python, matplotlib http://matplotlib.sourceforge.net/ is fairly awesome too.


  • Moderators, Science, Health & Environment Moderators, Social & Fun Moderators, Society & Culture Moderators Posts: 60,098 Mod ✭✭✭✭Tar.Aldarion


    Thanks guys! :)


  • Registered Users Posts: 7,157 ✭✭✭10-10-20


    I'm lovin' RRDTool at the moment.

    http://oss.oetiker.ch/rrdtool/

    It's a round-robin database with a simple output method.
    Code example here:
    http://owfs.org/index.php?page=garden

    log_soil_moisture_w_11_2009.png


Advertisement