Solved

The perl module Spreadsheet::ParseExcel called, works too slowly !

Posted on 2013-01-23
7
356 Views
Last Modified: 2013-03-21
Hi evrybody,

Tanks for your help
my problems are below :
1-/ I'm calling Spreadsheet::ParseExcel v0.59 module in my script. It seems to work very slowly and i dont know why.
2-/ How to change the field separator from "," to ";" for Spreadsheet::ParseExcel v0.59 module.

I'm joining the code and input file.
Here is the command : livraison_ENV.pl -a $INR_PRM -r INT -f S2ID2.BULL.Demandes_livraisons_TMA.xls :
-a : input file directory
-r : INT (Integration environment) ==> no change
-f : Name of input file

#!/usr/bin/perl

### Mise en place de library ###
#use lib '/opt/hrdev/bao/dev_bao/e-sda/exploit/sources';
use lib '/opt/hrdev/bao/utils/lib';
use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Getopt::Std;
#use Getopt::Simple;
use File::Copy;
use MIME::Lite;
use POSIX qw(strftime);

### Mise a jour variables ###
my @ENTITES;
my $oExcel;
my $ORDRE;
my $ENTITES;
my $LIGNE;
my $oBook;
my $CODES;
my $FEUILLE;
my $oWkS;
our @CODES;
our @LIGNE;


our @champs;
our @cles;
my $dir_prm;
my $param;
my $rapport;
my $par;
my $lignes;
my $cod;
our $fich_xls;
our $param_rep;
our $param_ent;
our $param_man;
our $status;
our %opts=();
our @LECTURE;
our @CODE;
our @CODE_M;
our $i;

@ARGV or die "Usage $0 FICHIER\n";

### Mise en place des parametres ###

getopts( "a:f:d:r:e:m:c:", \%opts );

        if (defined $opts{a}) { $dir_prm = $opts{a} } else { $dir_prm=""; }
        if (defined $opts{f}) { $fich_xls = $opts{f} } else { $fich_xls=""; }
        if (defined $opts{d}) { $param_rep = $opts{d} } else { $param_rep=""; }
        if (defined $opts{r}) { $rapport = $opts{r} } else { $rapport="no"; }
        if (defined $opts{e}) { $param_ent = $opts{e} } else { $param_ent=""; }
        if (defined $opts{m}) { $param_man = $opts{m} } else { $param_man=""; }
        if (defined $opts{c}) { $status = $opts{c} } else { $status=""; }

     open my $out_fh, '>>', $dir_prm."/param_liv_rep_OUT.txt" or die "impossible de creer 'param_liv_rep_OUT.txt' $!";

# ------------------------- FIN OPTIONS -------------------------------------------

#print {$out_fh} strftime("%Y%m%d%H%M\n", localtime);
my $temps = strftime("%Y%m%d%H%M\n", localtime);

# ------------------ SAISIE MANUELLE DES DONNEES ET CREATION DES CARTES PARAMETRES HRACCESS ------------------------

@cles = keys(%opts);
#-------------- CREATION DU RAPPORT DE FIN DE LIVRAISON -------------------------

if ( ( grep /r/,@cles ) && ( grep /f/,@cles ) ) {
        $rapport =~ tr [a-z] [A-Z];

        #==================== EXTRACTION DU FICHIER EXCEL =======================

        if (( $rapport =~ "INT" ) || ( $rapport =~ "INTEGRATION" )) {

                        ### Extraction du fichier excel ###

          chdir "$dir_prm/fic" or die "\n--- ERR : Impossible de se deplacer dans le repertoire fic ---\n";
          ( -f $fich_xls ) or die " \n--- ERR : Le fichier Excel n\'est pas present --- \n";
          print "\n --- Extraction des donnees du fichier excel ...\n";

          $oExcel = new Spreadsheet::ParseExcel;
          $oBook = $oExcel->Parse($fich_xls);

          foreach $oWkS (@{$oBook->{Worksheet}}) {
            $ORDRE ++;
            ($FEUILLE=$oWkS->{Name}) =~ s/ //g ;
            if ( grep /Entites/,$FEUILLE) { $ENTITES=(xls2csv( ${fich_xls}, "${ORDRE}-A3:W500"))};
#           if ( grep /Codes/,$FEUILLE) { $CODES=(xls2csv( ${fich_xls}, "${ORDRE}-A1:W500"))};
          }

          @ENTITES=split /\n/,$ENTITES;
#         @CODES=split /\n/,$CODES;

#         join(';', @ENTITES);

          for $LIGNE ( grep /prochaine.livraison/i,@ENTITES) {
            @LIGNE=split ',',$LIGNE;
            if (( grep /prochaine.livraison/i,$LIGNE[10] ) && ( ! $LIGNE[13] ) && ( ! $LIGNE[14] ) && ( ! $LIGNE[15] )) {
              print $LIGNE;
              print "\n";
            }
          }

          print "\nFin de la saisie.\n";
        }
}
S2ID2.BULL.Demandes-livraisons-T.xls
0
Comment
Question by:papfal
  • 4
  • 3
7 Comments
 

Author Comment

by:papfal
ID: 38813763
Hi,
When I try to send mail after the Spreadsheet module execution I get that error  :
open |/usr/lib/sendmail -t -oi -oem: Ne peut allouer de la mémoire (can't allow enough memory) at ./livraison_ENV.pl line 277.

I think that the module takes a lot of memory.

Thanks for some feedback

Seydina
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38814885
You should profile your script to see where it's spending its time.  With that info you can begin to refactor/optimize those sections.

Devel::NYTProf - Powerful fast feature-rich Perl source code profiler
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38814891
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:papfal
ID: 38839352
Hello,

I installed the Devil-NYTProf module and execute it well.
Spreadsheet module consumes 99% of the execution time, actually.
Is there any tool which can refactor/optimze official modules ?
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38840034
Spreadsheet module consumes 99% of the execution time
That's pretty vague.

You need to determine which line(s) of code is being executed which is utilizing that amount time and how much time it takes for each execution of that statement.  Is the line being executed fairly quickly but a large number of times?  Is the statement making a bunch of system calls?

Taking a second look at your code, I'd say the first step might be to simply clean it up.  It's very messy.

Declaring all of your vars at the top is almost always a bad design.  You should declare your vars in the smallest scope that they require and as close as possible to where they are needed/used.  You should only use global vars when absolutely necessary and I don't see any need for them in this script.  So, change all of your 'our' declarations to 'my'.

Your initialization of your getops params (i.e., the if defined statements) is very kludgy.

Your use of grep on a scalar is odd and inefficient.  Use index() instead of grep().
0
 
LVL 28

Accepted Solution

by:
FishMonger earned 500 total points
ID: 38841136
Here's a short test script that I ran which only parses the worksheet and outputs the results.  My test run only took a few seconds to complete.  Note that I reduced the number of rows to match what was actually being used in your posted example file.

#!/usr/bin/perl

use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Data::Dumper;

print scalar localtime, $/;

my $xls_file = 'S2ID2.BULL.Demandes-livraisons-T.xls';
my $xls2csv  = xls2csv($xls_file, '1-A3:W348');

print scalar localtime, $/;
print Dumper $xls2csv;

Open in new window

0
 

Author Comment

by:papfal
ID: 39006476
You're right,

My code is now running well. I've used the refactor module to identify the lines which were abnormally long.

I pulled out the pool parsing lines and simplify the code by translating all our vars on my vars ...

But when I make the sum of the second line of the ouput file form refactoring I get the double of the real execution time of the code. Is it right ?

Anyway this helped me to resolve my issues.

Thanks
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

On Microsoft Windows, if  when you click or type the name of a .pl file, you get an error "is not recognized as an internal or external command, operable program or batch file", then this means you do not have the .pl file extension associated with …
A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now