Link to home
Start Free TrialLog in
Avatar of Seydina Fall
Seydina Fall

asked on

The perl module Spreadsheet::ParseExcel called, works too slowly !

Hi evrybody,

Tanks for your help
my problems are below :
1-/ I'm calling Spreadsheet::ParseExcel v0.59 module in my script. It seems to work very slowly and i dont know why.
2-/ How to change the field separator from "," to ";" for Spreadsheet::ParseExcel v0.59 module.

I'm joining the code and input file.
Here is the command : livraison_ENV.pl -a $INR_PRM -r INT -f S2ID2.BULL.Demandes_livraisons_TMA.xls :
-a : input file directory
-r : INT (Integration environment) ==> no change
-f : Name of input file

#!/usr/bin/perl

### Mise en place de library ###
#use lib '/opt/hrdev/bao/dev_bao/e-sda/exploit/sources';
use lib '/opt/hrdev/bao/utils/lib';
use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Getopt::Std;
#use Getopt::Simple;
use File::Copy;
use MIME::Lite;
use POSIX qw(strftime);

### Mise a jour variables ###
my @ENTITES;
my $oExcel;
my $ORDRE;
my $ENTITES;
my $LIGNE;
my $oBook;
my $CODES;
my $FEUILLE;
my $oWkS;
our @CODES;
our @LIGNE;


our @champs;
our @cles;
my $dir_prm;
my $param;
my $rapport;
my $par;
my $lignes;
my $cod;
our $fich_xls;
our $param_rep;
our $param_ent;
our $param_man;
our $status;
our %opts=();
our @LECTURE;
our @CODE;
our @CODE_M;
our $i;

@ARGV or die "Usage $0 FICHIER\n";

### Mise en place des parametres ###

getopts( "a:f:d:r:e:m:c:", \%opts );

        if (defined $opts{a}) { $dir_prm = $opts{a} } else { $dir_prm=""; }
        if (defined $opts{f}) { $fich_xls = $opts{f} } else { $fich_xls=""; }
        if (defined $opts{d}) { $param_rep = $opts{d} } else { $param_rep=""; }
        if (defined $opts{r}) { $rapport = $opts{r} } else { $rapport="no"; }
        if (defined $opts{e}) { $param_ent = $opts{e} } else { $param_ent=""; }
        if (defined $opts{m}) { $param_man = $opts{m} } else { $param_man=""; }
        if (defined $opts{c}) { $status = $opts{c} } else { $status=""; }

     open my $out_fh, '>>', $dir_prm."/param_liv_rep_OUT.txt" or die "impossible de creer 'param_liv_rep_OUT.txt' $!";

# ------------------------- FIN OPTIONS -------------------------------------------

#print {$out_fh} strftime("%Y%m%d%H%M\n", localtime);
my $temps = strftime("%Y%m%d%H%M\n", localtime);

# ------------------ SAISIE MANUELLE DES DONNEES ET CREATION DES CARTES PARAMETRES HRACCESS ------------------------

@cles = keys(%opts);
#-------------- CREATION DU RAPPORT DE FIN DE LIVRAISON -------------------------

if ( ( grep /r/,@cles ) && ( grep /f/,@cles ) ) {
        $rapport =~ tr [a-z] [A-Z];

        #==================== EXTRACTION DU FICHIER EXCEL =======================

        if (( $rapport =~ "INT" ) || ( $rapport =~ "INTEGRATION" )) {

                        ### Extraction du fichier excel ###

          chdir "$dir_prm/fic" or die "\n--- ERR : Impossible de se deplacer dans le repertoire fic ---\n";
          ( -f $fich_xls ) or die " \n--- ERR : Le fichier Excel n\'est pas present --- \n";
          print "\n --- Extraction des donnees du fichier excel ...\n";

          $oExcel = new Spreadsheet::ParseExcel;
          $oBook = $oExcel->Parse($fich_xls);

          foreach $oWkS (@{$oBook->{Worksheet}}) {
            $ORDRE ++;
            ($FEUILLE=$oWkS->{Name}) =~ s/ //g ;
            if ( grep /Entites/,$FEUILLE) { $ENTITES=(xls2csv( ${fich_xls}, "${ORDRE}-A3:W500"))};
#           if ( grep /Codes/,$FEUILLE) { $CODES=(xls2csv( ${fich_xls}, "${ORDRE}-A1:W500"))};
          }

          @ENTITES=split /\n/,$ENTITES;
#         @CODES=split /\n/,$CODES;

#         join(';', @ENTITES);

          for $LIGNE ( grep /prochaine.livraison/i,@ENTITES) {
            @LIGNE=split ',',$LIGNE;
            if (( grep /prochaine.livraison/i,$LIGNE[10] ) && ( ! $LIGNE[13] ) && ( ! $LIGNE[14] ) && ( ! $LIGNE[15] )) {
              print $LIGNE;
              print "\n";
            }
          }

          print "\nFin de la saisie.\n";
        }
}
S2ID2.BULL.Demandes-livraisons-T.xls
Avatar of Seydina Fall
Seydina Fall

ASKER

Hi,
When I try to send mail after the Spreadsheet module execution I get that error  :
open |/usr/lib/sendmail -t -oi -oem: Ne peut allouer de la mémoire (can't allow enough memory) at ./livraison_ENV.pl line 277.

I think that the module takes a lot of memory.

Thanks for some feedback

Seydina
Avatar of FishMonger
You should profile your script to see where it's spending its time.  With that info you can begin to refactor/optimize those sections.

Devel::NYTProf - Powerful fast feature-rich Perl source code profiler
Hello,

I installed the Devil-NYTProf module and execute it well.
Spreadsheet module consumes 99% of the execution time, actually.
Is there any tool which can refactor/optimze official modules ?
Spreadsheet module consumes 99% of the execution time
That's pretty vague.

You need to determine which line(s) of code is being executed which is utilizing that amount time and how much time it takes for each execution of that statement.  Is the line being executed fairly quickly but a large number of times?  Is the statement making a bunch of system calls?

Taking a second look at your code, I'd say the first step might be to simply clean it up.  It's very messy.

Declaring all of your vars at the top is almost always a bad design.  You should declare your vars in the smallest scope that they require and as close as possible to where they are needed/used.  You should only use global vars when absolutely necessary and I don't see any need for them in this script.  So, change all of your 'our' declarations to 'my'.

Your initialization of your getops params (i.e., the if defined statements) is very kludgy.

Your use of grep on a scalar is odd and inefficient.  Use index() instead of grep().
ASKER CERTIFIED SOLUTION
Avatar of FishMonger
FishMonger
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
You're right,

My code is now running well. I've used the refactor module to identify the lines which were abnormally long.

I pulled out the pool parsing lines and simplify the code by translating all our vars on my vars ...

But when I make the sum of the second line of the ouput file form refactoring I get the double of the real execution time of the code. Is it right ?

Anyway this helped me to resolve my issues.

Thanks