Solved

The perl module Spreadsheet::ParseExcel called, works too slowly !

Posted on 2013-01-23
7
365 Views
Last Modified: 2013-03-21
Hi evrybody,

Tanks for your help
my problems are below :
1-/ I'm calling Spreadsheet::ParseExcel v0.59 module in my script. It seems to work very slowly and i dont know why.
2-/ How to change the field separator from "," to ";" for Spreadsheet::ParseExcel v0.59 module.

I'm joining the code and input file.
Here is the command : livraison_ENV.pl -a $INR_PRM -r INT -f S2ID2.BULL.Demandes_livraisons_TMA.xls :
-a : input file directory
-r : INT (Integration environment) ==> no change
-f : Name of input file

#!/usr/bin/perl

### Mise en place de library ###
#use lib '/opt/hrdev/bao/dev_bao/e-sda/exploit/sources';
use lib '/opt/hrdev/bao/utils/lib';
use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Getopt::Std;
#use Getopt::Simple;
use File::Copy;
use MIME::Lite;
use POSIX qw(strftime);

### Mise a jour variables ###
my @ENTITES;
my $oExcel;
my $ORDRE;
my $ENTITES;
my $LIGNE;
my $oBook;
my $CODES;
my $FEUILLE;
my $oWkS;
our @CODES;
our @LIGNE;


our @champs;
our @cles;
my $dir_prm;
my $param;
my $rapport;
my $par;
my $lignes;
my $cod;
our $fich_xls;
our $param_rep;
our $param_ent;
our $param_man;
our $status;
our %opts=();
our @LECTURE;
our @CODE;
our @CODE_M;
our $i;

@ARGV or die "Usage $0 FICHIER\n";

### Mise en place des parametres ###

getopts( "a:f:d:r:e:m:c:", \%opts );

        if (defined $opts{a}) { $dir_prm = $opts{a} } else { $dir_prm=""; }
        if (defined $opts{f}) { $fich_xls = $opts{f} } else { $fich_xls=""; }
        if (defined $opts{d}) { $param_rep = $opts{d} } else { $param_rep=""; }
        if (defined $opts{r}) { $rapport = $opts{r} } else { $rapport="no"; }
        if (defined $opts{e}) { $param_ent = $opts{e} } else { $param_ent=""; }
        if (defined $opts{m}) { $param_man = $opts{m} } else { $param_man=""; }
        if (defined $opts{c}) { $status = $opts{c} } else { $status=""; }

     open my $out_fh, '>>', $dir_prm."/param_liv_rep_OUT.txt" or die "impossible de creer 'param_liv_rep_OUT.txt' $!";

# ------------------------- FIN OPTIONS -------------------------------------------

#print {$out_fh} strftime("%Y%m%d%H%M\n", localtime);
my $temps = strftime("%Y%m%d%H%M\n", localtime);

# ------------------ SAISIE MANUELLE DES DONNEES ET CREATION DES CARTES PARAMETRES HRACCESS ------------------------

@cles = keys(%opts);
#-------------- CREATION DU RAPPORT DE FIN DE LIVRAISON -------------------------

if ( ( grep /r/,@cles ) && ( grep /f/,@cles ) ) {
        $rapport =~ tr [a-z] [A-Z];

        #==================== EXTRACTION DU FICHIER EXCEL =======================

        if (( $rapport =~ "INT" ) || ( $rapport =~ "INTEGRATION" )) {

                        ### Extraction du fichier excel ###

          chdir "$dir_prm/fic" or die "\n--- ERR : Impossible de se deplacer dans le repertoire fic ---\n";
          ( -f $fich_xls ) or die " \n--- ERR : Le fichier Excel n\'est pas present --- \n";
          print "\n --- Extraction des donnees du fichier excel ...\n";

          $oExcel = new Spreadsheet::ParseExcel;
          $oBook = $oExcel->Parse($fich_xls);

          foreach $oWkS (@{$oBook->{Worksheet}}) {
            $ORDRE ++;
            ($FEUILLE=$oWkS->{Name}) =~ s/ //g ;
            if ( grep /Entites/,$FEUILLE) { $ENTITES=(xls2csv( ${fich_xls}, "${ORDRE}-A3:W500"))};
#           if ( grep /Codes/,$FEUILLE) { $CODES=(xls2csv( ${fich_xls}, "${ORDRE}-A1:W500"))};
          }

          @ENTITES=split /\n/,$ENTITES;
#         @CODES=split /\n/,$CODES;

#         join(';', @ENTITES);

          for $LIGNE ( grep /prochaine.livraison/i,@ENTITES) {
            @LIGNE=split ',',$LIGNE;
            if (( grep /prochaine.livraison/i,$LIGNE[10] ) && ( ! $LIGNE[13] ) && ( ! $LIGNE[14] ) && ( ! $LIGNE[15] )) {
              print $LIGNE;
              print "\n";
            }
          }

          print "\nFin de la saisie.\n";
        }
}
S2ID2.BULL.Demandes-livraisons-T.xls
0
Comment
Question by:papfal
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
7 Comments
 

Author Comment

by:papfal
ID: 38813763
Hi,
When I try to send mail after the Spreadsheet module execution I get that error  :
open |/usr/lib/sendmail -t -oi -oem: Ne peut allouer de la mémoire (can't allow enough memory) at ./livraison_ENV.pl line 277.

I think that the module takes a lot of memory.

Thanks for some feedback

Seydina
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38814885
You should profile your script to see where it's spending its time.  With that info you can begin to refactor/optimize those sections.

Devel::NYTProf - Powerful fast feature-rich Perl source code profiler
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38814891
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:papfal
ID: 38839352
Hello,

I installed the Devil-NYTProf module and execute it well.
Spreadsheet module consumes 99% of the execution time, actually.
Is there any tool which can refactor/optimze official modules ?
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38840034
Spreadsheet module consumes 99% of the execution time
That's pretty vague.

You need to determine which line(s) of code is being executed which is utilizing that amount time and how much time it takes for each execution of that statement.  Is the line being executed fairly quickly but a large number of times?  Is the statement making a bunch of system calls?

Taking a second look at your code, I'd say the first step might be to simply clean it up.  It's very messy.

Declaring all of your vars at the top is almost always a bad design.  You should declare your vars in the smallest scope that they require and as close as possible to where they are needed/used.  You should only use global vars when absolutely necessary and I don't see any need for them in this script.  So, change all of your 'our' declarations to 'my'.

Your initialization of your getops params (i.e., the if defined statements) is very kludgy.

Your use of grep on a scalar is odd and inefficient.  Use index() instead of grep().
0
 
LVL 28

Accepted Solution

by:
FishMonger earned 500 total points
ID: 38841136
Here's a short test script that I ran which only parses the worksheet and outputs the results.  My test run only took a few seconds to complete.  Note that I reduced the number of rows to match what was actually being used in your posted example file.

#!/usr/bin/perl

use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Data::Dumper;

print scalar localtime, $/;

my $xls_file = 'S2ID2.BULL.Demandes-livraisons-T.xls';
my $xls2csv  = xls2csv($xls_file, '1-A3:W348');

print scalar localtime, $/;
print Dumper $xls2csv;

Open in new window

0
 

Author Comment

by:papfal
ID: 39006476
You're right,

My code is now running well. I've used the refactor module to identify the lines which were abnormally long.

I pulled out the pool parsing lines and simplify the code by translating all our vars on my vars ...

But when I make the sum of the second line of the ouput file form refactoring I get the double of the real execution time of the code. Is it right ?

Anyway this helped me to resolve my issues.

Thanks
0

Featured Post

On Demand Webinar - Networking for the Cloud Era

This webinar discusses:
-Common barriers companies experience when moving to the cloud
-How SD-WAN changes the way we look at networks
-Best practices customers should employ moving forward with cloud migration
-What happens behind the scenes of SteelConnect’s one-click button

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question