Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

The perl module Spreadsheet::ParseExcel called, works too slowly !

Posted on 2013-01-23
7
Medium Priority
?
370 Views
Last Modified: 2013-03-21
Hi evrybody,

Tanks for your help
my problems are below :
1-/ I'm calling Spreadsheet::ParseExcel v0.59 module in my script. It seems to work very slowly and i dont know why.
2-/ How to change the field separator from "," to ";" for Spreadsheet::ParseExcel v0.59 module.

I'm joining the code and input file.
Here is the command : livraison_ENV.pl -a $INR_PRM -r INT -f S2ID2.BULL.Demandes_livraisons_TMA.xls :
-a : input file directory
-r : INT (Integration environment) ==> no change
-f : Name of input file

#!/usr/bin/perl

### Mise en place de library ###
#use lib '/opt/hrdev/bao/dev_bao/e-sda/exploit/sources';
use lib '/opt/hrdev/bao/utils/lib';
use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Getopt::Std;
#use Getopt::Simple;
use File::Copy;
use MIME::Lite;
use POSIX qw(strftime);

### Mise a jour variables ###
my @ENTITES;
my $oExcel;
my $ORDRE;
my $ENTITES;
my $LIGNE;
my $oBook;
my $CODES;
my $FEUILLE;
my $oWkS;
our @CODES;
our @LIGNE;


our @champs;
our @cles;
my $dir_prm;
my $param;
my $rapport;
my $par;
my $lignes;
my $cod;
our $fich_xls;
our $param_rep;
our $param_ent;
our $param_man;
our $status;
our %opts=();
our @LECTURE;
our @CODE;
our @CODE_M;
our $i;

@ARGV or die "Usage $0 FICHIER\n";

### Mise en place des parametres ###

getopts( "a:f:d:r:e:m:c:", \%opts );

        if (defined $opts{a}) { $dir_prm = $opts{a} } else { $dir_prm=""; }
        if (defined $opts{f}) { $fich_xls = $opts{f} } else { $fich_xls=""; }
        if (defined $opts{d}) { $param_rep = $opts{d} } else { $param_rep=""; }
        if (defined $opts{r}) { $rapport = $opts{r} } else { $rapport="no"; }
        if (defined $opts{e}) { $param_ent = $opts{e} } else { $param_ent=""; }
        if (defined $opts{m}) { $param_man = $opts{m} } else { $param_man=""; }
        if (defined $opts{c}) { $status = $opts{c} } else { $status=""; }

     open my $out_fh, '>>', $dir_prm."/param_liv_rep_OUT.txt" or die "impossible de creer 'param_liv_rep_OUT.txt' $!";

# ------------------------- FIN OPTIONS -------------------------------------------

#print {$out_fh} strftime("%Y%m%d%H%M\n", localtime);
my $temps = strftime("%Y%m%d%H%M\n", localtime);

# ------------------ SAISIE MANUELLE DES DONNEES ET CREATION DES CARTES PARAMETRES HRACCESS ------------------------

@cles = keys(%opts);
#-------------- CREATION DU RAPPORT DE FIN DE LIVRAISON -------------------------

if ( ( grep /r/,@cles ) && ( grep /f/,@cles ) ) {
        $rapport =~ tr [a-z] [A-Z];

        #==================== EXTRACTION DU FICHIER EXCEL =======================

        if (( $rapport =~ "INT" ) || ( $rapport =~ "INTEGRATION" )) {

                        ### Extraction du fichier excel ###

          chdir "$dir_prm/fic" or die "\n--- ERR : Impossible de se deplacer dans le repertoire fic ---\n";
          ( -f $fich_xls ) or die " \n--- ERR : Le fichier Excel n\'est pas present --- \n";
          print "\n --- Extraction des donnees du fichier excel ...\n";

          $oExcel = new Spreadsheet::ParseExcel;
          $oBook = $oExcel->Parse($fich_xls);

          foreach $oWkS (@{$oBook->{Worksheet}}) {
            $ORDRE ++;
            ($FEUILLE=$oWkS->{Name}) =~ s/ //g ;
            if ( grep /Entites/,$FEUILLE) { $ENTITES=(xls2csv( ${fich_xls}, "${ORDRE}-A3:W500"))};
#           if ( grep /Codes/,$FEUILLE) { $CODES=(xls2csv( ${fich_xls}, "${ORDRE}-A1:W500"))};
          }

          @ENTITES=split /\n/,$ENTITES;
#         @CODES=split /\n/,$CODES;

#         join(';', @ENTITES);

          for $LIGNE ( grep /prochaine.livraison/i,@ENTITES) {
            @LIGNE=split ',',$LIGNE;
            if (( grep /prochaine.livraison/i,$LIGNE[10] ) && ( ! $LIGNE[13] ) && ( ! $LIGNE[14] ) && ( ! $LIGNE[15] )) {
              print $LIGNE;
              print "\n";
            }
          }

          print "\nFin de la saisie.\n";
        }
}
S2ID2.BULL.Demandes-livraisons-T.xls
0
Comment
Question by:papfal
  • 4
  • 3
7 Comments
 

Author Comment

by:papfal
ID: 38813763
Hi,
When I try to send mail after the Spreadsheet module execution I get that error  :
open |/usr/lib/sendmail -t -oi -oem: Ne peut allouer de la mémoire (can't allow enough memory) at ./livraison_ENV.pl line 277.

I think that the module takes a lot of memory.

Thanks for some feedback

Seydina
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38814885
You should profile your script to see where it's spending its time.  With that info you can begin to refactor/optimize those sections.

Devel::NYTProf - Powerful fast feature-rich Perl source code profiler
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:papfal
ID: 38839352
Hello,

I installed the Devil-NYTProf module and execute it well.
Spreadsheet module consumes 99% of the execution time, actually.
Is there any tool which can refactor/optimze official modules ?
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38840034
Spreadsheet module consumes 99% of the execution time
That's pretty vague.

You need to determine which line(s) of code is being executed which is utilizing that amount time and how much time it takes for each execution of that statement.  Is the line being executed fairly quickly but a large number of times?  Is the statement making a bunch of system calls?

Taking a second look at your code, I'd say the first step might be to simply clean it up.  It's very messy.

Declaring all of your vars at the top is almost always a bad design.  You should declare your vars in the smallest scope that they require and as close as possible to where they are needed/used.  You should only use global vars when absolutely necessary and I don't see any need for them in this script.  So, change all of your 'our' declarations to 'my'.

Your initialization of your getops params (i.e., the if defined statements) is very kludgy.

Your use of grep on a scalar is odd and inefficient.  Use index() instead of grep().
0
 
LVL 28

Accepted Solution

by:
FishMonger earned 1500 total points
ID: 38841136
Here's a short test script that I ran which only parses the worksheet and outputs the results.  My test run only took a few seconds to complete.  Note that I reduced the number of rows to match what was actually being used in your posted example file.

#!/usr/bin/perl

use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Data::Dumper;

print scalar localtime, $/;

my $xls_file = 'S2ID2.BULL.Demandes-livraisons-T.xls';
my $xls2csv  = xls2csv($xls_file, '1-A3:W348');

print scalar localtime, $/;
print Dumper $xls2csv;

Open in new window

0
 

Author Comment

by:papfal
ID: 39006476
You're right,

My code is now running well. I've used the refactor module to identify the lines which were abnormally long.

I pulled out the pool parsing lines and simplify the code by translating all our vars on my vars ...

But when I make the sum of the second line of the ouput file form refactoring I get the double of the real execution time of the code. Is it right ?

Anyway this helped me to resolve my issues.

Thanks
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Six Sigma Control Plans

824 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question