Solved

The perl module Spreadsheet::ParseExcel called, works too slowly !

Posted on 2013-01-23
7
361 Views
Last Modified: 2013-03-21
Hi evrybody,

Tanks for your help
my problems are below :
1-/ I'm calling Spreadsheet::ParseExcel v0.59 module in my script. It seems to work very slowly and i dont know why.
2-/ How to change the field separator from "," to ";" for Spreadsheet::ParseExcel v0.59 module.

I'm joining the code and input file.
Here is the command : livraison_ENV.pl -a $INR_PRM -r INT -f S2ID2.BULL.Demandes_livraisons_TMA.xls :
-a : input file directory
-r : INT (Integration environment) ==> no change
-f : Name of input file

#!/usr/bin/perl

### Mise en place de library ###
#use lib '/opt/hrdev/bao/dev_bao/e-sda/exploit/sources';
use lib '/opt/hrdev/bao/utils/lib';
use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Getopt::Std;
#use Getopt::Simple;
use File::Copy;
use MIME::Lite;
use POSIX qw(strftime);

### Mise a jour variables ###
my @ENTITES;
my $oExcel;
my $ORDRE;
my $ENTITES;
my $LIGNE;
my $oBook;
my $CODES;
my $FEUILLE;
my $oWkS;
our @CODES;
our @LIGNE;


our @champs;
our @cles;
my $dir_prm;
my $param;
my $rapport;
my $par;
my $lignes;
my $cod;
our $fich_xls;
our $param_rep;
our $param_ent;
our $param_man;
our $status;
our %opts=();
our @LECTURE;
our @CODE;
our @CODE_M;
our $i;

@ARGV or die "Usage $0 FICHIER\n";

### Mise en place des parametres ###

getopts( "a:f:d:r:e:m:c:", \%opts );

        if (defined $opts{a}) { $dir_prm = $opts{a} } else { $dir_prm=""; }
        if (defined $opts{f}) { $fich_xls = $opts{f} } else { $fich_xls=""; }
        if (defined $opts{d}) { $param_rep = $opts{d} } else { $param_rep=""; }
        if (defined $opts{r}) { $rapport = $opts{r} } else { $rapport="no"; }
        if (defined $opts{e}) { $param_ent = $opts{e} } else { $param_ent=""; }
        if (defined $opts{m}) { $param_man = $opts{m} } else { $param_man=""; }
        if (defined $opts{c}) { $status = $opts{c} } else { $status=""; }

     open my $out_fh, '>>', $dir_prm."/param_liv_rep_OUT.txt" or die "impossible de creer 'param_liv_rep_OUT.txt' $!";

# ------------------------- FIN OPTIONS -------------------------------------------

#print {$out_fh} strftime("%Y%m%d%H%M\n", localtime);
my $temps = strftime("%Y%m%d%H%M\n", localtime);

# ------------------ SAISIE MANUELLE DES DONNEES ET CREATION DES CARTES PARAMETRES HRACCESS ------------------------

@cles = keys(%opts);
#-------------- CREATION DU RAPPORT DE FIN DE LIVRAISON -------------------------

if ( ( grep /r/,@cles ) && ( grep /f/,@cles ) ) {
        $rapport =~ tr [a-z] [A-Z];

        #==================== EXTRACTION DU FICHIER EXCEL =======================

        if (( $rapport =~ "INT" ) || ( $rapport =~ "INTEGRATION" )) {

                        ### Extraction du fichier excel ###

          chdir "$dir_prm/fic" or die "\n--- ERR : Impossible de se deplacer dans le repertoire fic ---\n";
          ( -f $fich_xls ) or die " \n--- ERR : Le fichier Excel n\'est pas present --- \n";
          print "\n --- Extraction des donnees du fichier excel ...\n";

          $oExcel = new Spreadsheet::ParseExcel;
          $oBook = $oExcel->Parse($fich_xls);

          foreach $oWkS (@{$oBook->{Worksheet}}) {
            $ORDRE ++;
            ($FEUILLE=$oWkS->{Name}) =~ s/ //g ;
            if ( grep /Entites/,$FEUILLE) { $ENTITES=(xls2csv( ${fich_xls}, "${ORDRE}-A3:W500"))};
#           if ( grep /Codes/,$FEUILLE) { $CODES=(xls2csv( ${fich_xls}, "${ORDRE}-A1:W500"))};
          }

          @ENTITES=split /\n/,$ENTITES;
#         @CODES=split /\n/,$CODES;

#         join(';', @ENTITES);

          for $LIGNE ( grep /prochaine.livraison/i,@ENTITES) {
            @LIGNE=split ',',$LIGNE;
            if (( grep /prochaine.livraison/i,$LIGNE[10] ) && ( ! $LIGNE[13] ) && ( ! $LIGNE[14] ) && ( ! $LIGNE[15] )) {
              print $LIGNE;
              print "\n";
            }
          }

          print "\nFin de la saisie.\n";
        }
}
S2ID2.BULL.Demandes-livraisons-T.xls
0
Comment
Question by:papfal
  • 4
  • 3
7 Comments
 

Author Comment

by:papfal
ID: 38813763
Hi,
When I try to send mail after the Spreadsheet module execution I get that error  :
open |/usr/lib/sendmail -t -oi -oem: Ne peut allouer de la mémoire (can't allow enough memory) at ./livraison_ENV.pl line 277.

I think that the module takes a lot of memory.

Thanks for some feedback

Seydina
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38814885
You should profile your script to see where it's spending its time.  With that info you can begin to refactor/optimize those sections.

Devel::NYTProf - Powerful fast feature-rich Perl source code profiler
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38814891
0
Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

 

Author Comment

by:papfal
ID: 38839352
Hello,

I installed the Devil-NYTProf module and execute it well.
Spreadsheet module consumes 99% of the execution time, actually.
Is there any tool which can refactor/optimze official modules ?
0
 
LVL 28

Expert Comment

by:FishMonger
ID: 38840034
Spreadsheet module consumes 99% of the execution time
That's pretty vague.

You need to determine which line(s) of code is being executed which is utilizing that amount time and how much time it takes for each execution of that statement.  Is the line being executed fairly quickly but a large number of times?  Is the statement making a bunch of system calls?

Taking a second look at your code, I'd say the first step might be to simply clean it up.  It's very messy.

Declaring all of your vars at the top is almost always a bad design.  You should declare your vars in the smallest scope that they require and as close as possible to where they are needed/used.  You should only use global vars when absolutely necessary and I don't see any need for them in this script.  So, change all of your 'our' declarations to 'my'.

Your initialization of your getops params (i.e., the if defined statements) is very kludgy.

Your use of grep on a scalar is odd and inefficient.  Use index() instead of grep().
0
 
LVL 28

Accepted Solution

by:
FishMonger earned 500 total points
ID: 38841136
Here's a short test script that I ran which only parses the worksheet and outputs the results.  My test run only took a few seconds to complete.  Note that I reduced the number of rows to match what was actually being used in your posted example file.

#!/usr/bin/perl

use warnings;
use strict;
use Spreadsheet::ParseExcel;
use Spreadsheet::ParseExcel::Utility qw(xls2csv);
use Data::Dumper;

print scalar localtime, $/;

my $xls_file = 'S2ID2.BULL.Demandes-livraisons-T.xls';
my $xls2csv  = xls2csv($xls_file, '1-A3:W348');

print scalar localtime, $/;
print Dumper $xls2csv;

Open in new window

0
 

Author Comment

by:papfal
ID: 39006476
You're right,

My code is now running well. I've used the refactor module to identify the lines which were abnormally long.

I pulled out the pool parsing lines and simplify the code by translating all our vars on my vars ...

But when I make the sum of the second line of the ouput file form refactoring I get the double of the real execution time of the code. Is it right ?

Anyway this helped me to resolve my issues.

Thanks
0

Featured Post

Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Need Helping Editing Perl Script for PowerChute Network Shutdown 3 82
iSeries PERL Scripts 7 157
batch script for automated email 12 109
Perl modules on linux ec2 3 104
Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…

786 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question