speed up slow Perl code with SQL statements

Posted on 2007-03-30
Last Modified: 2010-03-05
This is the guts of the code that I have written, however, when displayed in Firefox it takes a while to display. This seems to be down to the SQL hits i think, is there a way to optimize it as best as possible? I tried a JOIN which works, but is there any other ways to help speed it up?

I feel the speed is slowing at the @filescursor query.

I'm using use strict;

@groupscursor = $db->query("select distinct(filename),groups.receiverid,groups.filelimit,groups.theowner,,,receivers.extension from hols.groups left join hols.receivers on groups.receiverid = order by theowner asc, receiverid asc");

if ($groupscursor[1] ne "0E0") {
      while ($groupsreturned = $groupscursor[0]->fetchrow_hashref) {
            $receiverid            = $groupsreturned->{receiverid};
            $filename            = $groupsreturned->{filename};
            $theowner            = $groupsreturned->{theowner};
            $receivername      = $groupsreturned->{name};
            $extension            = $groupsreturned->{extension};      

            next if ($receivername eq $tmpreceiver and ($filename.".".$extension) eq $tmpfile);

            @filescursor = $db->query("select count(id) as results,id,filename,completed,max(created) as datecreated from picco.files where theowner = '$theowner' and filename = '$filename' group by filename");            

            if ($filescursor[1] ne "0E0") {
                  while ($filesreturned = $filescursor[0]->fetchrow_hashref) {
                        $fileid            = $filesreturned->{id};
                        $created      = $filesreturned->{datecreated};
                        $completed      = $filesreturned->{completed};

                        $total += $filesreturned->{results};

                        ($offers) = @{($db->query("SELECT count(id) as offers FROM picco.offers WHERE fileid = '$fileid'"))[0]->fetchrow_arrayref};

                        if ($theowner ne $tmpsite) {                  
                              push (@list, ["", qq|<b>| . $theowner . qq|</b>|, "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;"]);
                              $total += 1;
                        $tmpsite = $theowner;
                        $colour = "black";

                        if ($completed eq "N") {
                              $colour = "red";

                        push (@list,      [
                                                      qq|<font color='$colour'>| . $receivername . qq|</font>|,
                                                      qq|<font color='$colour'>| . ($filename .".". $extension) . qq|</font>|,
                                                      qq|<font color='$colour'>| . $groupsreturned->{filelimit} . qq|</font>|,
                                                      qq|<font color='$colour'>| . $groupsreturned->{active} . qq|</font>|,
                                                      qq|<font color='$colour'>| . $created . qq|</font>|,
                                                      qq|<font color='$colour'>| . $completed . qq|</font>|,
                                                      qq|<font color='$colour'>| . $offers . qq|</font>|
                        $tmpfile = ($filename.".".$extension);
                        $tmpreceiver = $receivername;
            @filescursor = undef;
@groupscursor = undef;
Question by:crmpicco
1 Comment

Accepted Solution

mzalfres earned 125 total points
ID: 18881503
I believe your problem lies not in the PERL code. Try to compare 'clean' SQL query run with your script. Run the same query using your database command line interface and see whether it works significantly faster. I'd rather look at database and try eg. add indexes for fields which are used as a search / join keys. (in this case - 'filename' field.)


Marek ZJ.

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
work on ods spreadsheet with perl in ubuntu 4 75
PERL - Find newest folder 12 118
Vb script to unzip a files and rename the files 12 96
Export Variables in Perl 3 46
I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (,  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…

910 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now