Solved

speed up slow Perl code with SQL statements

Posted on 2007-03-30
1
177 Views
Last Modified: 2010-03-05
This is the guts of the code that I have written, however, when displayed in Firefox it takes a while to display. This seems to be down to the SQL hits i think, is there a way to optimize it as best as possible? I tried a JOIN which works, but is there any other ways to help speed it up?

I feel the speed is slowing at the @filescursor query.

I'm using use strict;

@groupscursor = $db->query("select distinct(filename),groups.receiverid,groups.filelimit,groups.theowner,groups.active,receivers.name,receivers.extension from hols.groups left join hols.receivers on groups.receiverid = receivers.id order by theowner asc, receiverid asc");

if ($groupscursor[1] ne "0E0") {
      while ($groupsreturned = $groupscursor[0]->fetchrow_hashref) {
            $receiverid            = $groupsreturned->{receiverid};
            $filename            = $groupsreturned->{filename};
            $theowner            = $groupsreturned->{theowner};
            $receivername      = $groupsreturned->{name};
            $extension            = $groupsreturned->{extension};      

            next if ($receivername eq $tmpreceiver and ($filename.".".$extension) eq $tmpfile);


            @filescursor = $db->query("select count(id) as results,id,filename,completed,max(created) as datecreated from picco.files where theowner = '$theowner' and filename = '$filename' group by filename");            

            if ($filescursor[1] ne "0E0") {
                  while ($filesreturned = $filescursor[0]->fetchrow_hashref) {
                                                
                        $fileid            = $filesreturned->{id};
                        $created      = $filesreturned->{datecreated};
                        $completed      = $filesreturned->{completed};

                        $total += $filesreturned->{results};

                        ($offers) = @{($db->query("SELECT count(id) as offers FROM picco.offers WHERE fileid = '$fileid'"))[0]->fetchrow_arrayref};

                        if ($theowner ne $tmpsite) {                  
                              push (@list, ["", qq|<b>| . $theowner . qq|</b>|, "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;"]);
                              $total += 1;
                        }
                        $tmpsite = $theowner;
                        
                        $colour = "black";

                        if ($completed eq "N") {
                              $colour = "red";
                        }                        

                        push (@list,      [
                                                      "&nbsp;",                        
                                                      "&nbsp;",
                                                      qq|<font color='$colour'>| . $receivername . qq|</font>|,
                                                      qq|<font color='$colour'>| . ($filename .".". $extension) . qq|</font>|,
                                                      qq|<font color='$colour'>| . $groupsreturned->{filelimit} . qq|</font>|,
                                                      qq|<font color='$colour'>| . $groupsreturned->{active} . qq|</font>|,
                                                      qq|<font color='$colour'>| . $created . qq|</font>|,
                                                      qq|<font color='$colour'>| . $completed . qq|</font>|,
                                                      qq|<font color='$colour'>| . $offers . qq|</font>|
                                                ]);
                        $tmpfile = ($filename.".".$extension);
                        $tmpreceiver = $receivername;
                  }
            }
            @filescursor = undef;
      }
}
@groupscursor = undef;
0
Comment
Question by:crmpicco
1 Comment
 
LVL 7

Accepted Solution

by:
mzalfres earned 125 total points
ID: 18881503
I believe your problem lies not in the PERL code. Try to compare 'clean' SQL query run with your script. Run the same query using your database command line interface and see whether it works significantly faster. I'd rather look at database and try eg. add indexes for fields which are used as a search / join keys. (in this case - 'filename' field.)

Regards,

Marek ZJ.
0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Problem with one substitution modifying another 3 77
Convert grep lines to perl 6 75
How to get all the API from website? 11 122
Regex rule to match two different url 5 75
On Microsoft Windows, if  when you click or type the name of a .pl file, you get an error "is not recognized as an internal or external command, operable program or batch file", then this means you do not have the .pl file extension associated with …
A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

680 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question