Solved

speed up slow Perl code with SQL statements

Posted on 2007-03-30
1
178 Views
Last Modified: 2010-03-05
This is the guts of the code that I have written, however, when displayed in Firefox it takes a while to display. This seems to be down to the SQL hits i think, is there a way to optimize it as best as possible? I tried a JOIN which works, but is there any other ways to help speed it up?

I feel the speed is slowing at the @filescursor query.

I'm using use strict;

@groupscursor = $db->query("select distinct(filename),groups.receiverid,groups.filelimit,groups.theowner,groups.active,receivers.name,receivers.extension from hols.groups left join hols.receivers on groups.receiverid = receivers.id order by theowner asc, receiverid asc");

if ($groupscursor[1] ne "0E0") {
      while ($groupsreturned = $groupscursor[0]->fetchrow_hashref) {
            $receiverid            = $groupsreturned->{receiverid};
            $filename            = $groupsreturned->{filename};
            $theowner            = $groupsreturned->{theowner};
            $receivername      = $groupsreturned->{name};
            $extension            = $groupsreturned->{extension};      

            next if ($receivername eq $tmpreceiver and ($filename.".".$extension) eq $tmpfile);


            @filescursor = $db->query("select count(id) as results,id,filename,completed,max(created) as datecreated from picco.files where theowner = '$theowner' and filename = '$filename' group by filename");            

            if ($filescursor[1] ne "0E0") {
                  while ($filesreturned = $filescursor[0]->fetchrow_hashref) {
                                                
                        $fileid            = $filesreturned->{id};
                        $created      = $filesreturned->{datecreated};
                        $completed      = $filesreturned->{completed};

                        $total += $filesreturned->{results};

                        ($offers) = @{($db->query("SELECT count(id) as offers FROM picco.offers WHERE fileid = '$fileid'"))[0]->fetchrow_arrayref};

                        if ($theowner ne $tmpsite) {                  
                              push (@list, ["", qq|<b>| . $theowner . qq|</b>|, "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;", "&nbsp;"]);
                              $total += 1;
                        }
                        $tmpsite = $theowner;
                        
                        $colour = "black";

                        if ($completed eq "N") {
                              $colour = "red";
                        }                        

                        push (@list,      [
                                                      "&nbsp;",                        
                                                      "&nbsp;",
                                                      qq|<font color='$colour'>| . $receivername . qq|</font>|,
                                                      qq|<font color='$colour'>| . ($filename .".". $extension) . qq|</font>|,
                                                      qq|<font color='$colour'>| . $groupsreturned->{filelimit} . qq|</font>|,
                                                      qq|<font color='$colour'>| . $groupsreturned->{active} . qq|</font>|,
                                                      qq|<font color='$colour'>| . $created . qq|</font>|,
                                                      qq|<font color='$colour'>| . $completed . qq|</font>|,
                                                      qq|<font color='$colour'>| . $offers . qq|</font>|
                                                ]);
                        $tmpfile = ($filename.".".$extension);
                        $tmpreceiver = $receivername;
                  }
            }
            @filescursor = undef;
      }
}
@groupscursor = undef;
0
Comment
Question by:crmpicco
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 
LVL 7

Accepted Solution

by:
mzalfres earned 125 total points
ID: 18881503
I believe your problem lies not in the PERL code. Try to compare 'clean' SQL query run with your script. Run the same query using your database command line interface and see whether it works significantly faster. I'd rather look at database and try eg. add indexes for fields which are used as a search / join keys. (in this case - 'filename' field.)

Regards,

Marek ZJ.
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question