Link to home
Start Free TrialLog in
Avatar of shragi
shragiFlag for India

asked on

Perl efficient DB Call

I wrote the below script to update the file.
The script works but I had a small concern with the script I wrote, it makes multiple database calls.
Is there any way you think I can modify this and make just one DB call.

This is what the script currently does:

1)      Go thru each line of the file and get the 3 fields that are required ($fields[6], $fields[8], $fields[12])
2)      Run query and get the results using the 3 fields
3)      Print output file line by line by appending the results at the end of each line.

#!/usr/bin/perl

use 5.006;
use strict;
use warnings;
use Pod::Usage;
use Getopt::Long;
use Cwd;
use DateTime;
use DBD::mysql;
use DBI;

my $helpme = 0;
my $man = 0;
my $inputFileName = 'C:\\temp\\test.txt';
my $outputFileName = 'C:\\temp\\output.txt';
my $dsn = 'DBI:ODBC:Driver={SQL Server}';
my $host = 'testdb';
my $database = 'MYAPP';
my $user = 'admin';
my $auth = 'p@ssword';


GetOptions('help' => \$helpme, 'man' => \$man, 'infile=s' => \$inputFileName, 'outfile=s' => \$outputFileName) or pod2usage(2);

pod2usage(1) if $helpme;
pod2usage(-verbose => 2) if $man;

die "No input file name specified!" if !$inputFileName;
die "No output file name specified!" if !$outputFileName;
my $dbh = DBI->connect("$dsn;Server=$host;Database=$database", $user, $auth,{ RaiseError => 1, AutoCommit => 1}) || die "Database connection not made: $DBI::errstr"; 

open(INFILE, "<", $inputFileName) || die "Could not open input file!";
open(OUTFILE, ">", $outputFileName) || die "Could not open/create output file!";

while (<INFILE>) {
    chomp;
   my @fields = split /\s*\|\s*/; 
   my $sql = "SELECT e.id1, d.id2, s.id3   
              FROM  Employee e 	
              JOIN  Department d 
                      ON d.did = '$fields[12]' AND d.status ='A'
              JOIN  Student s  
                      ON s.sid= '$fields[6]' AND s.status ='A'	 
              Where e.eid= '$fields[8]' AND e.status ='A'";
   my $sth = $dbh->prepare($sql);
   $sth->execute();
   while (my @row = $sth->fetchrow_array) {
    print OUTFILE $_."|".$row[0]."|".$row[1]."|".$row[2]."\n";
   }    
}
close INFILE;
close OUTFILE;

Open in new window


Thanks,
Avatar of FishMonger
FishMonger
Flag of United States of America image

The first thing I'd recommend is to use placeholders in the sql statement and move the prepare statement to just before the loop.

Can you post a reasonable sample of the input file?
Avatar of shragi

ASKER

Sample file is something like the attached.
Avatar of shragi

ASKER

well if i want to append at the end of the file I can see that I create the sql first and execute it at the end, but i need to modify each line
ASKER CERTIFIED SOLUTION
Avatar of FishMonger
FishMonger
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I did not do any testing, but your join statements look a little fishy.  Do they actually do what you need?
FishMonger is correct in most cases.  However, there are databases where using placeholders is less efficient than executing a static sql each time (Sybase is (or at least was) one).
FishMonger is correct in most cases.
What?  You mean that I'm not perfect?
:-)  lol  ;-)