Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 209
  • Last Modified:

Repeated data inserted into db

hi ..
i have a perl script that extract the specific keywords from text files .

the text files extracted each time is stored in the computer approx 1 year
and whenever a new file come in, the keywords is being inserted into the db

Now, i don't want repeated keywords from the files being inserted into the db each time there's a new file coming in.

i use crontab to run my FTP script(for transfering of files) and perl script (for inserting keywords into db).
I also using mysql as my db and debian linux 3.0 as OS

this is my perl script

Perl Script

use DBI;
use File::Spec::Functions qw(curdir rel2abs);
use File::Find;
 my( $dbh, $sth, $query );

 $dbh = DBI->connect( 'dbi:mysql:rvsi');
 $query = 'insert into yield ( job, date, tray, lot, dir, report ) values ( ?, ?, ?, ?, ?, ?)';
 $sth = $dbh->prepare( $query );

find(
sub{
 return unless -f;
   
 $fname = rel2abs( $_ );
 print "$fname\n";

 @ARGV = ($_);
 my ( $job, $tray, $date, $lot, $report);

 while(<>) {
  if( /^\s*(\S+)\s*Report/ ){
    $report ="$1 report";
    print $report;
  } elsif( /Job\s*:\s*(\S+)\s*Start Date\s*:\s*(\S+)/ ) {
    $job = $1;
    $date = $2;
    print "$job\n";
    print "$date\n";
  } elsif( /Tray\s*:\s*(\S+)/ ) {
    $tray = $1;
    print "$tray\n";
  } elsif( /Lot\s*:\s*(\S+)/ ) {
    $lot = $1;
    print "$lot\n";

  }
 

if( $job && $date && $tray && $lot && $report ){
    # We have a full record
   
    my $fname = rel2abs( $ARGV);
    $sth->execute($job, $date, $tray, $lot, $fname, $report);

    $job = $date = $tray = $lot =  '';


  }
 }
},".");


Thanks.. :>

0
TPoly
Asked:
TPoly
  • 2
2 Solutions
 
fukengruvenCommented:
I'm not sure if this is the best way, but it would depend on the number of keywords. It sounds like you're going to have to perform some comparisons to see if they are already present in the database.

If you have an obscene amount of keywords then you might consider organizing the database differently (maybe seperated alphabetically) to keep from suffering a performance hit.

I would read in the words from the file and arrange them alphabetically in an array and then iterate thru the array and compare them to the query results I got from the database. Like I said, I would probably redesign the database so you wouldn't have to iterate thru the entire database every time.
0
 
davorgCommented:
So in some cases you can do an update instead of an insert.

In fact, as you're using MySQL, you can use "replace" and it will do an insert or an update as appropriate. You just need to ensure that that you have the right primary keys on the table.
0
 
TPolyAuthor Commented:
hi... i've tried the "replace"  
it still doesnt work
my primary key is the auto-increment serial number
is that consider as the correct primary key?
0
 
TPolyAuthor Commented:
yea ok i found out the problem
it's my PK

thanks for ya help.... it works :)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now