Solved

Repeated data inserted into db

Posted on 2004-08-13
4
199 Views
Last Modified: 2010-03-05
hi ..
i have a perl script that extract the specific keywords from text files .

the text files extracted each time is stored in the computer approx 1 year
and whenever a new file come in, the keywords is being inserted into the db

Now, i don't want repeated keywords from the files being inserted into the db each time there's a new file coming in.

i use crontab to run my FTP script(for transfering of files) and perl script (for inserting keywords into db).
I also using mysql as my db and debian linux 3.0 as OS

this is my perl script

Perl Script

use DBI;
use File::Spec::Functions qw(curdir rel2abs);
use File::Find;
 my( $dbh, $sth, $query );

 $dbh = DBI->connect( 'dbi:mysql:rvsi');
 $query = 'insert into yield ( job, date, tray, lot, dir, report ) values ( ?, ?, ?, ?, ?, ?)';
 $sth = $dbh->prepare( $query );

find(
sub{
 return unless -f;
   
 $fname = rel2abs( $_ );
 print "$fname\n";

 @ARGV = ($_);
 my ( $job, $tray, $date, $lot, $report);

 while(<>) {
  if( /^\s*(\S+)\s*Report/ ){
    $report ="$1 report";
    print $report;
  } elsif( /Job\s*:\s*(\S+)\s*Start Date\s*:\s*(\S+)/ ) {
    $job = $1;
    $date = $2;
    print "$job\n";
    print "$date\n";
  } elsif( /Tray\s*:\s*(\S+)/ ) {
    $tray = $1;
    print "$tray\n";
  } elsif( /Lot\s*:\s*(\S+)/ ) {
    $lot = $1;
    print "$lot\n";

  }
 

if( $job && $date && $tray && $lot && $report ){
    # We have a full record
   
    my $fname = rel2abs( $ARGV);
    $sth->execute($job, $date, $tray, $lot, $fname, $report);

    $job = $date = $tray = $lot =  '';


  }
 }
},".");


Thanks.. :>

0
Comment
Question by:TPoly
  • 2
4 Comments
 

Assisted Solution

by:fukengruven
fukengruven earned 100 total points
ID: 11791596
I'm not sure if this is the best way, but it would depend on the number of keywords. It sounds like you're going to have to perform some comparisons to see if they are already present in the database.

If you have an obscene amount of keywords then you might consider organizing the database differently (maybe seperated alphabetically) to keep from suffering a performance hit.

I would read in the words from the file and arrange them alphabetically in an array and then iterate thru the array and compare them to the query results I got from the database. Like I said, I would probably redesign the database so you wouldn't have to iterate thru the entire database every time.
0
 
LVL 8

Accepted Solution

by:
davorg earned 400 total points
ID: 11791721
So in some cases you can do an update instead of an insert.

In fact, as you're using MySQL, you can use "replace" and it will do an insert or an update as appropriate. You just need to ensure that that you have the right primary keys on the table.
0
 

Author Comment

by:TPoly
ID: 11806824
hi... i've tried the "replace"  
it still doesnt work
my primary key is the auto-increment serial number
is that consider as the correct primary key?
0
 

Author Comment

by:TPoly
ID: 11807523
yea ok i found out the problem
it's my PK

thanks for ya help.... it works :)
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
how to extract data from a table into a fixed length file 10 53
perl script help 12 103
compress files in RAR using perl 13 79
Formatting stings with pack and printf in perl 5 67
A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

943 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now