troubleshooting Question

Perl Script - Import Multiple CSV Files MySQL with Update Query In Between

Avatar of Joel Buhr
Joel BuhrFlag for United States of America asked on
LinuxPerlMySQL Server
9 Comments1 Solution589 ViewsLast Modified:
I have never messed with PERL but understand it is probably what I want to be using to import due to the file sizes I am dealing with.

Environment : Linux MariaDB server.

I have 7 data files of roughly 24 million rows each.

Normally I am importing these via SSH mysql command after I move the ".csv" files over to the server running the following command for each file.

LOAD DATA LOCAL INFILE '/home/APR17/20161003_FULL FILE_01.csv' INTO TABLE FD_MASTER FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES;

Then I run this query to code the batch that got loaded.

UPDATE FD_MASTER
SET LoadInputFile = '01'
WHERE LoadInputFile IS NULL

So my question is how do I create a PERL script that will load each of the 7 files one at a time, and in between each load run the Update Query.

It would be nice if it at the end provided a simple report of what it did including how long each segment took to load.

After this there are a series of queries that I also run on that Master file which I would like to also add to this process, but those can wait.
ASKER CERTIFIED SOLUTION
Join our community to see this answer!
Unlock 1 Answer and 9 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 9 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros