troubleshooting Question

Upload data from a url to PHP / mySQL

Avatar of birwin
birwinFlag for Canada asked on
PHPMySQL Server
4 Comments1 Solution308 ViewsLast Modified:
I need to load data from a url and dump it into a mySQL database using PHP. I will be croning this process, so I need it to happen with no intervention. The data is in .csv format. The file is large, approximately 75 MB.

I tried a test with this code, which I know works with smaller files.
<?php
set_time_limit(600);
$file = file_get_contents('http://www.xxxxxx.com/download_inventory/somefile.csv', FILE_USE_INCLUDE_PATH);

// some PHP to process $file
?>
Even with the set_time_limit it appears to time out. I assume that my variable $file can't hold 75 MB, so I need some way to write it to my server's drive, without having to hold the entire file in a variable.

My intention was to then use PHP to process the data and dump it into a database, but there may be a better way. Should I be writing it to the drive and then using LOAD DATA LOCAL INFILE to dump it to the database? If yes, what is the code to write the data from the remote url to my server's drive?  (note: I have a dedicated server, so I have full access) Is there a way to use mySQL to dump it directly into the database?

So I am looking for the best way to automate daily getting the data from the url and updating my database with it.

I'd appreciate your help.
ASKER CERTIFIED SOLUTION
f_o_o_k_y

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 1 Answer and 4 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 4 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros