We help IT Professionals succeed at work.
Get Started

Upload data from a url to PHP / mySQL

birwin asked
Last Modified: 2012-05-07
I need to load data from a url and dump it into a mySQL database using PHP. I will be croning this process, so I need it to happen with no intervention. The data is in .csv format. The file is large, approximately 75 MB.

I tried a test with this code, which I know works with smaller files.
$file = file_get_contents('http://www.xxxxxx.com/download_inventory/somefile.csv', FILE_USE_INCLUDE_PATH);

// some PHP to process $file
Even with the set_time_limit it appears to time out. I assume that my variable $file can't hold 75 MB, so I need some way to write it to my server's drive, without having to hold the entire file in a variable.

My intention was to then use PHP to process the data and dump it into a database, but there may be a better way. Should I be writing it to the drive and then using LOAD DATA LOCAL INFILE to dump it to the database? If yes, what is the code to write the data from the remote url to my server's drive?  (note: I have a dedicated server, so I have full access) Is there a way to use mySQL to dump it directly into the database?

So I am looking for the best way to automate daily getting the data from the url and updating my database with it.

I'd appreciate your help.
Watch Question
This problem has been solved!
Unlock 1 Answer and 4 Comments.
See Answer
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE