Question

Upload .CSV file into MySql DB using PHP

Asked by: ARC_UM

I am new at this & is looking for a csv file upload script using php. The file before being uploaded need to go through some validation (eg, file type & check the various column for integer & date type) before inserting into DB. once validation is done, the file will be inserted into 2 tables in DB.
Can someone please help?

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2009-04-19 at 17:30:12ID24336221
Tags

php

,

csv

,

file

,

upload

,

DB

,

MySql

Topics

MySQL Server

,

Programming Languages

,

Web Development Software

Participating Experts
3
Points
500
Comments
23

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. testing a csv upload
    I have written the following script to upload the given csv into a table of the following structure id | prod_id | att_cid | att_sid | att_count The main issue is I need to provide the logic that will error out and print an error mssg is someone tries to upload 2 rows with ...
  2. export mysql to csv with php
    I am using the following script to download the results of my query to .csv. The file opens perfectly in ms excel but when I try to open it in notepad every row looks like this "RGM215","Men's Reggaeton “Perro” Logo Tee","Black","XXL",...
  3. Trouble with first line of CSV file when using PHP to import …
    I have a PHP script which takes a .csv file on my webserver and imports it into a MySQL database. The script is run daily and deletes the .csv file after import. A new .csv file is uploaded every morning. The issue I'm having is this CSV file is uploaded from another souce wh...
  4. Upload CSV and Update Selected columns in MySQL table
    Help please Experts... I have a CSV file with 19 columns and I have a MySQL database that I need to update from the CSV file. Within the CSV is a column (19) that relates to a column (1) in the MySQL table. What I need to do is upload the file from desktop and use two entrie...
  5. csv to mysql using php
    Hi, I have a csv file.I need a php script which import the csv into the mysql table already created in a database. Instead of giving the file path in a script ,i need a browse buttom from which i can select the csv file. Thanks, uma

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: segurahPosted on 2009-04-19 at 18:06:46ID: 24181375

More that a cvs uploades you probably need and ETL (extraction, transforming and loading) see at : http://en.wikipedia.org/wiki/Extract,_transform,_load

 

by: ARC_UMPosted on 2009-04-19 at 19:24:45ID: 24181590

i need the script that will reae the contents of the csv file  & store it into the 2 tables in MySql DB

 

by: theGhost_k8Posted on 2009-04-19 at 21:20:50ID: 24181867

 

by: dazweejaPosted on 2009-04-20 at 01:52:31ID: 24182821

This should get you pretty close:

<?php
$conn = mysql_connect("localhost", "mysql_user", "mysql_password");
 
if (!$conn) {
    echo "Unable to connect to DB: " . mysql_error();
    exit;
}
  
if (!mysql_select_db("mydbname")) {
    echo "Unable to select mydbname: " . mysql_error();
    exit;
}
 
$handle = @fopen("inputfile.csv", "r");
 
if ($handle) {
    while (!feof($handle)) {
        $buffer = fgets($handle, 4096);
        $values = explode(",", $buffer);
		if(count($values == 2)) { // make sure the line is valid
			$col1 = trim($values[0]);
			$col2 = trim($values[1]);
			if(
				($col1 != '') &&
				(is_int($col1)) &&
				($col2 != '') &&
				(is_string($col2))
			) { // sample validation routine
				$sql = "INSERT INTO table_name
					(column_name1, column_name2)
					VALUES
					($col1, $col2);";
	
				$result = mysql_query($sql);
 
				if (!$result) {
					echo "Could not successfully run query ($sql) from DB: " . mysql_error();
					exit; // comment this out if you'd like to continue after errors
				}
			}
			else {
				echo "Data is invalid";
			}
		}
		else {
			echo "Incorrect number of values on line";
		}
    }
    fclose($handle);
}
 
mysql_free_result($result);
?>
                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:

Select allOpen in new window

 

by: dazweejaPosted on 2009-04-20 at 01:57:50ID: 24182842

Of course, change line 20 to match the number of values you expect on each line of your csv and duplicate the code in lines 29-39 for the second table.

 

by: ARC_UMPosted on 2009-04-20 at 16:44:36ID: 24189723

hi dazweeja, thanks for the suggestion. i have the csv file uploaded on the server -- will your code be able to read the file from there. what is 4096 in line 18?

 

by: dazweejaPosted on 2009-04-20 at 17:04:52ID: 24189800

It will be able to read files on the same server - just give it the correct file path. It will also be able to read files on other servers, ie. using a URL as the path, but only if allow_url_fopen is On for that server (the default). See my comments here:

http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_24334585.html?cid=1066#a24183619

4096 is the number of bytes to reads into memory for each line. This value should handle most situations - it's at least 2048 characters per line - so is commonly used. You can make this value smaller if your lines are short. I think you'd only bother if memory/performance of your website was critical.

 

by: ARC_UMPosted on 2009-04-20 at 22:40:07ID: 24191149

it doesn't save the records in DB table

 

by: ARC_UMPosted on 2009-04-20 at 23:04:16ID: 24191251

there is also this error i forgot to mention earlier:
Warning: mysql_free_result(): supplied argument is not a valid MySQL result resource in /home1/uploader.php on line 89

 

by: dazweejaPosted on 2009-04-20 at 23:09:31ID: 24191280

My mistake, that line doesn't need to be there so get rid of it. It only applies if you are using SQL SELECT statements and it's in the wrong place anyway. I accidentally copied from another piece of code I had.

Are you getting any other error messages?

Are you sure it is opening your csv file correctly? If you change '@fopen' to 'fopen', you should see an error message on the screen or in your PHP error log if it's unable to open the CSV file. The '@' symbol is there to suppress error messages but at the coding stage, you probably want to see them. When the site is live, you would probably want to change it back to '@fopen' and put this at line 51:

else {
   echo "Problem opening csv file";
}

If the file is opening correctly, have you changed the SQL to match your table structure? As a debugging step, insert this line at line 33 to make sure your SQL command is what you expect:

echo $sql;

In the development stage, you might want to put echo statements (or error_log statements) all through your code to see exactly where the problem is or use a PHP debugger like XDebug or ZendDebugger.

 

by: dazweejaPosted on 2009-04-20 at 23:11:38ID: 24191287

To clarify, I meant delete the line (should be line 89 in your code):

mysql_free_result($result);

 

by: ARC_UMPosted on 2009-04-21 at 17:15:05ID: 24200502

i am receiving the following error:
Resource id #13INSERT INTO `gipps_student_info` (SUBMITTER_ID, PROJECT_ID, SCHOOL_NAME, STUDENT_CODE) VALUES (4, gipps, 200, 2002);Could not successfully run query (INSERT INTO `gipps_student_info` (SUBMITTER_ID, PROJECT_ID, SCHOOL_NAME, STUDENT_CODE) VALUES (4, gipps, 200, 2002);) from DB: Unknown column 'gipps' in 'field list'

i dont need to add values in sequential order as in the DB table since i am mentioning which columns the values go under? any idea how to solve the problem above?

in my csv if I have row headers, how do i omit while inserting in DB

 

by: ARC_UMPosted on 2009-04-21 at 17:25:55ID: 24200553

ERROR MSG:
Resource id #13INSERT INTO `gipps_student_info` (SUBMITTER_ID, SCHOOL_NAME, STUDENT_CODE) VALUES (4, 200, 2002)Could not successfully run query (INSERT INTO `gipps_student_info` (SUBMITTER_ID, SCHOOL_NAME, STUDENT_CODE) VALUES (4, 200, 2002)) from DB: Duplicate entry '2002' for key 2

looks like it is reading the first line over again

 

by: ARC_UMPosted on 2009-04-21 at 17:31:07ID: 24200574

disregard the prev message

i get this error but data is also inserted in the DB:
Resource id #13INSERT INTO `gipps_student_info` (SUBMITTER_ID, SCHOOL_NAME, STUDENT_CODE) VALUES (4, 200, 2002)1Data is invalidINSERT INTO `gipps_student_info` (SUBMITTER_ID, SCHOOL_NAME, STUDENT_CODE) VALUES (4, 200, 2003)1Data is invalidINSERT INTO `gipps_student_info` (SUBMITTER_ID, SCHOOL_NAME, STUDENT_CODE) VALUES (, , )Could not successfully run query (INSERT INTO `gipps_student_info` (SUBMITTER_ID, SCHOOL_NAME, STUDENT_CODE) VALUES (, , )) from DB: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ' , )' at line 1

how can i solve this bit?

 

by: dazweejaPosted on 2009-04-21 at 17:43:23ID: 24200631

If you are inserting a value into a column and the datatype of that column in your database is not numeric, eg. char, varchar, text, etc., you have to put quotation marks around your values in your SQL. You can also put quotation marks around numeric data as well but it's optional.

For example:

$sql = "INSERT INTO `gipps_student_info` (SUBMITTER_ID, SCHOOL_NAME, STUDENT_CODE) VALUES ('" . $submitter_id . "','" . $school_name . "','" . $student_code . "');";
 
echo $sql;

                                              
1:
2:
3:

Select allOpen in new window

 

by: dazweejaPosted on 2009-04-21 at 17:46:09ID: 24200643

Also, your last message seems to suggest that an INSERT is being attempted even when the data is empty or not valid. Have a look at your validation code and put in checks like ($school_name != "") if necessary.

 

by: ARC_UMPosted on 2009-04-21 at 18:37:12ID: 24200849

i've uploaded the code as well as a sample csv template i want to insert. i couldn't upload .csv file so you may need to save it as .csv file. users may not have to insert all fields, some may be kept blank.
when reading i want the insertion to exclude/skip the heading row in the csv file

$handle = @fopen($target_path, "r");
 
if ($handle) {
    while (!feof($handle)) 
    {
        $buffer = fgets($handle, 4096);
        $values = explode(",", $buffer);
                if(count($values == 10)) 
                { 
                        $col1 = trim($values[0]);
                        $col2 = trim($values[1]);
                        $col3 = trim($values[2]);
                        $col4 = trim($values[3]);
                        $col5 = trim($values[4]);
                        $col6 = trim($values[5]);
                        $col7 = trim($values[6]);
                        $col8 = trim($values[7]);
                        $col9 = trim($values[8]);
                        $col10 = trim($values[9]);                 
						
			$sql = "INSERT INTO `$_SESSION[student_info]` (SCHOOL_NAME, STUDENT_CODE, STUDENT_FIRST_NAME, STUDENT_LAST_NAME, YEAR_LEVEL, CLASS, GENDER, TEACHER_TITLE, TEACHER_FIRST_NAME, TEACHER_LAST_NAME, PROJECT_ID, SUBMITTER_ID) 
				VALUES ($col1, $col2, $col3, $col4, $col5, $col6, $col7, $col8, $col9, $col10, '$_SESSION[project_id]','$_SESSION[teachersid]');";
			$result = mysql_query($sql);
			
                        if (!$result) {
                		echo "Could not successfully run query ($sql) from DB: " . mysql_error();
                       		exit; 
                                }
                        
                        else {
                                echo "Data is invalid";
                        }
                }
                else {
                	echo "Incorrect number of values on line";
                }
    }
    fclose($handle);
}
?>

                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:

Select allOpen in new window

 

by: ARC_UMPosted on 2009-04-21 at 18:59:14ID: 24200927

the code inserts the data in teh DB but has the following error:
Data is invalidData is invalidData is invalidData is invalidData is invalid

it inserts an extra blank record for student.
How can i skip reading teh header in the csv file?

 

by: dazweejaPosted on 2009-04-21 at 19:06:08ID: 24200952

I won't be able to test this until I get home but start by putting quotation marks around all the values in line 22:

VALUES ('$col1', '$col2', '$col3',...

It won't hurt if the variables are blank.

If you want to skip the first line, you could do:

    $header_line = true;
    while (!feof($handle)) 
    {
        $buffer = fgets($handle, 4096);
        if($header_line) {
           $header_line = false;
           continue;
        }
        $values = explode(",", $buffer);
                if(count($values == 10))

                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:

Select allOpen in new window

 

by: ARC_UMPosted on 2009-04-21 at 19:40:18ID: 24201063

sorry i put the wrong code in. it inserts properly & there is no error, BUT INSERTS A BLANK RECORD AFTER FINISHING INSERTING FROM THE CSV FILE.
I'LL TRY YOUR CODE TO SKIP THE FIRST LINE

 

by: ARC_UMPosted on 2009-04-21 at 21:49:12ID: 24201479

it's not reading header, but inserting a blank record

 

by: dazweejaPosted on 2009-04-21 at 22:18:59ID: 24201561

Of these columns, is there one that can't be left blank? That is, can you identify one column that you wouldn't want to insert the values into the database if it's not given?

For example, if the data is useless without a SCHOOL_NAME, you could wrap lines 21-31 of your code above with:

if($col1 != "") {
// put code here
}

Otherwise, you could wrap these lines in:

if(($col1 != "") && ($col2 != "") && ($col3 != "")) { // etc.
}

With all the values so it would need at least one value to be given before it inserted a row.

Lastly, lines 10-19 could be condensed into:

$values = array_map("trim", $values);

 

by: ARC_UMPosted on 2009-04-21 at 22:31:39ID: 24201602

if i have more problems I'll let you know. Thanks a lot for your help. I already put 1 column to be not empty before you said :)

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...