Solved

MySQL Infile Load Data

Posted on 2010-08-17
18
647 Views
Last Modified: 2012-05-10
I have a large raw data that in the csv format (acutally a comma, not semi colon)

I have the first line and the last line should be ignored because they are header.
also it has a date field too....i do not know how to import it...

Suppose I have the folliwng data

Anyone can help me how to write the infile script?
STR,G_A-1_20100818_0001.csv
"12345678901","1234567","12345678901234","","12345678901234","","1001","1234","","100","18-08-2010 01:20:23","123456789012","123456","123456","","123456789012","","1009","","","MOB123ABC123A12345678ABC1234ACBD2","","","","","9","",""
END,G_A-1_20100818_0001.csv, 000001

Open in new window

0
Comment
Question by:mawingho
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 11
  • 7
18 Comments
 
LVL 8

Accepted Solution

by:
kingjely earned 500 total points
ID: 33461277

Hi, if your using mysql Query browser, you can

load data local infile
'drive:\path\filename.csv'
into table Tablename
fields terminated by ','
lines terminated by '\r\n';


You can optionally ignore lines with in this,
[IGNORE number LINES], where you put in the first and last linenumber.

Have a look here,  
http://dev.mysql.com/doc/refman/5.0/en/load-data.html

Kj



0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461286
Also, Mawingho,
before you load the data infile, you have to have created the table that will hold the data.

Have you done that?
0
 

Author Comment

by:mawingho
ID: 33461376
Yes, created.
However, the number of line is different from different files....
so how can I just ignore the last line?
Also how to handle date format and also the number value?
Because the field "" will be determined as string rather than integer...
so it returns an error...
0
Space-Age Communications Transitions to DevOps

ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.

 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461458
As far as im aware, You can't skip lines at the end of the file (only at the beginning).

You'll need to either alter the file before you load it or
delete the last line after you load the data infile.

For deleting the last line once it has been loaded,

If you have an auto-increment primary key column, you can:
DELETE FROM yourtable
ORDER BY primarykey_column DESC LIMIT 1;





0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461478

Also, you need '/' not '\' for the drive path.

load data local infile
'drive:/path/filename.csv'
into table Tablename
fields terminated by ','
lines terminated by '\r\n';
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461494
Hi, second part of the next question,

So you want to remove ALL the "" around all fields as they are all Numbers or date fields, is this correct?
0
 

Author Comment

by:mawingho
ID: 33461500
yes
0
 

Author Comment

by:mawingho
ID: 33461503
I have created a table with all VARCHAR fields now....
I will convert them back to number later...
0
 

Author Comment

by:mawingho
ID: 33461538
1. By the way, my friends said if I use MyISAM engine, there are 3 files will be stored in my directory...
2. also, I specified the datafile store in D:\
but I discovered that it stored in C:\ now and I do not know where it is stored...
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461541

So now you have your table with all fields with ALL "",
Which you would like to remove from the table now?

You can update the fields, and replace the " with  nothing ''

UPDATE table SET FIELD1= replace(FIELD1,'"','');
UPDATE table SET FIELD2= replace(FIELD2,'"','');
UPDATE table SET FIELD3= replace(FIELD3,'"','');
UPDATE table SET FIELD4= replace(FIELD4,'"','');
UPDATE table SET FIELD5= replace(FIELD5,'"','');
UPDATE table SET FIELD6= replace(FIELD6,'"','');
UPDATE table SET FIELD7= replace(FIELD7,'"','');

0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461556

Try using '/'s not '\'s in the drive path.



a few comments above, comment, # 33461494


Also, you need '/' not '\' for the drive path.

load data local infile
'drive:/path/filename.csv'
into table Tablename
fields terminated by ','
lines terminated by '\r\n';
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461660



2. also, I specified the datafile store in D:\
but I discovered that it stored in C:\ now and I do not know where it is stored...

You may not need

load data LOCAL infile

you may just need

load data infile - local will go to your local machine.

Also, make sure your  drive:/path/filename.csv  is using '/'  not '\'

0
 

Author Comment

by:mawingho
ID: 33461702
but I already load all the data into my database, what should I do now?
0
 

Author Comment

by:mawingho
ID: 33461709
how to move those data to D: now?
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461733


In regards to what exactly?
If you loaded the data,

Can you select fields from yourtable?



Go to Mysql query browser, and connect to your database


SELECT * from YOURTABLE


0
 

Author Comment

by:mawingho
ID: 33461736
yes, I can
but I do not want those data store in my C:\
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461759

Why do you think it is stored on your C:\  ?

Do you know where it is stored?
Maybe you can look in

C:\bin

if you have mysql command installed

If not, you could go to

START>SEARCH and search for the file name, on your C:\ to see if it exists?

0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461788

if your refering to this question..

< 1. By the way, my friends said if I use MyISAM engine, there are 3 files will be stored in my directory...

I am not particularly familliar with MyISAM, I use innoDB so you may need to ask a new Question to continue, someone else may beable to help you further with this ;)

Kj

0

Featured Post

U.S. Department of Agriculture and Acronis Access

With the new era of mobile computing, smartphones and tablets, wireless communications and cloud services, the USDA sought to take advantage of a mobilized workforce and the blurring lines between personal and corporate computing resources.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

These days, all we hear about hacktivists took down so and so websites and retrieved thousands of user’s data. One of the techniques to get unauthorized access to database is by performing SQL injection. This article is quite lengthy which gives bas…
When it comes to protecting Oracle Database servers and systems, there are a ton of myths out there. Here are the most common.
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

738 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question