Solved

MySQL Infile Load Data

Posted on 2010-08-17
18
631 Views
Last Modified: 2012-05-10
I have a large raw data that in the csv format (acutally a comma, not semi colon)

I have the first line and the last line should be ignored because they are header.
also it has a date field too....i do not know how to import it...

Suppose I have the folliwng data

Anyone can help me how to write the infile script?
STR,G_A-1_20100818_0001.csv

"12345678901","1234567","12345678901234","","12345678901234","","1001","1234","","100","18-08-2010 01:20:23","123456789012","123456","123456","","123456789012","","1009","","","MOB123ABC123A12345678ABC1234ACBD2","","","","","9","",""

END,G_A-1_20100818_0001.csv, 000001

Open in new window

0
Comment
Question by:mawingho
  • 11
  • 7
18 Comments
 
LVL 8

Accepted Solution

by:
kingjely earned 500 total points
ID: 33461277

Hi, if your using mysql Query browser, you can

load data local infile
'drive:\path\filename.csv'
into table Tablename
fields terminated by ','
lines terminated by '\r\n';


You can optionally ignore lines with in this,
[IGNORE number LINES], where you put in the first and last linenumber.

Have a look here,  
http://dev.mysql.com/doc/refman/5.0/en/load-data.html

Kj



0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461286
Also, Mawingho,
before you load the data infile, you have to have created the table that will hold the data.

Have you done that?
0
 

Author Comment

by:mawingho
ID: 33461376
Yes, created.
However, the number of line is different from different files....
so how can I just ignore the last line?
Also how to handle date format and also the number value?
Because the field "" will be determined as string rather than integer...
so it returns an error...
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461458
As far as im aware, You can't skip lines at the end of the file (only at the beginning).

You'll need to either alter the file before you load it or
delete the last line after you load the data infile.

For deleting the last line once it has been loaded,

If you have an auto-increment primary key column, you can:
DELETE FROM yourtable
ORDER BY primarykey_column DESC LIMIT 1;





0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461478

Also, you need '/' not '\' for the drive path.

load data local infile
'drive:/path/filename.csv'
into table Tablename
fields terminated by ','
lines terminated by '\r\n';
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461494
Hi, second part of the next question,

So you want to remove ALL the "" around all fields as they are all Numbers or date fields, is this correct?
0
 

Author Comment

by:mawingho
ID: 33461500
yes
0
 

Author Comment

by:mawingho
ID: 33461503
I have created a table with all VARCHAR fields now....
I will convert them back to number later...
0
 

Author Comment

by:mawingho
ID: 33461538
1. By the way, my friends said if I use MyISAM engine, there are 3 files will be stored in my directory...
2. also, I specified the datafile store in D:\
but I discovered that it stored in C:\ now and I do not know where it is stored...
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461541

So now you have your table with all fields with ALL "",
Which you would like to remove from the table now?

You can update the fields, and replace the " with  nothing ''

UPDATE table SET FIELD1= replace(FIELD1,'"','');
UPDATE table SET FIELD2= replace(FIELD2,'"','');
UPDATE table SET FIELD3= replace(FIELD3,'"','');
UPDATE table SET FIELD4= replace(FIELD4,'"','');
UPDATE table SET FIELD5= replace(FIELD5,'"','');
UPDATE table SET FIELD6= replace(FIELD6,'"','');
UPDATE table SET FIELD7= replace(FIELD7,'"','');

0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461556

Try using '/'s not '\'s in the drive path.



a few comments above, comment, # 33461494


Also, you need '/' not '\' for the drive path.

load data local infile
'drive:/path/filename.csv'
into table Tablename
fields terminated by ','
lines terminated by '\r\n';
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461660



2. also, I specified the datafile store in D:\
but I discovered that it stored in C:\ now and I do not know where it is stored...

You may not need

load data LOCAL infile

you may just need

load data infile - local will go to your local machine.

Also, make sure your  drive:/path/filename.csv  is using '/'  not '\'

0
 

Author Comment

by:mawingho
ID: 33461702
but I already load all the data into my database, what should I do now?
0
 

Author Comment

by:mawingho
ID: 33461709
how to move those data to D: now?
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461733


In regards to what exactly?
If you loaded the data,

Can you select fields from yourtable?



Go to Mysql query browser, and connect to your database


SELECT * from YOURTABLE


0
 

Author Comment

by:mawingho
ID: 33461736
yes, I can
but I do not want those data store in my C:\
0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461759

Why do you think it is stored on your C:\  ?

Do you know where it is stored?
Maybe you can look in

C:\bin

if you have mysql command installed

If not, you could go to

START>SEARCH and search for the file name, on your C:\ to see if it exists?

0
 
LVL 8

Assisted Solution

by:kingjely
kingjely earned 500 total points
ID: 33461788

if your refering to this question..

< 1. By the way, my friends said if I use MyISAM engine, there are 3 files will be stored in my directory...

I am not particularly familliar with MyISAM, I use innoDB so you may need to ask a new Question to continue, someone else may beable to help you further with this ;)

Kj

0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Join & Write a Comment

Composite queries are used to retrieve the results from joining multiple queries after applying any filters. UNION, INTERSECT, MINUS, and UNION ALL are some of the operators used to get certain desired results.​
Load balancing is the method of dividing the total amount of work performed by one computer between two or more computers. Its aim is to get more work done in the same amount of time, ensuring that all the users get served faster.
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now