Solved

Remove Duplicate Lines When Parsing CSV

Posted on 2007-12-04
8
995 Views
Last Modified: 2008-02-01
I am using Java to read lines from a text file.  In this case, it is a CSV.  I parse the file and process each line based on my application.  I noticed that sometimes the file contains duplicate lines.  Instead of processing the duplicate line multiple times, I would like to somehow remove the duplicate lines from the CSV and then process each line in the file.  Can someone instruct me on the one of the more efficient ways of doing this?  Thanks
0
Comment
Question by:pcarrollnf
8 Comments
 
LVL 86

Accepted Solution

by:
CEHJ earned 125 total points
ID: 20405022
If it's not too big you can save the file to a Set<String>. That will ensure uniqueness
0
 

Author Comment

by:pcarrollnf
ID: 20405043
That's would be an issue.  These CSV files can become large and may contain thousands of lines.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 20405108
What OS are you using?
0
Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

 

Author Comment

by:pcarrollnf
ID: 20405193
Windows 2000, 2003
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 20405968
Get a Windows port of textutils from http://gnuwin32.sourceforge.net. You can then do

cat orig.csv | sort | uniq >uniq.csv

Doubt if you'll get much more efficient than that
0
 
LVL 9

Expert Comment

by:brunoguimaraes
ID: 20406102
There is a software called Clippy that does what you want.

http://www.snapfiles.com/get/clippy.html
0
 
LVL 92

Expert Comment

by:objects
ID: 20406402
you don't need to store the entire csv in memory, just the unique keys.
That will allow you to check each line as you process it
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 20781397
:-)
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
servlet example 11 42
Opening PDF on button click and fill new document 2 40
hibernate insert example 13 29
What's wrong with this code? 4 23
For customizing the look of your lightweight component and making it look opaque like it was made of plastic.  This tip assumes your component to be of rectangular shape and completely opaque.   (CODE)
For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
Viewers learn how to read error messages and identify possible mistakes that could cause hours of frustration. Coding is as much about debugging your code as it is about writing it. Define Error Message: Line Numbers: Type of Error: Break Down…
This tutorial will introduce the viewer to VisualVM for the Java platform application. This video explains an example program and covers the Overview, Monitor, and Heap Dump tabs.

825 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question