Remove duplicate lines from a text file
Posted on 2008-06-24
I am trying to remove duplicate lines from a text file. To make things difficult the lines contain non unique timestamps but a unique reference number. Some of the duplicates amount to 10 lines whereas others can only be 2 lines.
1. Here are some examples of duplicates lines: <timestamp>,<reference>,<error message>
08:47:22,95847170050,Problem inputting data.
08:47:29,95847170050,Problem inputting data.
08:47:35,95847170050,Problem inputting data.
08:53:28, 96672540040, More problems inputting data.
08:53:35, 96672540040, More problems inputting data.
08:53:41, 96672540040, More problems inputting data.
I want to delete all but the most recent duplicate line.
I am new to java so can you tell what the best way of doing this is?
Thank you in advance.