What is the fastest way to get only distinct (unique) rows out of it? So that in this case result (in another file) would be:
(values of input file are not sorted - also there's no need for output file to be sorted)
I really need the fastest way to do this. The files I need to process like this are about 1GB long so I really need the most optimized solution to handle this.