Solved

Eliminate Duplicate Rows in CSV with PowerShell

Posted on 2013-10-24
2
1,018 Views
Last Modified: 2013-10-24
I'm looking for a way to import a CSV file in PowerShell and remove all duplicate entries. I am *not* looking for unique entries. If they're in the CSV more than once, I want all mention of them gone. Using the below example, I want the script to return only people who have one job, and to completely skip over users that have 2 or more jobs.

Here's an example CSV:
ID,LastName,FirstName,JobCode,JobTitle,DeptCode,Dept
1001,Doe,John,100,Programmer,199,IT
1002,Smith,Janice,110,Technician,199,IT
1003,Jordan,Michael,150,Janitor,102,Housekeeping
1003,Jordan,Michael,123,Cook,101,Cafeteria
1004,Johnson,Abigail,142,Nurse,180,Employee Health

Open in new window

The desired resulting CSV is:
ID,LastName,FirstName,JobCode,JobTitle,DeptCode,Dept
1001,Doe,John,100,Programmer,199,IT
1002,Smith,Janice,110,Technician,199,IT
1004,Johnson,Abigail,142,Nurse,180,Employee Health

Open in new window

I'm stumped here, I don't even have a good guess. I'm no farther than "Import-CSV"

Any help would be much appreciated.
0
Comment
Question by:Cacophony777
2 Comments
 
LVL 40

Accepted Solution

by:
footech earned 500 total points
ID: 39598406
$data = Import-Csv test.csv | Group id | Where {$_.Count -eq 1} | Select -expandProperty group

Open in new window

You can then output the data back to a .CSV or whatever.
0
 
LVL 1

Author Closing Comment

by:Cacophony777
ID: 39598508
Nice! That was easier than I thought it would be... I've never used the Group-Object command before. Thanks!
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Windows 10 came with  a lot of built in applications, Some organisations leave them there, some will control them using GPO's. This Article is useful for those who do not want to have any applications in their image (example:me).
The Nano Server Image Builder helps you create a custom Nano Server image and bootable USB media with the aid of a graphical interface. Based on the inputs you provide, it generates images for deployment and creates reusable PowerShell scripts that …
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question