• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1425
  • Last Modified:

Eliminate Duplicate Rows in CSV with PowerShell

I'm looking for a way to import a CSV file in PowerShell and remove all duplicate entries. I am *not* looking for unique entries. If they're in the CSV more than once, I want all mention of them gone. Using the below example, I want the script to return only people who have one job, and to completely skip over users that have 2 or more jobs.

Here's an example CSV:
ID,LastName,FirstName,JobCode,JobTitle,DeptCode,Dept
1001,Doe,John,100,Programmer,199,IT
1002,Smith,Janice,110,Technician,199,IT
1003,Jordan,Michael,150,Janitor,102,Housekeeping
1003,Jordan,Michael,123,Cook,101,Cafeteria
1004,Johnson,Abigail,142,Nurse,180,Employee Health

Open in new window

The desired resulting CSV is:
ID,LastName,FirstName,JobCode,JobTitle,DeptCode,Dept
1001,Doe,John,100,Programmer,199,IT
1002,Smith,Janice,110,Technician,199,IT
1004,Johnson,Abigail,142,Nurse,180,Employee Health

Open in new window

I'm stumped here, I don't even have a good guess. I'm no farther than "Import-CSV"

Any help would be much appreciated.
0
Cacophony777
Asked:
Cacophony777
1 Solution
 
footechCommented:
$data = Import-Csv test.csv | Group id | Where {$_.Count -eq 1} | Select -expandProperty group

Open in new window

You can then output the data back to a .CSV or whatever.
0
 
Cacophony777Author Commented:
Nice! That was easier than I thought it would be... I've never used the Group-Object command before. Thanks!
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now