Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1615
  • Last Modified:

Eliminate Duplicate Rows in CSV with PowerShell

I'm looking for a way to import a CSV file in PowerShell and remove all duplicate entries. I am *not* looking for unique entries. If they're in the CSV more than once, I want all mention of them gone. Using the below example, I want the script to return only people who have one job, and to completely skip over users that have 2 or more jobs.

Here's an example CSV:
ID,LastName,FirstName,JobCode,JobTitle,DeptCode,Dept
1001,Doe,John,100,Programmer,199,IT
1002,Smith,Janice,110,Technician,199,IT
1003,Jordan,Michael,150,Janitor,102,Housekeeping
1003,Jordan,Michael,123,Cook,101,Cafeteria
1004,Johnson,Abigail,142,Nurse,180,Employee Health

Open in new window

The desired resulting CSV is:
ID,LastName,FirstName,JobCode,JobTitle,DeptCode,Dept
1001,Doe,John,100,Programmer,199,IT
1002,Smith,Janice,110,Technician,199,IT
1004,Johnson,Abigail,142,Nurse,180,Employee Health

Open in new window

I'm stumped here, I don't even have a good guess. I'm no farther than "Import-CSV"

Any help would be much appreciated.
0
Cacophony777
Asked:
Cacophony777
1 Solution
 
footechCommented:
$data = Import-Csv test.csv | Group id | Where {$_.Count -eq 1} | Select -expandProperty group

Open in new window

You can then output the data back to a .CSV or whatever.
0
 
Cacophony777Author Commented:
Nice! That was easier than I thought it would be... I've never used the Group-Object command before. Thanks!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now