[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now


Eliminate Duplicate Rows in CSV with PowerShell

Posted on 2013-10-24
Medium Priority
Last Modified: 2013-10-24
I'm looking for a way to import a CSV file in PowerShell and remove all duplicate entries. I am *not* looking for unique entries. If they're in the CSV more than once, I want all mention of them gone. Using the below example, I want the script to return only people who have one job, and to completely skip over users that have 2 or more jobs.

Here's an example CSV:
1004,Johnson,Abigail,142,Nurse,180,Employee Health

Open in new window

The desired resulting CSV is:
1004,Johnson,Abigail,142,Nurse,180,Employee Health

Open in new window

I'm stumped here, I don't even have a good guess. I'm no farther than "Import-CSV"

Any help would be much appreciated.
Question by:Cacophony777
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
LVL 41

Accepted Solution

footech earned 2000 total points
ID: 39598406
$data = Import-Csv test.csv | Group id | Where {$_.Count -eq 1} | Select -expandProperty group

Open in new window

You can then output the data back to a .CSV or whatever.

Author Closing Comment

ID: 39598508
Nice! That was easier than I thought it would be... I've never used the Group-Object command before. Thanks!

Featured Post

Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to prepare an HTML email signature template file containing dynamic placeholders for users' Azure AD data. Furthermore, it explains how to use this file to remotely set up a department-wide email signature policy in Office …
The Nano Server Image Builder helps you create a custom Nano Server image and bootable USB media with the aid of a graphical interface. Based on the inputs you provide, it generates images for deployment and creates reusable PowerShell scripts that …
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an antispam), the admini…
Please read the paragraph below before following the instructions in the video — there are important caveats in the paragraph that I did not mention in the video. If your PaperPort 12 or PaperPort 14 is failing to start, or crashing, or hanging, …

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question