We help IT Professionals succeed at work.

powershell to find duplicate records in a text file and give me a sum of each duplicate record

fireburn11 asked
1 Endorsement
Last Modified: 2012-03-27
hi all

I have a text file that contains 14000 IP addresses. there are 5680 unique IPs in the file.

I need a powershell script to look into the text file and tell me the sum of duplicate records for each unique IP.

basically, i am thinking a script that read the first line and compare it to the rest of IPs in the file and if there is a match, sum+=1, at the end, gives me the total sum of match, then move on to the second record , compare it with all rest of the IPs...

Can someone help?

Watch Question

Simple way is to copy the text to an Excel file and follow the steps to find the unique IP address
1.Select the cells
2.Go to Data and click Filter
3.Click Advanced filter
4.Copy to another location
5.Choose another area to copy the unique IP address
6.Click on Unique records only
7.Click O.K
Dale HarrisProfessional Services Engineer

I'm thinking a simple sort-object -unique would do it for powershell.  But would that give you the occurrences of each match?  No.  The output will be huge and ugly.

What exactly are you going for here?

Maybe we can help if you give us some backstory.


I am going after the total occurences of each unique record.
Its really a DNS even log, we need to know in 3 minutes, how many times a IP has requested DNS to our DC.

So I have the DNS event log with all the client IPs, I need to know the total occurences of each IP.

Dale HarrisProfessional Services Engineer
This one is on us!
(Get your first solution completely free - no credit card required)


awesome! Thanks for the excel tip! works beautifully.
Dale HarrisProfessional Services Engineer

I'm glad it worked for you Fireburn!  If you have any other issues, please let us know how we can help.  At this point, I recommend you assign points and let this be a possible solution for someone in the future.

"Batchelor", Developer and EE Topic Advisor
Top Expert 2015
This one is on us!
(Get your first solution completely free - no credit card required)

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.