Solved

is it possible to normalize a rss feed to remove duplicate links?

Posted on 2013-12-16
6
252 Views
Last Modified: 2014-01-11
I have a rss feed from twitter list.

This often has duplicates - tweets that have the same link

How can I remove the tweets from the rss that have the dupes?
0
Comment
Question by:finnstone
  • 3
  • 3
6 Comments
 
LVL 52

Expert Comment

by:Scott Fell, EE MVE
ID: 39723175
I don't know perl but I think I would do this client side anyway.  Create an array of links.  For each post, first find the links and see if they are in the current array.   If not, add the link to the array.  If the link is in the array, then don't use it.  If there are not that many tweets you are checking, there may be no difference client/serverside.
0
 

Author Comment

by:finnstone
ID: 39723195
does not have to be perl, i didnt know where to post this
0
 
LVL 52

Expert Comment

by:Scott Fell, EE MVE
ID: 39723213
What language do you work in?  I'm sure there will be multiple ways to do this in any language.  

You would want to look at a post, then find each string starting, "<a " and ending in "</a>".  Then from the complete string <a href="http://mypage.com/link">Link</a> take just the mypage.com/link and add that to your array.

Then for each post look for a link, if the, "mypage.com/link" matches what is in the array, then do not use the link or post.  

I think you will find this harder though because the same link may be shortened with different url shorteners. If that is not a problem, this should work.
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:finnstone
ID: 39723216
I do not. I am  going to hire someone and am budget insensitive.
0
 

Author Comment

by:finnstone
ID: 39723217
Yes, but I think there is a system that could return true URL
0
 
LVL 52

Accepted Solution

by:
Scott Fell,  EE MVE earned 500 total points
ID: 39723228
>Yes, but I think there is a system that could return true URL
There are going to be multiple ways to do this in any given language.  No matter what it will include some sort of search for a string with given parameters.

How you end up using it and displaying and how many tweets you will end up checking will help determine the best solution.

If you want to see the least amount of code, you would use regex.  But even regex may be slightly different from language to language.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Send SMS with a web app to a SMS Gateway EMG ( Enterprise Messaging Gateway) 4 58
LAMP problem identifier tool ? 9 110
site launch date and last modified date 3 83
WEB Farm 6 64
If I have to fix slow responding website my first thoughts are server side optimizations: the database may not be optimized or caching is not enabled, or things like that. We often overlook another major part of our web application: the client. We o…
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
The purpose of this video is to demonstrate how to set up the WordPress backend so that each page automatically generates a Mailchimp signup form in the sidebar. This will be demonstrated using a Windows 8 PC. Tools Used are Photoshop, Awesome…
Learn how to set-up PayPal payment integration in your Wufoo form. Allow your users to remit payment through PayPal upon completion of your online form. This is helpful for collecting membership payments, customer payments, donations, and more.

896 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now