Recommendations for web scraping based on certain criteria?
Well, experts, I have a bit of a challenge ...
I'm unfamiliar with what is available in the data collection/web scraping arena (either free or chargeable). Bottom line, this is the type of information I need to get:
Photography-related websites or blogs (NOT photographers) which meet certain SEO criteria (a high traffic him of visitors would be a good example). I know there are numerous ways to gauge traffic (Alexa rank, back links, etc.). The ideal information, although I have no idea how it could be obtained, would be the number of visitors (either monthly, annually, etc.). The other critical piece of information is an e-mail address by which I could contact each website or blog (typically found on most websites under one or more the following categories: support, information, contact, etc.).
The ultimate goal is to assemble a list of at least several hundred (I would hope something more like several thousand would be more likely) websites that meet the criteria. I guess the minimum criteria would be: URL, brief website description, some indication of traffic rank, and e-mail address. The other criteria are harder to define for purposes of this post, but since I'm just trying to get a handle on this whole web scraping-data collection area, I don't want to muddy the waters with difficult to understand selection criteria.
I've done numerous searches, all of which have not resulted in anything close to what I need. My hope is that someone at EE is aware of an online or standalone software package which could supply most or all of what I need. As another option, I suppose purchasing an e-mail list is an option; however, I have never done that either so I don't know where to start.
I do not program so any solution involving that, would not work in my case. I also don't have the time or money to have custom programs developed to accomplish this, (unless my idea of what it would require is much more than what it actually would take).
I can't help but believe that somewhere, someone has developed this type of software tool, but I have no idea where to even start looking. Any suggestions or guidance would be greatly appreciated. Thank you.
If anyone has a suggestion as to a better zone to identify, please let me know because I don't understand what half of the zones mean anyway.
I'm unfamiliar with what is available in the data collection/web scraping arena (either free or chargeable). Bottom line, this is the type of information I need to get:
Photography-related websites or blogs (NOT photographers) which meet certain SEO criteria (a high traffic him of visitors would be a good example). I know there are numerous ways to gauge traffic (Alexa rank, back links, etc.). The ideal information, although I have no idea how it could be obtained, would be the number of visitors (either monthly, annually, etc.). The other critical piece of information is an e-mail address by which I could contact each website or blog (typically found on most websites under one or more the following categories: support, information, contact, etc.).
The ultimate goal is to assemble a list of at least several hundred (I would hope something more like several thousand would be more likely) websites that meet the criteria. I guess the minimum criteria would be: URL, brief website description, some indication of traffic rank, and e-mail address. The other criteria are harder to define for purposes of this post, but since I'm just trying to get a handle on this whole web scraping-data collection area, I don't want to muddy the waters with difficult to understand selection criteria.
I've done numerous searches, all of which have not resulted in anything close to what I need. My hope is that someone at EE is aware of an online or standalone software package which could supply most or all of what I need. As another option, I suppose purchasing an e-mail list is an option; however, I have never done that either so I don't know where to start.
I do not program so any solution involving that, would not work in my case. I also don't have the time or money to have custom programs developed to accomplish this, (unless my idea of what it would require is much more than what it actually would take).
I can't help but believe that somewhere, someone has developed this type of software tool, but I have no idea where to even start looking. Any suggestions or guidance would be greatly appreciated. Thank you.
If anyone has a suggestion as to a better zone to identify, please let me know because I don't understand what half of the zones mean anyway.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
I'm not aware of any specific products that do this - but it seems odd that sites which aggregate and sell traffic data (like comScore) wouldn't provide these sorts of search tools. Seems like an obvious need if you're looking to identify traffic levels or competitors in a particular industry sector. Did you try contacting them to make sure they can't meet this need?
Doug
Doug
ASKER
Doug,
I did not contact them yet. I base my conclusion on going through their website and reviewing their products/services. But I will contact them to find out for sure. Thanks again.
I did not contact them yet. I base my conclusion on going through their website and reviewing their products/services. But I will contact them to find out for sure. Thanks again.
ASKER
Unfortunately, I was right about them. However, I did find somebody through oDesk who has experience working with the Firefox add-on" SE0 quake, which will provide most of the information I need… I think.
ASKER
I think I understand what you're saying. however, I'm not sure how to get everything except the e-mail addresses. I looked at the comscore site and I couldn't find any category or product which correlated with what I am looking for. Do you know of any online or downloadable software products which do this?
Thanks