How to extract links from Amazon.com search results in Java

Hello All,

I am trying to extract urls for products with a certain brand, say "Toshiba" after searching for them in Amazon.com.

Steps:

1. Got to Amazon.com
2. Click on Electronics, search by brand and click on "Toshiba". This lists all the products in Toshiba
3. enter a specific tv model
4. extract url

I am using a crawler for steps 1 and 2 to gather all the urls. For steps 3 and 4, I am thinking about using lucene or a data structure to grab a specific url out of it.

Any suggestions about which GPL licensed crawler to use and technique to parse the search results?

Please let me know. Sample Code would be helpful as well.
pkrish80Asked:
Who is Participating?
 
CEHJConnect With a Mentor Commented:
Try to use available apis wherever possible
0
 
CEHJCommented:
Pretty sure Amazon has an api that you can use to avoid scraping
0
 
pkrish80Author Commented:
I wanted to add the ability to search for buy.com, eBay and other sites as well
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
pkrish80Author Commented:
ok. What if I need to do this for buy.com? I don't think there are any apis available for buy.com
0
 
CEHJCommented:
Well then you'd need to scrape it. Try HttpUnit
0
 
Derek JensenCommented:
Yeah, or www.screen-scraper.com is may fav...it lets you do everything all within itself...but I'm not sure what your end goal is, so it may not suit your needs. :-)
0
 
pkrish80Author Commented:
Using a combination of APIs and scraping
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.