How to extract links from search results in Java

Hello All,

I am trying to extract urls for products with a certain brand, say "Toshiba" after searching for them in


1. Got to
2. Click on Electronics, search by brand and click on "Toshiba". This lists all the products in Toshiba
3. enter a specific tv model
4. extract url

I am using a crawler for steps 1 and 2 to gather all the urls. For steps 3 and 4, I am thinking about using lucene or a data structure to grab a specific url out of it.

Any suggestions about which GPL licensed crawler to use and technique to parse the search results?

Please let me know. Sample Code would be helpful as well.
Who is Participating?
CEHJConnect With a Mentor Commented:
Try to use available apis wherever possible
Pretty sure Amazon has an api that you can use to avoid scraping
pkrish80Author Commented:
I wanted to add the ability to search for, eBay and other sites as well
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

pkrish80Author Commented:
ok. What if I need to do this for I don't think there are any apis available for
Well then you'd need to scrape it. Try HttpUnit
Derek JensenCommented:
Yeah, or is may lets you do everything all within itself...but I'm not sure what your end goal is, so it may not suit your needs. :-)
pkrish80Author Commented:
Using a combination of APIs and scraping
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.