Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

Troubleshooting
Research
Professional Opinions
Ask a Question
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

troubleshooting Question

Best Scraping/Crawling/Spidering Tools

Avatar of Anupama_2005
Anupama_2005Flag for India asked on
Software
1 Comment1 Solution1347 ViewsLast Modified:
.

Hi All,
         We have a tough task of developing 10,000 scrapers for scraping/crawling/spidering 10,000 different websites in a very short period of time, say 1 -2 months.

So we are looking out for paid/free best website scraping tools that are available in the market.

The tool basically need to do the following:

•      Should be configurable.

•      Should crawl/spider/scrape the given website by taking and posting the required parameters for a given search criteria.

•      Should capture the results (from the website for a given search query) and return them in the form of text/xml files or can insert that data into the database.

•      Should work on Windows (First Preference) / Linux Servers.



If any one have good idea on these tools then please let us know the tool names and where we can get those tools. It would be of great help to us.


Many Thanks and Best Regards.
ASKER CERTIFIED SOLUTION
Avatar of Sohel Rana
Sohel RanaFlag of Australia image

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Commented:
This problem has been solved!
Unlock 1 Answer and 1 Comment.
See Answers