Powerful web crawler/spider application

Posted on 2006-03-26
Last Modified: 2008-01-09
I am looking for the most powerful web crawler program I can find at any cost. I would like to spider a list of websites from a mysql database and return, metas, text, etc, back to the database. I would also like it be able to handle as many different types of docs as possible. Has anyone had good experiences with any of them? I have used a number of freeware versions, but they are far too unstable... many returning only 75% of sites back. If it helps, I don't need to go beyond one page with the crawler either.

Thanks for the input,
Question by:cbonnock
    LVL 4

    Accepted Solution

    Here's a couple>>

    $99.99 - WA Site Import 2.0

    $39.95 - SurfOffline 1.4.
    30-day Trial period
    LVL 4

    Expert Comment

    "If it helps, I don't need to go beyond one page with the crawler either." Try this>>>

    $19.90 - iPageSaver 1.0
    Free to Try

    Author Comment

    These are not quite what I am looking for...

    Featured Post

    Highfive + Dolby Voice = No More Audio Complaints!

    Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

    Join & Write a Comment

    Suggested Solutions

    If your app took Google’s lash recently, here are the 5 most likely reasons.
    The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
    The viewer will learn common shortcuts with easy ways to remember them. The viewer will then learn where to find all of the keyboard shortcuts, how to create/change them, and how to speed up their workflow.
    The viewer will learn how to create multiple layers to apply various filters and how to delete areas from each layer’s filter.

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now