Solved

Web crawlers and bots tutorial

Posted on 2002-03-20
2
1,616 Views
Last Modified: 2012-08-13
Can anyone point me to a good tutorial on how to write web bots and crawlers in java.

Thanks.
0
Comment
Question by:cvcarson
2 Comments
 
LVL 4

Accepted Solution

by:
pellep earned 100 total points
ID: 6884228
I'm not sure how 'smart' of a crawler you aim to develop, but i think you may be in over your head on this one. I have a friend that works for an ISP who wanted to develop their own search-engine. They had a team of 10 developers working for nearly 4 months before they cancelled the project as beeing too large and costly. All the engines out there (google, webcrawler etc) use their own 'intelligence' and they are not at all keen on letting anyone else in on how they do it. If you have a vague concept of what a link crawler or meta crawler does, you pretty much have your tutorial there. The basic idea is pretty simple, but the implementations can be infinitly complex. I may be wrong, of course. Someone might come up with 'the comprehensive guide to developing search-engines', but I doubt it.
0
 

Expert Comment

by:polkadot
ID: 11982945
Actually there are a few books out there I like that could give you a good intro. Programming Spiders, Bots, and Aggregators in Java by Jeff Heaton, is one.
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Android studio getdrawable(int) is deprecated 4 82
groupNoAdj 7 95
Apps blocked by Java 9 79
hibernate example for saving data 19 37
After being asked a question last year, I went into one of my moods where I did some research and code just for the fun and learning of it all.  Subsequently, from this journey, I put together this article on "Range Searching Using Visual Basic.NET …
This was posted to the Netbeans forum a Feb, 2010 and I also sent it to Verisign. Who didn't help much in my struggles to get my application signed. ------------------------- Start The idea here is to target your cell phones with the correct…
Viewers learn about the scanner class in this video and are introduced to receiving user input for their programs. Additionally, objects, conditional statements, and loops are used to help reinforce the concepts. Introduce Scanner class: Importing…
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now