Solved

Web crawlers and bots tutorial

Posted on 2002-03-20
2
1,612 Views
Last Modified: 2012-08-13
Can anyone point me to a good tutorial on how to write web bots and crawlers in java.

Thanks.
0
Comment
Question by:cvcarson
2 Comments
 
LVL 4

Accepted Solution

by:
pellep earned 100 total points
ID: 6884228
I'm not sure how 'smart' of a crawler you aim to develop, but i think you may be in over your head on this one. I have a friend that works for an ISP who wanted to develop their own search-engine. They had a team of 10 developers working for nearly 4 months before they cancelled the project as beeing too large and costly. All the engines out there (google, webcrawler etc) use their own 'intelligence' and they are not at all keen on letting anyone else in on how they do it. If you have a vague concept of what a link crawler or meta crawler does, you pretty much have your tutorial there. The basic idea is pretty simple, but the implementations can be infinitly complex. I may be wrong, of course. Someone might come up with 'the comprehensive guide to developing search-engines', but I doubt it.
0
 

Expert Comment

by:polkadot
ID: 11982945
Actually there are a few books out there I like that could give you a good intro. Programming Spiders, Bots, and Aggregators in Java by Jeff Heaton, is one.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
create a gui in perl 3 72
maven example error 3 52
Strange loading of website behaviour 3 32
replace a word with other 1 40
Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Viewers will learn about the different types of variables in Java and how to declare them. Decide the type of variable desired: Put the keyword corresponding to the type of variable in front of the variable name: Use the equal sign to assign a v…
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now