building a bot

Posted on 2008-10-08
Medium Priority
Last Modified: 2013-12-08
suppose i want to build a web bot that will scan only the main pages of all websites in some country.
where do i start from?
Question by:Sasha-N
  • 3
  • 2
LVL 13

Accepted Solution

Onthrax earned 2000 total points
ID: 22673735
It depends on the language you want to write this in.

The correct search term you need to google your answer is:
'creating web spider'

For example a tutorial in .NET:

Or in Java:

Hope this helps

Author Comment

ID: 22673946
so i dont need special hardware and i can just run it from home server?
how much time do you think will take to scan all the net domains?
LVL 13

Expert Comment

ID: 22676379
Correct. Although I imagine it will take a long long time with only your own machine to spider ALL the net domains.

For example Google has many many server parks consisting of hundreds of computers in every country.

Author Comment

ID: 22677001
suppose i want to scan all the sites in russia- with .ru domain
will it take about a month? or about an year?
LVL 13

Expert Comment

ID: 22677091
I really couldn't make an estimate m8. It all depends on a lot of factors.

For example:
- The method you will be using to spider domains. e.g. loop through all possible domains like a.ru, b.ru, ab.ru, abc.ru etc. or spider a few .ru sites and fetch links from those etc.
- The capacity of your machine. A pentium 1 would take longer than a powerfull Quad core.
- The bandwidth available and it's speed
- The size of the webpages you will be spidering. A single page with only a few words will be faster done than a site with a huge page and a lot of backpages.

Imagine google not having indexed the entire internet yet with all their machines. It's a huge job..


Featured Post

Get 10% Off Your First Squarespace Website

Ready to showcase your work, publish content or promote your business online? With Squarespace’s award-winning templates and 24/7 customer service, getting started is simple. Head to Squarespace.com and use offer code ‘EXPERTS’ to get 10% off your first purchase.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

An ASP.NET Web Form User Control is not newly introduced in ASP.NET. In fact, it was an old technology yet still playing a role to generate web content, especially when we want to use it to have a better and easy way to control part of the web conte…
I recently worked on a Wordpress site that utilized the popular ContactForm7 (https://contactform7.com/) plug-in that only sends an email and does not save data. The client wanted the data saved to a custom CRM database. This is my solution.
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This video teaches users how to migrate an existing Wordpress website to a new domain.

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question