Solved

web crawler vs mirroring web site?

Posted on 2011-09-21
2
413 Views
Last Modified: 2013-12-06
I am thinking they are same thing but
I am trying to write up a crawler that will download everything from the given URL.
The web crawler only looks for the hyperlinked <a> tag, is that correct?
What if the web folder/subfolders which are not linked on the URL?
Hoes it find those?
I guess I don't understand what crawler exactly does.
Can you explain this?
0
Comment
Question by:dkim18
  • 2
2 Comments
 
LVL 47

Expert Comment

by:for_yan
ID: 36575128
0
 
LVL 47

Accepted Solution

by:
for_yan earned 500 total points
ID: 36575193

You can read also this about mirrors:
http://www8.org/w8-papers/4c-server/mirror/mirror.html

I was thinking that by crawler we mean a program which tries to cover many web sites maybe related to some topic
and index the links so that effective serach becomes available (I guess Google has the ultimate web crawler)

Mirror is something which is focused on one particular site and makes the full copy of it -
and has quite different purpose of giveing access to this site to seom categorties of users
having additional copy just in case,ecetc., etc - see article about mirrors

I think mirrors can often be by arrangements with the mirrored site,
whereas crawlers do not need to be

At least that is my understaniding
 
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
The article shows the basic steps of integrating an HTML theme template into an ASP.NET MVC project
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question