Solved

web crawler vs mirroring web site?

Posted on 2011-09-21
2
414 Views
Last Modified: 2013-12-06
I am thinking they are same thing but
I am trying to write up a crawler that will download everything from the given URL.
The web crawler only looks for the hyperlinked <a> tag, is that correct?
What if the web folder/subfolders which are not linked on the URL?
Hoes it find those?
I guess I don't understand what crawler exactly does.
Can you explain this?
0
Comment
Question by:dkim18
  • 2
2 Comments
 
LVL 47

Expert Comment

by:for_yan
ID: 36575128
0
 
LVL 47

Accepted Solution

by:
for_yan earned 500 total points
ID: 36575193

You can read also this about mirrors:
http://www8.org/w8-papers/4c-server/mirror/mirror.html

I was thinking that by crawler we mean a program which tries to cover many web sites maybe related to some topic
and index the links so that effective serach becomes available (I guess Google has the ultimate web crawler)

Mirror is something which is focused on one particular site and makes the full copy of it -
and has quite different purpose of giveing access to this site to seom categorties of users
having additional copy just in case,ecetc., etc - see article about mirrors

I think mirrors can often be by arrangements with the mirrored site,
whereas crawlers do not need to be

At least that is my understaniding
 
0

Featured Post

Networking for the Cloud Era

Join Microsoft and Riverbed for a discussion and demonstration of enhancements to SteelConnect:
-One-click orchestration and cloud connectivity in Azure environments
-Tight integration of SD-WAN and WAN optimization capabilities
-Scalability and resiliency equal to a data center

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Online collaboration is quickly becoming embedded in the workplace, and its benefits are tangible. See what the current landscape looks like and what the future holds for collaboration tools and the future of work.
The article shows the basic steps of integrating an HTML theme template into an ASP.NET MVC project
Viewers learn how to read error messages and identify possible mistakes that could cause hours of frustration. Coding is as much about debugging your code as it is about writing it. Define Error Message: Line Numbers: Type of Error: Break Down…
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.

792 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question