Solved

web crawler vs mirroring web site?

Posted on 2011-09-21
2
420 Views
Last Modified: 2013-12-06
I am thinking they are same thing but
I am trying to write up a crawler that will download everything from the given URL.
The web crawler only looks for the hyperlinked <a> tag, is that correct?
What if the web folder/subfolders which are not linked on the URL?
Hoes it find those?
I guess I don't understand what crawler exactly does.
Can you explain this?
0
Comment
Question by:dkim18
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
2 Comments
 
LVL 47

Expert Comment

by:for_yan
ID: 36575128
0
 
LVL 47

Accepted Solution

by:
for_yan earned 500 total points
ID: 36575193

You can read also this about mirrors:
http://www8.org/w8-papers/4c-server/mirror/mirror.html

I was thinking that by crawler we mean a program which tries to cover many web sites maybe related to some topic
and index the links so that effective serach becomes available (I guess Google has the ultimate web crawler)

Mirror is something which is focused on one particular site and makes the full copy of it -
and has quite different purpose of giveing access to this site to seom categorties of users
having additional copy just in case,ecetc., etc - see article about mirrors

I think mirrors can often be by arrangements with the mirrored site,
whereas crawlers do not need to be

At least that is my understaniding
 
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

With the withdrawal of support for Windows Server 2003 this summer, many clients face the issue of moving away from their 2003 installs. There are a few options out there that many people/companies are selling. But the clients I have, haven't wanted…
Online collaboration is quickly becoming embedded in the workplace, and its benefits are tangible. See what the current landscape looks like and what the future holds for collaboration tools and the future of work.
Viewers will learn about basic arrays, how to declare them, and how to use them. Introduction and definition: Declare an array and cover the syntax of declaring them: Initialize every index in the created array: Example/Features of a basic arr…
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question