Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

web crawler vs mirroring web site?

Posted on 2011-09-21
2
Medium Priority
?
436 Views
Last Modified: 2013-12-06
I am thinking they are same thing but
I am trying to write up a crawler that will download everything from the given URL.
The web crawler only looks for the hyperlinked <a> tag, is that correct?
What if the web folder/subfolders which are not linked on the URL?
Hoes it find those?
I guess I don't understand what crawler exactly does.
Can you explain this?
0
Comment
Question by:dkim18
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
2 Comments
 
LVL 47

Expert Comment

by:for_yan
ID: 36575128
0
 
LVL 47

Accepted Solution

by:
for_yan earned 2000 total points
ID: 36575193

You can read also this about mirrors:
http://www8.org/w8-papers/4c-server/mirror/mirror.html

I was thinking that by crawler we mean a program which tries to cover many web sites maybe related to some topic
and index the links so that effective serach becomes available (I guess Google has the ultimate web crawler)

Mirror is something which is focused on one particular site and makes the full copy of it -
and has quite different purpose of giveing access to this site to seom categorties of users
having additional copy just in case,ecetc., etc - see article about mirrors

I think mirrors can often be by arrangements with the mirrored site,
whereas crawlers do not need to be

At least that is my understaniding
 
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many of the companies I’ve worked with have embraced cloud solutions due to their desire to “get out of the datacenter business.” The ability to achieve better security and availability, and the speed with which they are able to deploy, is far grea…
During and after that shift to cloud, one area that still poses a struggle for many organizations is what to do with their department file shares.
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.
Suggested Courses

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question