Solved

project to keep track of webpages

Posted on 2011-09-22
4
302 Views
Last Modified: 2012-05-12
I want to develop a project in java & may be in Linux to download web pages n keep track of those web pages which are changing.

First how to download the pages?

Next how do I come to know whether a web page is changing or not?
For example , if i have www.google.com in my database and if I had subscribed for that page, so how do I come to know that google has changed ?

what are the parameters that I should look that I come to know whether google has changed when the server periodically downloads it ?

Do I need to calculate the md5 of the page?

Ideas please>>>>>>
0
Comment
Question by:sriramvemaraju2000
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 
LVL 86

Accepted Solution

by:
CEHJ earned 500 total points
ID: 36582219
Very few pages these days will be static. In the rare cases where they are, and where the web server supports it, you can check

http://www.feedthebot.com/ifmodified.html

Otherwise, yes, a checksumming approach would be one way. If the pages are small enough to hold in memory twice you can do a string comparison - it'll be faster
0
 

Author Comment

by:sriramvemaraju2000
ID: 36582333
can you tell me how to do this in java? what is the best approach?
particularly how to get the web pages and how to track the modifications?
I think this will be useful for security purposes.. I can tell whether my page has hacked or not..
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 36584115
Begin with a java open source web crawler and take it from there - that's what i'd do

http://java-source.net/open-source/crawlers
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 36585698
:)
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Make the most of your online learning experience.
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
This tutorial will introduce the viewer to VisualVM for the Java platform application. This video explains an example program and covers the Overview, Monitor, and Heap Dump tabs.
Suggested Courses

617 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question