Solved

project to keep track of webpages

Posted on 2011-09-22
4
300 Views
Last Modified: 2012-05-12
I want to develop a project in java & may be in Linux to download web pages n keep track of those web pages which are changing.

First how to download the pages?

Next how do I come to know whether a web page is changing or not?
For example , if i have www.google.com in my database and if I had subscribed for that page, so how do I come to know that google has changed ?

what are the parameters that I should look that I come to know whether google has changed when the server periodically downloads it ?

Do I need to calculate the md5 of the page?

Ideas please>>>>>>
0
Comment
Question by:sriramvemaraju2000
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 
LVL 86

Accepted Solution

by:
CEHJ earned 500 total points
ID: 36582219
Very few pages these days will be static. In the rare cases where they are, and where the web server supports it, you can check

http://www.feedthebot.com/ifmodified.html

Otherwise, yes, a checksumming approach would be one way. If the pages are small enough to hold in memory twice you can do a string comparison - it'll be faster
0
 

Author Comment

by:sriramvemaraju2000
ID: 36582333
can you tell me how to do this in java? what is the best approach?
particularly how to get the web pages and how to track the modifications?
I think this will be useful for security purposes.. I can tell whether my page has hacked or not..
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 36584115
Begin with a java open source web crawler and take it from there - that's what i'd do

http://java-source.net/open-source/crawlers
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 36585698
:)
0

Featured Post

Secure Your Active Directory - April 20, 2017

Active Directory plays a critical role in your company’s IT infrastructure and keeping it secure in today’s hacker-infested world is a must.
Microsoft published 300+ pages of guidance, but who has the time, money, and resources to implement? Register now to find an easier way.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
This article will inform Clients about common and important expectations from the freelancers (Experts) who are looking at your Gig.
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

740 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question