Solved

how to download a webpage including images,applets,..

Posted on 2004-09-15
12
156 Views
Last Modified: 2012-05-05
Hi,

How can I write a routine that will download a web page and store it on my hard disk.  It also has to download all associated objects including Images,Applets, etc ?

for eg:if i type "www.yahoo.com" ,i should download the complete webpage including all.

i have tried with the following code i can not able to get the images

                        URL url=new URL("http://www.kumudam.com");
                        InputStream conn=url.openStream();
              BufferedReader input = new BufferedReader(new InputStreamReader(conn));
                    String line ;
                        byte data[]= new byte[1000];
                        int size = input.read();
                        System.out.println(size);
                        while ((line = input.readLine()) != null)
                  
                            System.out.println(line);
                            input.close();
Thanks in advance
0
Comment
Question by:vihar123
  • 4
  • 3
  • 2
  • +1
12 Comments
 
LVL 92

Accepted Solution

by:
objects earned 250 total points
ID: 12071200
you'll need to parse the html, here's an example showing how to extract all the links.

http://www.javaalmanac.com/egs/javax.swing.text.html/GetLinks.html

You can then download the data pointed to in the link.
0
 
LVL 21

Expert Comment

by:MogalManic
ID: 12071580
objects is right.

When a browser access a page, it loads the page and all of its assocated objects through a series of GETs.  The first get is the HTML of course.  Then the browser parses the HTML and as it encounters a tag that refers to an external resource it issues another GET to retrieve and render the resource.  Your code would have to do something simalar.
0
 
LVL 18

Expert Comment

by:armoghan
ID: 12071997
and when you get the image links as described by the link given by objects,
you can save the images to local file system like

http://forum.java.sun.com/thread.jsp?thread=433352&forum=31&message=1940583

basically you would be looking for
HTML.Attribute.IMG

you will need to find the <APPLET tag as well for downloading applets


0
 

Author Comment

by:vihar123
ID: 12083034
hi objects,

i have seen the same code and tried i am getting javax.swing.text.changedcharsetexception,i cannot able to rectify.

pls help me  :)
0
 
LVL 92

Expert Comment

by:objects
ID: 12089306
doc.putProperties("IgnoreCharacterSet", new Boolean(true));
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:vihar123
ID: 12224618
hi,
this HTML parser is not working for all websites.pls help me out :)
0
 
LVL 92

Expert Comment

by:objects
ID: 12224634
yes if it not standard html, or uses recent features it will have problems.
0
 

Author Comment

by:vihar123
ID: 12228311
hi objects,
what to do in this case? any idea..
0
 
LVL 92

Expert Comment

by:objects
ID: 12232177
you need to look at a different parser, perhaps a commercial one.
0
 
LVL 18

Assisted Solution

by:armoghan
armoghan earned 250 total points
ID: 12234430
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
eclipse compiler vs Installed JREs option 3 77
Java SE 8u111  Lot of stuff broke 11 54
Error in @AspectJ Based AOP with Spring 2 5
servlet example 17 12
For customizing the look of your lightweight component and making it look lucid like it was made of glass. Or: how to make your component more Apple-ish ;) This tip assumes your component to be of rectangular shape and completely opaque. (COD…
By the end of 1980s, object oriented programming using languages like C++, Simula69 and ObjectPascal gained momentum. It looked like programmers finally found the perfect language. C++ successfully combined the object oriented principles of Simula w…
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now