Solved

retriving information using wget command in Java

Posted on 2013-06-06
11
320 Views
Last Modified: 2013-12-24
Hello, I am trying to write a program using the wget command in java that goes to a website picks a link in that website and click it than goes to the second link clicks it and goes to the third link and grabs a particular string.

An example would be to go to http://www.w3schools.com/ click on "learn SQL" link than click on the "SQL SELECT TOP" link and grab a particular string from that page. I am not sure how to write it.
0
Comment
Question by:yescobar2012
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 3
  • 2
11 Comments
 
LVL 4

Expert Comment

by:BAKADY
ID: 39226523
I don't think wget is your best solution. it isn't available at macs or windows by default.
Use Apache Java Frameworks to powered your application.

use a httpclient to make page requests like:

http://hc.apache.org/httpcomponents-client-ga/index.html

and a html parse to load links like this:

http://tika.apache.org/1.3/parser.html

Regards
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39226540
BAKADY is right - wget is not the right tool. HttpClient though is not really the right tool - it's too low level. You'll find you have to write much less code with something like HtmlUnit
0
 

Author Comment

by:yescobar2012
ID: 39227037
Oh I see... how about HTMLParser
0
Space-Age Communications Transitions to DevOps

ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.

 
LVL 86

Expert Comment

by:CEHJ
ID: 39227224
EDITED
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39227232
Oh I see... how about HTMLParser
What's that?
0
 

Author Comment

by:yescobar2012
ID: 39227315
It is similar to this parsing i found some good example of what i was looking for

http://www.mkyong.com/java/jsoup-html-parser-hello-world-examples/
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39227340
You can use that but there's no need to go so low-level
0
 

Author Comment

by:yescobar2012
ID: 39229783
you had recommended to use HtmlUnit, if I use HtmlUnit can i navigate thru a website  or traverse the website? As what I had mention in my question.

An example would be to go to http://www.w3schools.com/ click on "learn SQL" link than click on the "SQL SELECT TOP" link and grab a particular string from that page. I am not sure how to write it.

Would you have any working examples I can take a look at that navigate thru a website (from a website navigates to a child website and a child website and grabs a String?
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 250 total points
ID: 39229873
if I use HtmlUnit can i navigate thru a website  or traverse the website?
Yes - or i would not have mentioned it ;)

http://htmlunit.sourceforge.net/gettingStarted.html
0
 
LVL 4

Assisted Solution

by:BAKADY
BAKADY earned 250 total points
ID: 39233374
you need a Framework - learn is a hard work - or Basis-Knowledge about HTTP and HTML and build it from scratch.

if you know what you need and what you are doing, it isn't complicate. Something like a little http/https proxy can works with only 300 lines code - including comments and no line bigger that 50 chars.

good luck
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39275453
:)
0

Featured Post

How Do You Stack Up Against Your Peers?

With today’s modern enterprise so dependent on digital infrastructures, the impact of major incidents has increased dramatically. Grab the report now to gain insight into how your organization ranks against your peers and learn best-in-class strategies to resolve incidents.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
ejb wildfly example 2 74
how many cores in note3? 5 78
Java pass by reference 3 21
Powershell- Registry HIVE 6 23
In this post we will learn how to make Android Gesture Tutorial and give different functionality whenever a user Touch or Scroll android screen.
Digital marketing agencies have encountered both the opportunities and difficulties that emerge from working with a wide-ranging organizations.
The Bounty Board allows you to request an article or video on any technical topic, or fulfill a bounty request to earn points. Watch this video to learn how to use the Bounty Board to get the content you want, earn points, and browse submitted bount…
Articles on a wide range of technology and professional topics are available on Experts Exchange. These resources are written by members, for members, and can be written about any topic you feel passionate about. Learn how to best write an article t…

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question