Improve company productivity with a Business Account.Sign Up

x
?
Solved

How to fetch  live data from website ?

Posted on 2012-03-22
7
Medium Priority
?
692 Views
Last Modified: 2012-03-24
How to  fetch  live data from website  ( without RSS feed / Webservice )?  

Do you  use any API ?

Example:  Live cricket scores


I want to write a standalone java program which will fetch  live data from  site. How do I start ?  What are the things I require ?

Could you throw some light .
0
Comment
Question by:cofactor
7 Comments
 
LVL 30

Expert Comment

by:IanTh
ID: 37751683
you need a database so the web site works and you can view the data remotely

Do you use any api

that depends on your setup I expect it will be linux and you can install mysql its usually installed anyway although it can be installed in windows

for remote view
see
http://www.softpedia.com/get/Internet/Servers/Database-Utils/Remote-MySQL-Viewer.shtml
0
 
LVL 40

Accepted Solution

by:
Gurvinder Pal Singh earned 2000 total points
ID: 37751860
you can invoke that URL using HTTPURLConnection
http://docs.oracle.com/javase/tutorial/networking/urls/readingWriting.html

once you have got the content of that URL, then you can parse it using Xpath
http://www.rgagnon.com/javadetails/java-0550.html
http://htmlparser.sourceforge.net/
0
 
LVL 40

Expert Comment

by:Gurvinder Pal Singh
ID: 37751864
you may have to use jTidy to purify HTML
http://jtidy.sourceforge.net/
0
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

 

Author Comment

by:cofactor
ID: 37752007
@gurvinder372
>>>you can invoke that URL using HTTPURLConnection

Is not that involve a  loop ?  something like this ...

while(true)
{
//HTTPURLConnection ...get data from site
//parse and get  required data
  Thread.sleep(30 secs)
}


OR   there  is some other smart and better approach exists I'm missing here ?


I  know a good package which  is capable of doing this kind of work neatly.  
http://httpunit.sourceforge.net/doc/faq.html

But I'm not sure whether you'll be using a loop to hit server every 30 secs interval to fetch latest updates  or  there is  better  a way ?


Please post you  comments
0
 
LVL 40

Expert Comment

by:Gurvinder Pal Singh
ID: 37752211
smarter approach would be to use web-services exposed by that site to use their data.

for example, google services, or facebook services, or twitter services
0
 
LVL 47

Expert Comment

by:for_yan
ID: 37755883
This is a simple working example of implementation of getting RSS feed form
the list of sites

It uses ROME API, let me find the appropriate link

import java.io.*;
import java.net.URL;
import java.util.ArrayList;
import java.util.Date;
import java.util.Iterator;
import java.util.StringTokenizer;

import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.feed.synd.SyndFeed;
import com.sun.syndication.io.SyndFeedInput;
import com.sun.syndication.io.XmlReader;

public class Reader {

 static String [] urls =  {"http://rss.cnn.com/rss/cnn_topstories.rss", "http://www.nytimes.com/services/xml/rss/nyt/World.xml",
 "http://www.nytimes.com/services/xml/rss/nyt/US.xml","http://newsrack.in/crawled.feeds/frontline.rss.xml"
};
public static void main(String[] args) throws Exception {                              

//URL url = new URL("http://viralpatel.net/blogs/feed");

  //  URL url = new URL("http://rss.cnn.com/rss/cnn_topstories.rss");
XmlReader reader = null;

try {

    while(true) {

long now = (new java.util.Date()).getTime();

    System.out.println("checking at " + (new Date(now)).toString());    
long week_before = now - (24L*3600L*7L*1000L);


ArrayList list = new ArrayList();

DataInputStream in = new DataInputStream(new FileInputStream("C:\\temp\\test\\visited.txt"));
PrintStream psout = new PrintStream(new FileOutputStream("C:\\temp\\test\\visited1.txt"));

String buff = null;

while((buff=in.readLine()) != null){
StringTokenizer t = new StringTokenizer(buff);
String ttime = t.nextToken();
if(Long.parseLong(ttime) <week_before)continue;
String llink = t.nextToken().trim();
list.add(llink);
psout.println(ttime + " " + llink);

}

          for (int jj=0; jj<urls.length; jj++){
            URL  url = new URL(urls[jj]);
reader = new XmlReader(url);
SyndFeed feed = new SyndFeedInput().build(reader);
//System.out.println("Feed Title: "+ feed.getAuthor());

for (Iterator i = feed.getEntries().iterator(); i.hasNext();) {
SyndEntry entry = (SyndEntry) i.next();
    String title = entry.getTitle();
     String link = entry.getUri().trim();
     if(list.contains(link))continue;
     Date date = entry.getPublishedDate();
// Problem here -->         **     SyndEntry source = item.getSource();
     String description;
     if (entry.getDescription()== null){
       description = "";
     } else {
       description = entry.getDescription().getValue();
     }
     String cleanDescription = description.replaceAll("\\<.*?>","").replaceAll("\\s+", " ");
        System.out.println(title);
System.out.println(link);
    System.out.println(cleanDescription);



//System.out.println(entry.getTitle());
//System.out.println(entry.getContents());
    System.out.println("");
     System.out.println("");
     psout.println(now + " " + link);
}

}

in.close();
psout.close();
File f0 = new File("C:\\temp\\test\\visited.txt");
File f1 = new File("C:\\temp\\test\\visited1.txt");
f0.delete();
f1.renameTo(f0);

Thread.currentThread().sleep(600000);

    }


} finally {
if (reader != null)
reader.close();
}
}
}
                                            

Open in new window

0
 
LVL 47

Expert Comment

by:for_yan
ID: 37755889
0

Featured Post

The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
Viewers learn about the scanner class in this video and are introduced to receiving user input for their programs. Additionally, objects, conditional statements, and loops are used to help reinforce the concepts. Introduce Scanner class: Importing…

606 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question