Solved

How to use ColdFusion to connect to Twitter's new streaming API

Posted on 2010-08-17
5
442 Views
Last Modified: 2014-03-07
Is it possible to connect to the new Twitter streaming API using ColdFusion?

I'm referring to the service documented here: http://dev.twitter.com/pages/streaming_api

I've been able to connect to it using curl.exe and a batch file and data file (see below). Is it possible to get ColdFusion to do the same thing, just capturing the JSON data that's being sent? Once it's captured, it should be easy enough to de-serialize the JSON with CF functions and go to town.

Test.bat: (replace "AnyTwitterAccount:Password" with a twitter account and password)

curl -d @following.txt http://stream.twitter.com/1/statuses/filter.json -u AnyTwitterAccount:Password


following.txt:

follow=12,13,15,16,20,87,48811565,7021362

I'm looking for the mechanism to establish the connection, and capture the JSON data that streams through. After that, no problem.

Many thanks!
0
Comment
Question by:ckryanco
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
5 Comments
 
LVL 10

Expert Comment

by:Mr_Nil
ID: 33531502
You need to use a client that can handle streaming data.  See here for connecting to the streaming api: http://dev.twitter.com/pages/streaming_api_concepts#connecting

A quick look at the curl documentation didn't seem to suggestion either way whether it is or isn't capable of handling streaming data in this fashion.

The streaming API would be more useful on the client side and not on the server side.  eg. making a persistent AJAX connection to stream content into a div on a web page (as a simplistic example)

Server side access to twitter you would be better sticking to the standard API and using CFHTTP or CFINVOKE to make request to the API. If you need help with that - take a look at http://riaforge.org/ There are at least 3 twitter APIs for ColdFusion on there.
0
 

Author Comment

by:ckryanco
ID: 33533495
Thanks. The standard Twitter API works perfectly -- but the 150-requests-per-hour rate limit imposed by Twitter applies there (so a Twitter staffer suggested the streaming API, since it isn't rate-limited, for now at least...)

There's no obvious way to to use ColdFusion to maintain a streaming http connection, and I don't want to install a whole new development platform like PHP or Python (which can do this), just for this one project.

Hence the question about whether there's a way to accomplish this in ColdFusion. Thanks.
0
 
LVL 10

Accepted Solution

by:
Mr_Nil earned 500 total points
ID: 33536920
I'm not sure why a staffer at Twitter would suggest that you use the streaming API for server side applications. They should have pointed you at getting your application white listed : http://dev.twitter.com/pages/rate-limiting#whitelisting

However, you could use cfexecute to to run the curl command and handle the response, BUT....
Unless you are calling the template with a CFExecute from a scheduled task the client (browser) request to the template will attempt to hold open the connection for as long as the browser will let you and only as long as the request timeout setting in the coldfusion administrator or cfsetting tag.

If you are calling the template with cfexecute with a scheduled task then any client side requests will have to wait for the scheduled task to run and complete before they can access any data that is retrieved.

Both of these issues will negate any perceived benefit that you might get from using the streaming API.

Something you could potentially do, but will be exceptionally intensive on your server, would be to let a curl bat file to run continuously as a service, appending data to a file.  You could then use BlazeDS, which is built into ColdFusion 9, and an event gateway, to create a longpolling client side application that could send data back to the client in near-realtime.  That seems entirely overkill when you could just build a javascript, flash/flex, html5 application that accesses the streaming api directly.

You could also try using cfhttp with a request timeout of in excess of 30 seconds (the streaming API connection timeout), but this still has the problem that you would have with the above suggestions as well.

I really would recommend that you think about what it is that you are trying to achieve before progressing further down this line.  

Please note, that switching to PHP or Python will not solve the problem you have if you are going to use these as server side technologies. For example the following quote is from an article on using Python's pycurl to access the streaming api : "You should make sure that it is set to something that is higher than the streaming API's 30-second keep-alive interval so that the connection isn't dropped." http://arstechnica.com/open-source/guides/2010/04/tutorial-use-twitters-new-real-time-stream-api-in-python.ars

0
 

Author Comment

by:ckryanco
ID: 33538285
Thanks again. The Twitter staffer suggested it after we applied for whitelisting, saying that Twitter is being "VERY stingy" about granting whitelisting these days because their system has been running so close to capacity for so long. (We still officially haven't heard back from Twitter about our whitelisting application...)

The streaming API was suggested as a possible way around the rate-limiting problem. But it's sure looking ugly... (This is for an application for news websites that captures tweets from groups of selected Twitter accounts of broad community interest, stores them on the local servers, and automatically posts them directly on the news websites -- such as all the Tweets from the local city/county governments.) It works perfectly using the standard Twitter API -- but Twitter won't even consider whitelisting unless the application is in production. Grrrrr....

0
 
LVL 10

Expert Comment

by:Mr_Nil
ID: 33541438
Have you tried "just" replacing the calls to the normal api with cfhttp calls to the streaming api, treating it as if it were a normal REST api rather than a streaming API.  It might work...

You probably need to thread the requests and give them a 30 second timeout with a fall back to some previously stored data if you don't get anything back. That kind of thing.

Just a thought that went through my mind... probably holds no water...
0

Featured Post

Connect further...control easier

With the ATEN CE624, you can now enjoy a high-quality visual experience powered by HDBaseT technology and the convenience of a single Cat6 cable to transmit uncompressed video with zero latency and multi-streaming for dual-view applications where remote access is required.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A web service (http://en.wikipedia.org/wiki/Web_service) is a software related technology that facilitates machine-to-machine interaction over a network. This article helps beginners in creating and consuming a web service using the ColdFusion Ma…
Article by: kevp75
Hey folks, 'bout time for me to come around with a little tip. Thanks to IIS 7.5 Extensions and Microsoft (well... really Windows 8, and IIS 8 I guess...), we can now prime our Application Pools, when IIS starts. Now, though it would be nice t…
Come and listen to Percona CEO Peter Zaitsev discuss what’s new in Percona open source software, including Percona Server for MySQL (https://www.percona.com/software/mysql-database/percona-server) and MongoDB (https://www.percona.com/software/mongo-…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…

691 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question