Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Writing Twitter data to CSV file with Python

Posted on 2013-01-17
5
Medium Priority
?
2,050 Views
Last Modified: 2013-01-18
Hi there,

I'm using Python and the Twitter API to scrape tweets that use a particular hashtag. The end goal is to scrape 1500 tweets, determine which users tweeted the most, then list the top 10 users who tweeted the most (its for a small school assignment).

I've managed to get my code to the point where I have my top 10 users in a dictionary but can't seem to figure out how to export my data to a CSV file. I've found a few solutions on Stackoverflow but I couldn't get them to work.

Any help would be greatly appreciated. My code is attached. Thanks in advance!
0
Comment
Question by:Adam
  • 3
  • 2
5 Comments
 
LVL 29

Expert Comment

by:pepr
ID: 38791956
(You forgot to attach your solution, anyway...)

You should definitely use the standard csv module. Try the following:
#!python2
import csv
fname_output = 'twitter_users.csv'

# The simulated dictionary.
d = {'user1': 'msg1 or whatever',
     'user2': 'msg2',
     'user3': 'msg3'
    }

# Open the otput file for writing in binary mode.
with open(fname_output, 'wb') as f:
    
    # Wrap the file object by the writer.
    writer = csv.writer(f)
    
    # Write the header line to the output if needed.
    writer.writerow(('USER', 'DATA'))
    
    # Loop through the dictionary and get the tuples.
    # You may want to sort it somehow. Not shown here.
    for t in d.iteritems():
        writer.writerow(t)
        
        
# Note: When using the 'with' construct, the output file is closed automatically.

Open in new window


When using Python 3, the file open slightly differs because of the difference in how strings are treated.
0
 
LVL 1

Author Comment

by:Adam
ID: 38792074
Thanks for the reply pepr. Really strange, I definitely had the code attached when I submitted the question. I'll post it instead of attaching it. If you could take a look at what I have so far, it would really be helpful.

In the meantime, I will try to work in your sample code with mine and see what I come up with. Thanks.

import twitter, json, operator, csv
#Construct Twitter API object
searchApi = twitter.Twitter(domain="search.twitter.com")
# Hashtag query
query = "#deathstar"
# Dictionary of tweet details
tweeters = dict()

# Loop through 15 pages of tweets = 1500 tweets
for i in range(1,16):
	response = searchApi.search(q=query, rpp=100, page=i)
	tweets = response['results']
	for item in tweets:
		tweet = json.loads(json.dumps(item))
		user = tweet['from_user_name']
		# If user tweets more then once, count the number of times
		if user in tweeters:
			tweeters[user] += 1
		else:
			tweeters[user] = 1

# Sort users from most tweets to least
sorted_tweeters = sorted(tweeters.iteritems(), key=operator.itemgetter(1), reverse=True)
# Print the top 10 users who tweeted the most
print sorted_tweeters[0:10]

Open in new window

0
 
LVL 29

Accepted Solution

by:
pepr earned 2000 total points
ID: 38792178
I do not know what twitter module you have. Anyway, the sorted_tweeters should be fine for the purpose. Try to add:
with open(fname_output, 'wb') as f:
    writer = csv.writer(f)
    writer.writerow(('USER', 'DATA'))  # only if you want it
    for t in sorted_tweeters[0:10]:
        writer.writerow(t)

Open in new window


A side note: You will probably observe less problem in future when using 4 spaces for one indentation instead of tabs (no tabs in sources). But it is your decision ;)
0
 
LVL 1

Author Closing Comment

by:Adam
ID: 38792402
Thanks for your help pepr! This worked great. Cheers!
0
 
LVL 29

Expert Comment

by:pepr
ID: 38792408
:)
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this blog, I will share you some basic tips for content marketing and to rank your website on Google.
There are many other benefits to creating an inbound marketing strategy. Let’s take a look at five of the biggest and how they'll affect your business.
Learn the basics of while and for loops in Python.  while loops are used for testing while, or until, a condition is met: The structure of a while loop is as follows:     while <condition>:         do something         repeate: The break statement m…
Learn how to set-up PayPal payment integration in your Wufoo form. Allow your users to remit payment through PayPal upon completion of your online form. This is helpful for collecting membership payments, customer payments, donations, and more.

886 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question