Solved

Writing Twitter data to CSV file with Python

Posted on 2013-01-17
5
1,495 Views
Last Modified: 2013-01-18
Hi there,

I'm using Python and the Twitter API to scrape tweets that use a particular hashtag. The end goal is to scrape 1500 tweets, determine which users tweeted the most, then list the top 10 users who tweeted the most (its for a small school assignment).

I've managed to get my code to the point where I have my top 10 users in a dictionary but can't seem to figure out how to export my data to a CSV file. I've found a few solutions on Stackoverflow but I couldn't get them to work.

Any help would be greatly appreciated. My code is attached. Thanks in advance!
0
Comment
Question by:Adam
  • 3
  • 2
5 Comments
 
LVL 28

Expert Comment

by:pepr
ID: 38791956
(You forgot to attach your solution, anyway...)

You should definitely use the standard csv module. Try the following:
#!python2
import csv
fname_output = 'twitter_users.csv'

# The simulated dictionary.
d = {'user1': 'msg1 or whatever',
     'user2': 'msg2',
     'user3': 'msg3'
    }

# Open the otput file for writing in binary mode.
with open(fname_output, 'wb') as f:
    
    # Wrap the file object by the writer.
    writer = csv.writer(f)
    
    # Write the header line to the output if needed.
    writer.writerow(('USER', 'DATA'))
    
    # Loop through the dictionary and get the tuples.
    # You may want to sort it somehow. Not shown here.
    for t in d.iteritems():
        writer.writerow(t)
        
        
# Note: When using the 'with' construct, the output file is closed automatically.

Open in new window


When using Python 3, the file open slightly differs because of the difference in how strings are treated.
0
 
LVL 1

Author Comment

by:Adam
ID: 38792074
Thanks for the reply pepr. Really strange, I definitely had the code attached when I submitted the question. I'll post it instead of attaching it. If you could take a look at what I have so far, it would really be helpful.

In the meantime, I will try to work in your sample code with mine and see what I come up with. Thanks.

import twitter, json, operator, csv
#Construct Twitter API object
searchApi = twitter.Twitter(domain="search.twitter.com")
# Hashtag query
query = "#deathstar"
# Dictionary of tweet details
tweeters = dict()

# Loop through 15 pages of tweets = 1500 tweets
for i in range(1,16):
	response = searchApi.search(q=query, rpp=100, page=i)
	tweets = response['results']
	for item in tweets:
		tweet = json.loads(json.dumps(item))
		user = tweet['from_user_name']
		# If user tweets more then once, count the number of times
		if user in tweeters:
			tweeters[user] += 1
		else:
			tweeters[user] = 1

# Sort users from most tweets to least
sorted_tweeters = sorted(tweeters.iteritems(), key=operator.itemgetter(1), reverse=True)
# Print the top 10 users who tweeted the most
print sorted_tweeters[0:10]

Open in new window

0
 
LVL 28

Accepted Solution

by:
pepr earned 500 total points
ID: 38792178
I do not know what twitter module you have. Anyway, the sorted_tweeters should be fine for the purpose. Try to add:
with open(fname_output, 'wb') as f:
    writer = csv.writer(f)
    writer.writerow(('USER', 'DATA'))  # only if you want it
    for t in sorted_tweeters[0:10]:
        writer.writerow(t)

Open in new window


A side note: You will probably observe less problem in future when using 4 spaces for one indentation instead of tabs (no tabs in sources). But it is your decision ;)
0
 
LVL 1

Author Closing Comment

by:Adam
ID: 38792402
Thanks for your help pepr! This worked great. Cheers!
0
 
LVL 28

Expert Comment

by:pepr
ID: 38792408
:)
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Need a python script 5 87
How to quickly make up my portfolio? 4 33
Wordpress Plug-in? - Need a landing page for a fund raiser 2 31
How to use pyenv 4 16
This story has been written with permission from the scammed victim, a valued client of mine – identity protected by request.
Every business owner understands the significance of online customer reviews and the impact it can have on sales and revenues. With technology advancing at such a rapid pace, getting online reviews has never been easier, especially when many regions…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
An overview of how to create reports in Adobe Analytics (formerly Omniture Site Catalyst) using pageNames, events, eVars and props. This video will show you how to install the Omniture Debugger tool so can see (and test) what is being passed int…

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now