Solved

Writing Twitter data to CSV file with Python

Posted on 2013-01-17
5
1,572 Views
Last Modified: 2013-01-18
Hi there,

I'm using Python and the Twitter API to scrape tweets that use a particular hashtag. The end goal is to scrape 1500 tweets, determine which users tweeted the most, then list the top 10 users who tweeted the most (its for a small school assignment).

I've managed to get my code to the point where I have my top 10 users in a dictionary but can't seem to figure out how to export my data to a CSV file. I've found a few solutions on Stackoverflow but I couldn't get them to work.

Any help would be greatly appreciated. My code is attached. Thanks in advance!
0
Comment
Question by:Adam
  • 3
  • 2
5 Comments
 
LVL 29

Expert Comment

by:pepr
ID: 38791956
(You forgot to attach your solution, anyway...)

You should definitely use the standard csv module. Try the following:
#!python2
import csv
fname_output = 'twitter_users.csv'

# The simulated dictionary.
d = {'user1': 'msg1 or whatever',
     'user2': 'msg2',
     'user3': 'msg3'
    }

# Open the otput file for writing in binary mode.
with open(fname_output, 'wb') as f:
    
    # Wrap the file object by the writer.
    writer = csv.writer(f)
    
    # Write the header line to the output if needed.
    writer.writerow(('USER', 'DATA'))
    
    # Loop through the dictionary and get the tuples.
    # You may want to sort it somehow. Not shown here.
    for t in d.iteritems():
        writer.writerow(t)
        
        
# Note: When using the 'with' construct, the output file is closed automatically.

Open in new window


When using Python 3, the file open slightly differs because of the difference in how strings are treated.
0
 
LVL 1

Author Comment

by:Adam
ID: 38792074
Thanks for the reply pepr. Really strange, I definitely had the code attached when I submitted the question. I'll post it instead of attaching it. If you could take a look at what I have so far, it would really be helpful.

In the meantime, I will try to work in your sample code with mine and see what I come up with. Thanks.

import twitter, json, operator, csv
#Construct Twitter API object
searchApi = twitter.Twitter(domain="search.twitter.com")
# Hashtag query
query = "#deathstar"
# Dictionary of tweet details
tweeters = dict()

# Loop through 15 pages of tweets = 1500 tweets
for i in range(1,16):
	response = searchApi.search(q=query, rpp=100, page=i)
	tweets = response['results']
	for item in tweets:
		tweet = json.loads(json.dumps(item))
		user = tweet['from_user_name']
		# If user tweets more then once, count the number of times
		if user in tweeters:
			tweeters[user] += 1
		else:
			tweeters[user] = 1

# Sort users from most tweets to least
sorted_tweeters = sorted(tweeters.iteritems(), key=operator.itemgetter(1), reverse=True)
# Print the top 10 users who tweeted the most
print sorted_tweeters[0:10]

Open in new window

0
 
LVL 29

Accepted Solution

by:
pepr earned 500 total points
ID: 38792178
I do not know what twitter module you have. Anyway, the sorted_tweeters should be fine for the purpose. Try to add:
with open(fname_output, 'wb') as f:
    writer = csv.writer(f)
    writer.writerow(('USER', 'DATA'))  # only if you want it
    for t in sorted_tweeters[0:10]:
        writer.writerow(t)

Open in new window


A side note: You will probably observe less problem in future when using 4 spaces for one indentation instead of tabs (no tabs in sources). But it is your decision ;)
0
 
LVL 1

Author Closing Comment

by:Adam
ID: 38792402
Thanks for your help pepr! This worked great. Cheers!
0
 
LVL 29

Expert Comment

by:pepr
ID: 38792408
:)
0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Facebook marketing help 3 106
Advice on making a YouTube video that generates traffic 7 108
Python variable _ manually assigned 9 90
Google Analytics setup 5 51
When we want to run, execute or repeat a statement multiple times, a loop is necessary. This article covers the two types of loops in Python: the while loop and the for loop.
Preparing an email is something we should all take special care with – especially when the email is for somebody you may not know very well. The pressures of everyday working life stacked with a hectic office environment can make this a real challen…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…

808 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question