Writing Twitter data to CSV file with Python

Hi there,

I'm using Python and the Twitter API to scrape tweets that use a particular hashtag. The end goal is to scrape 1500 tweets, determine which users tweeted the most, then list the top 10 users who tweeted the most (its for a small school assignment).

I've managed to get my code to the point where I have my top 10 users in a dictionary but can't seem to figure out how to export my data to a CSV file. I've found a few solutions on Stackoverflow but I couldn't get them to work.

Any help would be greatly appreciated. My code is attached. Thanks in advance!
LVL 1
AdamAsked:
Who is Participating?
 
peprConnect With a Mentor Commented:
I do not know what twitter module you have. Anyway, the sorted_tweeters should be fine for the purpose. Try to add:
with open(fname_output, 'wb') as f:
    writer = csv.writer(f)
    writer.writerow(('USER', 'DATA'))  # only if you want it
    for t in sorted_tweeters[0:10]:
        writer.writerow(t)

Open in new window


A side note: You will probably observe less problem in future when using 4 spaces for one indentation instead of tabs (no tabs in sources). But it is your decision ;)
0
 
peprCommented:
(You forgot to attach your solution, anyway...)

You should definitely use the standard csv module. Try the following:
#!python2
import csv
fname_output = 'twitter_users.csv'

# The simulated dictionary.
d = {'user1': 'msg1 or whatever',
     'user2': 'msg2',
     'user3': 'msg3'
    }

# Open the otput file for writing in binary mode.
with open(fname_output, 'wb') as f:
    
    # Wrap the file object by the writer.
    writer = csv.writer(f)
    
    # Write the header line to the output if needed.
    writer.writerow(('USER', 'DATA'))
    
    # Loop through the dictionary and get the tuples.
    # You may want to sort it somehow. Not shown here.
    for t in d.iteritems():
        writer.writerow(t)
        
        
# Note: When using the 'with' construct, the output file is closed automatically.

Open in new window


When using Python 3, the file open slightly differs because of the difference in how strings are treated.
0
 
AdamAuthor Commented:
Thanks for the reply pepr. Really strange, I definitely had the code attached when I submitted the question. I'll post it instead of attaching it. If you could take a look at what I have so far, it would really be helpful.

In the meantime, I will try to work in your sample code with mine and see what I come up with. Thanks.

import twitter, json, operator, csv
#Construct Twitter API object
searchApi = twitter.Twitter(domain="search.twitter.com")
# Hashtag query
query = "#deathstar"
# Dictionary of tweet details
tweeters = dict()

# Loop through 15 pages of tweets = 1500 tweets
for i in range(1,16):
	response = searchApi.search(q=query, rpp=100, page=i)
	tweets = response['results']
	for item in tweets:
		tweet = json.loads(json.dumps(item))
		user = tweet['from_user_name']
		# If user tweets more then once, count the number of times
		if user in tweeters:
			tweeters[user] += 1
		else:
			tweeters[user] = 1

# Sort users from most tweets to least
sorted_tweeters = sorted(tweeters.iteritems(), key=operator.itemgetter(1), reverse=True)
# Print the top 10 users who tweeted the most
print sorted_tweeters[0:10]

Open in new window

0
 
AdamAuthor Commented:
Thanks for your help pepr! This worked great. Cheers!
0
 
peprCommented:
:)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.