Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Writing Twitter data to CSV file with Python

Posted on 2013-01-17
5
Medium Priority
?
2,217 Views
Last Modified: 2013-01-18
Hi there,

I'm using Python and the Twitter API to scrape tweets that use a particular hashtag. The end goal is to scrape 1500 tweets, determine which users tweeted the most, then list the top 10 users who tweeted the most (its for a small school assignment).

I've managed to get my code to the point where I have my top 10 users in a dictionary but can't seem to figure out how to export my data to a CSV file. I've found a few solutions on Stackoverflow but I couldn't get them to work.

Any help would be greatly appreciated. My code is attached. Thanks in advance!
0
Comment
Question by:Adam
  • 3
  • 2
5 Comments
 
LVL 29

Expert Comment

by:pepr
ID: 38791956
(You forgot to attach your solution, anyway...)

You should definitely use the standard csv module. Try the following:
#!python2
import csv
fname_output = 'twitter_users.csv'

# The simulated dictionary.
d = {'user1': 'msg1 or whatever',
     'user2': 'msg2',
     'user3': 'msg3'
    }

# Open the otput file for writing in binary mode.
with open(fname_output, 'wb') as f:
    
    # Wrap the file object by the writer.
    writer = csv.writer(f)
    
    # Write the header line to the output if needed.
    writer.writerow(('USER', 'DATA'))
    
    # Loop through the dictionary and get the tuples.
    # You may want to sort it somehow. Not shown here.
    for t in d.iteritems():
        writer.writerow(t)
        
        
# Note: When using the 'with' construct, the output file is closed automatically.

Open in new window


When using Python 3, the file open slightly differs because of the difference in how strings are treated.
0
 
LVL 1

Author Comment

by:Adam
ID: 38792074
Thanks for the reply pepr. Really strange, I definitely had the code attached when I submitted the question. I'll post it instead of attaching it. If you could take a look at what I have so far, it would really be helpful.

In the meantime, I will try to work in your sample code with mine and see what I come up with. Thanks.

import twitter, json, operator, csv
#Construct Twitter API object
searchApi = twitter.Twitter(domain="search.twitter.com")
# Hashtag query
query = "#deathstar"
# Dictionary of tweet details
tweeters = dict()

# Loop through 15 pages of tweets = 1500 tweets
for i in range(1,16):
	response = searchApi.search(q=query, rpp=100, page=i)
	tweets = response['results']
	for item in tweets:
		tweet = json.loads(json.dumps(item))
		user = tweet['from_user_name']
		# If user tweets more then once, count the number of times
		if user in tweeters:
			tweeters[user] += 1
		else:
			tweeters[user] = 1

# Sort users from most tweets to least
sorted_tweeters = sorted(tweeters.iteritems(), key=operator.itemgetter(1), reverse=True)
# Print the top 10 users who tweeted the most
print sorted_tweeters[0:10]

Open in new window

0
 
LVL 29

Accepted Solution

by:
pepr earned 2000 total points
ID: 38792178
I do not know what twitter module you have. Anyway, the sorted_tweeters should be fine for the purpose. Try to add:
with open(fname_output, 'wb') as f:
    writer = csv.writer(f)
    writer.writerow(('USER', 'DATA'))  # only if you want it
    for t in sorted_tweeters[0:10]:
        writer.writerow(t)

Open in new window


A side note: You will probably observe less problem in future when using 4 spaces for one indentation instead of tabs (no tabs in sources). But it is your decision ;)
0
 
LVL 1

Author Closing Comment

by:Adam
ID: 38792402
Thanks for your help pepr! This worked great. Cheers!
0
 
LVL 29

Expert Comment

by:pepr
ID: 38792408
:)
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explores several main social media platforms.
Choosing a core focus or particular set of features and options can be tough. To help out, we’re going to highlight a handful of things your business needs on one of your social media pages. In other words, if one of these is missing, you should imp…
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…
Learn how to set-up PayPal payment integration in your Wufoo form. Allow your users to remit payment through PayPal upon completion of your online form. This is helpful for collecting membership payments, customer payments, donations, and more.
Suggested Courses

564 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question