Solved

how to download  raw post data json python

Posted on 2015-01-02
25
200 Views
Last Modified: 2015-01-12
hello
i would like download playlist json files how to downloads this files?

1 load url www.somsite.com/playliste/1
2 download json raw http://somesite/index.php?option=com_play&view=playlist&format=raw



ty
0
Comment
Question by:Gaaara
  • 14
  • 11
25 Comments
 
LVL 45

Expert Comment

by:aikimark
ID: 40529221
Have you used urllib, mechanize, or request?
0
 

Author Comment

by:Gaaara
ID: 40529822
I want to know how to get files I am one thus begin wanted to know what to use and also to have some documentation with visible demo
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40529865
is there a specific web site you are opening?
0
 

Author Comment

by:Gaaara
ID: 40530824
no just get the playlis with a cron job for get a émail In every new links for 2 -3 web site To miss nothing
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40530825
please post an actual working URL that returns a JSON file.
0
 

Author Comment

by:Gaaara
ID: 40530829
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40530867
I'm getting a 404 on the link that is supposed to return JSON
0
 

Author Comment

by:Gaaara
ID: 40530902
the json link Is activated in the load of the page with a free account or premium account with cookie

exemple with curl copied with  firebug
curl 'http://animedigitalnetwork.fr/index.php?option=com_vodvideo&view=playlist&format=raw' -H 'Host: animedigitalnetwork.fr' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:34.0) Gecko/20100101 Firefox/34.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3' -H 'Accept-Encoding: gzip, deflate' -H 'X-Requested-With: XMLHttpRequest' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Referer: http://animedigitalnetwork.fr/video/naruto-shippuden' -H 'Cookie: _ga=GA1.2.2143731636.1420256453; 18acd9b63ecbf50de0b8c010c2b7289f=m9sbtbulg7mciu7b2f1gpt3337; _gat=1' --data 'playlist=265&season=&order=DESC'

Open in new window

the script need to collect all the cookie and get the file

i dont no have a 404 with any link ?
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40530919
you have signed up for an account with that site.  That is why your URLs get results.  Log out (sign off) and retry your URL to see what I'm seeing.
0
 

Author Comment

by:Gaaara
ID: 40530928
is ok for me :) i dont no have a 404 error What do you want to know about the site
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40530965
I like to test code prior to posting it.
0
 

Author Comment

by:Gaaara
ID: 40530986
If you have some difficulty post the code i test it :)
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 45

Expert Comment

by:aikimark
ID: 40532446
Let me rephrase my prior comment.  I don't feel comfortable posting untested code.  Maybe one of the other Python experts will feel better about your testing proposal.
0
 

Author Comment

by:Gaaara
ID: 40532863
mm test with my test compte of this site

user    testpseudo
pass   KytmEgsdKQa9

you have a vpn ?
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40533602
I was able to sign in, but your second link doesn't return anything.  I added &playlist=265 and didn't see any JSON.
0
 

Author Comment

by:Gaaara
ID: 40533773
0
 

Author Comment

by:Gaaara
ID: 40537189
I've requested that this question be deleted for the following reason:

...
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40536505
Looks like (rendered) HTML to me
0
 

Author Comment

by:Gaaara
ID: 40536602
with BeautifulSoup is possible to passing this part ?

	

    import urllib2
    from BeautifulSoup import BeautifulSoup
     
    page = urllib2.urlopen('http://animedigitalnetwork.fr/video/naruto-shippuden')
    soup = BeautifulSoup(page)
    first_div = soup.find('div',{"class" : "adn-video"})
    print  first_div 

Open in new window


i got this résult

<div class="adn-video"> <div class="adn-video_screenshot">
<img src="http://image.animedigitalnetwork.fr/license/claymore/tv/web/eps1_328x184.jpg" alt="Claymore 1" /><span class="adn_video_play-button"></span> </div><div class="adn-video_text"><div class="adn-video_title">
<h4>Claymore</h4><span>Épisode 1</span><div class="adn-rating mobile-hide" itemprop="aggregateRating" itemscope="itemscope" itemtype="http://schema.org/AggregateRating"><meta itemprop="ratingValue" content="4.6667" /><meta itemprop="ratingCount" content="10" /><div id="adn-rating"><ul class="adn-rating_empty"><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li></ul><ul class="adn-rating_rating"><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li></ul></div><p class="adn-rating-message"></p></div></div><div class="adn-video_link">
<a title="Claymore 1" href="/video/claymore/1849-episode-1-la-claymore">Voir la vidéo</a>
</div></div></div>

Open in new window


is not naruto !?
0
 
LVL 45

Assisted Solution

by:aikimark
aikimark earned 500 total points
ID: 40537116
There are several ways to parse the data out of HTML and beautifulsoup is certainly one of those ways.  What you have posted in your latest comment is HTML and not JSON.


Try this URL:
http://animedigitalnetwork.fr/video/naruto-shippuden

Then do a view source or inspect element to see the HTML.
0
 

Accepted Solution

by:
Gaaara earned 0 total points
ID: 40537186
is resolved

from bs4 import BeautifulSoup
import requests

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

# initialize session
session = requests.Session()

# getting playlist
response = session.get('http://animedigitalnetwork.fr/video/naruto-shippuden', headers=headers)
soup = BeautifulSoup(response.content)
playlist = soup.find('a', {'data-playlist': True})['data-playlist']

# getting list of videos
url = 'http://animedigitalnetwork.fr/index.php?option=com_vodvideo&view=playlist&format=raw'
response = session.post(url, data={
    'playlist': playlist,
    'season': '',
    'order': 'DESC'
}, headers=headers)

soup = BeautifulSoup(response.content)
for video in soup.select('div.adn-video'):
    print video.a.get('href')

Open in new window

0
 

Author Comment

by:Gaaara
ID: 40537190
is ok
0
 

Author Comment

by:Gaaara
ID: 40537192
I've requested that this question be closed as follows:

Accepted answer: 0 points for Gaaara's comment #a40537186
Assisted answer: 500 points for aikimark's comment #a40537116

for the following reason:

i give you the 500 points for the efforts
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40537797
There is no need to give me 'effort' points.  You can accept your comment as the solution.
0
 

Author Closing Comment

by:Gaaara
ID: 40544014
i give you the 500 points for the efforts
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Strings in Python are the set of characters that, once defined, cannot be changed by any other method like replace. Even if we use the replace method it still does not modify the original string that we use, but just copies the string and then modif…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now