Solved

Python - Internet access

Posted on 2014-09-30
2
261 Views
Last Modified: 2014-09-30
I am adapting the following code from https://docs.python.org/3/tutorial/ section 10.7 Internet access:

from urllib.request import urlopen:
    for line in urlopen('http://apps.rhs.org.uk/horticulturaldatabase/orchidregister/orchiddetails.asp?ID=171586'):
    for line in urlopen('http://plantilus.com/plantdb/RlczLepr/index.html'):
        line = line.decode('utf-8')
        if 'Genus' in line or 'Seed' in line:
        print(line)

Got expected results without any error.

When I changed to a different URL http://apps.rhs.org.uk/horticulturaldatabase/orchidregister/orchiddetails.asp?ID=171586

I got the UnicideDecodeError (invalid start byte)

Note: Similar code runs successfully in Perl.    I hope to be able to do this in Python.

Hope someone here could explain what the problem is.
pax
0
Comment
Question by:cpeters5
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 25

Accepted Solution

by:
clockwatcher earned 500 total points
ID: 40353854
I can't get to ID=171586 but checking another page that I can get to (ID=114368) and the problem appears to be that even though the page claims (based on it's meta charset tag) to be UTF-8 encoded, the Google Tag manager comment contains a non utf-8 character:

  <!-- Google Tag Manager ? Carat -->

from urllib.request import urlopen

# for line in urlopen('http://apps.rhs.org.uk/horticulturaldatabase/orchidregister/orchiddetails.asp?ID=171586'):
# for line in urlopen('http://plantilus.com/plantdb/RlczLepr/index.html'):

for line in urlopen('http://apps.rhs.org.uk/horticulturaldatabase/orchidregister/orchiddetails.asp?ID=114368'):
     try:
         line = line.decode('utf-8')
     except:
         line = line.decode('utf-8', 'replace')  # you also have the option to 'ignore' / skip the character
         print("Non UTF8 found in: {0}".format(line))

     if 'Genus' in line or 'Seed' in line:
         print(line)

Open in new window


In other words, change your decode to either:
line=line.decode('utf-8', 'ignore')

Open in new window

or
line = line.decode('utf-8', 'replace')

Open in new window


To handle pages that aren't truly utf-8 (like your orchid detail page).
0
 

Author Closing Comment

by:cpeters5
ID: 40353860
Clockwatcher, Thank you!
pax
0

Featured Post

[Webinar] Learn How Hackers Steal Your Credentials

Do You Know How Hackers Steal Your Credentials? Join us and Skyport Systems to learn how hackers steal your credentials and why Active Directory must be secure to stop them. Thursday, July 13, 2017 10:00 A.M. PDT

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The really strange introduction Once upon a time there were individuals who intentionally put the grass seeds to the soil with anticipation of solving their nutrition problems. Or they maybe only played with seeds and noticed what happened... Som…
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Learn the basics of if, else, and elif statements in Python 2.7. Use "if" statements to test a specified condition.: The structure of an if statement is as follows: (CODE) Use "else" statements to allow the execution of an alternative, if the …
Learn the basics of while and for loops in Python.  while loops are used for testing while, or until, a condition is met: The structure of a while loop is as follows:     while <condition>:         do something         repeate: The break statement m…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question