?
Solved

Python - Internet access

Posted on 2014-09-30
2
Medium Priority
?
264 Views
Last Modified: 2014-09-30
I am adapting the following code from https://docs.python.org/3/tutorial/ section 10.7 Internet access:

from urllib.request import urlopen:
    for line in urlopen('http://apps.rhs.org.uk/horticulturaldatabase/orchidregister/orchiddetails.asp?ID=171586'):
    for line in urlopen('http://plantilus.com/plantdb/RlczLepr/index.html'):
        line = line.decode('utf-8')
        if 'Genus' in line or 'Seed' in line:
        print(line)

Got expected results without any error.

When I changed to a different URL http://apps.rhs.org.uk/horticulturaldatabase/orchidregister/orchiddetails.asp?ID=171586

I got the UnicideDecodeError (invalid start byte)

Note: Similar code runs successfully in Perl.    I hope to be able to do this in Python.

Hope someone here could explain what the problem is.
pax
0
Comment
Question by:cpeters5
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 25

Accepted Solution

by:
clockwatcher earned 2000 total points
ID: 40353854
I can't get to ID=171586 but checking another page that I can get to (ID=114368) and the problem appears to be that even though the page claims (based on it's meta charset tag) to be UTF-8 encoded, the Google Tag manager comment contains a non utf-8 character:

  <!-- Google Tag Manager ? Carat -->

from urllib.request import urlopen

# for line in urlopen('http://apps.rhs.org.uk/horticulturaldatabase/orchidregister/orchiddetails.asp?ID=171586'):
# for line in urlopen('http://plantilus.com/plantdb/RlczLepr/index.html'):

for line in urlopen('http://apps.rhs.org.uk/horticulturaldatabase/orchidregister/orchiddetails.asp?ID=114368'):
     try:
         line = line.decode('utf-8')
     except:
         line = line.decode('utf-8', 'replace')  # you also have the option to 'ignore' / skip the character
         print("Non UTF8 found in: {0}".format(line))

     if 'Genus' in line or 'Seed' in line:
         print(line)

Open in new window


In other words, change your decode to either:
line=line.decode('utf-8', 'ignore')

Open in new window

or
line = line.decode('utf-8', 'replace')

Open in new window


To handle pages that aren't truly utf-8 (like your orchid detail page).
0
 

Author Closing Comment

by:cpeters5
ID: 40353860
Clockwatcher, Thank you!
pax
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

"The time has come," the Walrus said, "To talk of many things: Of sets--and lists--and dictionaries-- Of variable kinks-- And why you see it changing not-- And why so strange are strings." This part describes how variables and references (see …
Sequence is something that used to store data in it in very simple words. Let us just create a list first. To create a list first of all we need to give a name to our list which I have taken as “COURSE” followed by equals sign and finally enclosed …
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
Learn the basics of while and for loops in Python.  while loops are used for testing while, or until, a condition is met: The structure of a while loop is as follows:     while <condition>:         do something         repeate: The break statement m…
Suggested Courses

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question