Python - Internet access

Posted on 2014-09-30
Last Modified: 2014-09-30
I am adapting the following code from section 10.7 Internet access:

from urllib.request import urlopen:
    for line in urlopen(''):
    for line in urlopen(''):
        line = line.decode('utf-8')
        if 'Genus' in line or 'Seed' in line:

Got expected results without any error.

When I changed to a different URL

I got the UnicideDecodeError (invalid start byte)

Note: Similar code runs successfully in Perl.    I hope to be able to do this in Python.

Hope someone here could explain what the problem is.
Question by:cpeters5
LVL 25

Accepted Solution

clockwatcher earned 500 total points
Comment Utility
I can't get to ID=171586 but checking another page that I can get to (ID=114368) and the problem appears to be that even though the page claims (based on it's meta charset tag) to be UTF-8 encoded, the Google Tag manager comment contains a non utf-8 character:

  <!-- Google Tag Manager ? Carat -->

from urllib.request import urlopen

# for line in urlopen(''):
# for line in urlopen(''):

for line in urlopen(''):
         line = line.decode('utf-8')
         line = line.decode('utf-8', 'replace')  # you also have the option to 'ignore' / skip the character
         print("Non UTF8 found in: {0}".format(line))

     if 'Genus' in line or 'Seed' in line:

Open in new window

In other words, change your decode to either:
line=line.decode('utf-8', 'ignore')

Open in new window

line = line.decode('utf-8', 'replace')

Open in new window

To handle pages that aren't truly utf-8 (like your orchid detail page).

Author Closing Comment

Comment Utility
Clockwatcher, Thank you!

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
When we want to run, execute or repeat a statement multiple times, a loop is necessary. This article covers the two types of loops in Python: the while loop and the for loop.
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Learn the basics of if, else, and elif statements in Python 2.7. Use "if" statements to test a specified condition.: The structure of an if statement is as follows: (CODE) Use "else" statements to allow the execution of an alternative, if the …

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now