List Comprehension on list of list

I have a list of lists.  I would like to modify the nth element in each list of the list of lists using a list compresion.  I think of my list of lists as rows and columns of unequal length.  I want to do column operations on a list of lists.   This is easily done with the for ... in ... construct.  

Python 2.71 Example:

mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
# Assume that I want to modiy the 4th element in each list
for l in mylist:
     if len(l) >= 4 :  l[3] = l[3]**2

How can the same function be achieved using list comprehension?  Thanks.
dlnicho1Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

HonorGodSoftware EngineerCommented:
well consider that this:

x = [ y for y in mylist ]

The result is a new list, identical to the original

Now, consider that we can define a function to manipulate a function:

def fun( L, index, newValue ) :
   L[ index ] = newValue
   return L

And we can see how it works interactively:

>>> fun( [ 0, 1, 2, 3 ], 2, 'x' )
[0, 1, 'x', 3]

So, we could do something like:

>>> [ fun( y, 3, 'x' ) for y in mylist ]
[[0, 1, 2, 'x'], [2, 3, 4, 'x', 5, 6], [0, 0, 0, 'x', 2, 3]]

If you are comfortable with using lamda, we could use that instead of defining a "global" or "local" function

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
HonorGodSoftware EngineerCommented:
For example: ... see attached code


Since your "function" needs to include an "if" test, we can't use lambda


>>> mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
>>>
>>> def fun( L ) :
...   if len( L ) > 3 :
...     L[ 3 ] **= 2
...   return L
...
>>> [ fun( y ) for y in mylist ]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]
>>>

Open in new window

0
dlnicho1Author Commented:
Nicely done. However, it seems like the for .. in.. construct is simpler for this particular example.  In general when working with a list of lists or nested lists, are list comprehensions better (faster, more readable, clearer, less prone to errors) then the for ... in ... construct?

I was trying something like the list comprehension below but it did not work.  

test = [[x for x in y if not y.index(x) == 1 else x_something] for y in mylist]

However, y.index(x) returns the position x in the y list and the else statement has a syntax error.  This is not what I wanted.

What would the lambda solution look like?
0
Rowby Goren Makes an Impact on Screen and Online

Learn about longtime user Rowby Goren and his great contributions to the site. We explore his method for posing questions that are likely to yield a solution, and take a look at how his career transformed from a Hollywood writer to a website entrepreneur.

HonorGodSoftware EngineerCommented:
> are list comprehensions better (faster, more readable, clearer, less prone to errors) then the for ... in ... construct?

faster?                   Maybe
more readable?     Arguable based upon the readers level of experience
clearer?                 Again, arguable based upon the readers level of experience
less error prone?   I would think that it would depend upon the complexity of the expressions/tests being used

> What would the lambda solution look like?

  A lamba expression looks like:

g = lambda x: x**2

  which means that we don't have to have a (trivial) function definition of the form:

def f (x):
  return x**2

or even:

def f (x): return x**2

Note, however, that lambda expressions can not include an "IF" test!

So, in order for our "function" to only modify the third element of a list IF the list has more than 2 elements, we need to define a function.  
0
HonorGodSoftware EngineerCommented:
I keep going back to using the external function, ( e.g.,"fun() shown below) because of readability.

Don't you find this readable?

[ fun( y ) for y in mylist ]
>>> def fun( L ) :
...   if len( L ) > 3 :
...     L[ 3 ] **= 2
...   return L
...
>>> [ fun( y ) for y in mylist ]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]

Open in new window

0
HonorGodSoftware EngineerCommented:
Note, however, that the function, as written, modifies the original list!
This is because it doesn't make a copy of the list before the if test...
If we want / need to leave the original list unmodified, we should do
something like:
>>> mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
>>>
>>> def fun( L ) :
...   M = L[:]
...   if len( M ) > 3 :
...     M[ 3 ] **= 2
...   return M
...
>>> [ fun( y ) for y in mylist ]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]
>>>
>>> mylist
[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
>>>

Open in new window

0
dlnicho1Author Commented:
An excellent explanation.  Thanks for the help
0
dlnicho1Author Commented:
The function solution is clever and I like it very much.  However, it just seems easier to me (imho) that the for ... in construct is cleaner for walking a column in a list of lists.  Two lines of easily understandable code, even for a beginner

for l in mylist:
     if len(l) >= 4 :  l[3] = l[3]**2

Perhaps it depends on the complexity of the operation on the "column" of data.  I can certainly see a use for the function application if the future.  Thanks again.
0
HonorGodSoftware EngineerCommented:
You are very welcome.

Thanks for the grade & points.

Good luck & have a great day.
0
peprCommented:
To add...

Focus on the HonorGod's comment about copying.  Actually, the list comprehension could be renamed as "generator notation of the list".  In other languages, the comprehension is a kind of notation that prescribes how the NEW list is to be constructed (list, or set, or dictionary in Python).  The list comprehension always makes a copy of the list.  This way, the list comprehension cannot be used for modification of only some elements of the existing structure.

There is no need to replace the original code by a list comprehension.  It does not fit there; however, the list comprehension can be used in other cases.

To show it from another point of view, to understand what is behind.  Focus on the internals of the list comprehension (what is written inside the [] brackets).  This is called a generator expression in Python.  You can think about it as about a generator that produces some values, and the list is constructed out of the values.  You can even use the generator expression for the other purpose.  Try the following sample (not related to the code above):

a.py
lst = [1, 2, 3]
print lst

# When generator expression is used outside any construct
# that tries to iterate, it returns a generator object.
g = (x for x in lst)  
print g          # this is the generator object

# The iteration through the generator values means calling
# its .next() method.
print g.next()   # the generator returns an element
print g.next()   # next element
print g.next()
print g.next()   # no more elements means the exception

Open in new window


It prints the following:

c:\tmp\_Python\dlnicho1\Q_26927693>python a.py
[1, 2, 3]
<generator object <genexpr> at 0x00B9B2B0>
1
2
3
Traceback (most recent call last):
  File "a.py", line 14, in <module>
    print g.next()   # no more elements means the exception
StopIteration

Open in new window


The StopIteration is recognized by various constructs as the indication of the end of sequence of elements.

Now try the following sample and read the comments:

b.py
lst = [1, 2, 3]
print lst

# The lst is capable of iteration, so that the for loop
# can work with it.
for elem in lst:
    print elem

# There would be no point to use the generator expression
# instead of the lst directly in the for loop. Let's
# make it more "usefull"
g = (x**2 for x in lst)  
print g          # this is the generator object

# The for loop can work with generators. It internally
# calls the .next() until the StopIteration exception.
for elem in g:
    print elem
    
# The generator object cannot be reset. We have to construct
# it again. Now using it inside the list().
g = (x**2 for x in lst)  
print g          # this is the generator object
print list(g)    # the list was generated

# The generator expression can be directly used inside
# a function call -- no syntactic need for the enclosing
# parentheses.
print list( x**2 for x in lst )    # the list was generated

# The generator object cannot be reset. Now using only a minor
# syntactic change. The 'list(' was replaced by '[', the ')' 
# was replaced by ']'. (The spaces are not neccessary.)
print [ x**2 for x in lst ]      

# The same way you can use the generator expression 
# for generating a set...
print set( x**2 for x in lst )

# ... which can also be achieved using the set comprehension.
print { x**2 for x in lst }

Open in new window


It prints:

c:\tmp\_Python\dlnicho1\Q_26927693>python b.py
[1, 2, 3]
1
2
3
<generator object <genexpr> at 0x00B9B2D8>
1
4
9
<generator object <genexpr> at 0x00B9B300>
[1, 4, 9]
[1, 4, 9]
[1, 4, 9]
set([1, 4, 9])
set([1, 4, 9])

Open in new window

0
peprCommented:
To summarize, the list comprehension can be more related (via the generator expressions) to what is called generator functions -- i.e. the ones that use the "yield somethin" instead of "return something".
0
HonorGodSoftware EngineerCommented:
@pepr marvelous.  simply marvelous. ;-)  No fooling!
0
peprCommented:
Apriiiiil :)))
0
peprCommented:
No. It really works ;)  Thanks HonorGod.
0
HonorGodSoftware EngineerCommented:
One of the reasons that I appreciate your post is that I get to (read, must) use an earlier version of Python in which generator functions do not exist.  So I haven't had the pleasure of using them before.

Your update is going to encourage me to start reading that massive tomb
Programming Python 4th Edition
0
peprCommented:
To another part of the question.  Run the Python in the interactive mode and do "import this" -- and read The Zen of Python:

c:\tmp\_Python\dlnicho1\Q_26927693>python
Python 2.7 (r27:82525, Jul  4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>>

Open in new window


Notice the "In the face of ambiguity, refuse the temptation to guess."  This is for "Is comprehension faster than the for loop?"  Refusing temptation to guess here simply means "measure it".  There is the timeit standard module (http://docs.python.org/library/timeit.html) -- notice the examples at the end.

Actually, the comprehensions should be more efficient as the generator expressions are more strict than the general bodies of the for loops.  There is much less things that could be considered a general code.  There basically is only the iterated object and possibly some tests.  It also means that it can be compiled more efficiently to the intermediate code.
0
peprCommented:
Try also the nice "Dive into Python 3" by Mark Pilgrim -- http://diveintopython3.org/ 

It is for Python 3, but Python 2.7 implements majority of the new constructs, and the text can be easily used for the Python 2.7.  I like also the "light/funny" way Mark explains the things.

But you probably do not want to buy the printed version of the book.  The HTML version is excelent.  The printed content is the same, but the typesetting is extremely poor (in my opinion).  At least, have a look inside the book if you still want to buy it.
0
peprCommented:
Actually, it is possible to get a copy of the list from the question with modified elements via list comprehensios.  However, it is not readable -- at least.  Read the steps in comments:

c.py
mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
print mylist
for lst in mylist:
     if len(lst) >= 4:  
         lst[3] = lst[3]**2
print mylist     
print

# Let's start with easy copying of the whole lists.
mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
print mylist     
mylist2 = [ lst for lst in mylist ]
print mylist2     

# However, it is not the deep copy as the lists inside
# the mylist2 are shared -- i.e. identical with those 
# inside mylist.
print id(mylist[0]), id(mylist2[0])
mylist[0][0] = 'xxx'   # modified only the first one
print mylist           #
print mylist2          # but the second also changed
print

# Here the deep copy is done via copying the elements.
mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
print mylist     
mylist2 = [ [elem for elem in lst] for lst in mylist ]
print mylist2     

# ... different identities, however the number objects
# are still shared. Anyway, the elements are immutable,
# so that it looks as if they were not shared.
print id(mylist[0]), id(mylist2[0])
mylist[0][0] = 'xxx'   # modified only the first one
print mylist           #
print mylist2          # the second contains the original.
print

# Now the modification of every fourth element 
# of the internal list.
mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
print mylist     
mylist2 = [ [elem**2 if idx == 3 else elem for idx, elem in enumerate(lst)] for lst in mylist ]
print mylist2     
print '-' * 70         # separator

# Let's dissect the complicated list comprehension via formatting.
# Read the comments below from bottom up.
mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
print mylist     
mylist2 = [ 
            [
              elem**2 if idx == 3 else elem   # conditional expression based on index
              for idx, elem in enumerate(lst) # enumeration through the inner list 
            ]                                 # ... returns index and element
            for lst in mylist                 # outer hidden loop produces lists
          ]
print mylist2     

Open in new window


And it prints
c:\tmp\_Python\dlnicho1\Q_26927693>python c.py
[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]

[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
12167248 12167248
[['xxx', 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
[['xxx', 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]

[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
12166688 12169088
[['xxx', 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]

[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]
----------------------------------------------------------------------
[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]

Open in new window

0
peprCommented:
The time has come to measure...

d.py
import timeit

def fn0():
    mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
    return mylist


def fn1():
    mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
    for lst in mylist:
         if len(lst) >= 4:  
             lst[3] = lst[3]**2
    return mylist


def fn2():
    mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
    return [ [
                elem**2 if idx == 3 else elem 
                for idx, elem in enumerate(lst)
             ] 
             for lst in mylist 
           ]

def fn3():
    mylist = [[0,1,2,3], [2,3,4,4,5,6], [0,0,0,4,2,3]]
    mylist2 = []
    for lst in mylist:
        lst2 = []
        for idx, elem in enumerate(lst):
            lst2.append(elem**2 if idx == 3 else elem)
        mylist2.append(lst2)    
    return mylist2


mylist2 = fn0()
print mylist2

mylist2 = fn1()
print mylist2

mylist2 = fn2()
print mylist2

mylist2 = fn3()
print mylist2

t = timeit.Timer('fn0()', 'from __main__ import fn0')
print t.timeit()

t = timeit.Timer('fn1()', 'from __main__ import fn1')
print t.timeit()

t = timeit.Timer('fn2()', 'from __main__ import fn2')
print t.timeit()

t = timeit.Timer('fn3()', 'from __main__ import fn3')
print t.timeit()

Open in new window


It prints

c:\tmp\_Python\dlnicho1\Q_26927693>python d.py
[[0, 1, 2, 3], [2, 3, 4, 4, 5, 6], [0, 0, 0, 4, 2, 3]]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]
[[0, 1, 2, 9], [2, 3, 4, 16, 5, 6], [0, 0, 0, 16, 2, 3]]
1.75050544133
3.59242920539
10.6153309988
14.5245716984

Open in new window


The fn0 was used only to measure the time of function call, construction of the sample list and the return.  The fn1, fn2, and fn3 contain the same plus the compared algorithms.  It is clear here that the original solution is fastest.  However, it does something completely different than fn2 -- i.e. fn1 does no copying, only modification in situ.  Even though I tried to make fn3 functionally equal to fn2 a much as possible, it is clear that the list comprehension is faster.  The numbers are the seconds spent on my slow computer when repeating the call the default-number times.

0
HonorGodSoftware EngineerCommented:
@pepr: I enjoy reading your updates.  I almost always learn something new.  Thanks for taking the time to share your knowledge with everyone.
0
peprCommented:
:)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Python

From novice to tech pro — start learning today.