List of lists changes reflected across sublists unexpectedly | Python[Answered]

Problem:

I needed to create a list of lists in Python, so I typed the following:

my_list = [[1] * 4] * 3

The list looked like this:

[[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]]  

Then I changed one of the innermost values:

my_list[0][0] = 5

Now my list looks like this:

[[5, 1, 1, 1], [5, 1, 1, 1], [5, 1, 1, 1]]  

which is not what I wanted or expected. Can someone please explain what’s going on, and how to get around it?

List of lists changes reflected across sublists unexpectedly- Answer #1:

When you write [x]*3 you get, essentially, the list [x, x, x]. That is, a list with 3 references to the same x. When you then modify this single x it is visible via all three references to it:

x = [1] * 4
l = [x] * 3
print(f"id(x): {id(x)}")
# id(x): 140560897920048
print(
    f"id(l[0]): {id(l[0])}\n"
    f"id(l[1]): {id(l[1])}\n"
    f"id(l[2]): {id(l[2])}"
)
# id(l[0]): 140560897920048
# id(l[1]): 140560897920048
# id(l[2]): 140560897920048

x[0] = 42
print(f"x: {x}")
# x: [42, 1, 1, 1]
print(f"l: {l}")
# l: [[42, 1, 1, 1], [42, 1, 1, 1], [42, 1, 1, 1]]

To fix it, you need to make sure that you create a new list at each position. One way to do it is

[[1]*4 for _ in range(3)]

which will reevaluate [1]*4 each time instead of evaluating it once and making 3 references to 1 list.


You might wonder why * can’t make independent objects the way the list comprehension does. That’s because the multiplication operator * operates on objects, without seeing expressions. When you use * to multiply [[1] * 4] by 3, * only sees the 1-element list [[1] * 4] evaluates to, not the [[1] * 4 expression text. * has no idea how to make copies of that element, no idea how to reevaluate [[1] * 4], and no idea you even want copies, and in general, there might not even be a way to copy the element.

The only option * has is to make new references to the existing sublist instead of trying to make new sublists. Anything else would be inconsistent or require major redesigning of fundamental language design decisions.

In contrast, a list comprehension reevaluates the element expression on every iteration. [[1] * 4 for n in range(3)] reevaluates [1] * 4 every time for the same reason [x**2 for x in range(3)] reevaluates x**2 every time. Every evaluation of [1] * 4 generates a new list, so the list comprehension does what you wanted.

Incidentally, [1] * 4 also doesn’t copy the elements of [1], but that doesn’t matter, since integers are immutable. You can’t do something like 1.value = 2 and turn a 1 into a 2.

List of lists changes reflected across sublists unexpectedly- Answer #2:

size = 3
matrix_surprise = [[0] * size] * size
matrix = [[0]*size for _ in range(size)]

Live visualization:

Frames and Objects

List of lists changes reflected across sublists unexpectedly- Answer #2:

Actually, this is exactly what you would expect. Let’s decompose what is happening here:

You write

lst = [[1] * 4] * 3

This is equivalent to:

lst1 = [1]*4
lst = [lst1]*3

This means lst is a list with 3 elements all pointing to lst1. This means the two following lines are equivalent:

lst[0][0] = 5
lst1[0] = 5

As lst[0] is nothing but lst1.

To obtain the desired behavior, you can use a list comprehension:

lst = [ [1]*4 for n in range(3) ]

In this case, the expression is re-evaluated for each n, leading to a different list.

List of lists changes reflected across sublists unexpectedly- Answer #3:

[[1] * 4] * 3

or even:

[[1, 1, 1, 1]] * 3

Creates a list that references the internal [1,1,1,1] 3 times – not three copies of the inner list, so any time you modify the list (in any position), you’ll see the change three times.

It’s the same as this example:

>>> inner = [1,1,1,1]
>>> outer = [inner]*3
>>> outer
[[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]]
>>> inner[0] = 5
>>> outer
[[5, 1, 1, 1], [5, 1, 1, 1], [5, 1, 1, 1]]

where it’s probably a little less surprising.

List of lists changes reflected across sublists unexpectedly- Answer #4:

my_list = [[1]*4] * 3 creates one list object [1,1,1,1] in memory and copies its reference 3 times over. This is equivalent to obj = [1,1,1,1]; my_list = [obj]*3. Any modification to obj will be reflected at three places, wherever obj is referenced in the list. The right statement would be:

my_list = [[1]*4 for _ in range(3)]

or

my_list = [[1 for __ in range(4)] for _ in range(3)]

Important thing to note here is that the * operator is mostly used to create a list of literals. Although 1 is immutable, obj = [1]*4 will still create a list of 1 repeated 4 times over to form [1,1,1,1]. But if any reference to an immutable object is made, the object is overwritten with a new one.

This means if we do obj[1] = 42, then obj will become [1,42,1,1] not [42,42,42,42] as some may assume. This can also be verified:

>>> my_list = [1]*4
>>> my_list
[1, 1, 1, 1]

>>> id(my_list[0])
4522139440
>>> id(my_list[1])  # Same as my_list[0]
4522139440

>>> my_list[1] = 42  # Since my_list[1] is immutable, this operation overwrites my_list[1] with a new object changing its id.
>>> my_list
[1, 42, 1, 1]

>>> id(my_list[0])
4522139440
>>> id(my_list[1])  # id changed
4522140752
>>> id(my_list[2])  # id still same as my_list[0], still referring to value `1`.
4522139440

List of lists changes reflected across sublists unexpectedly- Answer #5:

These questions have a lot of answers, I am adding my answer to explain the same diagrammatically.

The way you created the 2D, creates a shallow list

    arr = [[0]*cols]*row

Instead, if you want to update the elements of the list, you should use

   rows, cols = (5, 5) 
   arr = [[0 for i in range(cols)] for j in range(rows)] 

Explanation:

One can create a list using :

   arr = [0]*N 

or

   arr = [0 for i in range(N)] 

In the first case all the indices of the array point to the same integer object

enter image description here

and when you assign a value to a particular index, a new int object is created, for eg arr[4] = 5 creates

enter image description here

Now let us see what happens when we create a list of list, in this case, all the elements of our top list will point to the same list

enter image description here

And if you update the value of any index a new int object will be created. But since all the top-level list indexes are pointing at the same list, all the rows will look the same. And you will get the feeling that updating an element is updating all the elements in that column.

enter image description here

Follow Programming Articles for more!

About ᴾᴿᴼᵍʳᵃᵐᵐᵉʳ

Linux and Python enthusiast, in love with open source since 2014, Writer at programming-articles.com, India.

View all posts by ᴾᴿᴼᵍʳᵃᵐᵐᵉʳ →