List changes unexpectedly after assignment. Why is this and how can I prevent it? | Python [Answered]

Problem:

While using new_list = my_list, any modifications to new_list changes my_list every time. Why is this, and how can I clone or copy the list to prevent it?

How to prevent list from changing after assignment in Python? Answer #1

With new_list = my_list, you don’t actually have two lists. The assignment just copies the reference to the list, not the actual list, so both new_list and my_list refer to the same list after the assignment.

To actually copy the list, you have various possibilities:

  • You can use the builtin list.copy() method (available since Python 3.3):new_list = old_list.copy()
  • You can slice it:new_list = old_list[:] Alex Martelli’s opinion (at least back in 2007) about this is, that it is a weird syntax and it does not make sense to use it ever. 😉 (In his opinion, the next one is more readable).
  • You can use the built in list() function:new_list = list(old_list)
  • You can use generic copy.copy():import copy new_list = copy.copy(old_list) This is a little slower than list() because it has to find out the datatype of old_list first.
  • If the list contains objects and you want to copy them as well, use generic copy.deepcopy():import copy new_list = copy.deepcopy(old_list) Obviously the slowest and most memory-needing method, but sometimes unavoidable.

Example:

import copy

class Foo(object):
    def __init__(self, val):
         self.val = val

    def __repr__(self):
        return 'Foo({!r})'.format(self.val)

foo = Foo(1)

a = ['foo', foo]
b = a.copy()
c = a[:]
d = list(a)
e = copy.copy(a)
f = copy.deepcopy(a)

# edit orignal list and instance 
a.append('baz')
foo.val = 5

print('original: %r\nlist.copy(): %r\nslice: %r\nlist(): %r\ncopy: %r\ndeepcopy: %r'
      % (a, b, c, d, e, f))

Result:

original: ['foo', Foo(5), 'baz']
list.copy(): ['foo', Foo(5)]
slice: ['foo', Foo(5)]
list(): ['foo', Foo(5)]
copy: ['foo', Foo(5)]
deepcopy: ['foo', Foo(1)]

How to prevent list from changing after assignment in Python? Answer #2

The first answer is already an excellent answer, but I thought I’d do a speed comparison of the various methods:

  1. 10.59 sec (105.9 µs/itn) – copy.deepcopy(old_list)
  2. 10.16 sec (101.6 µs/itn) – pure Python Copy() method copying classes with deepcopy
  3. 1.488 sec (14.88 µs/itn) – pure Python Copy() method not copying classes (only dicts/lists/tuples)
  4. 0.325 sec (3.25 µs/itn) – for item in old_list: new_list.append(item)
  5. 0.217 sec (2.17 µs/itn) – [i for i in old_list] (a list comprehension)
  6. 0.186 sec (1.86 µs/itn) – copy.copy(old_list)
  7. 0.075 sec (0.75 µs/itn) – list(old_list)
  8. 0.053 sec (0.53 µs/itn) – new_list = []; new_list.extend(old_list)
  9. 0.039 sec (0.39 µs/itn) – old_list[:] (list slicing)

So the fastest is list slicing. But be aware that copy.copy()list[:] and list(list), unlike copy.deepcopy() and the python version don’t copy any lists, dictionaries and class instances in the list, so if the originals change, they will change in the copied list too and vice versa.

(Here’s the script if anyone’s interested or wants to raise any issues:)

from copy import deepcopy

class old_class:
    def __init__(self):
        self.blah = 'blah'

class new_class(object):
    def __init__(self):
        self.blah = 'blah'

dignore = {str: None, unicode: None, int: None, type(None): None}

def Copy(obj, use_deepcopy=True):
    t = type(obj)

    if t in (list, tuple):
        if t == tuple:
            # Convert to a list if a tuple to
            # allow assigning to when copying
            is_tuple = True
            obj = list(obj)
        else:
            # Otherwise just do a quick slice copy
            obj = obj[:]
            is_tuple = False

        # Copy each item recursively
        for x in xrange(len(obj)):
            if type(obj[x]) in dignore:
                continue
            obj[x] = Copy(obj[x], use_deepcopy)

        if is_tuple:
            # Convert back into a tuple again
            obj = tuple(obj)

    elif t == dict:
        # Use the fast shallow dict copy() method and copy any
        # values which aren't immutable (like lists, dicts etc)
        obj = obj.copy()
        for k in obj:
            if type(obj[k]) in dignore:
                continue
            obj[k] = Copy(obj[k], use_deepcopy)

    elif t in dignore:
        # Numeric or string/unicode?
        # It's immutable, so ignore it!
        pass

    elif use_deepcopy:
        obj = deepcopy(obj)
    return obj

if __name__ == '__main__':
    import copy
    from time import time

    num_times = 100000
    L = [None, 'blah', 1, 543.4532,
         ['foo'], ('bar',), {'blah': 'blah'},
         old_class(), new_class()]

    t = time()
    for i in xrange(num_times):
        Copy(L)
    print 'Custom Copy:', time()-t

    t = time()
    for i in xrange(num_times):
        Copy(L, use_deepcopy=False)
    print 'Custom Copy Only Copying Lists/Tuples/Dicts (no classes):', time()-t

    t = time()
    for i in xrange(num_times):
        copy.copy(L)
    print 'copy.copy:', time()-t

    t = time()
    for i in xrange(num_times):
        copy.deepcopy(L)
    print 'copy.deepcopy:', time()-t

    t = time()
    for i in xrange(num_times):
        L[:]
    print 'list slicing [:]:', time()-t

    t = time()
    for i in xrange(num_times):
        list(L)
    print 'list(L):', time()-t

    t = time()
    for i in xrange(num_times):
        

How to prevent list from changing after assignment in Python? Answer #3

What are the options to clone or copy a list in Python?

In Python 3, a shallow copy can be made with:

a_copy = a_list.copy()

In Python 2 and 3, you can get a shallow copy with a full slice of the original:

a_copy = a_list[:]

Explanation

There are two semantic ways to copy a list. A shallow copy creates a new list of the same objects, a deep copy creates a new list containing new equivalent objects.

Shallow list copy

A shallow copy only copies the list itself, which is a container of references to the objects in the list. If the objects contained themselves are mutable and one is changed, the change will be reflected in both lists.

There are different ways to do this in Python 2 and 3. The Python 2 ways will also work in Python 3.

Python 2

In Python 2, the idiomatic way of making a shallow copy of a list is with a complete slice of the original:

a_copy = a_list[:]

You can also accomplish the same thing by passing the list through the list constructor,

a_copy = list(a_list)

but using the constructor is less efficient:

>>> timeit
>>> l = range(20)
>>> min(timeit.repeat(lambda: l[:]))
0.30504298210144043
>>> min(timeit.repeat(lambda: list(l)))
0.40698814392089844

Python 3

In Python 3, lists get the list.copy method:

a_copy = a_list.copy()

In Python 3.5:

>>> import timeit
>>> l = list(range(20))
>>> min(timeit.repeat(lambda: l[:]))
0.38448613602668047
>>> min(timeit.repeat(lambda: list(l)))
0.6309100328944623
>>> min(timeit.repeat(lambda: l.copy()))
0.38122922903858125

Making another pointer does not make a copy

Using new_list = my_list then modifies new_list every time my_list changes. Why is this?

my_list is just a name that points to the actual list in memory. When you say new_list = my_list you’re not making a copy, you’re just adding another name that points at that original list in memory. We can have similar issues when we make copies of lists.

>>> l = [[], [], []]
>>> l_copy = l[:]
>>> l_copy
[[], [], []]
>>> l_copy[0].append('foo')
>>> l_copy
[['foo'], [], []]
>>> l
[['foo'], [], []]

The list is just an array of pointers to the contents, so a shallow copy just copies the pointers, and so you have two different lists, but they have the same contents. To make copies of the contents, you need a deep copy.

Deep copies

To make a deep copy of a list, in Python 2 or 3, use deepcopy in the copy module:

import copy
a_deep_copy = copy.deepcopy(a_list)

To demonstrate how this allows us to make new sub-lists:

>>> import copy
>>> l
[['foo'], [], []]
>>> l_deep_copy = copy.deepcopy(l)
>>> l_deep_copy[0].pop()
'foo'
>>> l_deep_copy
[[], [], []]
>>> l
[['foo'], [], []]

And so we see that the deep copied list is an entirely different list from the original. You could roll your own function – but don’t. You’re likely to create bugs you otherwise wouldn’t have by using the standard library’s deepcopy function.

Don’t use eval

You may see this used as a way to deepcopy, but don’t do it:

problematic_deep_copy = eval(repr(a_list))
  1. It’s dangerous, particularly if you’re evaluating something from a source you don’t trust.
  2. It’s not reliable, if a subelement you’re copying doesn’t have a representation that can be eval’d to reproduce an equivalent element.
  3. It’s also less performant.

In 64 bit Python 2.7:

>>> import timeit
>>> import copy
>>> l = range(10)
>>> min(timeit.repeat(lambda: copy.deepcopy(l)))
27.55826997756958
>>> min(timeit.repeat(lambda: eval(repr(l))))
29.04534101486206

on 64 bit Python 3.5:

>>> import timeit
>>> import copy
>>> l = list(range(10))
>>> min(timeit.repeat(lambda: copy.deepcopy(l)))
16.84255409205798
>>> min(timeit.repeat(lambda: eval(repr(l))))
34.813894678023644

How to prevent list from changing after assignment in Python? Answer #4

Let’s start from the beginning and explore this question.

So let’s suppose you have two lists:

list_1 = ['01', '98']
list_2 = [['01', '98']]

And we have to copy both lists, now starting from the first list:

So first let’s try by setting the variable copy to our original list, list_1:

copy = list_1

Now if you are thinking copy copied the list_1, then you are wrong. The id function can show us if two variables can point to the same object. Let’s try this:

print(id(copy))
print(id(list_1))

The output is:

4329485320
4329485320

Both variables are the exact same argument. Are you surprised?

So as we know, Python doesn’t store anything in a variable, Variables are just referencing to the object and object store the value. Here object is a list but we created two references to that same object by two different variable names. This means that both variables are pointing to the same object, just with different names.

When you do copy = list_1, it is actually doing:

Enter image description here

Here in the image list_1 and copy are two variable names, but the object is same for both variable which is list.

So if you try to modify copied list then it will modify the original list too because the list is only one there, you will modify that list no matter you do from the copied list or from the original list:

copy[0] = "modify"

print(copy)
print(list_1)

Output:

['modify', '98']
['modify', '98']

So it modified the original list:

Now let’s move onto a Pythonic method for copying lists.

copy_1 = list_1[:]

This method fixes the first issue we had:

print(id(copy_1))
print(id(list_1))

4338792136
4338791432

So as we can see our both list having different id and it means that both variables are pointing to different objects. So what actually going on here is:

Enter image description here

Now let’s try to modify the list and let’s see if we still face the previous problem:

copy_1[0] = "modify"

print(list_1)
print(copy_1)

The output is:

['01', '98']
['modify', '98']

As you can see, it only modified the copied list. That means it worked.

Do you think we’re done? No. Let’s try to copy our nested list.

copy_2 = list_2[:]

list_2 should reference to another object which is copy of list_2. Let’s check:

print(id((list_2)), id(copy_2))

We get the output:

4330403592 4330403528

Now we can assume both lists are pointing different object, so now let’s try to modify it and let’s see it is giving what we want:

copy_2[0][1] = "modify"

print(list_2, copy_2)

This gives us the output:

[['01', 'modify']] [['01', 'modify']]

This may seem a little bit confusing, because the same method we previously used worked. Let’s try to understand this.

When you do:

copy_2 = list_2[:]

You’re only copying the outer list, not the inside list. We can use the id function once again to check this.

print(id(copy_2[0]))
print(id(list_2[0]))

The output is:

4329485832
4329485832

When we do copy_2 = list_2[:], this happens:

Enter image description here

It creates the copy of list, but only outer list copy, not the nested list copy. The nested list is same for both variable, so if you try to modify the nested list then it will modify the original list too as the nested list object is same for both lists.

What is the solution? The solution is the deepcopy function.

from copy import deepcopy
deep = deepcopy(list_2)

Let’s check this:

print(id((list_2)), id(deep))

4322146056 4322148040

Both outer lists have different IDs. Let’s try this on the inner nested lists.

print(id(deep[0]))
print(id(list_2[0]))

The output is:

4322145992
4322145800

As you can see both IDs are different, meaning we can assume that both nested lists are pointing different object now.

This means when you do deep = deepcopy(list_2) what actually happens:

Enter image description here

Both nested lists are pointing to different objects and they have a separate copy of the nested list now.

Now let’s try to modify the nested list and see if it solved the previous issue or not:

deep[0][1] = "modify"
print(list_2, deep)

It outputs:

[['01', '98']] [['01', 'modify']]

As you can see, it didn’t modify the original nested list, it only modified the copied list.

Hope you’ve got your answer.

Follow Programming Articles for more!

About ᴾᴿᴼᵍʳᵃᵐᵐᵉʳ

Linux and Python enthusiast, in love with open source since 2014, Writer at programming-articles.com, India.

View all posts by ᴾᴿᴼᵍʳᵃᵐᵐᵉʳ →