How do you find all the duplicates from a list in Python?
To find all the duplicate elements from a list using Python, use the following list comprehension:
[x for idx, x in enumerate(original_list) if x in original_list[idx+1:] and x not in original_list[:idx]]
The result from the above code will be a list of unique elements representing all the duplicate elements from the original list.
Here’s an example of how this looks in your Python console:
>>> my_list = [1, 1, 2, 3, 3, 3]
>>> [x for idx, x in enumerate(my_list) if x in my_list[idx+1:] and x not in my_list[:idx]]
Get List Of Duplicates
What if you want a list of all the duplicate entries?
To get a list of all the duplicate elements in a list using Python amend the code to
[x for idx, x in enumerate(original_list) if x in original_list[idx+1:]]
Similar to the previous article, where duplicates were removed from a list using Python , this solution uses the same technique: a list comprehension with an if condition as a filter on each element.
Here’s an example demonstrating how to get a list of duplicates using the same code previously mentioned:
As you can see from the above examples the first list comprehension with an if condition produces a
of all the duplicates in the list and the second list comprehension (without the
component) produces a list of the elements that
The beauty of this code is that it’s on one line and as it’s a list comprehension it produces a new list meaning that it doesn’t mutate or change the original list. In the example above,
[1, 1, 2, 3, 3, 3]
it hasn’t changed.
So how does this code work?
The list comprehension is a technique that enables easy creation of a list by combining the for loop one-liner with one-line if statement to produce a new list.
The main difference compared to
removing duplicates from a list
is seen in the
condition which filters the elements to be added into the new list.
This time the
condition checks if it
found in the remaining elements of the list
found in the preceding elements of the list. The reason for the
component of the
condition is that it helps to remove the doubling up of all duplicates populating the list – as is seen in the second list comprehension example, where the
statement is removed and as you can see all duplicates are therefore listed.
Python provides an easy and powerful way of being able to get all the unique duplicate elements in a list, or if needed each of the elements that are duplicated in a list. This is all done using the powerful list comprehension technique.
You might want to check our other post using this same technique where duplicates are removed from a list .