How do you find all the duplicates from a list in Python?
To find all the duplicate elements from a list using Python, use the following list comprehension:
[x for idx, x in enumerate(original_list) if x in original_list[idx+1:] and x not in original_list[:idx]]The result from the above code will be a list of unique elements representing all the duplicate elements from the original list.
Here’s an example of how this looks in your Python console:
>>> my_list = [1, 1, 2, 3, 3, 3]
>>> [x for idx, x in enumerate(my_list) if x in my_list[idx+1:] and x not in my_list[:idx]]
[1, 3]Get List Of Duplicates
What if you want a list of all the duplicate entries?
To get a list of all the duplicate elements in a list using Python amend the code to
[x for idx, x in enumerate(original_list) if x in original_list[idx+1:]]
.
Similar to the previous article, where duplicates were removed from a list using Python , this solution uses the same technique: a list comprehension with an if condition as a filter on each element.
Here’s an example demonstrating how to get a list of duplicates using the same code previously mentioned:
As you can see from the above examples the first list comprehension with an if condition produces a
unique list
of all the duplicates in the list and the second list comprehension (without the
and
component) produces a list of the elements that
are duplicates
.
The beauty of this code is that it’s on one line and as it’s a list comprehension it produces a new list meaning that it doesn’t mutate or change the original list. In the example above,
my_list
is still
[1, 1, 2, 3, 3, 3]
it hasn’t changed.
So how does this code work?
The list comprehension is a technique that enables easy creation of a list by combining the for loop one-liner with one-line if statement to produce a new list.
The main difference compared to
removing duplicates from a list
is seen in the
if
condition which filters the elements to be added into the new list.
This time the
if
condition checks if it
is
found in the remaining elements of the list
and
is
not
found in the preceding elements of the list. The reason for the
and
component of the
if
condition is that it helps to remove the doubling up of all duplicates populating the list – as is seen in the second list comprehension example, where the
and
statement is removed and as you can see all duplicates are therefore listed.
Summary
Python provides an easy and powerful way of being able to get all the unique duplicate elements in a list, or if needed each of the elements that are duplicated in a list. This is all done using the powerful list comprehension technique.
You might want to check our other post using this same technique where duplicates are removed from a list .
