How do you get a unique list of duplicates from an original list using Python? And how do you get a list of all the duplicate items in a list in Python?
To get a unique list of all the duplicate elements in a list using Python use the list comprehension:
[x for idx, x in enumerate(original_list) if x in original_list[idx+1:] and x not in original_list[:idx]]. To get a list of all the duplicate elements in a list using Python amend the code to
[x for idx, x in enumerate(original_list) if x in original_list[idx+1:]].
Similar to the previous article where duplicates were removed from a list using Python, this solution uses the same technique: a list comprehension with an if condition as a filter on each element.
Here’s an example demonstrating how to get a list of duplicates using the same code previously mentioned:
>>> my_list = [1, 1, 2, 3, 3, 3] >>> [x for idx, x in enumerate(my_list) if x in my_list[idx+1:] and x not in my_list[:idx]] [1, 3] >>> [x for idx, x in enumerate(my_list) if x in my_list[idx+1:]] [1, 3, 3]
As you can see from the above examples the first list comprehension with an if condition produces a unique list of all the duplicates in the list and the second list comprehension (without the
and component) produces a list of the elements that are duplicates.
The beauty of this code is that it’s on one line and as it’s a list comprehension it produces a new list meaning that it doesn’t mutate or change the original list. In the example above,
my_list is still
[1, 1, 2, 3, 3, 3] it hasn’t changed.
So how does this code work?
The list comprehension is a technique that enables easy creation of a list by combining the for loop one-liner with one-line if statement to produce a new list.
The main difference compared to removing duplicates from a list is seen in the
if condition which filters the elements to be added into the new list.
This time the
if condition checks if it is found in the remaining elements of the list and is not found in the preceding elements of the list. The reason for the
and component of the
if condition is that it helps to remove the doubling up of all duplicates populating the list – as is seen in the second list comprehension example, where the
and statement is removed and as you can see all duplicates are therefore listed.
Python provides an easy and powerful way of being able to get all the unique duplicate elements in a list, or if needed each of the elements that are duplicated in a list. This is all done using the powerful list comprehension technique.
You might want to check our other post using this same technique where duplicates are removed from a list.