The Python Sets: No Duplicates Allowed 🎯
The container everyone forgets exists, and then uses constantly once they remember. A set is a bag where nothing appears twice and order means nothing.
Duplicates simply vanish
visited = {"Melee", "Monkey", "Melee", "Booty", "Monkey"}
print(sorted(visited))
print(len(visited))['Booty', 'Melee', 'Monkey']
3A set has no order, and printing one raw shows its items in whatever arrangement the internal table happens to produce. That arrangement can differ between runs, between machines and between Python versions. Every example on this page sorts before printing, and you should too, whenever a human is going to read the output.
Curly braces like a dictionary, but with single values instead of pairs. Adding something already present does nothing at all, which is the entire point.
{{}} is an empty dictionary, not an empty set. Python had dictionaries first and they got the braces. For an empty set you must write set().
The one-line deduplicate
names = ["Otis", "Elaine", "Otis", "Guybrush", "Elaine", "Otis"]
unique = list(set(names))
print(sorted(unique))
print(f"{len(names)} entries, {len(unique)} unique")['Elaine', 'Guybrush', 'Otis']
6 entries, 3 unique
Note the sorted. Sets have no order, so list(set(...)) can
come back in any arrangement. If you need the original order preserved while removing
duplicates, use a dictionary instead, which keeps insertion order:
names = ["Otis", "Elaine", "Otis", "Guybrush", "Elaine"]
print(list(dict.fromkeys(names)))['Otis', 'Elaine', 'Guybrush']Membership, at speed
allowed = {"guybrush", "elaine", "otis"}
print("elaine" in allowed)
print("lechuck" in allowed)
allowed.add("meathook")
allowed.discard("otis") # no error if it is absent
print(sorted(allowed))True
False
['elaine', 'guybrush', 'meathook']
in on a set is effectively instant no matter how big the set is, exactly
like a dictionary key lookup and for the same reason. in on a list checks
items one at a time. If you are testing membership against thousands of items inside a
loop, converting the list to a set once can turn minutes into milliseconds.
This is the first optimisation worth learning, because it is not a trick, it is a correction. You were using the wrong container. Choosing the right data structure beats clever code almost every time, and it usually makes the code shorter too.
Set arithmetic
pirates = {"Guybrush", "LeChuck", "Meathook"}
governors = {"Elaine", "Guybrush"}
print(sorted(pirates | governors)) # union: everyone
print(sorted(pirates & governors)) # intersection: in both
print(sorted(pirates - governors)) # difference: pirates only
print(sorted(pirates ^ governors)) # symmetric difference: in one but not both['Elaine', 'Guybrush', 'LeChuck', 'Meathook']
['Guybrush']
['LeChuck', 'Meathook']
['Elaine', 'LeChuck', 'Meathook']| Operator | Method | Means |
|---|---|---|
| | .union() | in either |
& | .intersection() | in both |
- | .difference() | in the first only |
^ | .symmetric_difference() | in exactly one |
<= | .issubset() | all of these are in that |
These turn fiddly loops into one readable line. "Which users signed up but never logged
in" is signed_up - logged_in. "Which tags do these two articles share" is
a & b. Whenever you catch yourself writing a loop with an
if x in other_list inside it, stop and ask whether this is set arithmetic
wearing a disguise.
What sets cannot do
- No order. There is no
set[0]. If order matters, you want a list. - No duplicates, obviously. If you need to count occurrences, you
want
Counter. - Items must be immutable, for the same hashing reason as dictionary keys. Sets of tuples are fine, sets of lists are not.
Choosing a container
| If you need... | Use | Why |
|---|---|---|
| An ordered collection you will change | list | The default. Order and duplicates both kept |
| A fixed group that belongs together | tuple | Cannot be changed by accident, can be a dict key |
| Lookup by name or id | dict | Instant lookup, readable code |
| Uniqueness or fast membership | set | Duplicates gone, in is instant |
| Counting occurrences | Counter | A dict that starts every count at zero |
Common interests
Two people list their hobbies. Print what they share, what only the first one does, and the combined list, all sorted.
Reveal solution
alice = {"sailing", "insults", "cartography", "grog"}
bob = {"grog", "swordfighting", "sailing"}
print("Both: ", sorted(alice & bob))
print("Alice only:", sorted(alice - bob))
print("Together: ", sorted(alice | bob))Both: ['grog', 'sailing']
Alice only: ['cartography', 'insults']
Together: ['cartography', 'grog', 'insults', 'sailing', 'swordfighting']Unique words
Count how many distinct words appear in a sentence, ignoring case and full stops, and list any that appear more than once.
Reveal solution
text = "The dog saw the cat. The cat saw the dog."
words = text.lower().replace(".", "").split()
unique = set(words)
print(f"{len(words)} words, {len(unique)} distinct")
repeated = sorted(w for w in unique if words.count(w) > 1)
print("repeated:", repeated)10 words, 4 distinct
repeated: ['cat', 'dog', 'saw', 'the']Honest note: words.count(w) inside a loop scans the whole list every time, which is wasteful. For ten words nobody cares; for ten million you would use Counter. Knowing when you are allowed not to care is part of the job.
Which container?
Pick the right one for each, and say why.
- Every unique IP address that hit a web server today.
- The order players took their turns in.
- Looking up a product's price by its barcode.
- Checking whether a word is in a 300,000 word dictionary, a million times.
Reveal solution
- set. Unique is in the requirement.
- list. Order is the entire point, and repeats are possible.
- dict, keyed by barcode. Instant lookup by name.
- set. A list would do 300,000 comparisons per check, a million times over. A set does one. This is the difference between a program that finishes and one you kill after an hour.