Python itertools Explained: Lazy Loops, Combinations and Grouping, Tested
Python itertools gives you fast, memory-light building blocks for looping: infinite counters, chaining, grouping, combinations and batching. Learn each tool with outputs we actually ran, plus the traps that quietly return wrong answers.
itertoolsis built into Python. Its functions return lazy iterators, so they produce values only when you ask.groupbyonly groups neighbouring items. Sort by the same key first, or you get split groups.product,permutationsandcombinationsreplace nested loops and are handy for test matrices.- In our run, flattening 100 lists with
chain.from_iterabletook 0.045 ms against 0.522 ms forsum(lists, []).
Many Python loops follow the same handful of shapes: count forever, join several lists, group rows by a key, try every pair, or split work into batches. You can write each one by hand with nested for loops and temporary lists, but Python itertools already ships tested, fast versions of all of them. Once you know the names, your loops get shorter and use far less memory.
Every example in this guide was run with Python 3.13.6, and the outputs shown are copied from that run. If you are new to iterators, our guide to Python generators is a good companion, because generators and itertools work together naturally.
What is itertools?
itertools is a standard library module of iterator building blocks. You import it, call a function, and get back an iterator object rather than a list. The functions fall into three groups:
- Infinite iterators:
count,cycleandrepeat. - Iterators that end with the shortest input:
accumulate,chain,groupby,compress,pairwise,batched,takewhileand others. - Combinatoric iterators:
product,permutations,combinationsandcombinations_with_replacement.
import itertools as it
Why lazy iterators matter
A list stores every value in memory at once. An iterator stores only enough state to produce the next value. That is why you can build an infinite counter without your program freezing: nothing is computed until a loop, next() or list() pulls a value out.
We measured the difference. A list of one million integers took 8,448,728 bytes according to sys.getsizeof, which counts only the list’s pointer array. An islice object that yields the same million numbers took 72 bytes. The catch is that an iterator can be read once only, which we cover in the tee section.
count, cycle and repeat
These three never stop on their own, so pair them with islice, zip or a break.
print(list(it.islice(it.count(10, 5), 4)))
print(list(it.islice(it.cycle("AB"), 5)))
print(list(map(pow, range(4), it.repeat(2))))
[10, 15, 20, 25]
['A', 'B', 'A', 'B', 'A']
[0, 1, 4, 9]
count(start, step) is perfect for generating IDs. cycle loops a sequence forever, which suits round-robin work such as assigning tickets to a rota of testers. repeat feeds the same value to map without building a list of copies.
A neat trick combines count with a generator expression to find the first value that meets a rule:
first = next(x for x in it.count(1) if x * x > 2026)
print(first) # 46
accumulate: running totals
accumulate returns running results. By default it adds, but you can pass any two-argument function.
print(list(it.accumulate([5, 3, -2, 4])))
print(list(it.accumulate([3, 1, 4, 1, 5], max)))
[5, 8, 6, 10]
[3, 3, 4, 4, 5]
The first line is a running balance. The second is a running maximum, which is useful for tracking the best score so far or the peak response time in a load test.
chain: join iterables without copying
chain walks through several iterables one after another as if they were one. chain.from_iterable does the same for an iterable of iterables, which is the standard way to flatten one level of nesting.
print(list(it.chain([1, 2], (3,), "ab")))
print(list(it.chain.from_iterable([[1, 2], [3], [4, 5]])))
[1, 2, 3, 'a', 'b']
[1, 2, 3, 4, 5]
Notice that a string is itself iterable, so "ab" becomes two items. If you want the string as one item, wrap it in a list first.
groupby: the most misunderstood tool
groupby splits a sequence into runs of items that share a key. The word “runs” is the important part: it only groups items that sit next to each other. Look what happened with unsorted log levels:
logs = ["error", "error", "info", "error", "info", "info"]
print([(k, len(list(g))) for k, g in it.groupby(logs)])
print([(k, len(list(g))) for k, g in it.groupby(sorted(logs))])
[('error', 2), ('info', 1), ('error', 1), ('info', 2)]
[('error', 3), ('info', 3)]
The unsorted version produced four groups instead of two, and nothing warned us. This behaves very differently from SQL, where GROUP BY collects all matching rows wherever they are. If that comparison helps, see SQL GROUP BY and HAVING.
For records, sort and group with the same key function:
import operator
orders = [{"city": "Pune", "amt": 120}, {"city": "Delhi", "amt": 80}, {"city": "Pune", "amt": 50}]
key = operator.itemgetter("city")
print({k: sum(o["amt"] for o in g) for k, g in it.groupby(sorted(orders, key=key), key=key)})
{'Delhi': 80, 'Pune': 170}
Tip: each group is itself an iterator that becomes invalid once groupby moves on. Consume it straight away, for example with list(g) or sum(...), before the next group.
If your data is not already sorted and you only need totals, a plain Python dictionary or collections.Counter is often simpler and avoids the sort.
product, permutations and combinations
These replace nested loops when you need every arrangement of some inputs.
print(list(it.product(["chrome", "firefox"], ["win", "mac"])))
print(["".join(p) for p in it.permutations("ABC", 2)])
print(["".join(c) for c in it.combinations("ABCD", 2)])
print(["".join(c) for c in it.combinations_with_replacement("AB", 2)])
[('chrome', 'win'), ('chrome', 'mac'), ('firefox', 'win'), ('firefox', 'mac')]
['AB', 'AC', 'BA', 'BC', 'CA', 'CB']
['AB', 'AC', 'AD', 'BC', 'BD', 'CD']
['AA', 'AB', 'BB']
| Function | Order matters? | Repeats allowed? | Typical use |
|---|---|---|---|
product |
Yes | Yes (across inputs) | Browser and OS matrix |
permutations |
Yes | No | Orderings of steps |
combinations |
No | No | Unique pairs to compare |
combinations_with_replacement |
No | Yes | Multisets, dice totals |
Be careful with size. Counting the product of three ranges of 3, 4 and 5 items gave 60 tuples. Each extra dimension multiplies the total, so a matrix that looks small can explode. Count with sum(1 for _ in ...) or math.prod before you run anything expensive inside the loop.
pairwise and batched
pairwise (Python 3.10+) gives overlapping neighbours, which is ideal for differences between readings. batched (Python 3.12+) splits data into fixed-size chunks.
print(list(it.pairwise([1, 4, 9, 16])))
print(list(it.batched(range(7), 3)))
[(1, 4), (4, 9), (9, 16)]
[(0, 1, 2), (3, 4, 5), (6,)]
Note that the last batch is shorter. If you send batches to an API that expects exactly three items, handle that final chunk. batched is a clean way to insert rows into a database a few hundred at a time, or to respect an API rate limit.
Filtering: compress, takewhile, dropwhile, filterfalse
print(list(it.compress("ABCDE", [1, 0, 1, 0, 1])))
print(list(it.takewhile(lambda x: x < 5, [1, 3, 6, 2])))
print(list(it.dropwhile(lambda x: x < 5, [1, 3, 6, 2])))
print(list(it.filterfalse(lambda x: x % 2, range(8))))
['A', 'C', 'E']
[1, 3]
[6, 2]
[0, 2, 4, 6]
The surprise here is takewhile. It stops at the first failure, so the 2 at the end was never checked. dropwhile is the mirror image: once the condition fails, it yields everything after it, including the 2. Use them for sorted data or streams with a clear cut-off, such as reading log lines until a timestamp passes. For a general filter, use filter or a comprehension. The short functions passed here are Python lambda functions.
zip_longest and starmap
Built-in zip stops at the shortest input and silently drops the rest. zip_longest keeps going and fills the gaps:
print(list(zip("abc", [1, 2])))
print(list(it.zip_longest("abc", [1, 2], fillvalue=0)))
print(list(it.starmap(pow, [(2, 3), (3, 2), (10, 0)])))
[('a', 1), ('b', 2)]
[('a', 1), ('b', 2), ('c', 0)]
[8, 9, 1]
When comparing an expected list with an actual list in a test, zip can hide a missing final item. zip_longest exposes it. Since Python 3.10 you can also pass strict=True to zip to raise an error on unequal lengths. starmap unpacks each tuple into function arguments, which saves a lambda.
tee and iterator exhaustion
An iterator is used up after one pass. This is the most common itertools bug:
g = (x * x for x in range(3))
print(list(g), list(g)) # [0, 1, 4] []
a, b = it.tee(iter([1, 2, 3]))
print(list(a), list(b)) # [1, 2, 3] [1, 2, 3]
The second list(g) was empty, with no error. tee splits one iterator into independent copies, but it buffers values internally. If one copy runs far ahead of the other, memory grows, and at that point a plain list is usually simpler. Also, do not use the original iterator after calling tee.
Memory and speed in practice
We timed flattening 100 lists of 100 numbers, 200 times each. sum(lists, []) averaged 0.522 ms, while list(chain.from_iterable(lists)) averaged 0.045 ms, about eleven times faster. The sum version is slow because it builds a new, longer list at every step.
Your numbers will differ by machine, but the shape holds: itertools functions are implemented in C and avoid building temporary lists. The bigger win is usually memory. A pipeline of iterators can process a file of millions of lines while holding only one line at a time. If you prefer comprehension syntax, list comprehensions build lists eagerly, while generator expressions combine well with itertools.
Using itertools in testing
Testers get a lot out of this module. product builds configuration matrices for parametrised tests, such as every browser with every screen size. combinations gives every pair of features to test together, which is the starting point for pairwise testing. cycle rotates test accounts so parallel runs do not fight over one login. count generates unique order numbers for test data.
import pytest
MATRIX = list(it.product(["chrome", "firefox"], ["1280x800", "375x812"]))
@pytest.mark.parametrize("browser,size", MATRIX)
def test_home_page(browser, size):
...
This pytest snippet shows the pattern only; we did not run it against a browser. Keep the matrix small and deliberate, because four combinations become forty quickly. Our pytest fixtures guide shows how to share setup across these cases.
When to use itertools, and when not to
- Use it for streams, large files, generated sequences, combinations and batching, where laziness saves memory.
- Use it to replace nested loops whose only job is to build arrangements.
- Skip it when a list comprehension reads more clearly to your team. A clever one-liner nobody can debug is not a win.
- Skip it when you need to reuse the data many times; store a list instead of fighting exhaustion.
The official documentation also includes a “recipes” section with ready-made helpers built from these pieces. Read those once you are comfortable with the basics here.
Final thoughts
Python itertools rewards a small investment. Learn chain, groupby, product, pairwise and batched first, because they cover most everyday loops. Then remember the two traps that caught us in testing: sort before groupby, and never read an iterator twice. With those habits, your loops become shorter, faster and easier to review.
Frequently asked questions
What is itertools in Python?
itertools is a built-in module of fast iterator tools for counting, chaining, grouping, filtering and generating combinations. Its functions return lazy iterators instead of lists.
Why does itertools.groupby give duplicate groups?
groupby only groups consecutive items with the same key. Sort the data by the same key first. In our test, unsorted logs produced four groups instead of two.
What is the difference between permutations and combinations?
permutations treats AB and BA as different, while combinations treats them as the same pair. For 2 items from ABC, permutations gave 6 results and combinations would give 3.
Which Python version do I need for batched and pairwise?
pairwise was added in Python 3.10 and batched in Python 3.12. We ran every example in this guide on Python 3.13.6.
Is itertools faster than writing loops?
Often, yes, because the functions are implemented in C and avoid temporary lists. Flattening with chain.from_iterable was about eleven times faster than sum(lists, []) in our timing.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.