1. Home
  2. Blog
  3. Software Development

Software Development

Python itertools Explained: Lazy Loops, Combinations and Grouping, Tested

Python itertools gives you fast, memory-light building blocks for looping: infinite counters, chaining, grouping, combinations and batching. Learn each tool with outputs we actually ran, plus the traps that quietly return wrong answers.

10 min read
Python itertools explained: lazy loops, combinations and grouping, tested

Key takeaways

  • itertools is built into Python. Its functions return lazy iterators, so they produce values only when you ask.
  • groupby only groups neighbouring items. Sort by the same key first, or you get split groups.
  • product, permutations and combinations replace nested loops and are handy for test matrices.
  • In our run, flattening 100 lists with chain.from_iterable took 0.045 ms against 0.522 ms for sum(lists, []).

Many Python loops follow the same handful of shapes: count forever, join several lists, group rows by a key, try every pair, or split work into batches. You can write each one by hand with nested for loops and temporary lists, but Python itertools already ships tested, fast versions of all of them. Once you know the names, your loops get shorter and use far less memory.

Every example in this guide was run with Python 3.13.6, and the outputs shown are copied from that run. If you are new to iterators, our guide to Python generators is a good companion, because generators and itertools work together naturally.

What is itertools?

itertools is a standard library module of iterator building blocks. You import it, call a function, and get back an iterator object rather than a list. The functions fall into three groups:

  • Infinite iterators: count, cycle and repeat.
  • Iterators that end with the shortest input: accumulate, chain, groupby, compress, pairwise, batched, takewhile and others.
  • Combinatoric iterators: product, permutations, combinations and combinations_with_replacement.
import itertools as it

Why lazy iterators matter

A list stores every value in memory at once. An iterator stores only enough state to produce the next value. That is why you can build an infinite counter without your program freezing: nothing is computed until a loop, next() or list() pulls a value out.

We measured the difference. A list of one million integers took 8,448,728 bytes according to sys.getsizeof, which counts only the list’s pointer array. An islice object that yields the same million numbers took 72 bytes. The catch is that an iterator can be read once only, which we cover in the tee section.

count, cycle and repeat

These three never stop on their own, so pair them with islice, zip or a break.

print(list(it.islice(it.count(10, 5), 4)))
print(list(it.islice(it.cycle("AB"), 5)))
print(list(map(pow, range(4), it.repeat(2))))
[10, 15, 20, 25]
['A', 'B', 'A', 'B', 'A']
[0, 1, 4, 9]

count(start, step) is perfect for generating IDs. cycle loops a sequence forever, which suits round-robin work such as assigning tickets to a rota of testers. repeat feeds the same value to map without building a list of copies.

A neat trick combines count with a generator expression to find the first value that meets a rule:

first = next(x for x in it.count(1) if x * x > 2026)
print(first)   # 46

accumulate: running totals

accumulate returns running results. By default it adds, but you can pass any two-argument function.

print(list(it.accumulate([5, 3, -2, 4])))
print(list(it.accumulate([3, 1, 4, 1, 5], max)))
[5, 8, 6, 10]
[3, 3, 4, 4, 5]

The first line is a running balance. The second is a running maximum, which is useful for tracking the best score so far or the peak response time in a load test.

chain: join iterables without copying

chain walks through several iterables one after another as if they were one. chain.from_iterable does the same for an iterable of iterables, which is the standard way to flatten one level of nesting.

print(list(it.chain([1, 2], (3,), "ab")))
print(list(it.chain.from_iterable([[1, 2], [3], [4, 5]])))
[1, 2, 3, 'a', 'b']
[1, 2, 3, 4, 5]

Notice that a string is itself iterable, so "ab" becomes two items. If you want the string as one item, wrap it in a list first.

groupby: the most misunderstood tool

groupby splits a sequence into runs of items that share a key. The word “runs” is the important part: it only groups items that sit next to each other. Look what happened with unsorted log levels:

logs = ["error", "error", "info", "error", "info", "info"]
print([(k, len(list(g))) for k, g in it.groupby(logs)])
print([(k, len(list(g))) for k, g in it.groupby(sorted(logs))])
[('error', 2), ('info', 1), ('error', 1), ('info', 2)]
[('error', 3), ('info', 3)]

The unsorted version produced four groups instead of two, and nothing warned us. This behaves very differently from SQL, where GROUP BY collects all matching rows wherever they are. If that comparison helps, see SQL GROUP BY and HAVING.

For records, sort and group with the same key function:

import operator
orders = [{"city": "Pune", "amt": 120}, {"city": "Delhi", "amt": 80}, {"city": "Pune", "amt": 50}]
key = operator.itemgetter("city")
print({k: sum(o["amt"] for o in g) for k, g in it.groupby(sorted(orders, key=key), key=key)})
{'Delhi': 80, 'Pune': 170}

Tip: each group is itself an iterator that becomes invalid once groupby moves on. Consume it straight away, for example with list(g) or sum(...), before the next group.

If your data is not already sorted and you only need totals, a plain Python dictionary or collections.Counter is often simpler and avoids the sort.

product, permutations and combinations

These replace nested loops when you need every arrangement of some inputs.

print(list(it.product(["chrome", "firefox"], ["win", "mac"])))
print(["".join(p) for p in it.permutations("ABC", 2)])
print(["".join(c) for c in it.combinations("ABCD", 2)])
print(["".join(c) for c in it.combinations_with_replacement("AB", 2)])
[('chrome', 'win'), ('chrome', 'mac'), ('firefox', 'win'), ('firefox', 'mac')]
['AB', 'AC', 'BA', 'BC', 'CA', 'CB']
['AB', 'AC', 'AD', 'BC', 'BD', 'CD']
['AA', 'AB', 'BB']
Function Order matters? Repeats allowed? Typical use
product Yes Yes (across inputs) Browser and OS matrix
permutations Yes No Orderings of steps
combinations No No Unique pairs to compare
combinations_with_replacement No Yes Multisets, dice totals

Be careful with size. Counting the product of three ranges of 3, 4 and 5 items gave 60 tuples. Each extra dimension multiplies the total, so a matrix that looks small can explode. Count with sum(1 for _ in ...) or math.prod before you run anything expensive inside the loop.

pairwise and batched

pairwise (Python 3.10+) gives overlapping neighbours, which is ideal for differences between readings. batched (Python 3.12+) splits data into fixed-size chunks.

print(list(it.pairwise([1, 4, 9, 16])))
print(list(it.batched(range(7), 3)))
[(1, 4), (4, 9), (9, 16)]
[(0, 1, 2), (3, 4, 5), (6,)]

Note that the last batch is shorter. If you send batches to an API that expects exactly three items, handle that final chunk. batched is a clean way to insert rows into a database a few hundred at a time, or to respect an API rate limit.

Filtering: compress, takewhile, dropwhile, filterfalse

print(list(it.compress("ABCDE", [1, 0, 1, 0, 1])))
print(list(it.takewhile(lambda x: x < 5, [1, 3, 6, 2])))
print(list(it.dropwhile(lambda x: x < 5, [1, 3, 6, 2])))
print(list(it.filterfalse(lambda x: x % 2, range(8))))
['A', 'C', 'E']
[1, 3]
[6, 2]
[0, 2, 4, 6]

The surprise here is takewhile. It stops at the first failure, so the 2 at the end was never checked. dropwhile is the mirror image: once the condition fails, it yields everything after it, including the 2. Use them for sorted data or streams with a clear cut-off, such as reading log lines until a timestamp passes. For a general filter, use filter or a comprehension. The short functions passed here are Python lambda functions.

zip_longest and starmap

Built-in zip stops at the shortest input and silently drops the rest. zip_longest keeps going and fills the gaps:

print(list(zip("abc", [1, 2])))
print(list(it.zip_longest("abc", [1, 2], fillvalue=0)))
print(list(it.starmap(pow, [(2, 3), (3, 2), (10, 0)])))
[('a', 1), ('b', 2)]
[('a', 1), ('b', 2), ('c', 0)]
[8, 9, 1]

When comparing an expected list with an actual list in a test, zip can hide a missing final item. zip_longest exposes it. Since Python 3.10 you can also pass strict=True to zip to raise an error on unequal lengths. starmap unpacks each tuple into function arguments, which saves a lambda.

tee and iterator exhaustion

An iterator is used up after one pass. This is the most common itertools bug:

g = (x * x for x in range(3))
print(list(g), list(g))     # [0, 1, 4] []

a, b = it.tee(iter([1, 2, 3]))
print(list(a), list(b))     # [1, 2, 3] [1, 2, 3]

The second list(g) was empty, with no error. tee splits one iterator into independent copies, but it buffers values internally. If one copy runs far ahead of the other, memory grows, and at that point a plain list is usually simpler. Also, do not use the original iterator after calling tee.

Memory and speed in practice

We timed flattening 100 lists of 100 numbers, 200 times each. sum(lists, []) averaged 0.522 ms, while list(chain.from_iterable(lists)) averaged 0.045 ms, about eleven times faster. The sum version is slow because it builds a new, longer list at every step.

Your numbers will differ by machine, but the shape holds: itertools functions are implemented in C and avoid building temporary lists. The bigger win is usually memory. A pipeline of iterators can process a file of millions of lines while holding only one line at a time. If you prefer comprehension syntax, list comprehensions build lists eagerly, while generator expressions combine well with itertools.

Using itertools in testing

Testers get a lot out of this module. product builds configuration matrices for parametrised tests, such as every browser with every screen size. combinations gives every pair of features to test together, which is the starting point for pairwise testing. cycle rotates test accounts so parallel runs do not fight over one login. count generates unique order numbers for test data.

import pytest
MATRIX = list(it.product(["chrome", "firefox"], ["1280x800", "375x812"]))

@pytest.mark.parametrize("browser,size", MATRIX)
def test_home_page(browser, size):
    ...

This pytest snippet shows the pattern only; we did not run it against a browser. Keep the matrix small and deliberate, because four combinations become forty quickly. Our pytest fixtures guide shows how to share setup across these cases.

When to use itertools, and when not to

  • Use it for streams, large files, generated sequences, combinations and batching, where laziness saves memory.
  • Use it to replace nested loops whose only job is to build arrangements.
  • Skip it when a list comprehension reads more clearly to your team. A clever one-liner nobody can debug is not a win.
  • Skip it when you need to reuse the data many times; store a list instead of fighting exhaustion.

The official documentation also includes a “recipes” section with ready-made helpers built from these pieces. Read those once you are comfortable with the basics here.

Final thoughts

Python itertools rewards a small investment. Learn chain, groupby, product, pairwise and batched first, because they cover most everyday loops. Then remember the two traps that caught us in testing: sort before groupby, and never read an iterator twice. With those habits, your loops become shorter, faster and easier to review.

Frequently asked questions

What is itertools in Python?

itertools is a built-in module of fast iterator tools for counting, chaining, grouping, filtering and generating combinations. Its functions return lazy iterators instead of lists.

Why does itertools.groupby give duplicate groups?

groupby only groups consecutive items with the same key. Sort the data by the same key first. In our test, unsorted logs produced four groups instead of two.

What is the difference between permutations and combinations?

permutations treats AB and BA as different, while combinations treats them as the same pair. For 2 items from ABC, permutations gave 6 results and combinations would give 3.

Which Python version do I need for batched and pairwise?

pairwise was added in Python 3.10 and batched in Python 3.12. We ran every example in this guide on Python 3.13.6.

Is itertools faster than writing loops?

Often, yes, because the functions are implemented in C and avoid temporary lists. Flattening with chain.from_iterable was about eleven times faster than sum(lists, []) in our timing.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

BM25 Explained: How Keyword Search Ranks Results, With Python Code and Hybrid Search

BM25 explained with a pure Python implementation we ran: the formula, IDF, term saturation, the k1 and b parameters, why…

Read article →
Software Development

HTTP Methods Explained: GET, POST, PUT, PATCH, DELETE, Tested on a Live Server

HTTP methods explained with a small server we ran: GET, POST, PUT, PATCH, DELETE, HEAD and OPTIONS, safe vs idempotent,…

Read article →
DevOps

Git Bisect Explained: Find the Commit That Broke Your Code, Manually or Automatically

Git bisect explained with a tested repository: find the one bad commit among 41 in 5 test runs, automate it…

Read article →
Test Automation

pytest Markers Explained: Custom Marks, skip, skipif, xfail and -m Filters

pytest markers explained with a tested suite: register custom marks, run only smoke tests with -m, combine filters with and,…

Read article →
Software Testing

Positive vs Negative Testing: Differences, 24 Real Test Cases and a Checklist

Positive vs negative testing explained with a real signup form: 6 positive and 18 negative test cases run in pytest,…

Read article →
Databases & SQL

SQL ORDER BY Multiple Columns: ASC, DESC, NULLS LAST, Case and Custom Order

SQL ORDER BY with multiple columns explained with tested queries: mixed ASC and DESC, tie-breakers, NULLS FIRST and LAST, case-insensitive…

Read article →
Databases & SQL

SQL Running Total: SUM OVER, Per-Group Totals, Moving Sums and the RANGE Trap

How to calculate a SQL running total with SUM() OVER: per-group totals, daily cumulative sums, moving sums, percent of total,…

Read article →
Software Development

JavaScript Sort Array: Numbers, Objects, Strings and Why 10 Comes Before 5

How to sort an array in JavaScript correctly: numbers with a compare function, objects by one or more fields, strings…

Read article →
Software Development

Python String Methods: Cheat Sheet, Real Examples and the strip() Trap

Python string methods with tested examples: split, join, strip, replace, find, startswith, is-methods, casefold, partition and translate, plus the strip()…

Read article →
Software Development

Python zip() Explained: Loop Two Lists at Once, Unzip, strict=True and Traps

Python zip() pairs items from several lists in one loop. Tested examples for dicts, unzipping with *, strict=True, zip_longest, sliding…

Read article →
Artificial Intelligence

Cosine Similarity Explained: Formula, Worked Example and Python Code

Cosine similarity explained simply: the formula, a worked example by hand, pure Python and NumPy code, cosine vs Euclidean distance…

Read article →
Software Development

UTF-8 Encoding Explained: How Text Becomes Bytes and How to Fix Garbled Text

UTF-8 encoding explained with byte-level examples we ran in Python: how characters become 1 to 4 bytes, why café turns…

Read article →
DevOps

Git Squash Commits: 4 Easy Ways, Tested Step by Step (rebase -i, reset, merge)

Git squash commits explained with 4 tested methods: interactive rebase, reset –soft, merge –squash and fixup with autosquash. Before-and-after logs,…

Read article →
Test Automation

pytest parametrize Explained: 7 Patterns with Tested Examples

pytest parametrize explained with 7 tested patterns: basic cases, readable ids, stacked decorators, pytest.param with xfail and skip, indirect fixtures…

Read article →
Software Testing

Retesting vs Regression Testing: 8 Key Differences with a Real Bug Fix Example

Retesting vs regression testing explained with a real bug fix run in pytest: retesting confirmed the fix, the regression run…

Read article →
Databases & SQL

Delete Duplicate Rows in SQL: 5 Safe Methods, Tested Step by Step

How to delete duplicate rows in SQL safely: find them, preview them, then remove them with ROW_NUMBER, MIN(id), EXISTS, rowid…

Read article →
Databases & SQL

SQL RANK vs DENSE_RANK vs ROW_NUMBER: Differences with Tested Examples

SQL RANK vs DENSE_RANK vs ROW_NUMBER in one table: how each handles ties, top-N per group, the second-highest salary trap…

Read article →
Software Development

JavaScript Optional Chaining (?.) Explained: 8 Examples, ?? Defaults and Traps

JavaScript optional chaining stops “Cannot read properties of undefined” errors. 8 tested examples for properties, arrays and methods, ?? vs…

Read article →
Artificial Intelligence

RAG Chunking Strategies Explained: Fixed, Overlap, Sentence, Recursive and Headings, Measured

RAG chunking strategies compared on one document with tiktoken 0.14: fixed-size, overlap, sentence packing, recursive splitting and heading-based chunks, measuring…

Read article →
DevOps

Git Stash Explained: Save, Apply, Pop and Untracked Files, Tested

Git stash explained with commands we ran on Git 2.56: push with a message, untracked files, list and show, apply…

Read article →
Software Testing

Pairwise Testing Explained: How All-Pairs Cuts 216 Tests to 13, Tested

Pairwise testing explained with a generator we ran in Python 3.13: a checkout with 216 combinations covered by 13 tests,…

Read article →
Test Automation

Test Doubles Explained: Dummy, Stub, Spy, Mock and Fake, with Tested Python Code

Test doubles explained with one Python example run in pytest 9.2: dummy, stub, spy, mock and fake, when to use…

Read article →
Databases & SQL

SQL COALESCE Explained: Defaults, NULLIF, IFNULL vs ISNULL and Traps, Tested

SQL COALESCE explained with queries we ran in SQLite 3.50: first non-null values, empty strings, NULL in maths, NULLIF for…

Read article →
Databases & SQL

SQL Self Join Explained: Employee-Manager, Pairs and Row Comparisons, Tested

SQL self join explained with queries we ran in SQLite 3.50: employee and manager, earning more than the manager, unique…

Read article →
Artificial Intelligence

LLM Context Window Explained: Tokens, Limits and How to Fit More, Measured

The LLM context window explained with token counts we measured using tiktoken: words vs tokens for prose, code and Hindi,…

Read article →
DevOps

Git Cherry-Pick Explained: Copy the Commit You Need, Safely, Tested

Git cherry-pick explained with commands we ran in Git 2.56: copying a hotfix to a release branch, new hashes and…

Read article →
Test Automation

Snapshot Testing Explained: How It Works, When It Helps and When It Hurts

Snapshot testing explained with a pytest and syrupy demo we ran: creating snapshots, reading the diff after a code change,…

Read article →
Software Testing

Mutation Testing Explained: Why 100% Coverage Still Missed Our Bug

Mutation testing explained with a small mutation tool we built and ran: 14 mutants, a test suite with 100% line…

Read article →
Artificial Intelligence

Prompt Injection Explained: How AI Apps Get Tricked and How to Defend Them

Prompt injection explained for builders: direct and indirect attacks, why keyword filters fail (our test blocked 1 of 4), and…

Read article →
DevOps

DNS Records Explained: A, AAAA, CNAME, MX, TXT and TTL With Real Lookups

DNS records explained with lookups we ran: how resolution works, A and AAAA, CNAME rules, MX and null MX, TXT…

Read article →
DevOps

Feature Flags Explained: Safe Releases, Gradual Rollouts and Flag Hygiene

Feature flags explained with a rollout simulation we ran: flag types, percentage rollouts with stable hashing, kill switches, allow lists,…

Read article →
Test Automation

Test Data Generation With Python Faker: Realistic, Repeatable and Safe Data

Test data generation with Python Faker, run and measured: seeding for repeatable data, locales, unique values, domain-shaped records, edge cases…

Read article →
Software Testing

Shift-Left Testing Explained: Find Bugs Earlier Without Slowing the Team

Shift-left testing explained: what it means, why early defects are cheaper, practices from requirement reviews to CI checks, the tester's…

Read article →
Artificial Intelligence

LLM Structured Output Explained: Getting Reliable JSON From AI Models

LLM structured output explained with validation we ran in Pydantic: JSON mode and schemas, constrained decoding, the ways model output…

Read article →
Test Automation

API Contract Testing Explained: Catch Breaking Changes Before They Ship

API contract testing explained with a schema check we ran: what a contract is, consumer-driven contracts, Pact, OpenAPI, which changes…

Read article →
Software Testing

Risk-Based Testing Explained: Score Risks, Build a Matrix and Test What Matters

Risk-based testing explained step by step: identifying risks, scoring likelihood and impact, building a risk matrix, mapping risk to test…

Read article →