1. Home
  2. Blog
  3. Software Development

Software Development

Python Subprocess Explained: run, Capture Output, Timeouts and Safety

Python subprocess runs other programs from your code: git, ffmpeg, a shell script or another Python. Learn subprocess.run with captured output, error checking, timeouts, input, environment variables and live streaming with Popen, plus the shell injection we reproduced and the Windows encoding surprise, with every example run on Python 3.13.

9 min read
Python subprocess explained: run, capture output, timeouts and safety

Key takeaways

  • Python subprocess work should start with subprocess.run([...], capture_output=True, text=True, check=True).
  • Without check=True, a program that exited with code 3 raised no error at all.
  • With shell=True, the input report.txt & echo INJECTED ran a second command; passing a list of arguments kept it as one harmless string.
  • timeout=1 stopped a 5-second job after 1.01 seconds with TimeoutExpired.

Sooner or later every Python script needs to run another program: call git to read a commit hash, convert a video with ffmpeg, run a database backup tool, or start another Python script in isolation. The standard library’s Python subprocess module does this safely and portably, and since Python 3.5 one function, subprocess.run, covers almost every case.

We ran every example on Python 3.13.6 on Windows, using sys.executable (the current Python interpreter) as the child program so the examples work on any operating system. Output is copied from the run.

subprocess.run basics

import subprocess, sys
r = subprocess.run([sys.executable, "-c", "print('hello')"])
r.returncode    # 0

Pass the command as a list: the program first, then each argument as a separate item. run starts the program, waits for it to finish and returns a CompletedProcess object with the args, returncode, and (if captured) stdout and stderr. A return code of 0 conventionally means success; anything else is an error code chosen by the program.

Capturing output

r = subprocess.run(
    [sys.executable, "-c", "print('hello'); import sys; print('warn', file=sys.stderr)"],
    capture_output=True, text=True)
type(r).__name__, r.returncode, r.stdout, r.stderr
# ('CompletedProcess', 0, 'hello\n', 'warn\n')

capture_output=True collects standard output and standard error separately instead of printing them to your terminal. text=True decodes the bytes into strings. Note the trailing newline: use r.stdout.strip() or r.stdout.splitlines() before comparing. For a one-liner that returns only stdout, subprocess.check_output([sys.executable, "--version"], text=True) gave 'Python 3.13.6\n'.

Errors and check=True

r = subprocess.run([sys.executable, "-c", "import sys; sys.exit(3)"])
r.returncode    # 3 and no exception

try:
    subprocess.run([sys.executable, "-c", "import sys; print('oops', file=sys.stderr); sys.exit(3)"],
                   capture_output=True, text=True, check=True)
except subprocess.CalledProcessError as e:
    e.returncode, e.stderr     # (3, 'oops\n')
    # str(e): Command [...] returned non-zero exit status 3.

This is the most common subprocess bug: a failed command is silently ignored because nobody checked the return code. check=True raises CalledProcessError on any non-zero exit, carrying the return code and captured output, so your normal exception handling applies. Log e.stderr: it usually contains the real reason for the failure.

Timeouts

try:
    subprocess.run([sys.executable, "-c", "import time; time.sleep(5)"], timeout=1)
except subprocess.TimeoutExpired as e:
    print(e)    # ... timed out after 1 seconds
# raised after 1.01 s

Without a timeout, a hung child (a network call that never returns, a prompt waiting for input) hangs your program forever. With timeout, run kills the child and raises TimeoutExpired; it fired after 1.01 seconds in our test. Grandchild processes started by the child may survive the kill, so for complex trees use process groups or a job manager.

When the program does not exist

subprocess.run(["no-such-program-xyz"])
# FileNotFoundError: The system cannot find the file specified

A missing executable raises FileNotFoundError before anything runs, which is different from the program running and failing. Catch both if a tool may not be installed, or check first with shutil.which("git"), which returns the full path or None.

Sending input

r = subprocess.run([sys.executable, "-c", "import sys; print(sys.stdin.read().upper())"],
                   input="piped text", capture_output=True, text=True)
r.stdout.strip()    # 'PIPED TEXT'

input= writes data to the child’s standard input and closes it, which is how you feed text to filters such as grep, jq or a formatter without temporary files.

Environment variables and the working folder

env = dict(os.environ, GREETING="hi from env")
subprocess.run([sys.executable, "-c", "import os; print(os.environ['GREETING'])"],
               capture_output=True, text=True, env=env).stdout.strip()
# 'hi from env'

subprocess.run([sys.executable, "-c", "import os; print(os.path.basename(os.getcwd()))"],
               capture_output=True, text=True, cwd=os.path.dirname(sys.executable)).stdout.strip()
# 'Python313'

env= replaces the child’s whole environment, so build it from os.environ and add or override keys; passing only your new key removes PATH and breaks most programs. This is the safe way to hand secrets to a tool without putting them on the command line, as described in our environment variables guide. cwd= runs the child in another folder; a pathlib Path works there too.

shell=True and command injection

user_input = "report.txt & echo INJECTED"

subprocess.run(f"echo {user_input}", shell=True, capture_output=True, text=True).stdout.splitlines()
# ['report.txt ', 'INJECTED']          a second command ran

subprocess.run([sys.executable, "-c", "import sys; print(sys.argv[1:])", user_input],
               capture_output=True, text=True).stdout.strip()
# "['report.txt & echo INJECTED']"   one harmless argument

With shell=True, the command string is handed to the system shell (cmd.exe on Windows, /bin/sh elsewhere), which interprets &, ;, |, $() and backticks. Our “file name” contained & echo INJECTED, and the shell ran it as a second command. A real attacker would use & del or ; curl ... | sh. With a list and no shell, each item is passed directly to the program as one argument, and special characters mean nothing. Only use shell=True with fixed strings you wrote yourself, and if you must include values on Unix, quote them with shlex.quote.

Tip: to split a command line you copied from documentation into a list, use shlex.split("git log -n 5"), which gives ['git', 'log', '-n', '5'].

Redirecting and merging streams

subprocess.run([...], stdout=subprocess.DEVNULL).stdout          # None (output discarded)

r = subprocess.run([sys.executable, "-c", "import sys; print('out'); print('err', file=sys.stderr)"],
                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
r.stdout.split()    # ['err', 'out']

DEVNULL throws output away, and stderr=subprocess.STDOUT merges errors into the output stream, which is handy for logs. Notice the order: “err” came before “out” even though the program printed “out” first. Standard output is buffered when it goes to a pipe while standard error is not, so merged streams can appear out of order. For a Python child, pass -u or set PYTHONUNBUFFERED=1 to keep them in order. You can also pass an open file, for example stdout=open("build.log", "w").

The encoding trap on Windows

subprocess.run([sys.executable, "-c", "print('café')"], capture_output=True).stdout
# b'caf\xe9\r\n'

Without text=True you get raw bytes, and here the bytes were not UTF-8: the child Python wrote “é” as \xe9 using the Windows code page because its output was a pipe, and ended the line with \r\n. Decoding that as UTF-8 would raise an error. When the child’s encoding matters, pass encoding="utf-8" (and, for a Python child, set PYTHONUTF8=1 or PYTHONIOENCODING=utf-8 in its environment) so both sides agree.

Streaming output with Popen

p = subprocess.Popen([sys.executable, "-u", "-c",
                      "import time\nfor i in range(3):\n print('line', i); time.sleep(0.2)"],
                     stdout=subprocess.PIPE, text=True)
for line in p.stdout:
    print("streamed:", line.strip())
p.wait()
# streamed: line 0 / streamed: line 1 / streamed: line 2, then returncode 0

run waits until the program finishes. For long jobs where you want progress as it happens, or to run several programs at once, use Popen directly and read lines as they arrive. Always wait() (or use it as a context manager) so the process is cleaned up and the return code is collected. If you capture both stdout and stderr with Popen, use communicate() instead of reading them yourself, or a full pipe buffer can deadlock both processes.

Start-up cost

for _ in range(20): subprocess.run([sys.executable, "-c", "pass"])
# 20 Python start-ups: 0.95 s

Each new process costs time; starting Python took about 48 ms each here. Calling a tool once is fine; calling it 100,000 times in a loop is not. Batch the work into one call (most tools accept many files), or use a Python library instead. For parallel Python work, the multiprocessing guide covers worker pools, and for waiting on many subprocesses at once, asyncio has create_subprocess_exec.

Subprocess cheat sheet

Task Code
Run and fail on error run(cmd, check=True)
Get output as text run(cmd, capture_output=True, text=True).stdout
Limit run time run(cmd, timeout=30)
Feed input run(cmd, input="data", text=True)
Extra environment run(cmd, env={**os.environ, "KEY": "v"})
Different folder run(cmd, cwd="path")
Discard output run(cmd, stdout=subprocess.DEVNULL)
Live output Popen(cmd, stdout=PIPE, text=True)
Find a program shutil.which("git")

Log the command and result of every subprocess call with the logging module; it makes failures in scheduled jobs far easier to diagnose.

Final thoughts

Python subprocess is simple once you follow a few rules. Pass commands as lists, add check=True so failures raise, set a timeout, capture output with text=True and an explicit encoding when it matters, and avoid shell=True with any value you did not write yourself. In our tests those rules caught a silent exit code, stopped a hung job in one second and blocked an injected command.

Frequently asked questions

What is the difference between subprocess.run and Popen?

run starts a program, waits for it and returns the result in one call. Popen starts it and returns immediately, so you can stream output, interact with it or run several programs at the same time.

How do I get the output of a command in Python?

Call subprocess.run with capture_output=True and text=True, then read the stdout attribute of the result. Add check=True so a failing command raises an error instead of returning partial output.

Why is shell=True dangerous?

The command string is interpreted by the system shell, so characters like ampersands and semicolons in user input can start extra commands. Pass a list of arguments without the shell instead.

How do I set a timeout for a subprocess?

Pass timeout with a number of seconds. If the program runs longer, Python kills it and raises subprocess.TimeoutExpired, which you can catch and handle.

Should I use os.system or subprocess?

Use subprocess. os.system always uses the shell, cannot capture output and only returns an exit status, while subprocess gives you output, errors, timeouts and safe argument handling.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Confusion Matrix Explained: Precision, Recall, F1 and Thresholds in Python

Confusion matrix explained with Python: TP, FP, FN, TN, precision, recall, F1, the accuracy trap, threshold choice by cost and…

Read article →
Software Development

API Pagination: Offset vs Cursor vs Keyset, Tested on 1 Million Rows

API pagination explained: offset vs keyset and cursor pagination timed on 1 million rows, duplicates when data changes, the ties…

Read article →
DevOps

Git Submodule Explained: Add, Clone, Update and Remove Safely

Git submodule explained: add, clone, update –init, update –remote, the detached HEAD trap, foreach and clean removal, tested step by…

Read article →
Test Automation

Pytest tmp path Explained: tmp_path, tmp_path_factory and Temp Files

Pytest tmp_path explained: per-test temporary folders, tmp_path_factory for shared files, monkeypatch.chdir, legacy tmpdir, retention and basetemp.

Read article →
Software Testing

Test Case Prioritization: 6 Techniques Compared With APFD

Test case prioritization compared: risk, coverage, cost, history and random orders scored with APFD and time to find all faults…

Read article →
Databases & SQL

Full Text Search in SQL: FTS5, Ranking, Phrases and Speed

Full text search in SQL with SQLite FTS5: MATCH queries, phrases, prefixes, NEAR, stemming, BM25 ranking, snippets, two gotchas and…

Read article →
Databases & SQL

Generated Columns in SQL: Virtual vs Stored, Indexes and Limits

Generated columns in SQL explained: virtual vs stored, automatic updates, write errors, storage cost, indexing, JSON fields and support in…

Read article →
Software Development

Object Freeze in JavaScript: Shallow vs Deep, Seal and const

Object freeze in JavaScript explained: freeze vs seal vs const, strict mode errors, shallow vs deep freeze, frozen arrays, Map…

Read article →
Software Development

Python hashlib: SHA-256, File Hashes, HMAC and Password Hashing

Python hashlib explained: SHA-256 for strings and files, algorithm speeds, why passwords need PBKDF2 or scrypt, safe verification and HMAC…

Read article →
Software Development

Python UUID Explained: uuid1, uuid4, uuid5, uuid7 and Database Keys

Python UUID explained: uuid4, uuid5, uuid1 privacy, time-ordered uuid7, parsing, storage formats and UUIDs as database keys, tested on Python…

Read article →
Artificial Intelligence

Random Forest in Python: Bagging, Feature Sampling and OOB From Scratch

Random forest in Python from scratch: bootstrap sampling, feature sampling, number of trees, out-of-bag evaluation and impurity vs permutation importance.

Read article →
Software Development

MIME Types Explained: Content-Type, nosniff and a Common Types List

MIME types explained and tested in a browser: Content-Type, nosniff, scripts and stylesheets with wrong types, a common MIME types…

Read article →
DevOps

Git Clean Explained: Remove Untracked Files Without Regret

Git clean explained with dry runs: -n, -f, -d, -x, -X, exclusions, paths, nested repositories and interactive mode, plus a…

Read article →
Test Automation

Pytest approx: Comparing Floats With rel, abs and Collections

Pytest approx explained: default tolerances, the near-zero trap, rel and abs, lists and dicts, NaN, NumPy, and how it differs…

Read article →
Software Testing

Accessibility Testing: WCAG Checklist, Automated Tools and Manual Checks

Accessibility testing explained: WCAG basics, an axe-core scan of a broken and a fixed page, what automation missed, contrast checks…

Read article →
Databases & SQL

SQL UNPIVOT: Turn Columns Into Rows, 4 Ways Tested

SQL UNPIVOT explained: turn columns into rows with UNION ALL, CROSS JOIN with CASE, JSON functions or native UNPIVOT, with…

Read article →
Software Development

Intl NumberFormat in JavaScript: Currency, Compact, Percent and Units

Intl.NumberFormat in JavaScript: locales, currency, compact notation, percent, units, rounding, formatToParts, the non-breaking space trap and formatter reuse.

Read article →
Databases & SQL

SQL EXPLAIN Plan: How to Read Query Plans and Fix Slow Queries

SQL EXPLAIN plan explained: read scans, index searches, covering indexes and temporary sorts, with before and after timings for five…

Read article →
Artificial Intelligence

Decision Tree in Python: Gini, Entropy and Overfitting From Scratch

Decision tree in Python from scratch: Gini impurity, entropy, information gain, best splits, readable rules, and how depth and leaf…

Read article →
DevOps

Git Blame Explained: Find Who Changed a Line, and Why

Git blame explained with a real repository: ignore whitespace, line ranges and functions, ignore-revs files for formatting commits, moved code…

Read article →
Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark tutorial: the benchmark fixture, reading results, parametrized and pedantic benchmarks, saving baselines and failing the build on regressions.

Read article →
Software Testing

Defect Density: Formula, Worked Examples and How to Use It

Defect density explained: the formula, a five-module worked example, per KLOC vs per function point, severity weighting, defect removal efficiency…

Read article →
Databases & SQL

Case Insensitive Search in SQL: LOWER, LIKE, COLLATE and Indexes

Case insensitive search in SQL: LOWER, LIKE, COLLATE NOCASE and expression indexes compared with query plans, plus accented letters and…

Read article →
Databases & SQL

ON DELETE CASCADE in SQL: CASCADE vs SET NULL vs RESTRICT

ON DELETE CASCADE in SQL tested: CASCADE vs SET NULL vs RESTRICT, multi-level cascades, the SQLite foreign key pragma, row…

Read article →
Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression in Python from scratch: sigmoid, log loss, gradient descent, odds ratios, thresholds, sigmoid overflow and separable data, tested…

Read article →
DevOps

Git Worktree Explained: Work on Two Branches at Once

Git worktree explained: add, list, remove, prune, lock and detached worktrees, a hotfix workflow, and the errors you will hit,…

Read article →
Test Automation

Pytest ini Explained: addopts, markers, testpaths and Precedence

Pytest ini explained: testpaths, addopts, strict markers, xfail_strict, filterwarnings, minversion, config file precedence and command-line overrides, tested.

Read article →
Software Testing

Test Estimation Techniques: WBS, Three Point PERT and Monte Carlo

Test estimation techniques worked through: WBS, three point PERT, total uncertainty, a Monte Carlo check, percentage rules and Wideband Delphi.

Read article →
Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression in Python from scratch: closed form, NumPy least squares and gradient descent, R squared, RMSE, outliers and curved…

Read article →
DevOps

Git Remote Explained: Add, Set-URL, Rename, Remove and Push URLs

Git remote explained: add, set-url, separate push URLs, rename, remove and prune, plus how to avoid leaking tokens in remote…

Read article →
Test Automation

Pytest Mock: mocker.patch, spy, side_effect and autospec Explained

Pytest mock explained: mocker.patch, return_value, side_effect, spy, autospec and PropertyMock, with tested examples for APIs, clocks and retries.

Read article →
Software Testing

System Testing Explained: Types, Process and a Real End-to-End Example

System testing explained: types, entry and exit criteria, the process, and a real end-to-end test suite run against a small…

Read article →
Artificial Intelligence

Naive Bayes Classifier Explained: Build a Spam Filter in Python

Naive Bayes classifier explained by building a spam filter in Python: Bayes rule, the naive assumption, Laplace smoothing, log probabilities…

Read article →
DevOps

Git Branch Explained: Create, Switch, Rename, Delete and Recover

Git branch explained with tested commands: list, create, switch, rename, delete, merged checks, recovering deleted branches, remote tracking and naming.

Read article →
Test Automation

Pytest raises: Testing Exceptions, match and excinfo

Pytest raises explained with tested examples: match regexes, excinfo, custom and chained exceptions, parametrized error cases, pytest.warns and common mistakes.

Read article →
Software Testing

Non Functional Testing: Types, Metrics and a Real Load Test

Non functional testing explained: 10 key types, measurable requirements, latency percentiles and what a small real load test revealed about…

Read article →