Run external commands from Python without shell surprises

A maintenance job works when an operator pastes a command into a terminal, but fails when Python launches it. Sometimes the executable is not found. Sometimes a path containing spaces becomes two arguments. In the worst case, a value from a ticket or API is interpreted as shell syntax. These symptoms share one boundary: Python is creating a new process, and the program, arguments, environment, working directory, streams, exit status, and lifetime all need explicit contracts.

Start by recording what actually crossed the boundary

Do not debug a reconstructed command copied from a log. Preserve the argument vector as structured data, redact secrets, and capture the execution context:

from pathlib import Path
import os
import shutil

argv = ["rsync", "--archive", "--", "/srv/source files/", "/srv/backup/"]
print({
    "argv": argv,
    "cwd": str(Path.cwd()),
    "rsync": shutil.which(argv[0]),
    "path": os.environ.get("PATH", "").split(os.pathsep),
})

The -- ends option parsing for programs that support it, so a user-controlled path beginning with a dash is not mistaken for a flag. It does not replace validation, and it is program-specific.

Use the process API, not a command-shaped string

Current Python documentation recommends subprocess.run() for cases it can handle. Pass a sequence so Python starts the executable directly without asking a shell to parse spaces, quotes, dollar signs, semicolons, or redirections:

import subprocess

result = subprocess.run(
    ["rsync", "--archive", "--", "/srv/source files/", "/srv/backup/"],
    stdin=subprocess.DEVNULL,
    capture_output=True,
    text=True,
    encoding="utf-8",
    errors="replace",
    timeout=300,
    check=False,
)

if result.returncode != 0:
    raise RuntimeError(
        f"rsync failed rc={result.returncode}\n"
        f"stderr={result.stderr[-4000:]}"
    )

The accepted Stack Overflow answer by David Cournapeau now points to this interface. That updates older advice built around os.system() or subprocess.call(): run() returns a CompletedProcess, supports timeouts and capture directly, and keeps failure policy visible.

Decide what a failure means

With check=True, a nonzero exit raises CalledProcessError. That is useful when every nonzero status is exceptional:

try:
    completed = subprocess.run(
        ["git", "-C", "/srv/app", "status", "--porcelain=v1"],
        capture_output=True,
        text=True,
        timeout=15,
        check=True,
    )
except FileNotFoundError:
    # The executable could not be started.
    raise RuntimeError("git is not installed or PATH is wrong")
except subprocess.TimeoutExpired as exc:
    raise RuntimeError(f"git exceeded {exc.timeout} seconds")
except subprocess.CalledProcessError as exc:
    raise RuntimeError(
        f"git exited {exc.returncode}: {exc.stderr[-2000:]}"
    )

Do not collapse these into one “command failed” message. “Could not start,” “timed out,” “terminated by signal,” and “ran and returned an application error” lead to different repairs. On POSIX, a negative return code represents the signal that terminated the child.

Why shell=True changes the threat model

Without a shell, metacharacters are ordinary argument characters. With shell=True, the shell parses the string and the application becomes responsible for quoting. This is exploitable:

# Unsafe: project may contain shell syntax.
subprocess.run(f"tar -czf backup.tgz {project}", shell=True)

The safe form is an argument vector:

subprocess.run(
    ["tar", "-czf", "backup.tgz", "--", project],
    check=True,
    timeout=120,
)

Use a shell only when shell language is genuinely the feature—such as a reviewed pipeline or redirection—and keep the script constant. On POSIX, shlex.quote() can quote one shell token, but the Python documentation warns it is not guaranteed for non-POSIX shells. Quoting a whole command after concatenation is not a reliable repair.

Build pipelines without a shell

producer = subprocess.Popen(
    ["journalctl", "--unit", "worker.service", "--since", "-1 hour"],
    stdout=subprocess.PIPE,
    text=True,
)
consumer = subprocess.run(
    ["grep", "-F", "ERROR"],
    stdin=producer.stdout,
    capture_output=True,
    text=True,
    timeout=30,
)
assert producer.stdout is not None
producer.stdout.close()
producer_rc = producer.wait(timeout=30)

Inspect both return codes. If the consumer exits early, close the parent’s copy of the pipe so the producer can receive the appropriate signal. For complex streaming, cancellation, or many concurrent processes, use Popen.communicate() or asyncio.create_subprocess_exec(); calling wait() while unread pipes fill can deadlock.

Control environment, directory, and privileges

A service has a different PATH, home directory, locale, umask, and current directory than an interactive shell. Make dependencies explicit:

env = {
    "PATH": "/usr/local/bin:/usr/bin:/bin",
    "LANG": "C.UTF-8",
}
subprocess.run(
    ["/usr/bin/make", "deploy"],
    cwd="/srv/app",
    env=env,
    stdin=subprocess.DEVNULL,
    check=True,
    timeout=600,
)

An explicit environment also prevents accidental credential inheritance, but removing variables can break certificate discovery, proxies, or runtime libraries. Start from an allowlist only after testing the program’s documented needs. Do not solve permission errors by running the entire Python service as root.

Verification that catches production-only defects

1. Test an argument containing spaces, quotes, Unicode, a leading dash, and a literal semicolon.
2. Test missing executables, nonzero exits, signals, and timeouts separately.
3. Run under the same service account, working directory, environment, and filesystem mounts as production.
4. Exercise large stdout and stderr so pipe handling cannot deadlock.
5. Confirm logs retain the executable, redacted arguments, duration, and return code without leaking tokens.
6. Confirm a retry does not duplicate durable work.

Timeout is not transaction rollback

subprocess.run(timeout=...) kills and waits for the direct child before re-raising TimeoutExpired, but the child may already have changed files, sent requests, or created grandchildren. A process-group strategy may be required on POSIX, and even terminating every process cannot undo an external side effect.

Design the called operation with a staging directory, atomic rename, database transaction, idempotency key, or resumable checkpoint. On failure, keep the staging area long enough to diagnose it, then remove it through a reviewed recovery path. Never blindly rerun a timed-out billing, provisioning, or notification command.

Rollout and rollback

Introduce the wrapper behind a feature flag and compare old and new exit status, duration, and output classification in a non-mutating mode. If a deployment exposes quoting or environment regressions, roll back the caller while preserving its captured evidence. If the child made durable changes, reconcile those changes before retrying; code rollback alone is not recovery.

Direct execution is safer and more observable, but it gives up shell conveniences. Capturing output improves diagnostics but consumes memory, so stream unbounded output to a rotated file or log sink. Tight timeouts protect worker capacity but can kill legitimate slow work. Set them from measured latency and an operational deadline, not intuition.

A command that works in a terminal is only a clue. Production correctness comes from an explicit argument vector, execution context, failure policy, and recovery plan.

Sources: the Stack Overflow question by freshWoWer and accepted answer by David Cournapeau (CC BY-SA), and the current Python documentation for subprocess management and security considerations and POSIX shell quoting.