Create Python output directories without a check-then-create race

An export job needs /data/exports/customer-42/2026-07-27. A common implementation checks whether the path exists and then creates it. Under concurrent workers, another process can change the filesystem between those operations. The check does not reserve anything; it only reports a moment that has already passed.

Use one operation that expresses the intended result

from pathlib import Path

output_dir = Path("/data/exports/customer-42/2026-07-27")
output_dir.mkdir(parents=True, exist_ok=True)

Python documents that parents=True creates missing ancestors and exist_ok=True ignores FileExistsError only when the final path is already a directory. If the path exists as a regular file, creation still fails—which is the correct signal.

Do not reintroduce the race with exists()

# Fragile under concurrency
if not output_dir.exists():
    output_dir.mkdir(parents=True)

Worker A and worker B can both observe absence; one creates the directory and the other receives FileExistsError. More importantly, an attacker or misbehaving process can replace a checked component before use. Ask the filesystem to perform the desired operation and handle its result.

Report failures with their real cause

def ensure_directory(path: Path) -> Path:
    try:
        path.mkdir(parents=True, exist_ok=True)
    except FileExistsError as exc:
        raise NotADirectoryError(
            f"output path exists but is not a directory: {path}"
        ) from exc
    except PermissionError as exc:
        raise PermissionError(
            f"cannot create output directory: {path}"
        ) from exc
    except OSError as exc:
        raise RuntimeError(
            f"filesystem error creating {path}: {exc}"
        ) from exc

    if not path.is_dir():
        raise NotADirectoryError(path)
    return path

Do not catch OSError and continue. Read-only filesystems, full disks, invalid components, symlink loops and permission failures are not equivalent to “another worker created it.”

Understand permission and umask behavior

output_dir.mkdir(mode=0o750, parents=True, exist_ok=True)

The process umask modifies requested permissions. With parents=True, intermediate parents are created using default permissions rather than the final mode. Provision shared roots such as /data/exports ahead of time with the intended owner, group, setgid bit and access policy; do not expect application code to repair an entire directory tree safely.

install -d -o exporter -g dataops -m 2770 /data/exports

Keep untrusted identifiers inside the approved root

from pathlib import Path

base = Path("/data/exports").resolve(strict=True)
candidate = (base / customer_slug / run_id).resolve(strict=False)

if not candidate.is_relative_to(base):
    raise ValueError("output path escapes export root")

candidate.mkdir(parents=True, exist_ok=True)

Do not accept absolute customer paths or .. traversal. A lexical allow-list for identifiers is still valuable. If hostile local users can modify the tree, path resolution followed by creation is not a complete defense against symlink races; use an OS-level sandbox, private directory ownership, or descriptor-relative APIs appropriate to the threat model.

Create per-run directories without collisions

from tempfile import mkdtemp
from pathlib import Path

run_dir = Path(mkdtemp(prefix="run-", dir=base))
try:
    produce_export(run_dir)
finally:
    cleanup_if_required(run_dir)

For a unique workspace, mkdtemp atomically creates a new directory rather than relying on timestamps or random names generated separately.

Directory creation does not make output files atomic

import os
import tempfile
from pathlib import Path

def atomic_write(target: Path, data: bytes) -> None:
    target.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary_name = tempfile.mkstemp(
        prefix=f".{target.name}.",
        suffix=".tmp",
        dir=target.parent,
    )
    temporary = Path(temporary_name)
    try:
        with os.fdopen(fd, "wb") as handle:
            handle.write(data)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, target)
    except BaseException:
        temporary.unlink(missing_ok=True)
        raise

Writing directly to the final filename can expose partial content to another process. A temporary file in the same directory followed by os.replace gives atomic name replacement on the same filesystem. If durability across power loss matters, the directory itself may also require syncing according to platform semantics.

Coordinate when several workers target one logical file

exist_ok=True makes directory establishment cooperative; it does not serialize writes. If several workers produce result.csv, define an ownership rule, unique output names, a job-level lock or a compare-and-swap mechanism. “Last replacement wins” may be atomic yet logically wrong.

Test the failures, not only the happy path

def test_parallel_creation(tmp_path):
    target = tmp_path / "a" / "b" / "c"
    with ThreadPoolExecutor(max_workers=16) as pool:
        list(pool.map(lambda _: ensure_directory(target), range(100)))
    assert target.is_dir()

def test_file_blocks_directory(tmp_path):
    target = tmp_path / "not-a-directory"
    target.write_text("occupied")
    with pytest.raises(NotADirectoryError):
        ensure_directory(target)

Add permission-denied tests where the platform permits, path traversal tests, concurrent final-file tests and cleanup behavior. Avoid tests that pass only as root because root bypasses the failure the test intends to exercise.

Requirement	Approach
Shared nested directory	mkdir(parents=True, exist_ok=True)
Unique private workspace	tempfile.mkdtemp
No partial final file	Same-directory temp file plus os.replace
Controlled shared permissions	Provision root owner/group/mode outside job
Hostile path input	Allow-list, constrain root and strengthen OS boundary

“Exists” is an observation. “Create this directory if needed” is an operation. Concurrent automation should be built from operations whose outcomes the filesystem can enforce.

Sources: the solved Stack Overflow question (question by Parand; accepted answer by Blair Conrad, CC BY-SA), Python’s official Path.mkdir, tempfile, and os.replace documentation.