Copy files safely with Python’s shutil

shutil.copyfile(), shutil.copy(), and shutil.copy2() all copy file data, but they promise different destination and metadata behavior. In a production automation job, the real questions are: may an existing file be replaced, what metadata must survive, can readers observe a partial copy, and how will the job prove and recover from the change?

Choose the contract before the function

Function	Destination	Metadata intent
copyfile(src, dst)	dst must be the target filename	File content only
copy(src, dst)	Filename or existing directory	Content plus permission mode
copy2(src, dst)	Filename or existing directory	Content plus as much metadata as copystat() can preserve

All three can replace an existing regular destination file. None is a byte-for-byte backup of every filesystem property. Python’s documentation warns that owner, group, ACLs, resource forks, alternate data streams, and other platform-specific metadata may not be preserved. copy2() attempts more metadata and does not fail merely because every field cannot be retained.

Symlinks need an explicit decision. With the default follow_symlinks=True, the referent is copied. With False, supported functions can create or operate on a link instead. Special files such as devices and pipes are outside the ordinary file-copy contract.

Preflight both paths

Resolve paths from trusted configuration, not raw user input. Confirm the source is the kind of file the job expects and that source and destination are not the same file:

from pathlib import Path
import os

src = Path("/srv/export/report.csv")
dst = Path("/srv/publish/report.csv")

if not src.is_file() or src.is_symlink():
    raise RuntimeError("source must be a regular non-symlink file")

dst.parent.mkdir(parents=True, exist_ok=True)
if dst.exists() and os.path.samefile(src, dst):
    raise RuntimeError("source and destination are the same file")

Path.resolve() is useful for diagnostics but does not by itself defeat symlink races. For a hostile shared directory, path checks followed by path-based opens are not a security boundary. Use directory file descriptors, restrictive ownership and permissions, and operating-system-specific safe-open primitives where an attacker can rename components.

Measure free space and quota with headroom for the temporary copy, filesystem allocation, and concurrent writers. Confirm the overwrite policy. “Destination exists” should be a deliberate branch, not an accidental behavior inherited from shutil.

A recoverable replacement pattern

Copy to a unique temporary file in the destination directory, flush it, verify it, then replace the visible name:

from pathlib import Path
import os
import shutil
import tempfile

def publish_copy(src: Path, dst: Path) -> None:
    dst.parent.mkdir(parents=True, exist_ok=True)
    temp_name = None
    try:
        with tempfile.NamedTemporaryFile(
            dir=dst.parent,
            prefix=f".{dst.name}.",
            suffix=".tmp",
            delete=False,
        ) as temp:
            temp_name = Path(temp.name)

        shutil.copy2(src, temp_name)
        with temp_name.open("rb") as copied:
            os.fsync(copied.fileno())

        os.replace(temp_name, dst)
        temp_name = None

        directory_fd = os.open(dst.parent, os.O_RDONLY)
        try:
            os.fsync(directory_fd)
        finally:
            os.close(directory_fd)
    finally:
        if temp_name is not None:
            temp_name.unlink(missing_ok=True)

Placing the temporary file in the same directory is important: os.replace() is atomic when the platform and filesystem support atomic rename on that filesystem. Readers see either the old complete name or the new complete name, not a half-written destination. Atomic visibility is not the same as durable storage, which is why the example flushes the file and directory. Filesystems, network mounts, and operating systems differ; test the actual deployment.

The temporary file’s initial restrictive permissions may be changed by copy2(). If the destination needs a fixed service mode rather than the source mode, enforce and verify that policy after copying and before replacement. Never copy set-user-ID or other privileged mode bits casually.

Content verification without false confidence

For a stable source, compare sizes and a cryptographic digest when the cost is justified:

import hashlib

def sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as stream:
        for block in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(block)
    return digest.hexdigest()

Hashing the source after copying has a race: another process may modify it during or between reads. The strongest pattern is to make producers publish immutable, versioned source files, verify the version, then copy. If that cannot be done, open the source once, compare fstat() before and after, and coordinate with the producer; even then, the application’s concurrency contract matters.

A matching hash proves copied content, not correct ownership, access control, extended attributes, semantic validity, or provenance. Parse a CSV, archive, certificate, or configuration with a safe validator before promotion. Do not execute the copied file as a validation step.

Common failure modes

• Partial destination: copying directly over the live name lets readers see truncation or incomplete data. Use same-directory temporary promotion.
• Wrong basename: passing a directory to copy() or copy2() derives the basename; copyfile() does not. Construct and log the intended final path.
• Metadata surprise: timestamps can affect incremental jobs and caches; preserved permissions can be too broad. Define metadata field by field.
• Disk exhaustion: a failed copy can leave a temporary file. Clean only files created by this operation, never a broad glob.
• Cross-filesystem move: rename atomicity does not extend across filesystems. Ensure the temporary file is on the destination filesystem.
• Network filesystem semantics: atomicity, close-to-open consistency, locks, and durability vary. Validate with the mount and failure model in production.
• Same-file aliases: hard links and different path spellings can identify the same inode. Catch SameFileError and use samefile() when both paths exist.

Verification, observability, and retries

After promotion, open the final path and verify size, digest or format, required mode, and application readability under the service account. Emit structured metadata such as operation ID, approved source/destination identifiers, byte count, duration, and digest when policy permits. Do not log sensitive file contents or user-supplied raw paths.

Make retries idempotent. A content-addressed or versioned destination is easiest: if the expected digest is already present, report success. For a mutable fixed name, compare the expected current generation before replacement so a retry cannot overwrite a newer publisher’s result. Serialize writers with an application-level lock or generation check; atomic replace alone does not choose the right winner.

Rollback and disaster recovery

Before overwriting valuable data, preserve the previous generation under a unique, access-controlled name or rely on a tested snapshot/versioning system. Keep retention bounded. If post-promotion checks fail, restore only when the visible destination still matches the failed generation; otherwise another writer may have legitimately advanced it.

A rollback should use the same temporary-copy, verification, and atomic-promotion path. For critical data, test recovery from backup on a separate location and record recovery time. Replication is not automatically a backup: it can faithfully replicate an erroneous overwrite.

Attribution and current verification

This guide was prompted by Matt’s Stack Overflow question and Swati’s accepted answer, used under CC BY-SA. The accepted distinction among copyfile(), copy(), and copy2() remains valid. Current behavior, metadata limitations, symlink handling, and platform fast-copy behavior were independently verified against Python’s official shutil documentation, with atomic replacement checked against the official os.replace() reference and temporary-file behavior against tempfile.