Merge Python dictionaries without silently losing configuration

Two valid configuration dictionaries go into a deployment tool; one setting quietly disappears. Nothing crashed, the syntax was correct, and that is exactly why this failure reaches production. A dictionary merge is not merely concatenation: duplicate keys force a precedence decision, and Python will make that decision even when the application never documented it.

Reproduce the collision before choosing syntax

base = {"host": "api.internal", "timeout": 30, "tls": True}
override = {"timeout": 5, "region": "eu-west"}

print(base.keys() & override.keys())
# {'timeout'}

Record the source of each mapping, its key types, and whether nested values are dictionaries. A shallow merge only operates at the top level. If both sides contain "database", the later value replaces the entire earlier object; its missing children are not preserved.

The modern non-mutating merge

For Python 3.9 and later, the accepted Stack Overflow answer by Roushan uses the dictionary union operator:

effective = base | override

The current Python language documentation confirms that this creates a new dictionary and that values from the right operand win on duplicate keys. The inputs remain unchanged. That is a good default for layered configuration because the effective result is separate and testable.

For supported older Python versions, unpacking has the same right-wins behavior:

effective = {**base, **override}

base.update(override) and base |= override mutate base. Mutation can be intentional for an accumulator, but it is a production hazard when other code retains a reference to the supposed default mapping.

Turn precedence into a policy

Never let operand order be the only documentation. Reject collisions when keys are meant to be unique:

def merge_unique(left, right):
    overlap = left.keys() & right.keys()
    if overlap:
        raise ValueError(f"duplicate configuration keys: {sorted(overlap)!r}")
    return left | right

When overrides are expected, validate them before merging:

ALLOWED_OVERRIDES = {"timeout", "log_level"}
unexpected = override.keys() - ALLOWED_OVERRIDES
if unexpected:
    raise ValueError(f"override not permitted: {sorted(unexpected)!r}")

effective = base | override

This catches a misspelling such as timeuot, which otherwise becomes a second unused key while the original timeout stays active.

Shallow is not recursive

base = {"db": {"host": "db", "port": 5432, "tls": True}}
override = {"db": {"host": "db-canary"}}
effective = base | override
# effective["db"] is only {"host": "db-canary"}

Do not reach for an arbitrary “deep merge” helper without defining list handling, deletion, type conflicts, and maximum depth. For safety-critical configuration, parse layers into a typed schema and reject unknown fields. A small domain-specific merge is easier to audit than universal recursion.

Verification should prove more than the happy path

1. Assert both inputs are unchanged after a non-mutating merge.
2. Test duplicate keys and confirm the documented winner.
3. Test nested dictionaries and make the shallow replacement visible.
4. Reject unknown keys and incompatible value types.
5. Serialize the redacted effective configuration and compare it in staging.
6. Run on the oldest supported Python runtime; | is syntax only from 3.9.

Failure modes, recovery, and rollout

A common mistake is assigning the result of update(): it returns None. Another is merging untrusted input into options that control file paths, commands, or authorization. Allowlist those fields; merging does not validate them. Also avoid logging secrets while diagnosing precedence.

Roll out a new merge policy in comparison mode: compute the old and new effective mappings, redact secret values, and report only changed keys. Stop if protected settings differ. Keep the last known-good configuration artifact so rollback restores the complete mapping rather than merely reverting code. If a bad configuration already changed durable infrastructure, reconcile that state before retrying.

Copying a mapping costs memory proportional to its size; mutating can be cheaper but spreads state changes through shared references. In most deployment configuration, predictability is worth the copy. For very large mappings, measure and isolate ownership rather than selecting mutation by intuition.

The merge operator answers “which value wins?” with operand order. Production code must also answer “was that key allowed to compete?”

Sources: the Stack Overflow question by Carl Meyer and accepted answer by Roushan (CC BY-SA), the Python documentation for dictionary merge and update behavior, and PEP 584.