Find repeated literal dictionary keys

Before you begin

Python 3.12 or newer

What this script changesReads Python source and writes a new JSON result without changing or running analyzed code. Maximum 500 files and 2 MiB per file. Static findings are review aids; dynamic behavior and runtime imports are outside scope.

Set up your example

Save the complete code as python-duplicate-dict-keys.py in a working folder.
Save the provided sample files in that same folder; run commands from that folder.

Sample files in this download:

sample.py

Download the ZIP, extract it, and open a terminal inside the extracted folder. The source file is python-duplicate-dict-keys.py.

Run the script

python "python-duplicate-dict-keys.py" --source "sample.py" --output "analysis.json"

If your system uses python3 instead of python, use that command for Python examples. Keep paths containing spaces in quotes. Required inputs and optional settings are described in the code and setup steps.

What to expect

Flag repeated evaluable literal keys within dictionary displays. The clean fixture reports no duplicate dictionary keys.

The complete script

#!/usr/bin/env python3
'Find repeated literal dictionary keys'
import argparse
import json
from pathlib import Path

def write_json(path, value):
    data = json.dumps(value, indent=2, ensure_ascii=False, allow_nan=False) + "\n"
    with Path(path).open("x", encoding="utf-8", newline="") as handle:
        handle.write(data)

import ast
import tokenize
import io
import doctest
from collections import Counter, defaultdict

FUNCTIONS=(ast.FunctionDef,ast.AsyncFunctionDef)
SCOPES=(ast.FunctionDef,ast.AsyncFunctionDef,ast.ClassDef,ast.Lambda)
def functions(tree):
    return [n for n in ast.walk(tree) if isinstance(n,FUNCTIONS)]
def local_nodes(node):
    yield node
    for child in ast.iter_child_nodes(node):
        if not isinstance(child,SCOPES):
            yield from local_nodes(child)
def name(node):
    try: return ast.unparse(node)
    except (ValueError,TypeError): return type(node).__name__
def targets(node):
    return [n.id for n in ast.walk(node) if isinstance(n,ast.Name) and isinstance(n.ctx,ast.Store)]
def sources(value):
    root=Path(value)
    if root.is_file(): return [root]
    if not root.is_dir(): raise ValueError("--source must be a Python file or directory.")
    excluded={".git",".venv","venv","node_modules","__pycache__"}
    files=sorted(p for p in root.rglob("*.py") if p.is_file() and not any(x in excluded or x.startswith(".") for x in p.relative_to(root).parts))
    if len(files)>500: raise ValueError("At most 500 Python files per run.")
    return files


def analyze(tree,text):
    issues=[]
    for n in ast.walk(tree):
        if not isinstance(n,ast.Dict): continue
        seen={}
        for key in n.keys:
            if key is None: continue
            try:
                value=ast.literal_eval(key)
                hash(value)
            except (ValueError,TypeError): continue
            if value in seen: issues.append({"line":key.lineno,"key":name(key),"first_line":seen[value]})
            else: seen[value]=key.lineno
    return issues

def main():
    parser = argparse.ArgumentParser(description='Find repeated literal dictionary keys')
    parser.add_argument("--source",required=True,help="Python file or source directory")
    parser.add_argument("--output",required=True)
    args = parser.parse_args()
    reports=[]
    for path in sources(args.source):
        try:
            if path.stat().st_size>2*1024*1024: raise ValueError("Source exceeds 2 MiB.")
            with tokenize.open(path) as handle: text=handle.read()
            tree=ast.parse(text,filename=str(path))
            reports.append({"path":str(path),"result":analyze(tree,text)})
        except (OSError,SyntaxError,UnicodeError,ValueError) as exc:
            reports.append({"path":str(path),"error":str(exc)})
    write_json(args.output,reports)

if __name__ == "__main__":
    try:
        main()
    except (OSError, ValueError, KeyError, TypeError, ArithmeticError, IndexError, LookupError) as exc:
        raise SystemExit(str(exc))

Open source as plain text · Download source and sample files

How it works

The program parses source syntax without importing or executing it; per-file parse failures become error records.
Literal keys use Python equality and hashing, so 1 and True collide; dictionary unpacking and computed expressions are not resolved.
The clean fixture reports no duplicate dictionary keys.

Make it your own

--source

Use --source with a Python file or directory; directory mode skips common dependency and hidden folders.

Adapt the processing

Edit analyze to include the containing dictionary's line in each issue.

Change one setting at a time, run it again on sample data, and compare the output. The customization guide walks through this process.

If something goes wrong

A computed duplicate is missed.

Only safe literal evaluation is used; no user expressions are executed.

The destination already exists.

Choose a new --output path. The script deliberately refuses to replace an existing result.

Reference & next steps

These official references explain the underlying tools. The Scriptage example and guide are original instructional content. See verification notes for what has been checked and the limits of those checks.

Copies a template for your own notes. It does not send anything.