Detecting and Removing Secrets from Git History with Betterleaks and git-filter-repo

Have you ever wondered exactly what has entered your Git repository over time? Accidental commits of debug functions, incorrect comments, or copied files can usually be fixed with a follow-up commit. However, when sensitive data such as third-party service keys, integration credentials, database dumps, or entire /private/uploads directories are involved, simply committing a deletion does not remove the information from history.

Secrets committed to any repository, even a private one, remain accessible through git log or by checking out older commits. Anyone who can read the repository, clone it, fork it, or access connected CI/CD pipelines gains exposure. Contracts with clients frequently prohibit storing credentials or sensitive data in repositories for this reason, emphasizing the need for traceable password issuance and revocation processes.

The first step is to scan the repository thoroughly. The recommended tool is Betterleaks, which replaced earlier utilities such as gitleaks. It applies rule-based detection for patterns resembling AWS keys or GitHub tokens, uses BPE tokenization to reduce false positives, and automatically decodes Base64, hex, and URL-encoded content. Installation is available via Homebrew, Docker, or source build.

Running betterleaks git . --log-opts="--all" -v examines every commit, patch, and change across all branches. Results include details such as the rule ID, matched secret, author, commit message, and file path. Reports can be exported in JSON format for further processing. Manual review of findings is essential to distinguish real secrets from false positives.

After confirmation, git-filter-repo is used to rewrite history. Secrets extracted from the report are converted into replacement rules, and files matching dangerous extensions such as .sql, .env, or .bak are listed for removal. Before execution, secrets must be moved to external configuration files such as .env.local or settings.local.php so the application continues to function after cleanup.

The procedure is destructive and irreversible. A mirror backup must be created, all pull requests closed, and a clean clone prepared. The command git filter-repo --invert-paths --paths-from-file paths-to-remove.txt --replace-text secrets-to-replace.txt performs both file deletion and secret replacement in one pass. After verification with a new scan, aggressive garbage collection and a forced mirror push complete the process.

Every contributor and CI/CD pipeline must then perform a fresh clone rather than a pull or rebase, because commit hashes have changed. Even after these steps, old objects may survive in forks, backups, or caches, making secret rotation mandatory.