Best data anonymization tools for dataset licensing (2026)
The best open-source stack for preparing a dataset to license covers six jobs, and no single tool does all of them. For personal information in text, use Presidio with spaCy (or scrubadub for something smaller); for tables, ARX; for faces in video, deface; for secrets in code, gitleaks or TruffleHog

, then git-filter-repo to remove them from history; for third-party code, ScanCode; for near-duplicates, datasketch or text-dedup; and for documentation, a Hugging Face dataset card or a datasheet. Every one of them is free, and every detection tool misses things, so a human still reviews a sample. It means removing what the buyer must not receive and describing what they will. In practice that is six jobs: strip personal information, blur faces, remove credentials, separate code you don't own, collapse duplicates, and write down what the dataset is and how it was made.
Licensing raises the bar compared with internal use. Records leave your control, and the buyer's lawyers will ask how you de-identified them. Our guide to de-identifying company data before licensing covers the method. This post covers the tools.
We checked each project's license and status on its official repository on 7 November 2026. Licenses and maintenance change, so re-check before you adopt one. Tool Job License What it does Presidio PII in text, images, tables MIT Detects names, card numbers, phone numbers and more with NER, regex, rules and checksums, then masks or replaces them spaCy Named entity recognition MIT The NLP engine Presidio loads by default (en_core_web_lg) scrubadub PII in free text Apache-2.0 Small Python library: names, emails, phones, addresses (US, GB, CA), card numbers, US SSNs ARX Tables Apache-2.0 k-anonymity, l-diversity, t-closeness, differential privacy and re-identification risk analysis deface Faces in video and photos MIT Detects faces frame by frame and applies blur, solid boxes or mosaic gitleaks Secrets in git history MIT Scans every patch through git log -p, plus directories and stdin TruffleHog Secrets in git history AGPL-3.0 700+ detectors, with optional live verification of each credential detect-secrets Stopping new secrets Apache-2.0 Baseline file plus pre-commit hook, so only new secrets fail a commit git-filter-repo Removing data from history MIT Rewrites history to delete files or replace strings in every commit ScanCode Toolkit Third-party code Apache-2.0 Detects licenses, copyrights and package manifests in source and binaries datasketch Near-duplicate text MIT MinHash and LSH building blocks for your own pipeline text-dedup Near-duplicate text Apache-2.0 Ready scripts for MinHash, SimHash, suffix array and Bloom filter dedup dedupe Duplicate records MIT Fuzzy matching and entity resolution for structured rows Hugging Face dataset cards Documentation Format A README with YAML metadata: license, languages, structure, collection Datasheets for Datasets Documentation Paper A question set on motivation, composition, collection and intended uses Croissant Machine-readable metadata Apache-2.0 MLCommons format built on schema.org that describes files, structure and semantics Start with Presidio unless your data is tiny. It handles free text, images (including DICOM medical images) and structured data, and you can add your own recognizers for internal IDs such as customer or ticket numbers, which no generic model knows about.
One change matters as of November 2026: Presidio is no longer a Microsoft project. It moved to a community-governed GitHub organization, Data Privacy Stack, stays MIT-licensed, and new Docker images now ship from GitHub Container Registry instead of Microsoft's registry. Old links redirect, but update mcr.microsoft.com/presidio-* image references. A minimal run looks like this: from presidio_analyzer import AnalyzerEngine from presidio_anonymizer import AnonymizerEngine text = "Ticket 4411: call Dana Ruiz on 212-555-0143 about the refund." findings = AnalyzerEngine().analyze(text=text, language="en") print(AnonymizerEngine().anonymize(text=text, analyzer_results=findings).text) The default operator replaces each finding with its entity type, so a name becomes a PERSON placeholder.
The README itself warns that automated detection cannot guarantee it finds everything. Treat its output as a first pass and review a random sample by hand. scrubadub is simpler and fine for a quick script, but its last release (v2.0.1) dates from September 2023. For tables of customers or transactions, ARX is the better fit, because the risk there is rare combinations of ordinary fields (postcode, age, job title), not names in sentences.
Scan the full history, rotate what you find, then rewrite. A key deleted in a later commit is still in the old one, and a code buyer receives the whole history. gitleaks is the fast default. gitleaks git -v --log-opts="--all" .
scans every branch. TruffleHog adds verification: it tests each candidate against the provider's API and reports whether it is live. That is useful for triage, but it means the tool contacts those services with your credentials. --no-verification turns it off.
The AGPL-3.0 license matters if you plan to embed it in a product rather than run it. git-filter-repo does the rewrite. Since version 2.47 it has a --sensitive-data-removal flag that also reports which commits changed and what to clean up elsewhere. Use deface for faces, a MinHash tool for duplicates and ScanCode for code you don't own.
Each solves one narrow problem well. deface blurs faces only. Number plates, screens and paperwork in frame need another pass. Its output leaves out the audio track unless you pass --keep-audio, which is the right default for workplace footage.
For duplicates, DeepMind's Genie paper is a useful reminder of why curation matters: a curated set of about 30,000 hours, roughly a tenth of the raw video, trained a better model than the full set (Genie, arXiv). Near-duplicate support tickets and email threads inflate your volume count without adding new examples, so dedupe before you describe the dataset to anyone. For codebases, ScanCode lists vendored libraries and their licenses so you can exclude or document them. Our guide to selling source code for AI training explains why ownership gaps end code deals.
Write a dataset card even if the buyer never asks. Cover what the records are, the years they span, how they were collected, what you removed and with which tools, and what the buyer may not do with them. Datasheets for Datasets gives the questions; a Hugging Face card or Croissant file gives the format. The card also forces an honest inventory.
If you can't say where a field came from, a buyer can't license it. Records you never kept can't be documented at all, which is why a retention policy that keeps data sellable comes before any of these tools. Do these tools make data anonymous? Is gitleaks or TruffleHog better?
Can I use an LLM to redact PII instead? Who does the clean-up, the seller or the buyer? If you want to know whether your records are worth this preparation, our free estimate of what your company data is worth takes six questions and no files. Origin Data Partners is an independent data partner that helps companies get their data bought.
A buyer may pay Origin when a deal closes.
Key Takeaways
- โขEvery one of them is free, and every detection tool misses things, so a human still reviews a sample.
- โขThis story was reported by Dev.to, covering developments in the dev space.
- โขAI advancements continue to reshape industries โ read the full article on Dev.to for complete coverage.
๐ Continue reading the full article:
Read Full Article on Dev.to โShare this article



