Privacy filter: results on public benchmarks
Share of personal values found, per benchmark.
| Benchmark | Languages | Texts scored | Our privacy filter (strict mode) | Our privacy filter (loose mode) | Open model openai/privacy-filter (default settings) |
|---|---|---|---|---|---|
| REDACT (terms) | 12 EU | 480 | 99.3% | 99.1% | 89.0% |
| SPY | 1 | 1,000 | 99.6% | 98.7% | 86.8% |
| PII-TRACE* | 1 | 250 | 100% | 88.9% | 93.0% |
| Kiji* | 6 | 1,002 | 99.1% | 92.5% | 85.3% |
| Gretel finance* | 7 | 1,001 | 96.3% | 94.0% | 79.1% |
| Gretel English | 1 | 1,000 | 99.8% | 98.8% | 93.7% |
Loose mode looks for names and usernames, email addresses, phone numbers, postal addresses, dates of birth, personal links and IP addresses, bank and card numbers, ID, passport and VAT numbers, passwords and keys.
Strict mode looks for everything loose mode does, plus any date, codes, places, organisations and job titles.
Our filter also masks more text around personal data than the open model: about 2 to 3.5 times the personal data itself, against about 1 to 2 times.
* We used other samples from this benchmark while building our filter. The samples scored here were not used.
- REDACT: records from its evaluation split, EU languages only.
- SPY: English legal and medical questions.
- PII-TRACE (Perplexity): English multi-turn chat.
- Kiji (Dataiku): customer support messages.
- Gretel finance (Gretel): financial documents.
- Gretel English (Gretel): documents from many domains.