Evaluation
How well does the voice model work?
Headline results
| Test set | Result | Notes |
|---|---|---|
| English real-world speech (In-the-Wild, unseen speakers) | 2.9% EER | n = 1,000 genuine / 1,000 fake. Flagged synthetic: precision 99.9% (913/914), recall 91.3%. |
| Arabic speech (ArAD, voice conversion) | 2.85% EER | n = 484 genuine / 1,000 fake. Levantine dialect, short clips; Gulf dialects and Arabic TTS not tested. |
| WhatsApp-like audio (Opus 16 kbit/s, English) | 6.1% EER | n = 500 / 500. 24.6% of genuine clips come back “inconclusive”; 5% are wrongly flagged. |
Model: MMS-300M-AntiDeepfake (NII Yamagishi Lab), CPU inference. Thresholds: Likely genuine below 0.172 · likely synthetic above 0.859 · inconclusive in between — frozen on a speaker-disjoint In-the-Wild calibration split (600/600).
All locked test sets
EER = equal error rate. “Flagged” = labelled likely synthetic. Missed fakes = fakes wrongly labelled likely genuine.
| Dataset | Channel | n genuine / fake | EER | Precision | Recall | False positives | Missed fakes |
|---|---|---|---|---|---|---|---|
| In-the-Wild | clean | 1000 / 1000 | 2.90% | 99.9% | 91.3% | 0.1% | 1.3% |
| In-the-Wild | Opus 16 kbit/s | 500 / 500 | 6.10% | 94.9% | 92.4% | 5.0% | 1.4% |
| In-the-Wild | 8 kHz phone | 500 / 500 | 3.70% | 99.6% | 89.0% | 0.4% | 3.0% |
| ArAD (Arabic) | clean | 484 / 1000 | 2.85% | 100% | 77.4% | 0.0% | 5.5% |
| ArAD (Arabic) | Opus 16 kbit/s | 242 / 500 | 3.46% | 100% | 86.6% | 0.0% | 2.6% |
| ArAD (Arabic) | 8 kHz phone | 242 / 500 | 4.17% | 100% | 58.4% | 0.0% | 15.2% |
No clip failed to process in any set.
Why not the best-known model?
| Model | Where it looks good | Where it fails |
|---|---|---|
| AASIST (MIT licence) | 1.72% on ASVspoof 2019 LA (its own benchmark) | 39.6% on In-the-Wild · 42.1% on Arabic ArAD — near chance |
| Wav2Vec2-Small-AntiDeepfake | 4.7% on In-the-Wild | 19.4% on Arabic ArAD |
| MMS-300M-AntiDeepfake (selected) | 2.9% on In-the-Wild | 2.85% on Arabic ArAD |
AASIST is widely cited and permissively licensed, but its excellent benchmark score does not survive real-world audio. We selected on measured performance on data the models were not trained on — and published the failure.
What we do not claim
- Accuracy against modern zero-shot voice cloning (both test sets contain pre-2023 fakes).
- Accuracy for Gulf/Emirati Arabic, Arabic text-to-speech, or long WhatsApp voice notes.
- That a “likely genuine” result identifies the speaker. It does not.
- That detection replaces verification. Tumaini always requires independent verification before a payment.
Latency and licence
Measured through the live app (upload → VPS inference + transcription → result): about 6–9 s for an 11-second voice note on a shared 2-vCPU container.
Model weights: CC BY-NC-SA 4.0 (non-commercial). Suitable for this research prototype; a commercial launch needs a licensed or self-trained model.
Method, dataset licences and reproduction commands: docs/AI_EVALUATION.md, docs/DATASET_REGISTER.md and evaluation/ in the repository.