Three Times the Data, the Same Mistakes: A Manually Audited Evaluation of Seerie, an Offline Scam-Prevention Assistant for India

Tiwari, Jay

doi:10.5281/zenodo.23245616

Open access preprint · Zenodo · v349 · current

Three Times the Data, the Same Mistakes: A Manually Audited Evaluation of Seerie, an Offline Scam-Prevention Assistant for India

Jay Tiwari · jay@seerror.com

Seerror Technologies, Bengaluru, India · 9 October 2026

Official record · DOI 10.5281/zenodo.23245616

Zenodo · official now

Version 349
zenodo.org/records/23245616

Previous paper

v224 audit
36,669 conversations · 58/70 safe

—

Views

—

PDF downloads

87/100

Manual safe

107,575

Conversations

Download PDF Open PDF Seerie v3 model Zenodo / DOI Cite RIS / Zotero

Abstract

Seerie is a scam-prevention assistant for India that runs entirely on an Android phone, with no internet connection after the one-time model download and no conversation data leaving the device. This version is a newer model: a QLoRA fine-tune of Qwen3-1.7B on 107,575 conversations in English, Hinglish and Indic scripts (corpus v349), deployed as a 1.11 GB 4-bit GGUF model through llama.cpp. Every answer of the fine-tuned model and of the untuned base model to a 100-prompt suite was rated by hand. The fine-tuned model is safe on 87 of 100 prompts (95% CI 79–92) against 35 for the base model. On the 70 prompts shared with the previous model (36,669 conversations) it is no better — 59 against 58 safe — and it still fabricates legal sections. A held-out test of 34 unseen topics looks near-perfect (271 of 272 automatic passes, ROUGE-L 0.94), but 181 of the answers copy training replies word for word. The automatic harness again agrees with the manual rating barely above chance (Cohen’s κ = 0.12).

Main results

Hand ratings and checks reported for corpus v349. Source: Zenodo 10.5281/zenodo.23245616.
Measure	Result
Manual safe, fine-tuned	87 / 100 (95% CI 79–92)
Manual safe, untuned base	35 / 100
Shared 70 prompts vs v224	59 safe vs 58 safe
Held-out automatic passes	271 / 272 · ROUGE-L 0.94
Held-out answers copied from training	181
Harness vs manual (Cohen’s κ)	0.12

Three times the training data did not fix the failure mode of the earlier model. The new model is much safer than the untuned base, and it is not safer than v224 on the prompts both papers share. Fabricated legal sections remain.

Manual ratings for Seerie v349 on the 100-prompt suite — Manual ratings for the v349 model.

Training and held-out evaluation for Seerie v349 — Training and held-out view for corpus v349.

Records

Zenodo is the official citable record. Cite 10.5281/zenodo.23245616. The earlier v224 audit stays at 10.5281/zenodo.23110076. An arXiv identifier can be added later. It is not assigned yet.

Published 9 October 2026 (version 349, CC BY 4.0). The deposit includes the PDF, LaTeX source, the on-device GGUF, all 200 manually rated answers, held-out outputs, scripts, and figures.

Cite this paper

Tiwari, J. (2026). Three Times the Data, the Same Mistakes: A Manually Audited Evaluation of Seerie, an Offline Scam-Prevention Assistant for India (Version 349) [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23245616

@misc{tiwari2026seerie_v349,
  author       = {Tiwari, Jay},
  title        = {Three Times the Data, the Same Mistakes: A Manually Audited Evaluation of Seerie, an Offline Scam-Prevention Assistant for India},
  year         = {2026},
  month        = oct,
  note         = {Version 349},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.23245616},
  url          = {https://doi.org/10.5281/zenodo.23245616}
}

Full tables and verbatim answers are in the PDF.