LLM evaluation · Real-time NLP · Capital One
I built a generative complaint-detection model and, more importantly, the evaluation around it — anchored to a real ground-truth proxy instead of a convenient one. Status honesty: validated in production code, piloting in 2026 — not yet shipped.
This is a deliberately high-level account of proprietary work at Capital One — the problem, my role, and the approach, described qualitatively. It contains no confidential figures, internal systems, data, models, code, vendor or team names, or architecture.
From frustration phrases to policy-aware detection
The legacy approach was a hand-trained sentiment flag (with limited precision even on its own training data) that fired on frustration phrases. The replacement is an open-source GPT-based model used generatively: it produces a complaint rationale and an escalation prediction, and it detects more than frustration — policy-based escalation drivers are built into the detection logic.
The evaluation design (the part worth asking about)
There is no clean labeled set of “all complaints,” so I chose a real-world ground-truth proxy: known complaint escalations — calls escalated to a specialized support tier — over a month of production data. The legacy flag caught only a small fraction of them; the generative model captured a large majority, at a precision agents and QA can trust. The residual it cannot catch is structural: cases an agent escalates proactively from something they see on screen, with no customer-side trigger in the conversation. Knowing where the ceiling is — and why — is part of the result.
My role & status
Solo on this one: I implemented the model in production code and did the full analysis myself. The result earned an internal excellence award. To be precise about status: it is validated but not yet piloted — a pilot is planned for 2026, and I'm doing the streaming-pipeline work to bring real-time inference in-house.
Why it matters beyond the metric
Complaint capture is a compliance surface — a missed complaint is a missed obligation. Moving capture up sharply, at a precision agents and QA can trust, and with a documented account of what the system can't see, is the difference between a demo metric and an operable control.