Generative-AI product · Real-time NLP
At Capital One I lead the team that builds real-time, LLM-powered tools for our contact centers — systems that listen to a live call and help the agent while it's still happening. That work is proprietary; I can describe it, but a hiring manager can't click it. So I rebuilt the shape of it in public, solo, on a hobbyist stack — and you can talk into it right now.
The demo mid-call: a two-voice transcript, self-drafting notes, RAG-retrieved procedure docs, and a sub-second frustration alert. agentassistdemo.harrisonjansma.com
Why I built this
As a customer and an agent talk, a good copilot should transcribe the call, retrieve the right procedure, draft the notes, and flag a frustrated caller within about a second — so a supervisor can step in before the call goes sideways. That's the product I lead at work, and it's the most interesting kind of AI engineering I know: real people, real time, real stakes.
To show the shape of it without touching anything proprietary, I needed a fictional business to support. I used one I'd already built — Shopfolio, a little content back-office I made for my wife to help manage her Pinterest content. So the sample call is a creator whose custom-domain storefront is down, and the copilot helps the support agent work the problem. None of it touches Capital One's systems, data, models, or code — it's a clean-room rebuild of the idea.
What this is — and what it isn't
This isn't a machine-learning model I trained. The speech-to-text is Deepgram; the notes, document retrieval, and sentiment are OpenAI API calls. Those are off-the-shelf, general-purpose services anyone can call. What the project demonstrates is using assistive / generative-AI technologies intelligently — composing commodity models into a real-time product that fills an actual need. The skill on display is knowing which model to reach for and building the system around it, not inventing the model.
The architecture
One decision drove everything: a single WebSocket per session carries both the uplink audio and the downlink results. No separate audio channel, no realtime-database subscription for UI updates — one pipe. The browser streams mic (or the sample call's) audio; the worker relays it to Deepgram and fans each finalized line out to three jobs in parallel.
browser audio ──ws──▶ Node worker ──▶ Deepgram (streaming ASR + diarization)
│
each final line fans out, in parallel:
├──▶ LLM ▶ running call notes
├──▶ pgvector ▶ top-k procedure docs
└──▶ LLM ▶ sentiment ▶ frustration alert
│
results ──ws──▶ the three-panel UI
Web is Next.js + Tailwind. The worker is Node + ws. Data is Supabase Postgres with
pgvector. It deploys as two Railway services from one monorepo, behind Cloudflare.
The whole thing is about 1,500 lines — and the interesting part isn't the model calls, it's the
plumbing around them.
The parts that were actually hard
Cadence, so it doesn't cost a fortune. The naive version calls an LLM on every finalized sentence — expensive and jittery. Instead each job has a policy: notes regenerate only after enough new lines and a minimum interval, fed the previous draft plus only the new lines (so cost stays flat as a call runs long); retrieval de-dupes so the doc cards don't flicker; sentiment runs per utterance but the alert is rate-limited. That cadence layer is most of the value.
Retrieval that stays quiet. RAG is only useful if it says nothing when nothing's relevant. Retrieval embeds a rolling window of recent lines, does a cosine top-k against the corpus, and drops anything below a similarity threshold. Small talk surfaces nothing; "my certificate is stuck on pending" surfaces the SSL doc within one cycle; changing topic swaps the cards.
Two voices, cleanly separated. The sample call is a real agent-and-customer dialogue. Deepgram diarization tags who's speaking, so the transcript renders chat-style — and, importantly, the notes, retrieval, and sentiment analyze only the customer. The agent's lines are context, not signal.
A cached replay for a free, deterministic demo. Running the live pipeline on every visitor would cost money and depend on a warm speech socket. So the sample call was run through the real pipeline once, and every event — transcript, notes, doc hits, sentiment, alert, with its timing — was recorded. Clicking "play" replays that timeline in sync with the audio: no re-transcription, no LLM calls, deterministic and free. The microphone path is still fully live.
Measured latency
The number that matters for the product is transcript → sentiment ready, because that's what gates the frustration alert. Measured server-side (Deepgram final receipt to the sentiment score returning), it runs at a sub-second median across a call — fast enough that a supervisor gets pinged while the caller is still talking. Streaming ASR and a cosine scan over a small corpus are effectively instant by comparison; the LLM round-trip is the slow link, and it's still comfortably under a second at the median.
What I'd do differently at production scale
The demo runs one worker with an in-memory rate limiter, pushing results straight over the socket. That's exactly right for a portfolio piece and exactly wrong for a call center. At real scale the honest version is a durable event backbone (Kafka) instead of a socket, so transcripts and derived events become a stream that QA, analytics, and supervisor tooling all subscribe to; throughput and graceful degradation, not happy-path latency, become the hard problem; and you need offline eval sets and online guardrails so a "small" prompt change doesn't quietly wreck note quality across thousands of calls. Those are the parts you can't see in a weekend demo — but they're the parts the job is actually about. The demo is the shape; the scale is the work.
Stack
Next.js · Node + ws · Deepgram nova-2 · OpenAI
gpt-4o-mini + text-embedding-3-small · Supabase Postgres /
pgvector · Railway · Cloudflare. Pair-programmed end to end with
Claude Code, Anthropic's agentic
coding CLI. MIT-licensed — code linked below.