Ask Siri

A password-gated ElevenLabs Conversational AI agent, grounded only in my real resume, project history, and a red-teamed corpus. Ten minutes, real voice, zero tolerance for fabrication.

Classification

Voice AI · Conversational agents · RAG grounding · Applied trust & safety

Stack

Next.js · TypeScript · ElevenLabs Conversational AI · React SDK · Vercel

Subject

Ten minutes on the clock, a real cloned voice, and a hard rule: nothing invented, nothing guessed.

The Brief

For the past couple of years, AI and LLMs have been pigeon-holed into chatboxes, and that's not how humans were meant to communicate. Voice is the natural next step — it removes the friction that's held so many people back from actually talking to a computer instead of typing at one.

So instead of just saying that, I built the proof: a live agent visitors can actually talk to, grounded strictly in a corpus I control, that fails safe instead of bluffing when it doesn't know something.

Design Thesis

Grounded in what's real, honest about what isn't.

Every claim the agent makes has to trace to an uploaded document. If retrieval turns up nothing, it says so on the record, rather than smoothing over the gap. A fabricated personal claim here isn't a bug — it's the one failure mode this entire build exists to prevent.

The Architecture

One call, five stages, two systems.

A session flows left to right. Solid stages run in my own Next.js code. Dashed stages hand off entirely to ElevenLabs' infrastructure.

Stage 1

GATE

Password + a resumable 10-minute budget checked server-side, before any mic prompt or ElevenLabs call.

OUR CODE

Stage 2

SIGN

OUR CODE

Mints a short-lived ElevenLabs signed URL, as close to the moment it's used as possible.

OUR CODE

Stage 3

STREAM

Browser opens a direct WebSocket to ElevenLabs: two-way PCM audio. Our server isn't in that loop once it's live.

ELEVENLABS INFRASTRUCTURE

Stage 4

GROUND

CORE · RAG

Every answer retrieved from an uploaded knowledge base. Nothing from pretrained knowledge.

ELEVENLABS INFRASTRUCTURE

Stage 5

ESCALATE

Tool calls (log_unanswered, send_contact) hit Next.js webhooks that email the real Siri directly.

OUR CODE

GUARDRAILSSALARY · AVAILABILITY · OPINIONS ON NAMED PEOPLE — REFUSED EVERY TIMEEnforced by the agent's own rules and knowledge-base scoping, not a filter bolted on after the fact.

ON FILE

Answered

Grounded directly in the corpus, cited by document.

OFF LIMITS

Deflected

Salary, availability, opinions — refused by rule, every time.

GAP FOUND

Escalated

Logged and emailed to the real Siri, so the corpus gets better.

Solid ink = my own Next.js code Dashed oxblood = ElevenLabs infrastructure

View the full architecture diagram →

The Evidence

What actually shipped.

Knowledge-base documents

Resume, timeline, six project write-ups, FAQ, personal life — all real, all RAG-grounded.

10 min

Session cap per invite

A deliberate cost control, and the agent says so up front — not a trick.

60s

Silence timeout

Client-side watchdog, tracked independently of any ElevenLabs dashboard setting.

Agent tools wired to email

log_unanswered and send_contact, both curl-tested before the agent ever touched them.

What This Demonstrates

Three things, defended with the code.

Debug a phantom failure by trusting the evidence over the first theory.

The WebSocket handshake looked identical whether the call worked or not — DevTools showed a clean 101 Switching Protocols and a real conversation ID either way. The actual bug wasn't the network at all: Next.js's dev-mode Fast Refresh was remounting the component that owned the connection mid-handshake, orphaning it from the hook now on screen. Confirmed by running a production build, which has no Fast Refresh — it connected cleanly on the first try.

Design for a hard constraint: never let the model bluff.

The agent refuses to answer from general knowledge, full stop — if retrieval returns nothing, it says so. That rule got tested directly: a red-team pass against a fixed question set, run before anything shipped, hunting specifically for the one failure mode that matters here, a confident, fabricated answer.

Treat cost as a design constraint, not an afterthought.

Passwords carry a resumable, server-tracked ten-minute budget instead of a one-shot burn, synced via sendBeaconon tab close because a plain fetch has no delivery guarantee once a page starts unloading. A client-side silence watchdog ends an idle call automatically — ElevenLabs' own dashboard has no per-session override for that, so the guarantee had to live in the client.

FIELD NOTE · CONNECT

Rate-limited myself mid-debug: the same guard that stops password guessing also counts every retry while diagnosing a connection bug. Fixed by restarting the dev process, which clears the in-memory counter instantly.

FIELD NOTE · GROUND

There's a second, unrelated public Siri Rama — a dancer with a PhD in Fine Arts. The corpus explicitly warns the agent off any pretrained knowledge about that name, so it never answers as the wrong person.

FIELD NOTE · GATE

Signed URLs are short-lived by design, so they're minted the moment a visitor taps “begin,” not at page load or password entry.

EXHIBIT A — RESUMABLE BUDGET SYNC (TYPESCRIPT)

// Monotonically decreasing — a later report can never raise a

// password's remaining time back up.

function reportRemainingSeconds(password: string, reported: number) {

if (!SINGLE_USE_PASSWORDS.has(password)) return;

const current = budgets.get(password) ?? SESSION_SECONDS;

const next = Math.max(0, Math.min(current, Math.floor(reported))); // FIX

budgets.set(password, next);

}

Highlighted: a slow heartbeat landing after a newer one can't undo real usage — the budget can only ever go down.

Notes & Provenance

No visitor audio or transcript is used to train anything; conversations are transcribed for review, not resale.
The agent never draws on pretrained knowledge about “Siri Rama” — there's a second, unrelated public person with that name, and the corpus says so explicitly.
Every tool webhook fails closed: if the shared secret isn't set, the call is rejected, not silently allowed through.

Talk to it yourself →View the architecture diagram →← Back to all files