Phone agent for a US logistics company
Answers live customer calls about orders and warehouse status. Replies start in under 350 ms, and callers can only ever reach their own data.
- Role
- Lead Forward-Deployed AI Engineer (US Logistics/Warehouse Client via TechRAQ)
- Built with
- Python, FastAPI (SSE/WebSockets), Twilio, Vapi, Deepgram, Qdrant, Redis, Docker
Results
- <350msMedian LLM TTFTStreaming clause-level token piping to TTS
- <50msQdrant Vector LookupCollection-level HNSW + pre-filtered payloads
- 40%Token Spend ReductionRedis active context pruning across turns
- 0Unauthorized ExposuresOut-of-prompt AuthN/AuthZ payload binding
Sliding-window Redis passage pruning reduced multi-turn prompt growth by 40% over 8-minute logistics support calls, while cascaded streaming avoided the 3x–5x audio token cost of raw speech-to-speech models and preserved 100% deterministic RBAC over order tools.
The problem
End-to-end speech-to-speech models lacked deterministic control over intermediate tool calls and database permissions, while naive sequential STT → RAG → LLM → TTS pipelines exceeded 2.5+ seconds of turn latency, breaking natural phone conversation.
In more detail
Production telephony voice agent handling live customer order and warehouse inquiries with <350ms LLM Time-to-First-Token (TTFT) and zero-trust retrieval access control.
How it works
┌──────────────────────────────────────────────────────────────────────────────┐
│ PSTN CALLER (Twilio Media Stream) ──► Vapi / Deepgram Streaming ASR (WebSocket)│
└──────────────────────────────────────┬───────────────────────────────────────┘
│ Partial + Final Utterance Events
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ FastAPI STREAMING ORCHESTRATOR (Asyncio Event Loop • <350ms LLM TTFT) │
│ │
│ ┌──────────────────────────────┐ ┌─────────────────────────────────┐ │
│ │ Verified Session Claims │ │ Redis Active Session Memory │ │
│ │ {org_id, customer_id, scope} │ │ Sliding-Window Context Pruning │ │
│ └──────────────┬───────────────┘ └────────────────┬────────────────┘ │
│ │ Hard-Bound Outside Prompt │ -40% Token Spend │
│ ▼ ▼ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ Qdrant HNSW Retrieval (<50ms) │ │
│ │ MUST Filter: org_id == claim.org_id AND customer_id == claim.cust_id │ │
│ └───────────────────────────────────┬────────────────────────────────────┘ │
└──────────────────────────────────────┼───────────────────────────────────────┘
│ Grounded Context + Tool Guardrails
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ STREAMING LLM SYNTHESIS ──► Clause-Chunked SSE ──► Streaming TTS ──► Caller │
└──────────────────────────────────────────────────────────────────────────────┘Key decisions
Cascaded Streaming Pipeline (Twilio → Vapi → Deepgram → FastAPI)
Orchestrated Twilio PSTN telephony streams through Vapi and Deepgram Nova ASR into a low-latency FastAPI SSE/WebSocket orchestrator, piping clause-boundary token chunks directly into TTS synthesis while streaming full observability telemetry.
Collection-Level Qdrant Retrieval & Redis Context Pruning
Indexed live warehouse inventory and order state in Qdrant for sub-50ms retrieval; cached active session context in Redis with sliding-window passage pruning to eliminate redundant history and cut median multi-turn token spend by 40%.
Zero-Trust Retrieval Access Control (Out-of-Prompt AuthN/AuthZ)
Enforced strict caller authentication and authorization outside the LLM prompt by binding verified session claims (org_id, customer_id, order_scope) directly into Qdrant payload filters and deterministic tool execution guards—preventing prompt injection from leaking cross-customer order data.