A voice assistant in the Linux top bar
A GNOME extension that listens, transcribes speech on the laptop, hands it to an AI agent and speaks the reply. Speech in and out runs locally, and replies start in under 0.6 s.
- Role
- Systems & AI Engineer
- Built with
- GNOME Shell 50 (Wayland), D-Bus IPC, Whisper (Local STT), Kokoro ONNX (CPU TTS), Python / Bash, Stream-JSON Agent IPC
Results
- <0.6s first-sentence TTS
- 100% local Whisper STT
- Warm stream-json daemon (8s → 2.5s startup)
In more detail
System-level GNOME Shell 50 (Wayland) top-bar dynamic island extension communicating over session D-Bus with local Whisper speech-to-text, a warm stream-json headless AI agent process, persistent Markdown memory, and sentence-pipelined Kokoro ONNX CPU speech synthesis (<0.6s TTFA).
Key decisions
Priority-Driven Session D-Bus Dynamic Island Extension
Built a native GNOME Shell 50 extension exposing org.gnome.Shell.Extensions.Island over session D-Bus (Show, Hide, Copy, Clicked signals) with live audio level waveforms, timers, and expandable panels tested in headless Wayland sessions.
Local Whisper STT + Warm Agent Daemon + <0.6s Kokoro ONNX TTS
Recorded voice queries transcribe locally via GPU Whisper, route into a pre-warmed headless stream-json agent process with service-enforced Markdown memory (REMEMBER/FORGET directives), and synthesize sentence-by-sentence on CPU via Kokoro ONNX.