Sundry Shop Voice Assistant (Gemini Live)
Bahasa Malaysia voice assistant for kedai runcit owners over POS data.
A Bahasa Malaysia voice assistant for kedai runcit owners. The shop owner taps "Mula", taps the mic, and asks in casual Bahasa — "Hari ni jualan berapa?" — and hears the answer spoken back. No typing. No English required.
The use case is real because the alternative isn't. Most "AI for SMEs" demos assume the operator is comfortable typing in English. They aren't. Voice in their first language is the right interface for this audience.
Built on Gemini Live API with a function-calling bridge to MCP-style tools over the shop's POS dataset.
The LLM part was the easy part. The audio pipeline is where the real work lives:
- Browser captures mic at 16kHz PCM via an AudioWorklet
- Streams the PCM up via WebSocket to a FastAPI backend
- Backend forwards to Gemini Live; relays the model's
function_callto atool_bridgethat runs pandas queries on the POS CSV - Streams 24kHz PCM back down for playback
Gemini Live's function-calling worked smoothly with custom tools — the SDK handles it cleanly. The longer fight was the prompt: getting the model to consistently produce santai (relaxed) Bahasa, not formal textbook Malay, took more iteration than the entire backend.
For Malaysian SMEs whose phone time isn't English-keyboard time, this is the interface that lowers the barrier. Voice + local language + working POS data, in one tap.