Personal Voice Assistant

For business owners, missed calls are lost opportunities.

Most voice assistants, even from the big labs like OpenAI or Sesame Labs, still default to American voices. Designing an Australian conversational experience that’s instantly familiar would be surprisingly novelty. What if this novel experience could also do things for the business owner - book a meeting, send a callback request, and confirm their details - all before you even knew you missed the call?

I’d just spent a month building the pilot for an automated outbound calling system when the client pulled the pin, worried about being first to market with this kind of tech. Rather than shelve the idea, I pivoted my learnings to prove it could work in the wild.

Product Designer – Crafted the overall experience and conversation style so the assistant felt natural, local, and approachable over the arc of a conversation.

Developer – Built the booking logic, API integrations, and automations to connect Retell, Google Suite, n8n webhooks, and Twilio. Deployed the assistant, connected SMS alerts for real-time updates, and monitored the first month’s results.

1. Dialling In the Voice

First was to make the assistant sound like a human you’d actually want to talk to. I tinkered with every available voice parameter, finally landing on a parameter set and voice from ElevenLabs that stayed natural. This was the backbone, as if the voice felt off, nothing else would matter.

2. Designing the Conversational Arc

I mapped the entire flow including what info we needed, when we needed it, and how to get it without making the caller feel like they were in an interrogation. This included writing prompts for each section (aka conversational nodes), plotting where API calls would happen, adding fallbacks for when things went wrong, logging errors, and making sure we told the user something useful if we couldn’t deliver the ideal path. The happy flow had to sound natural, complete quickly, and work in a variety of real-world environments.

3. Integrating the Tech

I built the logic to take a caller’s request, turn it into a query the API could understand, then translate the returned availability into a friendly, conversational confirmation. As voice platforms and the Google APIs operated in GMT/PST, time conversion headaches also needed to be handled.

4. Breaking It (on Purpose)

I tested different humans, environments, tones, and styles. I created a suite of evals inside the voice platform to stress-test the system, pushing it to fail so I could fix its weak spots. As a nice-to-have technology, when I could reliably pass most evals it was good enough to go live.

5. Connecting the Dots & Launching

I wired up Twilio for routing, linked the flows to the business number, and launched the system so every missed call now turned into a productive interaction - logging all outcomes and sentiments for further analysis.

Voice Design ≠ Web Design

On a website, users can take their time, read at their own pace, and make decisions in silence. With voice, the interaction is live, immediate, and unforgiving. Every pause, mis-timed prompt, or awkward phrase is magnified because we know there’s no human on the other end, so our empathy is naturally lowered. This meant I had to think about pacing, anticipation, and reliable fallbacks far more than I would in any web UX.

Balance is Everything

A faster response might mean the API call never happened; better localisation might cost more to run; high-quality voices might add milliseconds that make the flow feel sluggish. I learned to identify the “sweet spots” where the experience still felt effortless for the user but remained efficient to operate. Some of this came down to clever sequencing, like running certain API calls in the background so the conversation didn’t stall, while others just required tinkering.

There’s an Art to Voice Info Validation

Confirming names and emails without frustrating the caller was (is) a real challenge. Accents, background noise, spelling, and special characters all add friction. There’s an art in framing a validation question so that it feels natural. I believe user-buy in (i.e. “Do you mind if I confirm that email?”) and signposting (i.e. “Great, let me read back your email letter-by-letter just to make sure it gets to you”) are two great ways to achieve this.

Voice Is Finally Growing Up

In 2018 we were talking about how smart speakers would make voice the new default interface, but the reality never matched the hype. Now, with the leaps in speech-to-text, text-to-speech, voice cloning, and LLM reasoning, the experience is starting to feel truly natural. It was validating to see that an Australian-first, localised voice could actually start to feel like a real, conversational partner. There’s a ways to go, but it can be done.