Welcome to the weekly Human AI Studio newsletter. Every week I share, insights, research and industry updates and article here.
This week I’ve been prototyping voice for Orbi, and I hit a real wall and I wanted to share in this issue.
The agent kept mixing up what I was saying. Some of it was me creating a task. Some of it was me asking for a recap of a task I already gave it. The words overlap a lot when you’re talking instead of typing, and the agent couldn’t always tell the difference.
When it gets this wrong, the whole experience breaks and that’s usually the moment people stop using the product.
What I keep coming back to: the agent has intelligence, but no harness. It understands language fine. It just doesn’t have a system around it deciding what to do with that understanding. Building that harness is turning out to be the real work.
It’s also changed how I think about building these things in general. You can’t just design a couple of screens and call it done anymore. If you actually want to know whether an agent experience works, you have to build it and use it.
The voice itself was its own problem too. Early on it sounded robotic, no personality, nothing natural about it. One of my design engineers who worked for OpenAI introduced me to OpenAI’s Whisper API. Latency and voice quality improved almost right away.
Then there’s the multi-modal side of it, which might be the hardest part. Listening, talking back, reading intent, visual feedback, audio feedback, all of that running together, staying in sync. That’s a different kind of design problem than anything I was dealing with before.
It’s made it pretty clear to me vibe coding an app and calling it done isn’t enough. You actually have to understand how agents work underneath it.
Thank you for reading
If you have a project in mind, book a 15 min call with me
Want to know how to build apps with AI, join our AI Academy