Skip to content
Blog

Architecture

Giving a chat-native agent a voice

Building one agent and teaching it to listen beats building two agents.

The questions people actually ask

Bruce is EGI’s enterprise execution agent. It works across every department of a company, on top of the systems they already run: ERP, CRM, inbox, calendar and the rest of their tools. Look at what people ask it day to day and most of it isn’t creative work. It’s questions about their own business:

  • “What’s our gross profit month to date?”
  • “Did the Acme order ship yet?”
  • “Any new quote requests since this morning?”
  • “Log a note on the account that I stopped by today.”

These are short questions with exact answers, usually asked between meetings or from the driver’s seat. People were already typing them one-handed. Voice is the natural way to ask them, so when reps started asking to talk to Bruce, the request made sense. The hard part was everything that happens around the answer.

What chat gives you for free

In the web app, a turn is more than a reply. While Bruce works, you watch it narrate each step and see which tool it is using. You can open any step and see exactly what ran. When it wants to change something in a system of record, it stops and shows an approval card, and nothing is written until you approve.

Under that screen sits the harness: the parts that make Bruce safe to point at a customer’s data.

One question to Bruce in chat
“What’s our gross profit month to date?”
What the rep sees
on the screen
  • Live narration
    Bruce says what it is doing at each step
  • Tool calls
    an ERP query, a CRM search, code in a sandbox
  • The work behind the answer
    open any step to see exactly what ran
  • Approval cards
    nothing changes in a system of record until you approve
What keeps it safe
underneath, in the harness
  • Tools per company
    each company only gets the tools it turned on
  • Permissions
    deny, ask or allow, set per company, role, person and agent
  • Write detection
    spots any change to a CRM or ERP before it runs
  • Approvals that hold
    they survive a restart and run exactly what was shown
  • Tenant isolation
    every read and write stays inside one company
What one chat turn gives a rep, and what sits underneath it.

All of it was designed around a screen. Voice has no buttons, and on a phone call there is no screen at all.

Two agents, or one agent with a voice

We tried both designs before choosing.

The first was a second agent built for voice, with its own tools, prompts and approval flow. It worked, but it meant running two agents. Every new feature had to be built twice and tested twice, and every change to permissions had to be copied into a second place where it could quietly drift. The voice agent would always be the weaker, less tested one.

The second was a voice layer on top of the agent we already had. The voice handles the conversation, and anything that needs real work goes down to the same Bruce that answers in chat. One agent, one set of permissions, one place to add features.

The rule we set

Voice is a layer on top of the harness, not a second agent.

Two designs we tried
Option A: two agents
Chat agent
tools, prompts, approvals
Voice agent
its own copy of all of it
Every new feature
built and tested twice
Option B: one agent, a voice layer
Voice layer
talks and listens
Hands the request down
as text
Bruce harness
the same one chat uses
ERP, CRM, sandbox
Both designs work on day one. Only one is still simple a year later.

Picking the voice model took longer than picking the design. We ran evals across the realtime voice models on the market, including OpenAI’s GPT-Live and ElevenLabs, until we found one that could hold a natural conversation, tell when a request needed Bruce, and hand it off cleanly. A model that sounds great but answers business questions from its own imagination is worse than no voice at all.

A mouth and a brain

The voice model runs the spoken conversation. Bruce does the work. A small turn controller in the browser sits between them.

  • The browser talks to the voice model directly, so audio never touches our servers.
  • When the voice model decides a request needs Bruce, the browser sends the user’s own words to Bruce, through the same path the chat box uses.
  • Bruce runs a normal turn: the same agent, tools, approvals and tenant boundaries as a typed one.
  • The turn controller watches Bruce work and decides what the voice should say out loud and what it only needs to know.
One spoken question, end to end
Repmic + speakerVoice modelrealtimeBrowser appturn controllerBruce agentsame as chatCRM + sandboxtools, approvals“What did Acme buylast quarter?”“Let me check.”this needs Brucethe rep’s wordstool calls, 20–40 sprogressa short update“Still pulling invoices.”resultsanswer + tableheadline to say,the rest to know“$889.89. The tableis on screen.”Audio stays between the browser and the voice model. Only text reaches Bruce.
Only text crosses between the voice and Bruce.

The link between the two is text in both directions. That one choice is what lets everything else stay the same.

Permissions live below the voice

We put the voice model and the turn controller above the harness on purpose. A spoken request reaches Bruce as the same kind of request as a typed one, from the same signed-in person. So it passes through the same permissions, the same policy checks and the same approvals, with nothing special for voice.

Voice model
Talks, listens and decides when a request needs Bruce. It can’t touch any business system.
voice layer
Turn controller
Decides what the voice says out loud and what it only needs to know.
voice layer
below this line, chat and voice are the same
Who is asking
The signed-in person, their company and their role
harness
Permissions and policy
What this person and this agent may do, and what needs a yes first
harness
Write detection and approvals
Every change to a system of record is caught and confirmed before it runs
harness
Tools and systems
ERP, CRM, inbox and calendar, scoped to one company
systems

Speaking a request never unlocks anything that typing it wouldn’t.

Everything that decides what can happen sits below the voice layer.

That ordering keeps role-based access intact. A role that can’t change the CRM can’t change it by voice either. A change that needs approval in chat needs it in voice. And when the harness improves, voice gets the improvement for free. We recently moved every tool call onto one shared set of checks, and voice needed no changes for it.

It also keeps costs down. Voice and chat share one conversation and one cached prompt, so moving from typing to talking doesn’t start Bruce over from scratch.

Turning a screen into speech

The plumbing was the easy part. The real work was deciding, item by item, what each thing on the screen becomes in speech.

On screen
What did Acme buy last quarter?
✓Searched the CRM for Acme
✓Pulled last quarter’s invoices from the ERP
Totalling orders in the sandbox
In voice
Bruce says
“Still pulling invoices.”

Knows the CRM search and the invoice pull are done, without reading them out.

Each thing the screen does has a spoken version. Pick one, or let it play.

Approvals by voice

By approvals by voice we mean this: you call Bruce and ask it to draft an email to a customer. Bruce reads the draft back to you, but it still waits for your clear go-ahead before sending, because a wrong email is a customer-facing problem. The same goes for a change in the CRM, an update in the ERP, or anything else that lands in a system of record.

This was one of our main pillars, and it got the most design time, because it is where voice could quietly weaken the system. The first request for it came from a rep who was driving and couldn’t tap the screen.

Approvals by voice
Email to a customer
“Email the customer a follow-up”
Bruce reads it back
the draft is on screen too
“Send it?”
a clear yes, or nothing
Same approval
as tapping the button
CRM or ERP change
“Move the deal to closed won”
Bruce says what will change
record, field, new value
“Should I go ahead?”
“yes?” is not a yes
Same approval
as tapping the button
Voice changes how a decision is collected. It never changes which decisions exist.
  • One path. A spoken yes goes through exactly the same approval as the Approve button. There is no voice-only shortcut to audit.
  • A strict ear. A question (“yes?”) or a yes with a change (“yes, but make it 300”) approves nothing. Bruce asks again.
  • Two meanings of cancel. “Cancel that” can mean “stop what you’re doing” or “cancel the order”. The first stops Bruce. The second is a business request, so it goes through the normal approval like any other change.

Seeing what the voice did

Each voice session is one trace, joined to every request it handed to Bruce. When an answer sounds wrong, we can see what the voice heard, what it passed down, what Bruce did and what came back. Voice failures don’t throw errors. They sound like a slightly wrong answer, and that trace is how we find them.

We also replay real sessions whenever we change the turn controller. That is how we caught a “go on” from the voice that split one request in two, and a voice that kept saying “still on it” while nothing was running. Those stories get their own post.

What we’d tell another team

  • Don’t give your voice model a second copy of your tools. Have it hand off to the agent you already trust.
  • Evaluate voice models on your real requests, not on how natural they sound in a demo.
  • Put permissions below the voice layer. The input device should never change what is allowed.
  • Go through what the screen did, item by item: progress, results, approvals, cancel, drafts.
  • Speak the headline and let the screen hold the table.

Voice turned out to be a new way to ask, not a new agent. That is why it was safe to ship, and we’re really proud of how it came out. To try it, go to app.bruce.sh and click the voice icon.

Next up: we hooked the same system up to a phone line, so you can call Bruce from a real phone number. That one gets its own post.

Thanks for reading.

Get new posts by email

One email when a new post goes up. No other mail, and you can unsubscribe from any email.

Other posts