Giving a voice AI hands: a WebRTC voice agent inside a web proxy

Built on Webfuse, Surfly's co-browsing platform. It's a floating orb you click and talk to, and it operates the live page for you - clicks buttons, fills forms, presses keys, picks from dropdowns, navigates. The voice and reasoning come from the Vapi Web SDK; the hands come from the Webfuse Automation API. A public, reusable reference extension - the source is on GitHub.

The problem

A browser voice agent needs two capability sets at the same time:

1. Network and CSP freedom - the Vapi SDK runs WebRTC through Daily.co, which fetches external JS bundles, opens wss:// signaling sockets, and spins up AudioWorklets from blob: URLs.
2. Browser media and DOM APIs - RTCPeerConnection, getUserMedia for the mic, AudioContext/AudioWorklet, and a real document.

The root cause of difficulty is that Webfuse is a web proxy, and it deliberately does not let an extension's host_permissions override the proxied page's Content Security Policy. That's the correct decision - overriding it would break the original document's security model - but it invalidates every obvious place to put the SDK:

• Inject the SDK <script> into the page -> runs in page context, inherits the page's CSP -> external fetch and blob worklets blocked.
• Import the SDK in the content script -> still tab context, still page CSP -> same blocks.
• Run the SDK in the background service worker -> CSP is fine, but service workers have no RTCPeerConnection, no getUserMedia, no AudioWorklet, no DOM -> "WebRTC not supported or suppressed".

No single naive context gives you both halves. This doesn't fail on a bug; it fails on the platform's security architecture. I tried four execution contexts before the working one.

The design

The resolution: run the SDK in the popup. A Webfuse extension popup is an extension page with its own document and its own CSP derived from host_permissions - so external fetch and blob worklets are allowed (unlike content scripts) and RTCPeerConnection, the mic, and AudioWorklet all exist (unlike service workers). It is the only context that has both halves. Everything else follows from that.

The popup is detached and styled as the floating orb. A standard Chrome popup closes the moment it loses focus - useless for a multi-minute voice call. Webfuse popups persist across navigation and can be detached and restyled, so the popup is the widget. No separate injected UI to keep in sync.

// Popup is the only context with extension CSP + WebRTC + DOM.
browser.browserAction.detachPopup();
browser.browserAction.setPopupStyles({
  backgroundColor: "transparent", border: "none", boxShadow: "none",
  position: "fixed", bottom: "0px", right: "0px",
  width: "224px", height: "224px", overflow: "visible",
  /* ... */
});
browser.browserAction.openPopup();

The assistant is configured fully inline - no dashboard. Model, transcriber, voice, system prompt, and the entire tool schema are defined in code and passed to vapi.start(). The tradeoff: I give up reusing dashboard-managed assistants, and in return the build is self-contained, version-controlled, and portable across accounts (it only needs a public key).

vapi.start({
  model: {
    provider: "openai", model: "gpt-4o",
    messages: [{ role: "system", content: systemPrompt }],
    tools: vapiTools as any[],
  },
  transcriber: { provider: "deepgram", model: "nova-2", language: "en" },
  voice: { provider: "vapi", voiceId: "Elliot" },
  firstMessage: "Hey! I can help you interact with this page. What would you like to do?",
  clientMessages: ["tool-calls", "transcript"],
});

The content script is a thin automation relay. Here's the catch that makes the split non-negotiable: the Webfuse Automation API is only exposed in tab context (the content script), but the SDK can't run there. So the two contexts each do only what they uniquely can - the popup decides what to do and the content script does it on the page.

// tools.ts (popup): send the call across to the page context
function delegateAutomation(scope: string, method: string, ...args: any[]) {
  return browser.tabs.sendMessage(0, { automationScope: scope, automationMethod: method, automationArgs: args });
}

// content.ts (tab): the only place webfuseSession.automation is available
browser.runtime.onMessage.addListener((message: any) => {
  if (message.automationScope && message.automationMethod) {
    const { automationScope: scope, automationMethod: method, automationArgs: args = [] } = message;
    return (browser.webfuseSession.automation as any)[scope][method](...args);
  }
});

The model sees stable targets, not raw HTML. The take_dom_snapshot tool returns interactive elements only, tagged with wf-id attributes that survive re-renders and cross shadow boundaries. That's a smaller payload to the model and a targeting scheme that doesn't break the moment the page re-renders - instead of feeding full-page HTML and brittle CSS selectors.

case "take_dom_snapshot": {
  const snapshot = await delegateAutomation("see", "domSnapshot", {
    webfuseIDs: true,      // stable wf-id targets, e.g. "2-1-54"
    crossShadow: true,     // pierce shadow DOM
    interactiveOnly: true, // only clickable/typeable elements -> smaller payload
  });
  return snapshot;
}

Tool results loop back into the live call. Tools run client-side and async; each result (or error) is sent back to the model as a system message, so the conversation keeps going and the assistant can recover from a failed action mid-call.

const result = await handleToolCall(name, params);
vapi?.send({
  type: "add-message",
  message: { role: "system", content: `[Tool "${name}" result]: ${result}` },
});

One smaller gotcha worth recording: Daily.co's signaling is a secure WebSocket, so wss://*.daily.co/* has to be whitelisted in host_permissions separately from the https:// entry, or the connection is silently blocked.

Results

• Four execution-context attempts before the working architecture: content-as-script-tag, content-as-import, background service worker, and finally the popup.
• Three contexts, one SDK location - the popup carries all the WebRTC, audio, and network; the content script is a thin relay; the background is a two-line opener.
• Six voice tools the model can call: take_dom_snapshot, click_element, type_text, press_key, select_option, navigate_to.
• Six host_permissions entries, including the non-obvious standalone wss://*.daily.co/*.
• Ships as three IIFE bundles from a single dependency (@vapi-ai/web), configured entirely in code.

The principle

When a platform sandboxes capabilities across execution contexts, stop hunting for the one context that does everything. Place each piece of work in the only context that can do it, and relay between them. The architecture isn't a compromise around the sandbox - it's the shape the sandbox was telling you to build all along.