GECX Chat SDK Product Guide

This guide explains what this repo is, what the product does, and how the pieces fit together. It is written for engineers, product managers, designers, CX teams, and coding agents who need the big picture before diving into the API reference.

The short version: GECX Chat SDK is a TypeScript headless chat SDK for building custom customer engagement AI experiences. Your app owns the interface. The SDK owns the chat runtime: sessions, streaming, typed messages, tools, authentication boundaries, transport, diagnostics, analytics, identity, and governance.

If you only remember one idea, remember this:

The SDK is not a drop-in chat widget. It is the engine beneath a chat experience your product team fully controls.

Table of Contents

• What This Product Is
• Why Headless Matters
• Who This Is For
• What Is In This Repo
• How The System Fits Together
• The Core Product Model
• Message Parts And Rich Content
• Tools And Actions
• Generative UI With A2UI
• Auth, Transport, And The Proxy Boundary
• Identity, Analytics, And Governance
• Long-Term Memory
• Voice And Multimodal
• Signals: Sentiment And Intent
• Agent Graphs
• Computer-Use
• Permissions
• Evaluation And Quality
• Demos And Developer Experience
• Safety Model
• How To Choose Your Integration Path
• Glossary
• Where To Go Next

Where to start: new to the SDK? Skim this guide for the big picture. Want the fastest path to a runnable project? Run pnpm dev:studio (or open start.gecx.chat) for the 4-step wizard — see Start Studio. Prefer a manual setup? Follow the Quickstart (5 min). Building with React? Jump to the React Quickstart (10 min). Want to see it running first? pnpm dev opens the showcase on http://localhost:3000 — no credentials needed.

What This Product Is

GECX Chat SDK is a prototype TypeScript monorepo for building chat experiences powered by Google Customer Engagement AI.

It gives developers the pieces that are annoying, risky, or repetitive to build from scratch:

• A ChatClient for configuring auth, transport, storage, tools, analytics, identity, memory, voice, signals, permissions, and governance.
• A ChatSession for one conversation lifecycle.
• A streaming transport contract with mock, HTTP, proxy/SSE, session API, and WebSocket options.
• A normalized message model built around typed message.parts.
• A tool registry for schema-validated client and server actions, plus bundled tools for memory and computer-use.
• Optional React hooks and components for teams that want a fast UI path.
• Mock-first testing utilities and a first-class gecx eval CLI so teams can build, test, and gate quality without credentials.
• A reference proxy server that keeps secrets and privileged operations out of browser code.
• Long-term memory adapters, a real-time voice subsystem with Gemini Live, sentiment/intent signal parts, and a PermissionManager for mic/camera/screen/geolocation.
• A typed agentGraph API for composing specialist agents over Google's Agent-to-Agent protocol.
• Pre-built analytics widgets and a self-hostable dashboard reference server.
• An A2UI catalog of 20 vetted composite surface presets installable via gecx add ui:<name>.
• JSON Schemas, recipes, CLI helpers, and agent-oriented assets for repeatable integrations.

It is called "headless" because the SDK does not force a visual widget, iframe, or rigid layout into your product. It gives you state, events, and typed content. You decide how the experience looks.

flowchart LR
  subgraph Product["Your product"]
    UI["Owned UI: page, drawer, panel, mobile view, kiosk, admin console"]
    UX["Owned UX: brand, layout, accessibility, approvals, routing"]
  end

  subgraph SDK["GECX Chat SDK"]
    Runtime["Runtime: ChatClient and ChatSession"]
    Messages["Typed message parts"]
    Tools["Tool registry"]
    State["State, streaming, recovery"]
    Safety["Redaction, validation, diagnostics"]
  end

  subgraph Backend["Backend boundary"]
    Proxy["Customer proxy"]
    ServerTools["Server tools"]
    Google["Google Customer Engagement AI"]
  end

  UI --> Runtime
  UX --> Messages
  Runtime --> State
  Runtime --> Tools
  Runtime --> Safety
  Runtime --> Proxy
  Proxy --> ServerTools
  Proxy --> Google

Why Headless Matters

Traditional web chat products often give you a prebuilt widget. That is useful when the main goal is speed and the widget's UI is acceptable. It becomes limiting when the chat experience needs to feel native to your product.

A headless SDK takes a different stance:

The tradeoff is intentional. A headless SDK asks you to build the surface, but rewards you with a chat experience that can become part of the product instead of floating beside it.

Who This Is For

This repo serves several audiences at once:

What Is In This Repo

The repo is a pnpm monorepo. The primary publishable package is gecx-chat; the rest of the workspace demonstrates, validates, or helps integrate it.

flowchart TB
  Root["headless-chat-sdk"]

  Root --> SDK["packages/gecx-chat<br/>Core SDK + gecx CLI (doctor, verify, scaffold, agent-pack, add)"]
  Root --> Create["packages/create-gecx-chat<br/>App generator (npm/pnpm/yarn create gecx-chat)"]
  Root --> MCP["packages/gecx-chat-mcp-server<br/>Agent-facing MCP server"]

  Root --> Showcase["apps/showcase<br/>Feature showcase on port 3000"]
  Root --> Coolaid["apps/agi-coolaid-stand<br/>Branded commerce demo on port 3001"]
  Root --> Retail["apps/applied-ai-retail<br/>Premium retail journey on port 3002"]
  Root --> Studio["apps/start-studio<br/>Onboarding wizard at start.gecx.chat on port 3003"]
  Root --> Dashboard["apps/dashboard-reference<br/>Self-hostable analytics dashboard server"]
  Root --> Proxy["apps/proxy-reference<br/>Reference proxy on port 8080"]

  Root --> Docs["docs<br/>Concepts, guides, reference, examples"]
  Root --> Recipes["recipes<br/>Copyable integration recipes"]
  Root --> Schemas["schemas<br/>JSON Schemas for public contracts"]
  Root --> Evals["evals<br/>Agent evaluation tasks"]

Main SDK package

packages/gecx-chat is the SDK source. Its public entry points are split so apps only import what they need:

Demo apps

• apps/showcase is the feature-by-feature playground. It demonstrates streaming, citations, tools, rich content, A2UI, handoff, diagnostics, analytics, governance, identity, voice, agent graphs, signals, computer-use, dashboards, and the A2UI catalog gallery.
• apps/agi-coolaid-stand is a branded storefront demo. It proves the SDK can live inside a real product UI rather than a generic widget.
• apps/applied-ai-retail is a premium retail journey where the whole journey lives inside the chat — browse, learn more, add to cart, checkout, track, return, and gift bundling. It also ships a /lab agentic-iteration playground, a /support triage graph, a /computer-use demo, and an /analytics "Conversational Commerce Intelligence" dashboard wired to the SDK's ProductAnalyticsEvent stream.
• apps/start-studio is the onboarding wizard at start.gecx.chat. A 4-step UI (template → tools → handoff → analytics) that downloads a runnable repo with a mock transport pre-seeded; install-to-first-conversation in under three minutes. Dogfoods the SDK's own ProductAnalytics for telemetry.
• apps/dashboard-reference is a self-hostable Next.js server. It ingests httpSink events and renders the five pre-built dashboard widgets (deflection, CSAT, AHT, agent-assist, GMV).
• apps/proxy-reference is a reference customer proxy. It shows the production boundary for tokens, streaming, uploads, server tools, deletion, export, right-to-erasure, voice token issuance, computer-use SSE, redaction, rate limiting, and audit logging.

Docs, schemas, recipes, and agents

• docs contains getting started guides, concepts, guides, reference pages, examples, demo walkthroughs, agent docs, and internal runbooks.
• schemas contains JSON Schemas mirroring the public TypeScript contracts.
• recipes contains copyable UI and transport patterns such as suggestion chips, product carousels, streaming indicators, citation previews, accessibility controls, tool call cards, and non-Google transport examples.
• packages/gecx-chat-mcp-server exposes docs, recipes, prompts, scaffolds, and validation tools to coding agents.

How The System Fits Together

The SDK is organized around three layers:

1. Runtime: session lifecycle, state machine, event normalization, tools, storage, identity, analytics, governance, and diagnostics.
2. Transport: how messages move between the app and a backend.
3. UI adapter: optional React helpers, plus enough framework-neutral state for any UI framework.

flowchart TB
  App["Your app UI"]
  React["Optional React adapter<br/>ChatProvider, useChatSession, MessagePart"]
  Core["Framework-neutral core<br/>createChatClient, ChatSession, store, events"]
  Normalizer["Message normalizer<br/>transport events -> ChatMessage parts"]
  Tools["Tool registry<br/>schemas, approval, timeouts"]
  Storage["Storage and identity<br/>session, local, memory, cross-tab sync"]
  Transport["Transport<br/>mock, HTTP, proxy/SSE, session API, WebSocket"]
  Backend["Backend or mock"]

  App --> React
  App --> Core
  React --> Core
  Core --> Normalizer
  Core --> Tools
  Core --> Storage
  Core --> Transport
  Transport --> Backend

A typical turn looks like this:

sequenceDiagram
  participant User
  participant UI as Your UI
  participant Session as ChatSession
  participant Transport as Transport
  participant Proxy as Customer Proxy
  participant AI as Google Customer Engagement AI

  User->>UI: Sends a message
  UI->>Session: session.sendText("Where is my order?")
  Session->>Session: Add pending user message
  Session->>Transport: stream(send request)
  Transport->>Proxy: POST /chat/stream
  Proxy->>AI: Forward with server-side auth
  AI-->>Proxy: Stream events
  Proxy-->>Transport: SSE events
  Transport-->>Session: response.started, text.delta, rich.payload, tool.call
  Session->>Session: Normalize events into typed message parts
  Session-->>UI: Store updates
  UI-->>User: Render streaming answer and rich content

In local development, the proxy and Google API can be replaced with mock transport:

sequenceDiagram
  participant UI as Your UI
  participant Session as ChatSession
  participant Mock as Mock transport

  UI->>Session: sendText("Show me backpacks")
  Session->>Mock: stream(send request)
  Mock-->>Session: Scripted streaming events
  Session-->>UI: Same message.parts contract as production

That mock-first design is important. Teams can design, test, and demo the experience before credentials, environments, or upstream APIs are ready.

The Core Product Model

ChatClient

ChatClient is the configured SDK instance. It owns shared concerns:

• Auth provider.
• Transport.
• Tool registry.
• Storage adapter.
• Capabilities.
• Rich content registry.
• Identity manager.
• Conversation registry.
• Data governance manager.
• Analytics configuration.
• Telemetry and diagnostics sinks.
• Recovery and multi-tab behavior.

Create one client when you want shared configuration:

import { createChatClient } from 'gecx-chat';
import { createMockTransport } from 'gecx-chat/testing';

const client = createChatClient({
  transport: createMockTransport(),
});

const session = await client.createSession();
await session.sendText('Hello');

ChatSession

ChatSession is one conversation. It owns:

• Session status.
• Messages.
• Streaming state.
• Current error.
• Tool lifecycle events.
• Upload state.
• Handoff status.
• Product analytics events.
• Trace timeline.
• Debug bundle generation.
• Session-scoped governance methods.

The session exposes events and a store so UI code can react to updates without knowing transport details.

Session lifecycle

Each session moves through a controlled state machine. That prevents invalid actions, such as sending a second message while a response is still streaming.

stateDiagram-v2
  [*] --> idle
  idle --> authenticating
  authenticating --> connecting
  connecting --> ready
  ready --> submitted
  submitted --> streaming
  streaming --> ready
  streaming --> waiting_for_tool
  waiting_for_tool --> streaming
  ready --> uploading
  uploading --> ready
  streaming --> disconnected
  disconnected --> recovering
  recovering --> ready
  ready --> ended
  ready --> expired
  ready --> error
  error --> recovering
  ended --> shutdown
  expired --> shutdown

Capabilities

Capabilities describe what the client can handle:

• Streaming.
• Rich content.
• File upload.
• Human handoff.
• Client tools.
• Citations.
• Suggestion chips.
• Markdown.
• A2UI.
• Per-family rich content versions and fallbacks.

Capabilities let the server or mock transport know what the host can render. If a response contains unsupported rich content, the SDK can downgrade to text or hide it according to the registry policy.

Message Parts And Rich Content

The SDK treats every message as an envelope with typed parts.

flowchart LR
  Message["ChatMessage"]
  Meta["id, role, status, createdAt, sessionId, turnIndex"]
  Parts["parts[]"]
  Text["text / text-delta / markdown"]
  Rich["citation / suggestion-chips / product-carousel / order-summary / custom"]
  Action["tool-call / tool-result"]
  Ops["file / agent-transfer / diagnostic / end-session / error"]
  Voice["audio-input / audio-output / transcript / audio-cue / vision"]
  Memory["memory-approval / memory-recall-result"]
  Signals["sentiment-signal / intent-signal"]
  Compute["computer-use-surface"]
  A2UI["a2ui-surface"]

  Message --> Meta
  Message --> Parts
  Parts --> Text
  Parts --> Rich
  Parts --> Action
  Parts --> Ops
  Parts --> Voice
  Parts --> Memory
  Parts --> Signals
  Parts --> Compute
  Parts --> A2UI

This is the core rendering contract. Instead of receiving one blob of HTML, the app receives structured data:

{
  id: 'msg_123',
  role: 'agent',
  status: 'streaming',
  parts: [
    { id: 'part_1', type: 'text-delta', delta: 'I found ' },
    { id: 'part_2', type: 'product-carousel', products: [...] },
    { id: 'part_3', type: 'suggestion-chips', chips: [...] }
  ]
}

Typed parts matter because they let each team render AI output with native product components:

The React adapter includes a MessagePart renderer with sensible defaults, but the product value is that you can replace any renderer with your own component.

Tools And Actions

Tools let the AI ask the host app to do something. The SDK supports two lanes:

• Client tools run in the browser and are useful for safe client-side operations.
• Server tools run behind the proxy and are used for privileged work such as refunds, address updates, payment operations, or database writes.

flowchart TB
  AI["AI asks for a tool"]
  Registry["Tool registry"]
  Known{"Registered tool?"}
  Schema{"Input matches schema?"}
  Approval{"Requires approval?"}
  Execute["Execute tool"]
  Result["Return result to model"]
  Deny["Fail closed"]

  AI --> Registry
  Registry --> Known
  Known -- No --> Deny
  Known -- Yes --> Schema
  Schema -- No --> Deny
  Schema -- Yes --> Approval
  Approval -- Yes --> User["Ask user or approval handler"]
  Approval -- No --> Execute
  User -- Approved --> Execute
  User -- Denied --> Deny
  Execute --> Result

Tool definitions include:

• A name.
• A description.
• A JSON Schema input contract.
• An execution function.
• Optional approval requirements.
• Optional timeout behavior.

Unknown tools fail closed. Invalid inputs fail before execution. Tools that require approval are denied if no approval handler is configured.

That shape is deliberate: AI can request actions, but the host app remains in control of validation, permission, execution, and user consent.

Bundled tools

A few tools ship with the SDK and register automatically when the relevant subsystem is configured:

The A2UI catalog (20 surface presets) is a related capability — host components installed via gecx add ui:<name> that the agent renders through A2UISurfacePart. See A2UI Catalog.

Generative UI With A2UI

Most rich content is best when the host app already knows the shape: product carousel, order summary, citation preview, suggestion chips.

A2UI is for a different case: the agent chooses the interaction layout at runtime.

Examples:

• A return workflow that asks for the exact missing fields.
• A troubleshooting checklist where steps appear as the agent narrows the issue.
• A product tuner with sliders and choices generated from the conversation.
• A guided support form that changes based on previous answers.

flowchart LR
  Agent["Agent emits A2UI frame"]
  Validate["SDK validates frame"]
  Catalog{"Catalog registered?"}
  Policy["Action policy<br/>allow, confirm, or block"]
  Render["Render with local React components"]
  Action["User action envelope"]
  Host["Host routes action"]

  Agent --> Validate
  Validate --> Catalog
  Catalog -- No --> Fallback["Fail closed with fallback"]
  Catalog -- Yes --> Render
  Render --> Policy
  Policy --> Action
  Action --> Host

A2UI safety rules are part of the product model:

• Frames are untrusted input.
• Catalogs must be registered by the host.
• Unsupported catalogs are rejected.
• Dangerous action names can require confirmation.
• Non-user-initiated actions can be blocked.
• Debug and trace payloads redact action context and data models by default.
• The SDK does not fetch remote catalogs or execute code from frames.

A2UI Catalog

Twenty vetted composite surface presets ship as the A2UI catalog. Each preset bundles a frame factory, a dataModel schema, an action allowlist, a host React renderer, a mock scenario, and a manifest with a sha256 integrity hash. Install one with:

gecx add ui:eta-promise
gecx add ui:appointment-picker
gecx ui:doctor             # verify installed components

Components fall into four tiers — Tier A (stable, basicCatalog primitives only), Tier B (stable, forms and multi-step), Tier C (stable, data-density), and Tier D (experimental, host-rendered for primitives outside basicCatalog). See A2UI Catalog for the mental model and Contributing to the catalog for the contribution flow.

Auth, Transport, And The Proxy Boundary

The product draws a hard line between browser-safe work and server-only work.

In local development, mock auth and mock transport are enough. In production, a customer proxy should sit between the browser and Google APIs.

flowchart LR
  Browser["Browser app<br/>gecx-chat runtime"]
  Token["POST /chat/token<br/>short-lived chat token"]
  Stream["POST /chat/stream<br/>SSE response stream"]
  Upload["POST /chat/upload<br/>multipart files"]
  ServerTool["POST /chat/tool-call<br/>privileged tools"]
  Governance["delete, export, forget-me"]
  Proxy["Customer proxy<br/>auth, redaction, rate limit, audit"]
  Google["Google Customer Engagement AI"]
  Systems["Customer systems<br/>orders, CRM, billing"]

  Browser --> Token
  Browser --> Stream
  Browser --> Upload
  Browser --> ServerTool
  Browser --> Governance
  Token --> Proxy
  Stream --> Proxy
  Upload --> Proxy
  ServerTool --> Proxy
  Governance --> Proxy
  Proxy --> Google
  Proxy --> Systems

The browser should not contain service account keys, long-lived credentials, privileged tokens, or direct upstream API secrets. The proxy handles those concerns server-side.

The reference proxy demonstrates:

• Short-lived token issuance.
• SSE streaming.
• Multipart upload forwarding.
• Server-side tool dispatch.
• Conversation deletion.
• Conversation export.
• Right-to-erasure.
• Health checks.
• CORS allowlists.
• Request redaction.
• Rate limiting.
• Structured audit logging.

Transport options are swappable:

Identity, Analytics, And Governance

The SDK is not only a send-message wrapper. It includes product-level systems needed for customer engagement experiences.

Identity and continuity

The identity system supports:

• Guest identities.
• Authenticated identities.
• Guest-to-authenticated upgrades.
• Conversation descriptors.
• Cross-tab synchronization.
• Cross-device imports from a backend.
• Sign-out cleanup.

flowchart LR
  Guest["Guest visitor"]
  Local["Local conversation history"]
  Login["User signs in"]
  Auth["Authenticated identity"]
  Remote["Remote conversation registry"]
  OtherDevice["Another device"]

  Guest --> Local
  Local --> Login
  Login --> Auth
  Auth --> Remote
  Remote --> OtherDevice

This lets a user start as a guest, sign in later, and keep continuity instead of losing the conversation.

Product analytics

Analytics are vendor-neutral. The SDK collects events and sends them only to sinks you configure. It does not send analytics to an external service by default.

Tracked journey signals include:

• Message impressions.
• Suggestion chip clicks.
• Tool approval requests and outcomes.
• Handoff requests, conversions, and completions.
• CSAT.
• Resolution and resolution without escalation.
• Session lifecycle, turn latency (TTFT, response duration), transport health, errors, tool execution timing, file uploads, and rich-content engagement.
• purchase_completed from chat.trackPurchase({ orderId, revenue, currency, items?, paymentMethod? }).
• Agent-graph routing decisions (agent_routed, agent_specialist_started/completed/failed, agent_graph_entered/exited).
• A2UI catalog install, render, and action events.
• Signal updates from sentiment and intent adapters.

Content is excluded by default. Metadata keys that look sensitive are redacted before events are stored or sent to sinks.

Four built-in sinks ship in the core package: httpSink, consoleSink, ga4Sink, and segmentSink. mixpanelSink ships as a first-party sink alongside them. Long-tail adapters (Amplitude, PostHog, Rudderstack, Datadog RUM, OpenTelemetry) live as copy-paste recipes under recipes/.

Five pre-built dashboard widgets — DeflectionTrend, CsatDistribution, AhtHistogram, AgentAssistRate, GmvBreakdown, plus a composite <MetricsDashboard> — ship under the gecx-chat/dashboards subpath. Recharts is an optional peer dependency, so consumers who never import this subpath see no install footprint. The self-hostable apps/dashboard-reference renders all five against httpSink events.

Data governance

Governance APIs cover:

• Consent posture.
• Local retention.
• Conversation export.
• Conversation deletion.
• Right-to-be-forgotten.
• Local state clearing.
• Expiration purges.
• Audit events.

flowchart TB
  Policy["DataGovernancePolicy"]
  Consent["Consent<br/>none, functional, analytics, all"]
  Retention["Retention<br/>session, ttl, forever"]
  Export["Export conversation"]
  Delete["Delete conversation"]
  Forget["Forget me"]
  Audit["Audit events"]

  Policy --> Consent
  Policy --> Retention
  Policy --> Export
  Policy --> Delete
  Policy --> Forget
  Policy --> Audit

These features are useful even in early prototypes because they make privacy and lifecycle expectations visible from the start.

Long-Term Memory

gecx-chat ships an opt-in long-term memory subsystem so the assistant can remember user facts across conversations. When ChatClientConfig.memory is omitted, nothing changes — no tools register, no interceptor installs, no extractor runs.

flowchart LR
  Chat["ChatSession"]
  Interceptor["MemoryInterceptor"]
  Store["MemoryStore"]
  Adapter["Adapter"]
  Tools["memory.save / update / delete / recall tools"]
  Extractor["Extractor (heuristic or LLM)"]

  Chat --> Interceptor
  Interceptor --> Store
  Store --> Adapter
  Interceptor --> Tools
  Chat --> Extractor
  Extractor --> Store

Three adapters ship — createLocalMemoryAdapter (browser-only, no setup), createServerMemoryAdapter (cross-device, requires a backend), and createHybridMemoryAdapter (local-first with server sync). Two read modes (inject adds saved facts as a system message; recall exposes a tool the model can call) and two write paths (tool-call by the model; automatic extraction from completed turns) cover the usual integration patterns. The React surface ships useMemory() and a <MemoryList> component.

See Memory and the Memory API reference. The applied retail demo wires a server-side memory store at apps/applied-ai-retail/src/lib/serverMemoryStore.ts — a good production reference.

Voice And Multimodal

Voice is a first-class peer to text. The same ChatMessage envelope carries voice parts, the same IdentityManager owns the session identity, and the same ChatGovernance exports and forgets transcripts and (optionally) audio bytes. Vision (image input + output) ships as a sibling part type.

Configuration is one line:

const client = createChatClient({
  // ...auth, transport
  voice: 'auto',  // mock in test, web-audio mock in dev, Gemini Live in prod/staging
});

'auto' picks a provider by environment. Explicit VoiceConfig gives full control. chat.voice is a lazy getter — configuring voice does not request the microphone or open a WebSocket; the provider factory only runs when something first reads session.voice.

The React adapter ships <VoiceToggle voice={chat.voice} /> with a mic-permission probe and inline remediation when access is blocked. firstAudioMs instrumentation surfaces a sub-400 ms latency target as a metrics event.

Bundled providers: createMockVoiceProvider (timer replay, zero browser deps), createWebAudioMockVoiceProvider (real AudioContext, synthetic tones), createGeminiLiveProvider (production WSS to Gemini Live). Hosts that run their own voice backend wire createVoiceWebSocketTransport.

See Voice and Multimodal for the mental model and Voice Integration for the integration walkthrough.

Signals: Sentiment And Intent

Two new message-part variants — SentimentSignalPart and IntentSignalPart — flow through the same ChatMessage.parts[] array as text. They let the host react to emotional state and inferred intent without inventing a parallel event channel.

Adapters trade latency for accuracy:

SignalRunner dispatches adapters; SignalEscalator evaluates declarative rules like frustration > 0.7 across two turns and calls HandoffController.requestTransfer() with stable idempotency keys. The handoff FSM is untouched.

React hooks useSentiment(messages) and useIntent(messages) return memoized snapshots of latest, history, and per-category/per-intent rollups.

See Signals and Sentiment and Intent.

Agent Graphs

Customers running production are hitting the limits of one monolithic system prompt and want to split work across specialists. The SDK ships a typed agentGraph API for composing router nodes and specialist agents under one router that picks the right specialist for each turn.

const triage = defineRouter({
  id: 'triage',
  intent: createHeuristicIntentClassifier({ /* ... */ }),
  routes: {
    returns: [/refund/i, /\breturn(s|ed|ing)?\b/i],
    billing: [/charge[ds]?/i, /bill(ing|ed)?/i],
    order: [/\border(ed|s)?\b/i, /shipping/i],
  },
});

const graph = agentGraph({
  entry: triage,
  specialists: { returns, billing, order },
});

const client = createChatClient({
  transport: createAgentGraphTransport({ graph, fallback: httpTransport }),
});

Specialists are reached over Google's Agent-to-Agent (A2A) protocol — the SDK ships an A2A client, not a server. Routing is in-process and observable (<200 ms p99). External (human) handoff is unchanged: routing to the reserved 'human' target delegates to the existing HandoffController.

Six new analytics events (agent_graph_entered/exited, agent_routed, agent_specialist_started/completed/failed) make routing decisions auditable. The DebugBundle gains an agentGraph snapshot. The wire format adds a 'bot_to_bot' value to the TransferType enum plus optional routeDecision and graphPath fields on AgentTransferPart and HandoffStatusEvent.

See Agent Graphs and Agent Graph guide. The applied retail /support route is a production-style reference.

Computer-Use

The SDK supports a sandboxed computer_use server tool that lets an agent drive a browser on the user's behalf — for tasks the host's APIs don't expose, like "look up my warranty status on the manufacturer's portal." It is default-off. Every invocation needs explicit governance.computerUse.enabled = true, an allowlist of hostnames, and a configured provider on the proxy.

flowchart TB
  Browser["ChatClient<br/>(governance, tool)"]
  Proxy["Customer proxy<br/>createComputerUseHandler"]
  Vendor["Hosted Chromium<br/>(Browserbase)"]
  Surface["ComputerUseSurface<br/>(sandboxed iframe + action log)"]

  Browser --> Surface
  Browser -- POST /chat/tool-call --> Proxy
  Proxy -- SSE screenshots --> Surface
  Proxy --> Vendor

Three independent enforcement layers gate every action: the SDK fast-fails against its local copy of the policy, the proxy reapplies the policy authoritatively, and the vendor browser is the real security boundary. Session-level approval reuses the standard ToolRegistry.tool.awaiting_approval flow; per-action high-risk approval (form submits, navigation, downloads) is a second tier inside the live session.

Every action emits a governance.computer_use.* audit event. A global kill switch — governance.computerUse.killSwitch = true plus an admin endpoint on the proxy — halts every session immediately. The bundled BrowserbaseProvider and MockProvider cover production and testing. See Computer-use and the threat model.

Permissions

Device permissions (microphone, camera, screen, geolocation) are a first-party concern. PermissionManager orchestrates requests across capabilities and platforms, syncs grants to governance consent flags, and emits audit events on every state change.

const client = createChatClient({
  // ...
  permissionProvider: createBrowserPermissionProvider(),
});

const stream = await client.permissions.ensure('microphone'); // throws on denial

Three bundled providers (createBrowserPermissionProvider, createMockPermissionProvider, createNoopPermissionProvider) cover the usual cases; native runtimes (Expo/RN, Capacitor) plug in their own adapter against the same contract.

request() never throws — errors are data. ensure() is the throwing convenience used by VoiceSession.start() and captureFromDevice(). Insecure context (HTTPS / localhost required) returns { status: 'unsupported', reason: 'insecure_context' } rather than collapsing into a generic denial.

Four new consent flags (microphone_capture, camera_capture, screen_capture, geolocation_capture) and two new audit kinds (permission_granted, permission_revoked) wire into ChatGovernance. React: usePermission, usePermissionManager, <PermissionPrompt>.

See Permissions and Permission Providers.

Evaluation And Quality

gecx eval runs scenario-replay tests against a ChatSession, scores them with deterministic and LLM-judge scorers, and emits a JSON report plus a regression gate that can block PRs in CI.

pnpm gecx eval ./apps/showcase/scenarios \
  --baseline ./apps/showcase/baseline.eval.json \
  --fail-on-regress \
  --json --output eval-report.json

Sixteen scorers ship — thirteen deterministic (message-contains, tool-called, tool-call-accuracy, latency-p95-under, error-code, handoff-triggered, no-handoff, no-error, etc.) and three LLM-judge (hallucination, helpfulness, tone) that skip cleanly when no provider key is configured. Three fetch-based judge adapters (Anthropic, OpenAI, Gemini) carry the LLM-judge path.

Scenarios are TypeScript or YAML files in a directory you choose; the showcase ships 22 reference scenarios and a committed baseline.eval.json. The GitHub Action recipe wires the runner into CI with a configurable failure threshold.

See Evaluation and the Eval CLI reference.

Demos And Developer Experience

The repo is designed to be experienced locally without credentials.

Fastest start: the onboarding wizard

apps/start-studio is the shortest route from "I want to try this" to a running chat. It is a 4-step wizard:

1. Template — Support Concierge or Commerce Concierge.
2. Tools — pick from a catalog of starter client tools.
3. Handoff — None, Mock, or a CCaaS adapter (Salesforce, Zendesk, ServiceNow, Genesys, Five9, Google CCAI).
4. Analytics — None, Console, GA4, Segment, or HTTP sink.

A live <ChatSession> in the right pane responds to wizard choices in real time. The Download button builds a zip client-side (Web Worker + fflate) and bundles the SDK as a tarball, so the generated repo installs offline. Target: under three minutes from landing to first conversation. See the Start Studio demo guide.

pnpm install --frozen-lockfile
pnpm dev:studio
# open http://localhost:3003

Manual local path

pnpm install --frozen-lockfile
pnpm build:packages
pnpm dev

Then open http://localhost:3000 for the showcase.

Showcase app

apps/showcase is the best place to understand SDK features one by one:

Coolaid Stand demo

apps/agi-coolaid-stand shows the SDK embedded in a branded customer-style storefront:

• Product catalog.
• Custom concierge panel.
• Streaming recommendations.
• Product carousels.
• Approval-gated cart tools.
• A2UI flavor tuner.
• Mock-first mode by default.
• Optional experimental live bridge.

This demo answers a practical question: "Can the SDK feel like our product instead of a vendor widget?" The answer is yes, because the UI is owned by the app.

Applied AI Retail

apps/applied-ai-retail is the most ambitious demo: a premium retail journey where the entire experience lives inside the chat. Routes include:

• /product, /cart, /checkout, /orders, /returns for the storefront chrome.
• /support — a production-style triage agentGraph with returns / billing / order specialists.
• /computer-use — the sandboxed agent-browsing flow in retail context.
• /lab — an agentic-iteration playground for prompt and recipe work.
• /analytics — "Conversational Commerce Intelligence" dashboard wired to the live ProductAnalyticsEvent stream.

It also wires a server-side memory store, an in-chat memory drawer, an A2UI gift-bundle moment, an SDK Inspector drawer, and in-UI "Connect GECX" (no env vars required).

Dashboard reference

apps/dashboard-reference is a self-hostable Next.js server that ingests httpSink events and renders the five pre-built dashboard widgets against a 5s SWR poll. See Hosted Dashboard for the deploy walkthrough and Dashboard Reference for the demo overview.

Proxy reference

apps/proxy-reference shows what the server boundary looks like when moving from mock mode to production-style deployments. It is intentionally plain Node.js so teams can adapt the hook functions to Cloud Run, Cloud Functions, Express, Fastify, or their own platform.

Beyond the original token, stream, upload, tool-call, deletion, export, and forget-me routes, it now hosts a /chat/voice-token route for Gemini Live and the computer-use SSE endpoints (/chat/computer-use/:sessionId/stream, /chat/computer-use/:sessionId/control) with an admin kill switch.

Start Studio (start.gecx.chat)

apps/start-studio is the onboarding wizard described above. Architecturally it is interesting for two reasons:

• The scaffold pipeline (src/lib/scaffold/) is a set of pure functions that emit { path, contents }[]. It runs identically in the Web Worker (production) and in Vitest (the generated-repo-compiles.test.ts suite runs pnpm install + pnpm typecheck against a real tmpdir to guarantee every combination compiles).
• The studio dogfoods the SDK's own ProductAnalytics HTTP-sink contract for telemetry, so the wizard_started → wizard_download_clicked interval is the measured time-to-first-conversation KPI. Events stream to Cloud Logging via stdout in the Cloud Run container.

CLI, generator, and recipes

Developer experience support includes:

• gecx doctor for checking token endpoints, CORS, deployment metadata, voice token endpoint, A2UI catalog integrity, and integration health.
• gecx scaffold for recommended integration structure. gecx scaffold --with-voice adds a voice page and a token route.
• gecx agent-pack init --all|--target <agent>|--mcp for installing agent-facing instructions in downstream projects.
• gecx list, gecx search, and gecx add for recipe discovery and copying.
• gecx add ui:<name> and gecx ui:doctor for A2UI catalog component installation and integrity verification.
• gecx eval <dir> for scenario-replay tests with deterministic and LLM-judge scorers.
• gecx lab init --template <commerce|support> to generate a /lab route powered by the @gecx/lab package.
• gecx walkthrough capture and gecx walkthrough verify for the cross-agent walkthrough artifact protocol.
• gecx validate for the PCI / PII lint pass (Luhn-valid card literals, CVV literals, unguarded payment-method fields).
• pnpm dlx gecx-chat-mcp-server --install to register the MCP server for Claude Code, Codex, or Antigravity.
• create-gecx-chat for generating a small app with support, commerce, or blank templates.

Agent integration

The repo includes assets for coding agents because headless UI work often involves composing SDK primitives into product-specific experiences. Tri-agent parity is a first-class goal: Claude Code, OpenAI Codex, and Google Antigravity all consume the same canonical skill source.

Agent-facing pieces include:

• docs/agent/llms.txt and docs/agent/llms-full.txt — canonical task summary for agents.
• docs/agent/agent-prompts.md — golden prompts for common integration tasks.
• docs/agent/agent-pack.md — downstream installation walkthrough.
• The canonical skill source at skills/headless-chat-vibe-coding/ syncs on pnpm install to .claude/skills/, .agents/skills/, and .agent/skills/. AGENTS.md, CLAUDE.md, and GEMINI.md are pointer files.
• An MCP server v1 (packages/gecx-chat-mcp-server) exposing 14 tools, 12 resource templates, and 14 prompts (10 general + 4 commerce). pnpm dlx gecx-chat-mcp-server --install writes the right config for each detected agent (.mcp.json, .codex/config.toml, mcp_servers.json).
• @gecx/lab — an agent-iteration primitive with a LabSurface React component, per-agent prompt variants, and copy/verify/walkthrough buttons. gecx lab init --template <commerce|support> generates the /lab route.
• A walkthrough artifact protocol (schemas/walkthrough.schema.json) plus gecx walkthrough capture / verify for cross-agent "show your work" replay.
• Automated Antigravity eval lane with pnpm agent-evals --report for cross-agent comparison.
• JSON Schemas under schemas/ for grounding model output against the public TypeScript surface.

Safety Model

The safety model is simple: treat model output, transport payloads, and browser inputs as untrusted unless validated.

flowchart TB
  Input["Untrusted input<br/>user, model, transport, files"]
  Schema["Schema validation"]
  Registry["Known registry lookup"]
  Policy["Host policy<br/>capabilities, approval, consent"]
  Redaction["Redaction<br/>tokens, secrets, PII-shaped keys"]
  Execute["Execute or render"]
  Reject["Reject or downgrade"]

  Input --> Schema
  Schema -- Invalid --> Reject
  Schema -- Valid --> Registry
  Registry -- Unknown --> Reject
  Registry -- Known --> Policy
  Policy -- Disallowed --> Reject
  Policy -- Allowed --> Redaction
  Redaction --> Execute

Important defaults and rules:

• Core SDK stays framework-neutral.
• React imports stay under packages/gecx-chat/src/react/.
• Mock transport is the default development path.
• Real Google API calls require an explicit live transport contract.
• Service account keys and privileged secrets stay out of browser code.
• Tokens are redacted from logs, state, trace events, and debug bundles.
• Rich content is rendered through typed message parts.
• Custom payloads are data, not executable code.
• HTML rendering is disabled by default.
• Client tools must have schemas.
• Unknown or invalid tools fail closed.
• Server tools run behind the proxy when privileged access is needed.
• A2UI catalogs must be registered by the host. Unsupported A2UI frames fail closed.
• A2UI catalog components carry sha256 integrity hashes; gecx ui:doctor reports drift.
• Product analytics excludes content by default.
• Voice and capture permissions never silently succeed. Every grant emits a governance.permission_granted audit event; revokes emit permission_revoked.
• Computer-use is default off. When enabled, allowlist enforcement runs at three independent layers (SDK fast-fail, proxy authoritative, vendor browser sandbox). Per-action high-risk approval is a second tier. A global kill switch halts every session immediately.
• Commerce governance fail-closes on PCI/PII. When strict mode is on, PAN/Luhn, CVV, SSN, email, phone, address, and EIN detections throw COMMERCE_PII_LEAK rather than leaking into transcripts, debug bundles, approval surfaces, or analytics.
• Agent-graph routing is observable. Every decision emits an agent_routed event with the matched rule and chosen specialist; the DebugBundle carries a recent-decisions snapshot.
• Signal adapters degrade gracefully. Optional peer dependencies dynamic-import; missing deps mean the adapter returns no emissions, not a crash.

How To Choose Your Integration Path

Use this decision map when starting a project:

flowchart TB
  Start["I want to add AI chat"]
  Mock["Start with mock transport"]
  React{"React app?"}
  ReactPath["Use gecx-chat/react<br/>ChatProvider and useChatSession"]
  CorePath["Use gecx-chat core<br/>subscribe to session.store"]
  Rich{"Need custom rich UI?"}
  Parts["Render typed message parts<br/>custom renderer registry"]
  Tools{"Need actions?"}
  ClientTools["Client tools for browser-safe work"]
  ServerTools["Server tools through proxy for privileged work"]
  Live{"Ready for live backend?"}
  Proxy["Deploy or adapt proxy-reference"]
  Validate["Run doctor, tests, e2e, and security review"]

  Start --> Mock
  Mock --> React
  React -- Yes --> ReactPath
  React -- No --> CorePath
  ReactPath --> Rich
  CorePath --> Rich
  Rich -- Yes --> Parts
  Rich -- No --> Tools
  Parts --> Tools
  Tools -- Browser-safe --> ClientTools
  Tools -- Privileged --> ServerTools
  ClientTools --> Live
  ServerTools --> Live
  Live -- No --> Mock
  Live -- Yes --> Proxy
  Proxy --> Validate

Recommended first integration

For most teams:

1. Build against mock transport.
2. Render message.parts with your own UI components.
3. Add one or two schema-backed client tools.
4. Add tests with gecx-chat/testing.
5. Move privileged actions behind the proxy.
6. Wire a token endpoint.
7. Run gecx doctor.
8. Add analytics, governance, and handoff once the core chat experience works.

When to use each feature

Glossary

Where To Go Next

Start with the path that matches your goal:

For a first hands-on pass, run:

pnpm install --frozen-lockfile
pnpm dev:studio    # or pnpm build:packages && pnpm dev

Then open the studio wizard at http://localhost:3003, or the showcase at http://localhost:3000 and start with the overview or the Wow Tour.