social-app/docs/protocols/ui/data-flow.md

前后端数据流通指南

本文档描述前端如何与后端 Agent 系统交互，以及如何渲染后端返回的 UI 数据。

---

1. 整体交互流程

┌─────────────────────────────────────────────────────────────────────────┐
│                              前端                                        │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌──────────────┐    POST /runs     ┌──────────────┐                │
│  │ 构造请求      │ ──────────────────▶│  后端        │                │
│  │ (多模态数据)  │                    │  (任务入队)  │                │
│  └──────────────┘◀────────────────── └──────────────┘ 202 Accepted   │
│         │                                                    taskId    │
│         │                                                            │
│         │ GET /runs/{threadId}/events                               │
│         │──────────────────────────────────────────────────────────▶   │
│         │                           SSE 事件流 ◀─────────────────    │
│         │                                                            │
│  ┌─────┴──────────────────────────────────────────────────────────┐   │
│  │  渲染事件                                                       │   │
│  │  - text.delta: 追加文本                                        │   │
│  │  - tool.result: 渲染 toolAgentOutput.ui_hints → UiSchema      │   │
│  │  - text.end: 渲染 workerAgentOutput.ui_hints → UiSchema       │   │
│  └────────────────────────────────────────────────────────────────┘   │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────┐
│                              后端                                        │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  POST /runs                                                            │
│    → 验证请求                                                          │
│    → 任务入队列 (Taskiq)                                              │
│                                                                         │
│  Worker 执行                                                           │
│    → emit 事件到 Redis Stream                                         │
│    → UiHints → ui_compiler.compile() → UiSchemaRenderer              │
│                                                                         │
│  持久化                                                                │
│    → AgentChatMessage (含 uiSchema)                                   │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

---

2. 多模态数据处理

2.1 前端上传附件流程

// 方式一：先上传附件，获取签名 URL
const formData = new FormData();
formData.append('threadId', threadId);
formData.append('file', imageFile);

const uploadResponse = await fetch('/api/v1/agent/attachments', {
  method: 'POST',
  body: formData,
});
const { attachment } = await uploadResponse.json();
// attachment.url 是临时访问 URL

// 方式二：直接使用已有的签名 URL（如果有）
const imageUrl = 'https://storage.example.com/...';

2.2 构造 RunAgentInput

const runInput = {
  threadId: threadId,
  runId: `run-${Date.now()}`,
  state: {},
  messages: [
    {
      id: `msg-${Date.now()}`,
      role: 'user',
      content: [
        { type: 'text', text: '这张图片里有什么？' },
        { 
          type: 'binary', 
          mimeType: 'image/png',
          url: attachment.url  // 签名 URL
        }
      ]
    }
  ],
  tools: [],  // 或传入工具定义
  context: [],
  forwardedProps: {}
};

2.3 发送请求

const response = await fetch('/api/v1/agent/runs', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(runInput)
});
// 返回 202 Accepted
const { taskId, threadId, runId, created } = await response.json();

---

3. 事件流渲染

3.1 订阅事件

class AgentEventHandler {
  constructor(threadId) {
    this.eventSource = new EventSource(`/api/v1/agent/runs/${threadId}/events`);
    this.setupListeners();
  }

  setupListeners() {
    this.eventSource.addEventListener('run.started', this.onRunStarted);
    this.eventSource.addEventListener('step.start', this.onStepStart);
    this.eventSource.addEventListener('text.delta', this.onTextDelta);
    this.eventSource.addEventListener('text.message.start', this.onTextStart);
    this.eventSource.addEventListener('tool.call.start', this.onToolStart);
    this.eventSource.addEventListener('tool.call.args', this.onToolArgs);
    this.eventSource.addEventListener('tool.call.result', this.onToolResult);
    this.eventSource.addEventListener('text.message.end', this.onTextEnd);
    this.eventSource.addEventListener('run.finished', this.onRunFinished);
    this.eventSource.addEventListener('run.error', this.onRunError);
  }
}

3.2 事件处理与渲染

text.delta - 文本增量

onTextDelta(event) {
  const data = JSON.parse(event.data);
  const { delta, messageId } = data.data;
  // 追加到对应消息的文本内容
  appendTextToMessage(messageId, delta);
}

tool.result - 工具结果

onToolResult(event) {
  const data = JSON.parse(event.data);
  const { toolAgentOutput, toolCallId } = data.data;
  
  if (toolAgentOutput?.ui_hints) {
    // 后端返回的是 UiHints，需要编译为 UiSchemaRenderer
    const uiSchema = compileUiHints(toolAgentOutput.ui_hints);
    renderUiSchema(uiSchema, toolCallId);
  } else {
    // 纯文本结果
    renderText(toolAgentOutput.result_summary, toolCallId);
  }
}

text.end - Worker 完成

onTextEnd(event) {
  const data = JSON.parse(event.data);
  const { workerAgentOutput, inputTokens, outputTokens, cost } = data.data;
  
  if (workerAgentOutput?.ui_hints) {
    // 渲染 UiHints → UiSchemaRenderer
    const uiSchema = compileUiHints(workerAgentOutput.ui_hints);
    renderUiSchema(uiSchema, messageId);
  } else {
    // 纯文本回复
    renderText(workerAgentOutput?.answer, messageId);
  }
  
  // 显示使用统计
  showUsageStats({ inputTokens, outputTokens, cost });
}

---

4. History 对话历史渲染

4.1 获取历史

const response = await fetch(
  `/api/v1/agent/history?threadId=${threadId}&before=${date}`
);
const { messages } = await response.json();

4.2 渲染历史消息

messages.forEach(msg => {
  switch (msg.role) {
    case 'user':
      renderUserMessage(msg);
      break;
    case 'assistant':
      if (msg.uiSchema) {
        // 直接渲染 UiSchemaRenderer（后端已编译好）
        renderUiSchema(msg.uiSchema);
      } else {
        // 纯文本
        renderText(msg.content);
      }
      break;
    case 'tool':
      // tool 结果也可能有 uiSchema
      if (msg.uiSchema) {
        renderUiSchema(msg.uiSchema);
      } else {
        renderText(msg.content);
      }
      break;
  }
});

---

5. UiSchemaRenderer 渲染

5.1 数据结构

后端 ui_compiler.compile() 输出的结构：

interface UiSchemaRenderer {
  version: "2.0";
  locale: string;
  status: "info" | "success" | "warning" | "error" | "pending";
  theme: "default" | "light" | "dark";
  meta?: {
    requestId?: string;
    toolId?: string;
    traceId?: string;
    userId?: string;
  };
  root: UiLayoutNode;
}

// 布局节点
interface UiStackNode {
  type: "stack";
  direction: "vertical" | "horizontal";
  gap?: number;
  appearance: "plain" | "card" | "section";
  status?: UiStatus;
  align?: LayoutAlign;
  justify?: LayoutJustify;
  wrap?: boolean;
  children: UiNode[];
}

interface UiGridNode {
  type: "grid";
  columns: number;
  gap?: number;
  appearance: LayoutAppearance;
  status?: UiStatus;
  children: UiNode[];
}

// 基础节点
type UiNode = 
  | UiTextNode      // 文本
  | UiIconNode      // 图标
  | UiBadgeNode     // 标签
  | UiButtonNode    // 按钮
  | UiKvNode        // 键值对
  | UiDividerNode;  // 分割线

// 文本节点
interface UiTextNode {
  type: "text";
  content: string;
  format: "plain" | "markdown";
  role: "title" | "subtitle" | "body" | "caption" | "code";
  status?: UiStatus;
  maxLines?: number;
  visible?: boolean;
}

// 按钮节点
interface UiButtonNode {
  type: "button";
  label: string;
  style: "primary" | "secondary" | "ghost" | "danger";
  disabled?: boolean;
  icon?: UiIconSpec;
  action: UiActionPayload;
}

// 键值对节点
interface UiKvNode {
  type: "kv";
  items: UiKvItem[];
  columns?: number;
}

interface UiKvItem {
  key: string;
  label?: string;
  value: any;
  copyable?: boolean;
}

// Action payloads
type UiActionPayload = 
  | { type: "navigation"; path: string; params?: Record<string, any> }
  | { type: "url"; url: string; target?: "_self" | "_blank" }
  | { type: "event"; event: string; payload?: Record<string, any> }
  | { type: "tool"; toolId: string; params?: Record<string, any> }
  | { type: "copy"; content: string; successMessage?: string }
  | { type: "payload"; payload: Record<string, any>; submitTo?: string };

5.2 统一渲染器实现

class UiSchemaRendererWidget extends StatelessWidget {
  final UiSchemaRenderer schema;
  
  @override
  Widget build(BuildContext context) {
    return renderNode(schema.root);
  }
  
  Widget renderNode(UiNode node) {
    switch (node.type) {
      case 'text':
        return renderTextNode(node as UiTextNode);
      case 'icon':
        return renderIconNode(node as UiIconNode);
      case 'badge':
        return renderBadgeNode(node as UiBadgeNode);
      case 'button':
        return renderButtonNode(node as UiButtonNode);
      case 'kv':
        return renderKvNode(node as UiKvNode);
      case 'divider':
        return renderDividerNode(node as UiDividerNode);
      case 'stack':
        return renderStackNode(node as UiStackNode);
      case 'grid':
        return renderGridNode(node as UiGridNode);
      default:
        return SizedBox.shrink();
    }
  }
}

5.3 事件渲染 vs History 渲染一致性

关键点：两者使用同一个渲染器

数据来源	数据格式	处理方式
events text.end	workerAgentOutput.ui_hints	compile → UiSchemaRenderer → 渲染
events tool.result	toolAgentOutput.ui_hints	compile → UiSchemaRenderer → 渲染
history assistant	msg.uiSchema	直接渲染
history tool	msg.uiSchema	直接渲染

编译函数（前端需要实现）：

// 将后端 UiHints 编译为 UiSchemaRenderer
function compileUiHints(hints) {
  // 这里可以调用后端的 compile 接口
  // 或者在前端实现相同的编译逻辑
  return fetch('/api/v1/agent/compile-ui-hints', {
    method: 'POST',
    body: JSON.stringify(hints)
  }).then(r => r.json());
}

---

6. UiHints 介绍（可选）

如果前端需要自己处理 UiHints，以下是结构：

UiHintsPayload

interface UiHintsPayload {
  version: "2.1";
  intent: "message" | "data" | "list" | "status" | "form" | "mixed";
  status: "info" | "success" | "warning" | "error" | "pending";
  
  title?: string;
  description?: string;
  body?: string;
  bodyFormat?: "plain" | "markdown";
  
  items?: UiHintKvItem[];
  listItems?: UiHintListItem[];
  sections?: UiHintSection[];
  actions?: UiHintAction[];
  icon?: UiHintIcon;
  meta?: Record<string, any>;
}

注意：History API 返回的消息已经包含了编译好的 uiSchema，前端不需要自己编译。

---

7. 总结

1. 多模态：前端上传附件 → 获取签名 URL → 构造 RunAgentInput → POST /runs
2. 实时事件：订阅 /runs/{threadId}/events → 处理 text.delta/tool.result/text.end → 渲染 UiSchema
3. 历史消息：GET /history → 遍历消息 → 有 uiSchema 则渲染，否则渲染纯文本
4. 一致性：events 和 history 使用同一个 UiSchemaRenderer 渲染组件