zl-q
433 lines
14 KiB
Markdown
433 lines
14 KiB
Markdown
# 前后端数据流通指南
|
||
|
||
本文档描述前端如何与后端 Agent 系统交互，以及如何渲染后端返回的 UI 数据。
|
||
|
||
---
|
||
|
||
## 1. 整体交互流程
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ 前端 │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ ┌──────────────┐ POST /runs ┌──────────────┐ │
|
||
│ │ 构造请求 │ ──────────────────▶│ 后端 │ │
|
||
│ │ (多模态数据) │ │ (任务入队) │ │
|
||
│ └──────────────┘◀────────────────── └──────────────┘ 202 Accepted │
|
||
│ │ taskId │
|
||
│ │ │
|
||
│ │ GET /runs/{threadId}/events │
|
||
│ │──────────────────────────────────────────────────────────▶ │
|
||
│ │ SSE 事件流 ◀───────────────── │
|
||
│ │ │
|
||
│ ┌─────┴──────────────────────────────────────────────────────────┐ │
|
||
│ │ 渲染事件 │ │
|
||
│ │ - text.delta: 追加文本 │ │
|
||
│ │ - tool.result: 渲染 toolAgentOutput.ui_hints → UiSchema │ │
|
||
│ │ - text.end: 渲染 workerAgentOutput.ui_hints → UiSchema │ │
|
||
│ └────────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ 后端 │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ POST /runs │
|
||
│ → 验证请求 │
|
||
│ → 任务入队列 (Taskiq) │
|
||
│ │
|
||
│ Worker 执行 │
|
||
│ → emit 事件到 Redis Stream │
|
||
│ → UiHints → ui_compiler.compile() → UiSchemaRenderer │
|
||
│ │
|
||
│ 持久化 │
|
||
│ → AgentChatMessage (含 uiSchema) │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 2. 多模态数据处理
|
||
|
||
### 2.1 前端上传附件流程
|
||
|
||
```javascript
|
||
// 方式一：先上传附件，获取签名 URL
|
||
const formData = new FormData();
|
||
formData.append('threadId', threadId);
|
||
formData.append('file', imageFile);
|
||
|
||
const uploadResponse = await fetch('/api/v1/agent/attachments', {
|
||
method: 'POST',
|
||
body: formData,
|
||
});
|
||
const { attachment } = await uploadResponse.json();
|
||
// attachment.url 是临时访问 URL
|
||
|
||
// 方式二：直接使用已有的签名 URL（如果有）
|
||
const imageUrl = 'https://storage.example.com/...';
|
||
```
|
||
|
||
### 2.2 构造 RunAgentInput
|
||
|
||
```javascript
|
||
const runInput = {
|
||
threadId: threadId,
|
||
runId: `run-${Date.now()}`,
|
||
state: {},
|
||
messages: [
|
||
{
|
||
id: `msg-${Date.now()}`,
|
||
role: 'user',
|
||
content: [
|
||
{ type: 'text', text: '这张图片里有什么？' },
|
||
{
|
||
type: 'binary',
|
||
mimeType: 'image/png',
|
||
url: attachment.url // 签名 URL
|
||
}
|
||
]
|
||
}
|
||
],
|
||
tools: [], // 或传入工具定义
|
||
context: [],
|
||
forwardedProps: {}
|
||
};
|
||
```
|
||
|
||
### 2.3 发送请求
|
||
|
||
```javascript
|
||
const response = await fetch('/api/v1/agent/runs', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(runInput)
|
||
});
|
||
// 返回 202 Accepted
|
||
const { taskId, threadId, runId, created } = await response.json();
|
||
```
|
||
|
||
---
|
||
|
||
## 3. 事件流渲染
|
||
|
||
### 3.1 订阅事件
|
||
|
||
```javascript
|
||
class AgentEventHandler {
|
||
constructor(threadId) {
|
||
this.eventSource = new EventSource(`/api/v1/agent/runs/${threadId}/events`);
|
||
this.setupListeners();
|
||
}
|
||
|
||
setupListeners() {
|
||
this.eventSource.addEventListener('run.started', this.onRunStarted);
|
||
this.eventSource.addEventListener('step.start', this.onStepStart);
|
||
this.eventSource.addEventListener('text.delta', this.onTextDelta);
|
||
this.eventSource.addEventListener('text.message.start', this.onTextStart);
|
||
this.eventSource.addEventListener('tool.call.start', this.onToolStart);
|
||
this.eventSource.addEventListener('tool.call.args', this.onToolArgs);
|
||
this.eventSource.addEventListener('tool.call.result', this.onToolResult);
|
||
this.eventSource.addEventListener('text.message.end', this.onTextEnd);
|
||
this.eventSource.addEventListener('run.finished', this.onRunFinished);
|
||
this.eventSource.addEventListener('run.error', this.onRunError);
|
||
}
|
||
}
|
||
```
|
||
|
||
### 3.2 事件处理与渲染
|
||
|
||
#### text.delta - 文本增量
|
||
|
||
```javascript
|
||
onTextDelta(event) {
|
||
const data = JSON.parse(event.data);
|
||
const { delta, messageId } = data.data;
|
||
// 追加到对应消息的文本内容
|
||
appendTextToMessage(messageId, delta);
|
||
}
|
||
```
|
||
|
||
#### tool.result - 工具结果
|
||
|
||
```javascript
|
||
onToolResult(event) {
|
||
const data = JSON.parse(event.data);
|
||
const { toolAgentOutput, toolCallId } = data.data;
|
||
|
||
if (toolAgentOutput?.ui_hints) {
|
||
// 后端返回的是 UiHints，需要编译为 UiSchemaRenderer
|
||
const uiSchema = compileUiHints(toolAgentOutput.ui_hints);
|
||
renderUiSchema(uiSchema, toolCallId);
|
||
} else {
|
||
// 纯文本结果
|
||
renderText(toolAgentOutput.result_summary, toolCallId);
|
||
}
|
||
}
|
||
```
|
||
|
||
#### text.end - Worker 完成
|
||
|
||
```javascript
|
||
onTextEnd(event) {
|
||
const data = JSON.parse(event.data);
|
||
const { workerAgentOutput, inputTokens, outputTokens, cost } = data.data;
|
||
|
||
if (workerAgentOutput?.ui_hints) {
|
||
// 渲染 UiHints → UiSchemaRenderer
|
||
const uiSchema = compileUiHints(workerAgentOutput.ui_hints);
|
||
renderUiSchema(uiSchema, messageId);
|
||
} else {
|
||
// 纯文本回复
|
||
renderText(workerAgentOutput?.answer, messageId);
|
||
}
|
||
|
||
// 显示使用统计
|
||
showUsageStats({ inputTokens, outputTokens, cost });
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 4. History 对话历史渲染
|
||
|
||
### 4.1 获取历史
|
||
|
||
```javascript
|
||
const response = await fetch(
|
||
`/api/v1/agent/history?threadId=${threadId}&before=${date}`
|
||
);
|
||
const { messages } = await response.json();
|
||
```
|
||
|
||
### 4.2 渲染历史消息
|
||
|
||
```javascript
|
||
messages.forEach(msg => {
|
||
switch (msg.role) {
|
||
case 'user':
|
||
renderUserMessage(msg);
|
||
break;
|
||
case 'assistant':
|
||
if (msg.uiSchema) {
|
||
// 直接渲染 UiSchemaRenderer（后端已编译好）
|
||
renderUiSchema(msg.uiSchema);
|
||
} else {
|
||
// 纯文本
|
||
renderText(msg.content);
|
||
}
|
||
break;
|
||
case 'tool':
|
||
// tool 结果也可能有 uiSchema
|
||
if (msg.uiSchema) {
|
||
renderUiSchema(msg.uiSchema);
|
||
} else {
|
||
renderText(msg.content);
|
||
}
|
||
break;
|
||
}
|
||
});
|
||
```
|
||
|
||
---
|
||
|
||
## 5. UiSchemaRenderer 渲染
|
||
|
||
### 5.1 数据结构
|
||
|
||
后端 `ui_compiler.compile()` 输出的结构：
|
||
|
||
```typescript
|
||
interface UiSchemaRenderer {
|
||
version: "2.0";
|
||
locale: string;
|
||
status: "info" | "success" | "warning" | "error" | "pending";
|
||
theme: "default" | "light" | "dark";
|
||
meta?: {
|
||
requestId?: string;
|
||
toolId?: string;
|
||
traceId?: string;
|
||
userId?: string;
|
||
};
|
||
root: UiLayoutNode;
|
||
}
|
||
|
||
// 布局节点
|
||
interface UiStackNode {
|
||
type: "stack";
|
||
direction: "vertical" | "horizontal";
|
||
gap?: number;
|
||
appearance: "plain" | "card" | "section";
|
||
status?: UiStatus;
|
||
align?: LayoutAlign;
|
||
justify?: LayoutJustify;
|
||
wrap?: boolean;
|
||
children: UiNode[];
|
||
}
|
||
|
||
interface UiGridNode {
|
||
type: "grid";
|
||
columns: number;
|
||
gap?: number;
|
||
appearance: LayoutAppearance;
|
||
status?: UiStatus;
|
||
children: UiNode[];
|
||
}
|
||
|
||
// 基础节点
|
||
type UiNode =
|
||
| UiTextNode // 文本
|
||
| UiIconNode // 图标
|
||
| UiBadgeNode // 标签
|
||
| UiButtonNode // 按钮
|
||
| UiKvNode // 键值对
|
||
| UiDividerNode; // 分割线
|
||
|
||
// 文本节点
|
||
interface UiTextNode {
|
||
type: "text";
|
||
content: string;
|
||
format: "plain" | "markdown";
|
||
role: "title" | "subtitle" | "body" | "caption" | "code";
|
||
status?: UiStatus;
|
||
maxLines?: number;
|
||
visible?: boolean;
|
||
}
|
||
|
||
// 按钮节点
|
||
interface UiButtonNode {
|
||
type: "button";
|
||
label: string;
|
||
style: "primary" | "secondary" | "ghost" | "danger";
|
||
disabled?: boolean;
|
||
icon?: UiIconSpec;
|
||
action: UiActionPayload;
|
||
}
|
||
|
||
// 键值对节点
|
||
interface UiKvNode {
|
||
type: "kv";
|
||
items: UiKvItem[];
|
||
columns?: number;
|
||
}
|
||
|
||
interface UiKvItem {
|
||
key: string;
|
||
label?: string;
|
||
value: any;
|
||
copyable?: boolean;
|
||
}
|
||
|
||
// Action payloads
|
||
type UiActionPayload =
|
||
| { type: "navigation"; path: string; params?: Record<string, any> }
|
||
| { type: "url"; url: string; target?: "_self" | "_blank" }
|
||
| { type: "event"; event: string; payload?: Record<string, any> }
|
||
| { type: "tool"; toolId: string; params?: Record<string, any> }
|
||
| { type: "copy"; content: string; successMessage?: string }
|
||
| { type: "payload"; payload: Record<string, any>; submitTo?: string };
|
||
```
|
||
|
||
### 5.2 统一渲染器实现
|
||
|
||
```dart
|
||
class UiSchemaRendererWidget extends StatelessWidget {
|
||
final UiSchemaRenderer schema;
|
||
|
||
@override
|
||
Widget build(BuildContext context) {
|
||
return renderNode(schema.root);
|
||
}
|
||
|
||
Widget renderNode(UiNode node) {
|
||
switch (node.type) {
|
||
case 'text':
|
||
return renderTextNode(node as UiTextNode);
|
||
case 'icon':
|
||
return renderIconNode(node as UiIconNode);
|
||
case 'badge':
|
||
return renderBadgeNode(node as UiBadgeNode);
|
||
case 'button':
|
||
return renderButtonNode(node as UiButtonNode);
|
||
case 'kv':
|
||
return renderKvNode(node as UiKvNode);
|
||
case 'divider':
|
||
return renderDividerNode(node as UiDividerNode);
|
||
case 'stack':
|
||
return renderStackNode(node as UiStackNode);
|
||
case 'grid':
|
||
return renderGridNode(node as UiGridNode);
|
||
default:
|
||
return SizedBox.shrink();
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### 5.3 事件渲染 vs History 渲染一致性
|
||
|
||
**关键点**：两者使用同一个渲染器
|
||
|
||
| 数据来源 | 数据格式 | 处理方式 |
|
||
|---------|---------|---------|
|
||
| events text.end | `workerAgentOutput.ui_hints` | compile → UiSchemaRenderer → 渲染 |
|
||
| events tool.result | `toolAgentOutput.ui_hints` | compile → UiSchemaRenderer → 渲染 |
|
||
| history assistant | `msg.uiSchema` | 直接渲染 |
|
||
| history tool | `msg.uiSchema` | 直接渲染 |
|
||
|
||
**编译函数**（前端需要实现）：
|
||
|
||
```javascript
|
||
// 将后端 UiHints 编译为 UiSchemaRenderer
|
||
function compileUiHints(hints) {
|
||
// 这里可以调用后端的 compile 接口
|
||
// 或者在前端实现相同的编译逻辑
|
||
return fetch('/api/v1/agent/compile-ui-hints', {
|
||
method: 'POST',
|
||
body: JSON.stringify(hints)
|
||
}).then(r => r.json());
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 6. UiHints 介绍（可选）
|
||
|
||
如果前端需要自己处理 UiHints，以下是结构：
|
||
|
||
### UiHintsPayload
|
||
|
||
```typescript
|
||
interface UiHintsPayload {
|
||
version: "2.1";
|
||
intent: "message" | "data" | "list" | "status" | "form" | "mixed";
|
||
status: "info" | "success" | "warning" | "error" | "pending";
|
||
|
||
title?: string;
|
||
description?: string;
|
||
body?: string;
|
||
bodyFormat?: "plain" | "markdown";
|
||
|
||
items?: UiHintKvItem[];
|
||
listItems?: UiHintListItem[];
|
||
sections?: UiHintSection[];
|
||
actions?: UiHintAction[];
|
||
icon?: UiHintIcon;
|
||
meta?: Record<string, any>;
|
||
}
|
||
```
|
||
|
||
**注意**：History API 返回的消息已经包含了编译好的 `uiSchema`，前端不需要自己编译。
|
||
|
||
---
|
||
|
||
## 7. 总结
|
||
|
||
1. **多模态**：前端上传附件 → 获取签名 URL → 构造 RunAgentInput → POST /runs
|
||
2. **实时事件**：订阅 /runs/{threadId}/events → 处理 text.delta/tool.result/text.end → 渲染 UiSchema
|
||
3. **历史消息**：GET /history → 遍历消息 → 有 uiSchema 则渲染，否则渲染纯文本
|
||
4. **一致性**：events 和 history 使用同一个 UiSchemaRenderer 渲染组件
|