feat(agent): 增强多模态链路与工具调用能力

This commit is contained in:
zl-q
2026-03-12 00:18:45 +08:00
parent 18db6c50e7
commit 21ba8e4a44
35 changed files with 2057 additions and 829 deletions
@@ -1,141 +0,0 @@
# Agent Multimodal Smoke Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** 完成 agent 三条主链路(runs/events/history)真实冒烟,并支持 RunAgentInput 图片信息在发送链路落 Supabase Storage、在 messages.metadata 持久化、在 history 返回中可渲染。
**Architecture:**`v1/agent` 服务层新增“用户消息持久化 + 图片附件上传”步骤:`enqueue_run` 时解析用户消息 content block,图片上传到 `config.storage.bucket`,将路径写入 `messages.metadata`。运行时继续通过 AgentScope pipeline 输出 AG-UI 事件,SSE 从 Redis stream 订阅,历史查询从 `messages` 回放并附带附件信息。
**Tech Stack:** FastAPI, SQLAlchemy AsyncSession, Supabase Storage Admin Client, Redis SSE stream, AG-UI, pytest/httpx。
---
### Task 1: 用户消息图片附件上传与落库
**Files:**
- Create: `backend/src/v1/agent/attachment_storage.py`
- Modify: `backend/src/v1/agent/service.py`
- Modify: `backend/src/v1/agent/repository.py`
- Test: `backend/tests/unit/v1/agent/test_service.py`
**Step 1: 写失败测试(RED**
```python
@pytest.mark.asyncio
async def test_enqueue_run_persists_user_message_with_uploaded_image_metadata() -> None:
...
```
**Step 2: 运行单测验证失败**
Run: `uv run pytest tests/unit/v1/agent/test_service.py::test_enqueue_run_persists_user_message_with_uploaded_image_metadata -q`
Expected: FAIL(缺少附件上传/metadata 持久化行为)
**Step 3: 最小实现(GREEN**
```python
class AgentAttachmentStorage:
async def upload_bytes(...):
...
class AgentService:
async def enqueue_run(...):
# 解析 user content blocks
# 上传图片到 storage
# repository 持久化 user message(metadata 包含 bucket/path)
...
```
**Step 4: 运行单测验证通过**
Run: `uv run pytest tests/unit/v1/agent/test_service.py::test_enqueue_run_persists_user_message_with_uploaded_image_metadata -q`
Expected: PASS
### Task 2: history 渲染附件路径
**Files:**
- Modify: `backend/src/v1/agent/repository.py`
- Test: `backend/tests/unit/v1/agent/test_repository.py`
**Step 1: 写失败测试(RED**
```python
@pytest.mark.asyncio
async def test_history_includes_user_message_attachments_from_metadata() -> None:
...
```
**Step 2: 运行测试验证失败**
Run: `uv run pytest tests/unit/v1/agent/test_repository.py::test_history_includes_user_message_attachments_from_metadata -q`
Expected: FAILhistory 尚未渲染 attachments
**Step 3: 最小实现(GREEN**
```python
if role == "user" and isinstance(metadata.get("attachments"), list):
payload["attachments"] = metadata["attachments"]
```
**Step 4: 运行测试验证通过**
Run: `uv run pytest tests/unit/v1/agent/test_repository.py::test_history_includes_user_message_attachments_from_metadata -q`
Expected: PASS
### Task 3: 真实冒烟 runs + SSE + history(含图片输入)
**Files:**
- Modify: `backend/tests/integration/v1/agent/test_sse_flow_live.py`
**Step 1: 写失败测试(RED**
```python
@pytest.mark.asyncio
@pytest.mark.live
async def test_agent_runs_events_history_live_with_image_input() -> None:
...
```
**Step 2: 运行 live 测试验证失败(实现前或环境不完整)**
Run: `AGENT_LIVE_INTEGRATION=1 AGENT_LIVE_EMAIL=... AGENT_LIVE_PASSWORD=... uv run pytest tests/integration/v1/agent/test_sse_flow_live.py::test_agent_runs_events_history_live_with_image_input -q -s`
Expected: FAIL(缺 metadata/path 或 history 不含附件)
**Step 3: 最小实现(GREEN**
```python
# live 测试流程:
# 1) 登录拿 token
# 2) POST /runs 发送 text + image(data)
# 3) SSE 订阅直到 RUN_FINISHED/RUN_ERROR
# 4) GET /runs/{thread_id}/history
# 5) SQL 校验 sessions/messages 字段与 metadata.attachments
```
**Step 4: 运行 live 测试验证通过**
Run: `AGENT_LIVE_INTEGRATION=1 AGENT_LIVE_EMAIL=... AGENT_LIVE_PASSWORD=... uv run pytest tests/integration/v1/agent/test_sse_flow_live.py::test_agent_runs_events_history_live_with_image_input -q -s`
Expected: PASS
### Task 4: 全量收口验证与安全门禁
**Files:**
- Modify (if needed): `backend/src/v1/agent/*`, `backend/tests/*`
**Step 1: 回归测试**
Run: `uv run pytest tests/unit/v1/agent tests/unit/core/agentscope tests/integration/v1/agent -q`
Expected: PASS
**Step 2: 静态检查**
Run: `uv run ruff check src/v1/agent src/core/agentscope tests/unit/v1/agent tests/integration/v1/agent`
Expected: PASS
Run: `uv run basedpyright src/v1/agent src/core/agentscope tests/unit/v1/agent tests/integration/v1/agent`
Expected: 0 errors
**Step 3: 评审门禁**
Run agents: `security-reviewer`, `refactor-cleaner`, `code-reviewer`
Expected: 无未解决 CRITICAL/HIGH
@@ -0,0 +1,69 @@
# Agent Multimodal Smoke Runbook
**Goal:** 固化 agent 三条主链路(runs/events/history)的真实冒烟标准与输入基线。
## 1. 覆盖范围
1. `POST /api/v1/agent/runs` - 接收多模态消息(文本+图片)
2. `GET /api/v1/agent/runs/{thread_id}/events` - SSE 事件流,事件名符合 AG-UI 标准(`RUN_STARTED``STEP_STARTED``TOOL_CALL_*``RUN_FINISHED`/`RUN_ERROR`
3. `GET /api/v1/agent/runs/{thread_id}/history` - 返回 `STATE_SNAPSHOT`,含 `attachments` metadata
4. `sessions/messages` 落库完整:message_count、tokens、cost、latency、title、metadata
5. tool result 存储:大 payload 写 storagemetadata 记录 `storage_bucket`/`storage_path`
6. storage bucket 来源:必须来自环境变量 `SOCIAL_STORAGE__BUCKET`
## 2. 固定测试输入
- 图片夹具:`backend/tests/fixtures/images/calendar_text_cn.png`
- 多模态消息:
- 文本:`"识别图片中的日历内容并调用 calendar.write 创建日程"`
- 图片:`{"type":"binary","data":"<base64>","mimeType":"image/png"}`
## 3. 账号与凭据
- 冒烟账号:`dagronl@126.com` / `123456`
- 通过环境变量注入:`AGENT_LIVE_EMAIL``AGENT_LIVE_PASSWORD`
## 4. 执行命令
```bash
AGENT_LIVE_INTEGRATION=1 \
AGENT_LIVE_EMAIL="dagronl@126.com" \
AGENT_LIVE_PASSWORD="123456" \
uv run pytest tests/integration/v1/agent/test_sse_flow_live.py::test_agent_runs_events_history_live_with_image_input -q -s
```
## 5. 结果记录模板
- `thread_id` / `run_id`
- `runs` 状态码与响应
- `events` 事件序列
- `history` 是否含 `attachments[].bucket/path/mimeType`
- `sessions` 字段:message_count / total_tokens / total_cost / status / title
- `messages` 字段:role / content / metadata / tokens / cost / latency
- `tool_result` 是否写 storage
## 6. 安全注意
- 禁止将密码/token 写入 git 跟踪文件
## 7. 已修复问题清单
| 问题 | 修复内容 |
|------|----------|
| bucket 写入失败回退 | 改为直接报错,禁止回退到硬编码 bucket |
| user.resolve 工具 | 新增按 email/name 解析 user_id |
| calendar.write 邀请参数 | 增加 invite 参数透传 |
| inbox_repository 缺失 | 修复 calendar runtime 依赖 |
| runtime 模型名拼接 | 修复无效 model name |
| 多模态透传 | runtime 透传 binary.data,不过滤为 `<omitted>` |
| sessions.title 生成 | 首条用户消息持久化时自动生成 |
| assistant latency 入库 | `messages.latency_ms` 列写入 |
| intent/execution 阶段消息落库 | 新增 `text.*``tool.result` 事件 |
| DIRECT_RESPONSE 早返回 | intent 判定后直接返回,不进入 report 阶段 |
## 8. 待修复问题(用户新增)
1. **意图/执行阶段 tokens/cost 入库** - 目前仅 report 阶段入库
2. **连续会话记忆测试** - 验证 session 是否从数据库读取历史上下文
3. **工具调用测试** - calendar 读/写/删/分享 + 用户查找 + 时间感知
4. **session 失败排查** - 找出最新失败原因并修复
@@ -1,583 +0,0 @@
# 日历邀请弹窗优化 Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** 优化日历邀请消息弹窗,显示完整信息(发送者名称 + 日历标题),使用公共弹窗组件替代所有旧弹窗代码
**Architecture:**
- 后端新增用户信息查询接口
- 前端创建公共弹窗组件 MessageActionSheet
- 删除所有旧的弹窗代码(好友请求、日历邀请),统一使用公共组件
**Tech Stack:** Flutter (Dart), FastAPI (Python)
---
### Task 1: 后端添加用户信息查询接口
**Files:**
- Modify: `backend/src/v1/users/router.py`
- Modify: `backend/src/v1/users/service.py`
- Modify: `backend/src/v1/users/repository.py`
**Step 1: 添加 repository 方法**
修改 `backend/src/v1/users/repository.py`,在 `UserRepository``SQLAlchemyUserRepository` 中已有 `get_by_user_id` 方法,确认存在。
**Step 2: 添加 service 方法**
修改 `backend/src/v1/users/service.py`,添加:
```python
async def get_user_by_id(self, user_id: UUID) -> UserBasicInfo:
from v1.friendships.schemas import UserBasicInfo
profile = await self._repository.get_by_user_id(user_id)
if not profile:
raise HTTPException(status_code=404, detail="User not found")
return UserBasicInfo(
id=str(profile.user_id),
username=profile.username,
avatar_url=profile.avatar_url,
)
```
**Step 3: 添加 router 接口**
修改 `backend/src/v1/users/router.py`,添加:
```python
@router.get("/{user_id}", response_model=UserBasicInfo)
async def get_user(
user_id: UUID,
service: Annotated[UserService, Depends(get_user_service)],
):
return await service.get_user_by_id(user_id)
```
**Step 4: 运行 lint 和 typecheck**
```bash
cd backend && uv run ruff check src/v1/users/ && uv run basedpyright src/v1/users/
```
**Step 5: 提交**
```bash
git add backend/src/v1/users/ && git commit -m "feat(users): add get user by id endpoint"
```
---
### Task 2: 前端添加用户 API 接口
**Files:**
- Modify: `apps/lib/features/users/data/users_api.dart`
- Modify: `apps/lib/core/di/injection.dart`
**Step 1: 添加 UserBasicInfo 类和 getById 方法**
修改 `apps/lib/features/users/data/users_api.dart`
```dart
class UserBasicInfo {
final String id;
final String username;
final String? avatarUrl;
UserBasicInfo({
required this.id,
required this.username,
this.avatarUrl,
});
factory UserBasicInfo.fromJson(Map<String, dynamic> json) {
return UserBasicInfo(
id: json['id'] as String,
username: json['username'] as String,
avatarUrl: json['avatar_url'] as String?,
);
}
}
class UsersApi {
final IApiClient _client;
static const _prefix = '/api/v1/users';
UsersApi(this._client);
// ... existing methods
Future<UserBasicInfo> getById(String userId) async {
final response = await _client.get('$_prefix/$userId');
return UserBasicInfo.fromJson(response.data);
}
}
```
**Step 2: 注册到 DI**
修改 `apps/lib/core/di/injection.dart`,添加:
```dart
sl.registerLazySingleton(() => UsersApi(sl<IApiClient>()));
```
**Step 3: 运行 flutter analyze**
```bash
cd apps && flutter analyze lib/features/users/
```
**Step 4: 提交**
```bash
git add apps/lib/features/users/ apps/lib/core/di/injection.dart && git commit -m "feat(users): add getById API and UserBasicInfo"
```
---
### Task 3: 创建公共弹窗组件 MessageActionSheet
**Files:**
- Create: `apps/lib/features/messages/ui/widgets/message_action_sheet.dart`
**Step 1: 创建弹窗组件**
创建 `apps/lib/features/messages/ui/widgets/message_action_sheet.dart`
```dart
import 'package:flutter/material.dart';
import '../../../../core/theme/design_tokens.dart';
import '../../../../shared/widgets/app_button.dart';
class MessageActionSheet extends StatelessWidget {
final String title;
final String? description;
final String? statusText;
final bool isReadOnly;
final VoidCallback? onAccept;
final VoidCallback? onDecline;
final IconData? icon;
final Color? iconColor;
const MessageActionSheet({
super.key,
required this.title,
this.description,
this.statusText,
this.isReadOnly = false,
this.onAccept,
this.onDecline,
this.icon,
this.iconColor,
});
@override
Widget build(BuildContext context) {
return Container(
width: double.infinity,
padding: const EdgeInsets.fromLTRB(24, 20, 24, 0),
decoration: const BoxDecoration(
color: AppColors.white,
borderRadius: BorderRadius.vertical(top: Radius.circular(24)),
),
child: Column(
mainAxisSize: MainAxisSize.min,
children: [
Container(
width: 40,
height: 4,
decoration: BoxDecoration(
color: AppColors.slate300,
borderRadius: BorderRadius.circular(2),
),
),
const SizedBox(height: 20),
if (icon != null) ...[
Container(
width: 72,
height: 72,
decoration: BoxDecoration(
color: (iconColor ?? AppColors.blue500).withValues(alpha: 0.1),
shape: BoxShape.circle,
),
child: Icon(icon, size: 32, color: iconColor ?? AppColors.blue500),
),
const SizedBox(height: 16),
],
Text(
title,
style: const TextStyle(
fontSize: 20,
fontWeight: FontWeight.w600,
color: AppColors.slate900,
),
textAlign: TextAlign.center,
),
if (description != null && description!.isNotEmpty) ...[
const SizedBox(height: 8),
Text(
description!,
style: const TextStyle(fontSize: 14, color: AppColors.slate500),
textAlign: TextAlign.center,
),
],
if (statusText != null) ...[
const SizedBox(height: 16),
Container(
padding: const EdgeInsets.symmetric(horizontal: 12, vertical: 6),
decoration: BoxDecoration(
color: AppColors.slate100,
borderRadius: BorderRadius.circular(16),
),
child: Text(
statusText!,
style: const TextStyle(fontSize: 14, color: AppColors.slate600),
),
),
],
const SizedBox(height: 24),
if (!isReadOnly) ...[
Row(
children: [
Expanded(
child: AppButton(
text: '拒绝',
isOutlined: true,
onPressed: () {
Navigator.pop(context);
onDecline?.call();
},
),
),
const SizedBox(width: AppSpacing.md),
Expanded(
child: AppButton(
text: '接受',
onPressed: () {
Navigator.pop(context);
onAccept?.call();
},
),
),
],
),
],
SizedBox(height: MediaQuery.of(context).padding.bottom + 12),
],
),
);
}
}
```
**Step 2: 运行 flutter analyze**
```bash
cd apps && flutter analyze lib/features/messages/ui/widgets/message_action_sheet.dart
```
**Step 3: 提交**
```bash
git add apps/lib/features/messages/ui/widgets/message_action_sheet.dart && git commit -m "feat(messages): add MessageActionSheet component"
```
---
### Task 4: 重构消息列表页面,使用公共组件并删除旧代码
**Files:**
- Modify: `apps/lib/features/messages/ui/screens/message_invite_list_screen.dart`
**Step 1: 添加依赖和字段**
在文件顶部添加:
```dart
import '../../../users/data/users_api.dart';
import '../widgets/message_action_sheet.dart';
```
`_MessageInviteListScreenState` 中添加:
```dart
late final UsersApi _usersApi;
```
`initState` 中添加:
```dart
_usersApi = sl<UsersApi>();
```
**Step 2: 添加获取日历邀请信息方法**
```dart
Future<(String calendarTitle, String senderName)?> _getCalendarInviteInfo(
InboxMessageResponse message,
) async {
if (message.scheduleItemId == null || message.senderId == null) {
return null;
}
try {
final calendar = await _calendarApi.getById(message.scheduleItemId!);
final sender = await _usersApi.getById(message.senderId!);
return (calendar.title, sender.username);
} catch (e) {
return null;
}
}
```
**Step 3: 替换日历邀请弹窗方法**
删除旧的 `_showCalendarInviteSheet` 方法,替换为:
```dart
Future<void> _showCalendarInviteSheet(InboxMessageResponse message) async {
final itemId = message.scheduleItemId;
if (itemId == null) return;
final info = await _getCalendarInviteInfo(message);
final title = info != null
? '${info.$2} 邀请你加入日历'
: '日历邀请';
final description = info?.$1;
if (!mounted) return;
showModalBottomSheet<void>(
context: context,
backgroundColor: Colors.transparent,
builder: (ctx) => MessageActionSheet(
title: title,
description: description,
icon: Icons.calendar_today,
iconColor: AppColors.blue500,
onAccept: () async {
try {
await _calendarApi.acceptSubscription(itemId);
await _inboxApi.markAsRead(message.id);
if (mounted) {
Toast.show(context, '已接受', type: ToastType.success);
_loadMessages();
}
} catch (e) {
if (mounted) {
Toast.show(context, '操作失败', type: ToastType.error);
}
}
},
onDecline: () async {
try {
await _calendarApi.rejectSubscription(itemId);
await _inboxApi.markAsRead(message.id);
if (mounted) {
Toast.show(context, '已拒绝', type: ToastType.success);
_loadMessages();
}
} catch (e) {
if (mounted) {
Toast.show(context, '操作失败', type: ToastType.error);
}
}
},
),
);
}
```
**Step 4: 添加已读日历邀请弹窗方法**
```dart
Future<void> _showCalendarInviteReadOnlySheet(InboxMessageResponse message) async {
final itemId = message.scheduleItemId;
if (itemId == null) return;
final info = await _getCalendarInviteInfo(message);
final title = info != null
? '${info.$2} 邀请你加入日历'
: '日历邀请';
final description = info?.$1;
final statusText = message.status.value == 'accepted' ? '已接受' : '已拒绝';
if (!mounted) return;
showModalBottomSheet<void>(
context: context,
backgroundColor: Colors.transparent,
builder: (ctx) => MessageActionSheet(
title: title,
description: description,
statusText: statusText,
isReadOnly: true,
icon: Icons.calendar_today,
iconColor: AppColors.blue500,
),
);
}
```
**Step 5: 替换好友请求弹窗方法**
删除旧的 `_showFriendRequestReadOnlySheet``_showFriendRequestActionSheet` 方法,替换为:
```dart
void _showFriendRequestSheet(MessageWithFriend item, {bool isReadOnly = false}) {
final message = item.message;
final friendRequest = item.friendRequest;
if (friendRequest == null) return;
final title = '${friendRequest.sender.username} 请求添加您为好友';
final description = message.content;
final statusText = isReadOnly
? (friendRequest.status == 'accepted'
? '已接受'
: friendRequest.status == 'rejected'
? '已拒绝'
: '已处理')
: null;
showModalBottomSheet<void>(
context: context,
backgroundColor: Colors.transparent,
isScrollControlled: true,
builder: (ctx) => MessageActionSheet(
title: title,
description: description,
statusText: statusText,
isReadOnly: isReadOnly,
icon: Icons.person_add_outlined,
iconColor: AppColors.emerald500,
onAccept: isReadOnly
? null
: () async {
await _processFriendRequest(item, accept: true);
},
onDecline: isReadOnly
? null
: () async {
await _processFriendRequest(item, accept: false);
},
),
);
}
```
**Step 6: 修改 _handleMessageTap 方法**
修改为调用新的统一方法:
```dart
case InboxMessageType.calendar:
final content = _parseCalendarContent(message.content);
if (content == null) return;
final type = content['type'] as String?;
if (type == 'invite') {
if (message.status.value == 'pending') {
await _showCalendarInviteSheet(message);
} else {
await _showCalendarInviteReadOnlySheet(message);
if (message.scheduleItemId != null && context.mounted) {
context.push('/calendar/events/${message.scheduleItemId}');
}
}
} else if (type == 'update') {
if (message.scheduleItemId != null) {
context.push('/calendar/events/${message.scheduleItemId}');
}
}
return;
case InboxMessageType.friendRequest:
if (item.friendRequest == null) {
Toast.show(context, '发送者信息加载失败,请下拉重试', type: ToastType.error);
return;
}
_showFriendRequestSheet(item, isReadOnly: message.isRead);
return;
```
**Step 7: 删除旧的 _FriendRequestSheet 类**
删除文件末尾的整个 `_FriendRequestSheet` 类(约605-749行)。
**Step 8: 运行 flutter analyze**
```bash
cd apps && flutter analyze lib/features/messages/ui/screens/message_invite_list_screen.dart
```
**Step 9: 提交**
```bash
git add apps/lib/features/messages/ && git commit -m "refactor(messages): use MessageActionSheet for all message types"
```
---
### Task 5: 删除日历消息卡片中的旧弹窗代码
**Files:**
- Modify: `apps/lib/features/messages/ui/widgets/calendar_message_card.dart`
**Step 1: 修改 CalendarInviteCard**
CalendarInviteCard 是用于列表展示的卡片,不需要显示弹窗。检查是否有不必要的硬编码,如果有则清理。
**Step 2: 运行 flutter analyze**
```bash
cd apps && flutter analyze lib/features/messages/ui/widgets/calendar_message_card.dart
```
**Step 3: 提交**
```bash
git add apps/lib/features/calendar_message_card.dart && git commit/messages/ui/widgets -f "chore(messages): clean up calendar message card"
```
---
### Task 6: 验证和测试
**Step 1: 运行完整测试**
```bash
cd apps && flutter test test/features/messages/
cd backend && uv run pytest tests/unit/v1/users/ -v
```
**Step 2: 手动测试场景**
1. 用户 A 发送日历邀请给用户 B
2. 用户 B 打开未读消息,点击日历邀请
3. 弹窗显示:"XXX 邀请你加入 [日历标题]"
4. 点击接受/拒绝
5. 用户 B 打开已读消息,点击日历邀请
6. 弹窗显示状态标签
7. 好友请求未读/已读都使用相同弹窗组件
---
## Summary
| Task | Description |
|------|-------------|
| 1 | 后端添加用户信息查询接口 `/api/v1/users/{user_id}` |
| 2 | 前端添加 UsersApi.getById 方法 |
| 3 | 创建公共弹窗组件 MessageActionSheet |
| 4 | 重构消息列表页面,删除旧弹窗代码,统一使用 MessageActionSheet |
| 5 | 清理日历消息卡片旧代码 |
| 6 | 验证测试 |
**Plan complete and saved to `docs/plans/2026-03-11-calendar-invite-sheet.md`. Two execution options:**
1. **Subagent-Driven (this session)** - I dispatch fresh subagent per task, review between tasks, fast iteration
2. **Parallel Session (separate)** - Open new session with executing-plans, batch execution with checkpoints
Which approach?