fix(agent): stabilize live e2e tool execution and loop isolation
This commit is contained in:
zl-q
@@ -0,0 +1,118 @@
|
||||
# Bug - 后端工具事件与前端中断稳定性
|
||||
|
||||
**日期**: 2026-03-08
|
||||
**范围**: `backend/src/core/agent`
|
||||
|
||||
## 状态
|
||||
|
||||
- [x] Bug 1 已修复: 后端工具调用事件未转发
|
||||
- [x] Bug 2 已修复: history 未过滤负 seq 内部消息
|
||||
- [ ] Bug 3 调查中: live 前端工具中断不稳定
|
||||
|
||||
---
|
||||
|
||||
## Bug 1 - 后端工具调用不转发事件给前端（已修复）
|
||||
|
||||
### 修复
|
||||
|
||||
- `run_service.py` 现在会消费 runtime 的 `tool_calls`（`target=backend`）并发出:
|
||||
- `TOOL_CALL_START`
|
||||
- `TOOL_CALL_ARGS`
|
||||
- `TOOL_CALL_END`
|
||||
- `TOOL_CALL_RESULT`
|
||||
- 同时落库 `role=TOOL` 消息，metadata 使用 `tool_result`。
|
||||
|
||||
### 验证
|
||||
|
||||
- `backend/tests/unit/core/agent/test_run_resume_service.py::test_run_service_executes_backend_calendar_tool_and_emits_result`
|
||||
|
||||
---
|
||||
|
||||
## Bug 2 - seq 设计缺陷与 history 暴露内部消息（已修复）
|
||||
|
||||
### 修复
|
||||
|
||||
- `SessionRepository.next_message_seq()` 支持 `mode`:
|
||||
- `public`: 仅基于正序号递增
|
||||
- `internal`: 基于负序号递减
|
||||
- `v1/agent/repository.py` history 查询增加 `seq > 0` 过滤。
|
||||
|
||||
### 验证
|
||||
|
||||
- `backend/tests/unit/v1/agent/test_repository.py::test_get_history_day_filters_out_negative_seq_messages`
|
||||
|
||||
---
|
||||
|
||||
## Bug 3 - live 前端工具中断不稳定（调查中）
|
||||
|
||||
### 现象
|
||||
|
||||
- `test_agent_live_front_tool_interrupt_resume_continue` 偶发或持续失败。
|
||||
- 失败点: `pending_tool_call_id` 为 `None`。
|
||||
|
||||
### 已采集证据
|
||||
|
||||
- 输入文本已明确要求调用工具。
|
||||
- 前端工具描述已注入到 prompt，且 execution 阶段可见工具列表。
|
||||
- 部分失败样本中，模型在 execution 输出里给出“需要审批”的文字/结构化说明，但没有真正触发工具调用事件。
|
||||
- 常见 execution_data 形态:
|
||||
- `tool_used/tool_name`
|
||||
- `approval_status/approval_required`
|
||||
- `target_route/target`
|
||||
- 但无真实 tool call 事件。
|
||||
|
||||
### 当前判断
|
||||
|
||||
- 问题不在“工具未注入”。
|
||||
- 主要是模型在 execution 阶段把“应调用工具”退化为“文本说明审批状态”，导致 runtime 无法拿到 pending call。
|
||||
|
||||
### 已做改进（非硬编码兜底）
|
||||
|
||||
- 提示词集中化到 `core/agent/prompt/runtime_stage_prompts.py`。
|
||||
- execution prompt 增加规则: 工具可满足请求时必须通过 runtime 工具接口调用，不可伪造工具结果文本。
|
||||
- pending 提取逻辑增强以兼容 `approval_required/target` 变体结构。
|
||||
- `DynamicRoutingTool._run` 改为接受 `**kwargs`，兼容 CrewAI 直接参数调用（之前仅收 `payload`，会导致 `unexpected keyword argument`）。
|
||||
- execution 阶段关闭 `output_pydantic` 强约束，避免 structured output 过早收敛影响 ReAct 工具动作循环。
|
||||
|
||||
### 最新验证（2026-03-08 晚）
|
||||
|
||||
- 前端中断 live 用例仍失败：
|
||||
- `AGENT_LIVE_E2E=1 uv run pytest backend/tests/e2e/test_agent_live_flow.py::test_agent_live_front_tool_interrupt_resume_continue -v -rs`
|
||||
- 结果：`pending_tool_call_id = null`
|
||||
- assistant 文本会声称“已触发审批/待确认”，但 runtime 仍未捕获真实 tool call。
|
||||
- 后端工具 live 用例本次环境未能执行到断言：
|
||||
- `AGENT_LIVE_E2E=1 uv run pytest backend/tests/e2e/test_agent_live_flow.py::test_agent_live_image_calendar_tool_persistence -v -rs`
|
||||
- `Tool result storage unavailable` 已定位并修复（测试初始化顺序问题，不是 Docker Storage 服务故障）
|
||||
- 当前新失败为业务断言：未创建 `schedule_items`
|
||||
- 非 live 证据：
|
||||
- `uv run pytest backend/tests/unit/core/agent/test_crewai_runtime_tools.py -q` PASS（验证 front tool kwargs 可进入 runtime）
|
||||
- `uv run pytest backend/tests/unit/core/agent/test_run_resume_service.py -q` PASS（后端工具链路单测通过）
|
||||
|
||||
### 后续建议
|
||||
|
||||
1. 为 live 失败样本继续沉淀 execution 原始输出分型统计。
|
||||
2. 评估在 execution stage 增加 CrewAI guardrail: 若 NEEDS_EXECUTION 且零 tool call，则判为无效输出并重试。
|
||||
3. 若仍不稳定，考虑升级模型或为关键路径启用更强结构化调用策略。
|
||||
4. 补充可观测性：在 execution 阶段记录“注入工具名列表 + Crew 原始 action 文本片段（脱敏）”，用于区分“未注入”与“注入后未 act”。
|
||||
|
||||
---
|
||||
|
||||
## 额外排查结论（CrewAI tools 与 Storage）
|
||||
|
||||
### A) CrewAI tools 机制对齐结论
|
||||
|
||||
- 官方 tools 文档要求 `BaseTool` 的 `args_schema` 与 `_run` 参数语义一致，示例为 `_run(self, argument: str)`。
|
||||
- CrewAI 执行器在 ReAct 模式下依赖 `Action / Action Input` 文本被 parser 解析后才会真正执行工具。
|
||||
- 我们此前 `_run(self, payload: dict)` 与实际运行时 kwargs 形态存在不匹配风险，已改为 `_run(self, **kwargs)` 兼容调用。
|
||||
- execution 阶段若过度强调“直接输出严格 JSON”，会与 ReAct 工具动作循环冲突，已在 prompt 中补充明确的 `Action` / `Action Input` 约束。
|
||||
|
||||
### B) Tool result storage unavailable 根因
|
||||
|
||||
- 根因不是 Supabase Docker Storage 宕机；`docker compose ps` 显示 `supabase-storage` healthy。
|
||||
- 真实原因是 live 测试在 `supabase_service.initialize()` 之前调用 `create_tool_result_storage()`，导致 admin client 尚未初始化而返回 `None`。
|
||||
- 已修复测试顺序：先初始化 Supabase，再创建 storage。
|
||||
|
||||
### C) 现阶段阻塞
|
||||
|
||||
- 后端图片场景还暴露出 AG-UI multimodal 输入兼容问题：`type=image` 不符合当前 `RunAgentInput`（期望 `binary`）。
|
||||
- 已修复为 `binary` 输入并在 `agui_input` 增加 `binary` 解析兼容；用例不再因 payload 校验失败而提前终止。
|
||||
@@ -0,0 +1,129 @@
|
||||
# Runtime Refactor and Prompt Centralization Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Refactor CrewAI runtime into reusable modules, centralize all prompt text under `core/agent/prompt`, and diagnose flaky front-tool interrupt behavior without adding hardcoded runtime heuristics.
|
||||
|
||||
**Architecture:** Keep `runtime.py` as a thin facade and move parsing/tool/prompt composition/stage execution into cohesive modules. Prompt strings (including stage contracts and injected tool-context instructions) are generated exclusively by prompt-module functions. Keep behavior equivalent by default; only add diagnostic observability for flaky live scenario analysis.
|
||||
|
||||
**Tech Stack:** Python 3.12, FastAPI backend, CrewAI, Pydantic v2, pytest, ruff, basedpyright.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add prompt module and centralize all runtime prompt text
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/core/agent/prompt/__init__.py`
|
||||
- Create: `backend/src/core/agent/prompt/runtime_stage_prompts.py`
|
||||
- Modify: `backend/src/core/agent/infrastructure/crewai/runtime.py`
|
||||
- Test: `backend/tests/unit/core/agent/test_crewai_runtime.py`
|
||||
|
||||
**Step 1: Write failing test**
|
||||
- Add unit test asserting runtime uses prompt builder output (not inline literals) for stage description/contract/tool context.
|
||||
|
||||
**Step 2: Run test to verify it fails**
|
||||
- Run: `uv run pytest backend/tests/unit/core/agent/test_crewai_runtime.py::test_runtime_uses_prompt_module_for_stage_descriptions -q`
|
||||
- Expected: FAIL because runtime still composes inline strings.
|
||||
|
||||
**Step 3: Implement prompt module**
|
||||
- Add prompt functions:
|
||||
- `build_stage_output_contract(stage: str) -> str`
|
||||
- `build_stage_task_description(...) -> str`
|
||||
- `build_intent_multimodal_prompt(...) -> str`
|
||||
- Use mainstream prompt structure: role/objective/context/constraints/output-format.
|
||||
- Keep rules non-hardcoded and behavior-oriented, avoid keyword-triggered branching rules.
|
||||
|
||||
**Step 4: Wire runtime to prompt functions**
|
||||
- Replace inline prompt strings in runtime with prompt-module function calls.
|
||||
- Ensure no prompt literals remain in runtime except minimal wiring labels.
|
||||
|
||||
**Step 5: Run tests**
|
||||
- Run: `uv run pytest backend/tests/unit/core/agent/test_crewai_runtime.py -q`
|
||||
- Expected: PASS.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Split runtime into reusable modules and keep facade stable
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/core/agent/infrastructure/crewai/runtime_models.py`
|
||||
- Create: `backend/src/core/agent/infrastructure/crewai/runtime_parsers.py`
|
||||
- Create: `backend/src/core/agent/infrastructure/crewai/runtime_tools.py`
|
||||
- Create: `backend/src/core/agent/infrastructure/crewai/runtime_stage_runner.py`
|
||||
- Modify: `backend/src/core/agent/infrastructure/crewai/runtime.py`
|
||||
- Modify: `backend/src/core/agent/infrastructure/crewai/__init__.py` (if needed)
|
||||
- Test: `backend/tests/unit/core/agent/test_crewai_runtime.py`
|
||||
|
||||
**Step 1: Write failing test**
|
||||
- Add/adjust unit test that imports `CrewAIRuntime` facade and verifies existing contract (`execute`, `map_events`, `is_registered_backend_tool`) still works after split.
|
||||
|
||||
**Step 2: Run test to verify it fails**
|
||||
- Run: `uv run pytest backend/tests/unit/core/agent/test_crewai_runtime.py::test_runtime_facade_contract_stable_after_refactor -q`
|
||||
- Expected: FAIL before module split wiring.
|
||||
|
||||
**Step 3: Extract models/parsers/tools/stage-runner**
|
||||
- Move Pydantic result models to `runtime_models.py`.
|
||||
- Move parse/normalize helpers to `runtime_parsers.py`.
|
||||
- Move tool normalization, routing tool class, pending-front-tool extraction to `runtime_tools.py`.
|
||||
- Move `_run_stage_with_crewai` + usage extraction to `runtime_stage_runner.py`.
|
||||
|
||||
**Step 4: Keep runtime facade thin**
|
||||
- `runtime.py` retains orchestration flow and public API only.
|
||||
- Import and compose extracted modules; no behavior change intended.
|
||||
|
||||
**Step 5: Run tests**
|
||||
- Run: `uv run pytest backend/tests/unit/core/agent/test_crewai_runtime.py -q`
|
||||
- Expected: PASS.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Diagnose front-tool interrupt instability with explicit observability
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/core/agent/infrastructure/crewai/runtime.py`
|
||||
- Modify: `backend/src/core/agent/infrastructure/crewai/runtime_stage_runner.py`
|
||||
- Modify: `backend/tests/e2e/test_agent_live_flow.py`
|
||||
- Modify: `docs/bugs/2026-03-08-backend-tool-no-events.md`
|
||||
|
||||
**Step 1: Add failing/diagnostic assertion in live test path**
|
||||
- Extend test to capture and print structured diagnostics when `pending_tool_call_id` is `None`:
|
||||
- intent/execution raw+structured output
|
||||
- tool payload injected into prompts
|
||||
- captured tool calls list
|
||||
|
||||
**Step 2: Run targeted live test for evidence**
|
||||
- Run: `AGENT_LIVE_E2E=1 uv run pytest backend/tests/e2e/test_agent_live_flow.py::test_agent_live_front_tool_interrupt_resume_continue -v -rs`
|
||||
- Expected: still flaky/fail, but with actionable diagnostics.
|
||||
|
||||
**Step 3: Analyze evidence and apply non-hardcoded fix**
|
||||
- If input ambiguity: refine test input prompt text under test fixture.
|
||||
- If tool-description injection issue: fix prompt-builder injection logic.
|
||||
- Do not add keyword heuristics in runtime branching.
|
||||
|
||||
**Step 4: Re-run live targeted test**
|
||||
- Same command as Step 2.
|
||||
- Expected: improved stability or clearly documented unresolved root cause.
|
||||
|
||||
**Step 5: Update bug doc**
|
||||
- Add root-cause findings and next actions under Bug 3 section.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Full verification and hygiene
|
||||
|
||||
**Files:**
|
||||
- Modify (if needed): `backend/tests/unit/core/agent/test_run_resume_service.py`
|
||||
|
||||
**Step 1: Run impacted unit suites**
|
||||
- `uv run pytest backend/tests/unit/core/agent/test_crewai_runtime.py -q`
|
||||
- `uv run pytest backend/tests/unit/core/agent/test_run_resume_service.py -q`
|
||||
|
||||
**Step 2: Run lint/type checks**
|
||||
- `uv run ruff check backend/src/core/agent/prompt backend/src/core/agent/infrastructure/crewai backend/tests/unit/core/agent/test_crewai_runtime.py backend/tests/e2e/test_agent_live_flow.py`
|
||||
- `uv run basedpyright backend/src/core/agent/prompt backend/src/core/agent/infrastructure/crewai backend/tests/unit/core/agent/test_crewai_runtime.py`
|
||||
|
||||
**Step 3: Optional live regression pack (if env ready)**
|
||||
- `AGENT_LIVE_E2E=1 uv run pytest backend/tests/e2e/test_agent_live_flow.py -m live -v -rs`
|
||||
|
||||
**Step 4: Report residual risk**
|
||||
- If live still flaky, report exact failure mode and captured diagnostics (no workaround heuristics).
|
||||
Reference in New Issue
Block a user