social-app/docs/protocols/agent-chat-messages.md

# Agent Chat Messages Protocol

> **NOTE**: This document is the single source of truth. All implementations must follow this specification.

## Overview

Chat messages in agent conversations with metadata for tracking and telemetry.

## Version

- **Current**: `1.0`
- **Status**: Active

---

## Message Metadata

```typescript
interface AgentChatMessageMetadata {
  run_id?: string;       // Unique run identifier
  stage?: string;       // Processing stage (e.g., "intent", "execution")
  latency_ms?: number;  // Processing latency in milliseconds
  message_id?: string;  // Message identifier
  [key: string]: any;   // Additional custom fields allowed
}
```

---

## Database Field

| Field | Type | Description |
|-------|------|-------------|
| metadata | jsonb | Message metadata including run_id, stage, latency_ms, message_id |

---

## JSON Examples

### Basic Message Metadata

```json
{
  "run_id": "run_1773286460762"
}
```

### Stage Completion Metadata

```json
{
  "run_id": "run_1773286460762",
  "stage": "intent",
  "latency_ms": 2610,
  "message_id": "intent-run_1773286460762"
}
```

### Tool Execution Metadata

```json
{
  "run_id": "run_1773287162123",
  "stage": "tool_execution",
  "latency_ms": 1500,
  "message_id": "tool_run_abc123",
  "tool_name": "calendar_create_event"
}
```

---

## User Message Attachments

### Overview

When a user sends a message with binary attachments (e.g., images), the frontend uploads the file to storage first, then sends a signed URL to the backend. The backend parses the signed URL to extract storage metadata and persists it with the message.

### Flow

```
Frontend                          Backend
────────────────────────────────────────────────────────────────
1. Upload file
   POST /api/v1/agent/attachments
   ──────────────────────────────>
   <──────────────────────────────
   {bucket, path, mime_type, url: signed_url}

2. Send message with binary block
   POST /api/v1/agent/run
   content: [
     {type: "text", text: "..."},
     {type: "binary", mimeType: "image/jpeg", url: signed_url}
   ]
   ──────────────────────────────>

3. Backend parses signed URL
   parse_signed_url(url) → {bucket, path}

4. Persist to database
   metadata.user_message_attachments = {bucket, path, mime_type}

5. Return history (GET /history)
   <──────────────────────────────
   messages: [{
     role: "user",
     content: [
       {type: "text", text: "..."},
       {type: "binary", mimeType: "image/jpeg", url: new_signed_url}
     ]
   }]
```

### Signed URL Format

Supabase signed URL format:
```
https://{project}.supabase.co/storage/v1/object/sign/{bucket}/{path}?token={jwt}
```

Backend parses to extract:
- `bucket`: URL path segment after `/sign/`
- `path`: Remaining path after bucket

### Metadata Schema

```typescript
interface UserMessageAttachments {
  bucket: string;      // Storage bucket name
  path: string;        // Object storage path
  mime_type: string;  // MIME type (e.g., "image/jpeg")
}

interface AgentChatMessageMetadata {
  // ... existing fields
  user_message_attachments?: UserMessageAttachments;
}
```

### Database Storage

| Field | Type | Description |
|-------|------|-------------|
| metadata | jsonb | Contains user_message_attachments with bucket, path, mime_type |

### Example

**Request (POST /run):**
```json
{
  "threadId": "thread-123",
  "runId": "run-456",
  "messages": [
    {
      "id": "msg-1",
      "role": "user",
      "content": [
        {"type": "text", "text": "帮我看看这张图"},
        {
          "type": "binary",
          "mimeType": "image/jpeg",
          "url": "https://xxx.supabase.co/storage/v1/object/sign/agent-files/agent-inputs/u/t/r/img.jpg?token=xxx"
        }
      ]
    }
  ]
}
```

**Stored Metadata:**
```json
{
  "user_message_attachments": {
    "bucket": "agent-files",
    "path": "agent-inputs/u/t/r/img.jpg",
    "mime_type": "image/jpeg"
  }
}
```

**History Response (GET /history):**
```json
{
  "messages": [
    {
      "id": "msg-1",
      "role": "user",
      "content": [
        {"type": "text", "text": "帮我看看这张图"},
        {
          "type": "binary",
          "mimeType": "image/jpeg",
          "url": "https://xxx.supabase.co/storage/v1/object/sign/agent-files/agent-inputs/u/t/r/img.jpg?token=yyy"
        }
      ]
    }
  ]
}
```