# Agent Architecture Simplification Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Simplify agent configuration by removing redundant user_agents table and renaming user_agent_catalog to system_agents

**Architecture:** Delete user_agents table (including memories.agent_id dependency), rename user_agent_catalog to system_agents, update all references in code

**Tech Stack:** Python 3.11+, SQLAlchemy, Alembic, PostgreSQL

---

## Prerequisites

- [ ] Current branch: dev
- [ ] No uncommitted changes
- [ ] Docker services running (Supabase local)

## Task 1: Create Database Migration

**Files:**
- Create: `backend/alembic/versions/20260304_simplify_agent_architecture.py`

**Step 1: Create migration file**

Run: `cd backend && uv run alembic revision -m "simplify_agent_architecture"`

Expected: New migration file created with revision ID

**Step 2: Write migration upgrade logic**

Edit the generated migration file with this complete upgrade function:

```python
def upgrade() -> None:
    # 1. Delete memories.agent_id dependencies
    op.drop_constraint("fk_memories_agent_id", "memories", type_="foreignkey")
    op.drop_constraint("chk_memory_type_agent_id", "memories", type_="check")
    op.execute("DROP INDEX IF EXISTS ix_memories_agent_type_status")
    op.drop_column("memories", "agent_id")

    # 2. Delete user_agents table
    _drop_rls("user_agents")
    
    op.drop_constraint("fk_user_agents_updated_by", "user_agents", type_="foreignkey")
    op.drop_constraint("fk_user_agents_created_by", "user_agents", type_="foreignkey")
    op.drop_constraint("fk_user_agents_llm_id", "user_agents", type_="foreignkey")
    op.drop_constraint("fk_user_agents_user_id", "user_agents", type_="foreignkey")
    op.drop_constraint("chk_agent_type", "user_agents", type_="check")
    op.drop_constraint("uq_user_agents_user_id_agent_type", "user_agents", type_="unique")
    
    op.execute("DROP INDEX IF EXISTS ix_user_agents_status")
    op.execute("DROP INDEX IF EXISTS ix_user_agents_agent_type")
    
    op.drop_table("user_agents")

    # 3. Rename user_agent_catalog to system_agents
    _drop_rls("user_agent_catalog")
    
    op.rename_table("user_agent_catalog", "system_agents")
    
    op.execute(
        "ALTER TABLE system_agents RENAME CONSTRAINT fk_user_agent_catalog_llm_id "
        "TO fk_system_agents_llm_id"
    )
    op.execute(
        "ALTER TABLE system_agents RENAME CONSTRAINT chk_user_agent_catalog_status "
        "TO chk_system_agents_status"
    )
    
    _enable_rls("system_agents")

    # 4. Update trigger
    op.execute("DROP TRIGGER IF EXISTS on_auth_user_created ON auth.users")
    op.execute("DROP FUNCTION IF EXISTS public.create_profile_for_new_user()")
    
    op.execute("""
        CREATE OR REPLACE FUNCTION public.create_profile_for_new_user()
        RETURNS trigger
        LANGUAGE plpgsql
        SECURITY DEFINER
        SET search_path = public
        AS $$
        BEGIN
            INSERT INTO public.profiles (id, username, avatar_url, bio, settings, created_at, updated_at)
            VALUES (
                NEW.id,
                COALESCE(
                    NEW.raw_user_meta_data ->> 'username',
                    split_part(NEW.email, '@', 1),
                    'user_' || substring(NEW.id::text, 1, 8)
                ),
                NULL,
                NULL,
                '{"agent_prompts": {}}'::jsonb,
                now(),
                now()
            )
            ON CONFLICT (id) DO NOTHING;
            
            RETURN NEW;
        END;
        $$
    """)
    
    op.execute("""
        CREATE TRIGGER on_auth_user_created
            AFTER INSERT ON auth.users
            FOR EACH ROW EXECUTE FUNCTION public.create_profile_for_new_user()
    """)

    # 5. Update existing profiles.settings
    op.execute("""
        UPDATE profiles 
        SET settings = jsonb_set(
            COALESCE(settings, '{}'::jsonb),
            '{agent_prompts}',
            '{}'::jsonb
        )
        WHERE NOT settings ? 'agent_prompts'
    """)
```

**Step 3: Write migration downgrade logic**

Add this complete downgrade function:

```python
def downgrade() -> None:
    # 1. Revert trigger
    op.execute("DROP TRIGGER IF EXISTS on_auth_user_created ON auth.users")
    op.execute("DROP FUNCTION IF EXISTS public.create_profile_for_new_user()")
    
    op.execute("""
        CREATE OR REPLACE FUNCTION public.create_profile_for_new_user()
        RETURNS trigger
        LANGUAGE plpgsql
        SECURITY DEFINER
        SET search_path = public
        AS $$
        BEGIN
            INSERT INTO public.profiles (id, username, avatar_url, bio, settings, created_at, updated_at)
            VALUES (
                NEW.id,
                COALESCE(
                    NEW.raw_user_meta_data ->> 'username',
                    split_part(NEW.email, '@', 1),
                    'user_' || substring(NEW.id::text, 1, 8)
                ),
                NULL,
                NULL,
                '{}'::jsonb,
                now(),
                now()
            )
            ON CONFLICT (id) DO NOTHING;

            INSERT INTO public.user_agents (id, user_id, llm_id, agent_type, config, status, created_by, updated_by)
            SELECT 
                gen_random_uuid(),
                NEW.id,
                uac.llm_id,
                uac.agent_type,
                uac.config,
                uac.status,
                NEW.id,
                NEW.id
            FROM public.user_agent_catalog uac;
            
            RETURN NEW;
        END;
        $$
    """)
    
    op.execute("""
        CREATE TRIGGER on_auth_user_created
            AFTER INSERT ON auth.users
            FOR EACH ROW EXECUTE FUNCTION public.create_profile_for_new_user()
    """)

    # 2. Revert rename: system_agents -> user_agent_catalog
    _drop_rls("system_agents")
    
    op.rename_table("system_agents", "user_agent_catalog")
    
    op.execute(
        "ALTER TABLE user_agent_catalog RENAME CONSTRAINT fk_system_agents_llm_id "
        "TO fk_user_agent_catalog_llm_id"
    )
    op.execute(
        "ALTER TABLE user_agent_catalog RENAME CONSTRAINT chk_system_agents_status "
        "TO chk_user_agent_catalog_status"
    )
    
    _enable_rls("user_agent_catalog")

    # 3. Recreate user_agents table
    op.create_table(
        "user_agents",
        sa.Column("id", sa.UUID(), nullable=False),
        sa.Column("user_id", sa.UUID(), nullable=False),
        sa.Column("llm_id", sa.UUID(), nullable=False),
        sa.Column("agent_type", sa.String(length=20), nullable=False),
        sa.Column(
            "config",
            postgresql.JSONB(astext_type=sa.Text()),
            server_default="{}",
            nullable=False,
        ),
        sa.Column("status", sa.String(length=20), nullable=False),
        sa.Column("created_by", sa.UUID(), nullable=True),
        sa.Column("updated_by", sa.UUID(), nullable=True),
        sa.Column(
            "created_at",
            sa.DateTime(timezone=True),
            server_default=sa.text("now()"),
            nullable=False,
        ),
        sa.Column(
            "updated_at",
            sa.DateTime(timezone=True),
            server_default=sa.text("now()"),
            nullable=False,
        ),
        sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True),
        sa.PrimaryKeyConstraint("id"),
    )
    
    op.create_unique_constraint(
        "uq_user_agents_user_id_agent_type", 
        "user_agents", 
        ["user_id", "agent_type"]
    )
    
    op.execute(
        "CREATE INDEX ix_user_agents_agent_type ON user_agents (agent_type)"
    )
    op.execute(
        "CREATE INDEX ix_user_agents_status ON user_agents (status)"
    )
    
    op.execute(
        "ALTER TABLE user_agents ADD CONSTRAINT chk_agent_type "
        "CHECK (agent_type IN ('INTENT_RECOGNITION', 'TASK_EXECUTION', 'RESULT_REPORTING'))"
    )
    
    op.create_foreign_key(
        "fk_user_agents_user_id",
        "user_agents",
        "users",
        ["user_id"],
        ["id"],
        referent_schema="auth",
        ondelete="CASCADE",
    )
    op.create_foreign_key(
        "fk_user_agents_llm_id",
        "user_agents",
        "llms",
        ["llm_id"],
        ["id"],
        ondelete="RESTRICT",
    )
    op.create_foreign_key(
        "fk_user_agents_created_by",
        "user_agents",
        "users",
        ["created_by"],
        ["id"],
        referent_schema="auth",
        ondelete="SET NULL",
    )
    op.create_foreign_key(
        "fk_user_agents_updated_by",
        "user_agents",
        "users",
        ["updated_by"],
        ["id"],
        referent_schema="auth",
        ondelete="SET NULL",
    )
    
    _enable_rls("user_agents")

    # 4. Recreate memories.agent_id
    op.add_column(
        "memories",
        sa.Column("agent_id", sa.UUID(), nullable=True)
    )
    
    op.create_foreign_key(
        "fk_memories_agent_id",
        "memories",
        "user_agents",
        ["agent_id"],
        ["id"],
        ondelete="CASCADE",
    )
    
    op.execute(
        "CREATE INDEX ix_memories_agent_type_status ON memories (agent_id, memory_type, status)"
    )
    
    op.execute(
        "ALTER TABLE memories ADD CONSTRAINT chk_memory_type_agent_id "
        "CHECK ((memory_type = 'work' AND agent_id IS NOT NULL) OR "
        "(memory_type = 'user' AND agent_id IS NULL))"
    )
```

**Step 4: Add helper functions**

Add these helper functions at the end of the migration file:

```python
def _enable_rls(table_name: str) -> None:
    for role in ["anon", "authenticated"]:
        for action in ["select", "insert", "update", "delete"]:
            op.execute(
                f"DROP POLICY IF EXISTS {role}_{action}_{table_name} ON {table_name}"
            )
    op.execute(f"ALTER TABLE {table_name} ENABLE ROW LEVEL SECURITY")
    for role in ["anon", "authenticated"]:
        op.execute(
            f"CREATE POLICY {role}_select_{table_name} ON {table_name} "
            f"FOR SELECT TO {role} USING (false)"
        )
        op.execute(
            f"CREATE POLICY {role}_insert_{table_name} ON {table_name} "
            f"FOR INSERT TO {role} WITH CHECK (false)"
        )
        op.execute(
            f"CREATE POLICY {role}_update_{table_name} ON {table_name} "
            f"FOR UPDATE TO {role} USING (false) WITH CHECK (false)"
        )
        op.execute(
            f"CREATE POLICY {role}_delete_{table_name} ON {table_name} "
            f"FOR DELETE TO {role} USING (false)"
        )


def _drop_rls(table_name: str) -> None:
    for role in ["anon", "authenticated"]:
        op.execute(f"DROP POLICY IF EXISTS {role}_delete_{table_name} ON {table_name}")
        op.execute(f"DROP POLICY IF EXISTS {role}_update_{table_name} ON {table_name}")
        op.execute(f"DROP POLICY IF EXISTS {role}_insert_{table_name} ON {table_name}")
        op.execute(f"DROP POLICY IF EXISTS {role}_select_{table_name} ON {table_name}")
    op.execute(f"ALTER TABLE {table_name} DISABLE ROW LEVEL SECURITY")
```

**Step 5: Verify migration file**

Check that all imports are correct:

```python
from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
```

**Step 6: Commit migration**

```bash
git add backend/alembic/versions/20260304_simplify_agent_architecture.py
git commit -m "feat(db): add migration to simplify agent architecture"
```

---

## Task 2: Delete UserAgents Model

**Files:**
- Delete: `backend/src/models/user_agents.py`
- Modify: `backend/src/models/__init__.py`

**Step 1: Remove import from models/__init__.py**

Edit `backend/src/models/__init__.py`:

Remove these lines:
```python
from models.user_agents import UserAgent
```

And remove `"UserAgent"` from `__all__` list.

**Step 2: Delete user_agents.py file**

```bash
rm backend/src/models/user_agents.py
```

**Step 3: Verify no other imports**

Run: `cd backend && grep -r "from models.user_agents" src/`

Expected: No results (or only in __init__.py which we already fixed)

**Step 4: Commit**

```bash
git add backend/src/models/user_agents.py backend/src/models/__init__.py
git commit -m "refactor(models): remove UserAgents model"
```

---

## Task 3: Rename UserAgentCatalog to SystemAgents

**Files:**
- Rename: `backend/src/models/user_agent_catalog.py` → `backend/src/models/system_agents.py`
- Modify: `backend/src/models/__init__.py`

**Step 1: Rename model file**

```bash
mv backend/src/models/user_agent_catalog.py backend/src/models/system_agents.py
```

**Step 2: Update class name in system_agents.py**

Edit `backend/src/models/system_agents.py`:

Change:
```python
class UserAgentCatalog(TimestampMixin, Base):
    __tablename__: str = "user_agent_catalog"
```

To:
```python
class SystemAgents(TimestampMixin, Base):
    __tablename__: str = "system_agents"
```

**Step 3: Update imports in models/__init__.py**

Edit `backend/src/models/__init__.py`:

Change:
```python
from models.user_agent_catalog import UserAgentCatalog
```

To:
```python
from models.system_agents import SystemAgents
```

And change `"UserAgentCatalog"` to `"SystemAgents"` in `__all__` list.

**Step 4: Commit**

```bash
git add backend/src/models/
git commit -m "refactor(models): rename UserAgentCatalog to SystemAgents"
```

---

## Task 4: Update Configuration Files

**Files:**
- Rename: `backend/src/core/config/static/database/user_agent_catalog.yaml`
  → `backend/src/core/config/static/database/system_agents.yaml`
- Modify: `backend/src/core/config/initial/init_data.py`

**Step 1: Rename YAML file**

```bash
mv backend/src/core/config/static/database/user_agent_catalog.yaml \
   backend/src/core/config/static/database/system_agents.yaml
```

**Step 2: Update init_data.py imports**

Edit `backend/src/core/config/initial/init_data.py`:

Change:
```python
from models.user_agent_catalog import UserAgentCatalog
```

To:
```python
from models.system_agents import SystemAgents
```

**Step 3: Update Pydantic models**

Change:
```python
class UserAgentCatalogSeed(BaseModel):
    agent_type: str
    llm_model_code: str
    status: str
    config: dict[str, Any]


class UserAgentCatalogYaml(BaseModel):
    agents: list[UserAgentCatalogSeed]
```

To:
```python
class SystemAgentsSeed(BaseModel):
    agent_type: str
    llm_model_code: str
    status: str
    config: dict[str, Any]


class SystemAgentsYaml(BaseModel):
    agents: list[SystemAgentsSeed]
```

**Step 4: Update path function**

Change:
```python
def _default_user_agent_catalog_path() -> Path:
    return (
        Path(__file__).resolve().parents[1]
        / "static"
        / "database"
        / "user_agent_catalog.yaml"
    )
```

To:
```python
def _default_system_agents_path() -> Path:
    return (
        Path(__file__).resolve().parents[1]
        / "static"
        / "database"
        / "system_agents.yaml"
    )
```

**Step 5: Update load function**

Change:
```python
def load_user_agent_catalog(catalog_path: Path | None = None) -> dict[str, Any]:
    path = catalog_path or _default_user_agent_catalog_path()
    with path.open("r", encoding="utf-8") as file:
        loaded = yaml.safe_load(file) or {}
    if not isinstance(loaded, dict):
        raise ValueError(f"Invalid user agent catalog format: {path}")
    raw_agents = loaded.get("agents", [])
    if not isinstance(raw_agents, list):
        raise ValueError(f"Invalid user agent catalog agents section: {path}")
    try:
        parsed = UserAgentCatalogYaml.model_validate({"agents": list(raw_agents)})
    except ValidationError as exc:
        raise ValueError(f"Invalid user agent catalog data: {path}") from exc

    return parsed.model_dump()
```

To:
```python
def load_system_agents(catalog_path: Path | None = None) -> dict[str, Any]:
    path = catalog_path or _default_system_agents_path()
    with path.open("r", encoding="utf-8") as file:
        loaded = yaml.safe_load(file) or {}
    if not isinstance(loaded, dict):
        raise ValueError(f"Invalid system agents format: {path}")
    raw_agents = loaded.get("agents", [])
    if not isinstance(raw_agents, list):
        raise ValueError(f"Invalid system agents agents section: {path}")
    try:
        parsed = SystemAgentsYaml.model_validate({"agents": list(raw_agents)})
    except ValidationError as exc:
        raise ValueError(f"Invalid system agents data: {path}") from exc

    return parsed.model_dump()
```

**Step 6: Update upsert function**

Change:
```python
async def _upsert_user_agent_catalog(
    session: AsyncSession,
    *,
    agent_type: str,
    llm_id: uuid.UUID,
    status: str,
    config: dict[str, Any],
) -> None:
    result = await session.execute(
        select(UserAgentCatalog).where(UserAgentCatalog.agent_type == agent_type)
    )
    catalog_entry = result.scalar_one_or_none()

    if catalog_entry is None:
        session.add(
            UserAgentCatalog(
                agent_type=agent_type,
                llm_id=llm_id,
                status=status,
                config=config,
            )
        )
    else:
        catalog_entry.llm_id = llm_id
        catalog_entry.status = status
        catalog_entry.config = config
```

To:
```python
async def _upsert_system_agents(
    session: AsyncSession,
    *,
    agent_type: str,
    llm_id: uuid.UUID,
    status: str,
    config: dict[str, Any],
) -> None:
    result = await session.execute(
        select(SystemAgents).where(SystemAgents.agent_type == agent_type)
    )
    catalog_entry = result.scalar_one_or_none()

    if catalog_entry is None:
        session.add(
            SystemAgents(
                agent_type=agent_type,
                llm_id=llm_id,
                status=status,
                config=config,
            )
        )
    else:
        catalog_entry.llm_id = llm_id
        catalog_entry.status = status
        catalog_entry.config = config
```

**Step 7: Update initialize function**

Change:
```python
async def initialize_user_agent_catalog() -> None:
    """Initialize user agent catalog from YAML."""
    catalog = load_user_agent_catalog()

    async with AsyncSessionLocal() as session:
        async with session.begin():
            for agent in catalog["agents"]:
                result = await session.execute(
                    select(Llm).where(Llm.model_code == agent["llm_model_code"])
                )
                llm = result.scalar_one_or_none()
                if llm is None:
                    raise RuntimeError(
                        f"LLM model '{agent['llm_model_code']}' not found for agent type '{agent['agent_type']}'"
                    )

                await _upsert_user_agent_catalog(
                    session,
                    agent_type=agent["agent_type"],
                    llm_id=llm.id,
                    status=agent["status"],
                    config=agent["config"],
                )

    logger.info("Initialized user agent catalog")
```

To:
```python
async def initialize_system_agents() -> None:
    """Initialize system agents from YAML."""
    catalog = load_system_agents()

    async with AsyncSessionLocal() as session:
        async with session.begin():
            for agent in catalog["agents"]:
                result = await session.execute(
                    select(Llm).where(Llm.model_code == agent["llm_model_code"])
                )
                llm = result.scalar_one_or_none()
                if llm is None:
                    raise RuntimeError(
                        f"LLM model '{agent['llm_model_code']}' not found for agent type '{agent['agent_type']}'"
                    )

                await _upsert_system_agents(
                    session,
                    agent_type=agent["agent_type"],
                    llm_id=llm.id,
                    status=agent["status"],
                    config=agent["config"],
                )

    logger.info("Initialized system agents")
```

**Step 8: Update initialize_data function**

Change:
```python
async def initialize_data() -> bool:
    """Initialize bootstrap data."""
    await initialize_llm_catalog()
    await initialize_user_agent_catalog()

    return True
```

To:
```python
async def initialize_data() -> bool:
    """Initialize bootstrap data."""
    await initialize_llm_catalog()
    await initialize_system_agents()

    return True
```

**Step 9: Commit**

```bash
git add backend/src/core/config/
git commit -m "refactor(config): rename user_agent_catalog to system_agents"
```

---

## Task 5: Run Migration

**Step 1: Run migration**

```bash
cd backend && uv run alembic upgrade head
```

Expected: Migration runs successfully

**Step 2: Verify tables**

Connect to database and check:
- `user_agents` table should NOT exist
- `system_agents` table should exist
- `memories.agent_id` column should NOT exist

**Step 3: Test downgrade (optional but recommended)**

```bash
cd backend && uv run alembic downgrade -1
```

Expected: Previous migration restored

**Step 4: Re-run upgrade**

```bash
cd backend && uv run alembic upgrade head
```

Expected: Migration runs successfully again

---

## Task 6: Run Tests and Linting

**Step 1: Run type checking**

```bash
cd backend && uv run basedpyright src/
```

Expected: No errors

**Step 2: Run linting**

```bash
cd backend && uv run ruff check src/
```

Expected: No errors

**Step 3: Run tests**

```bash
cd backend && uv run pytest tests/
```

Expected: All tests pass

**Step 4: Fix any failures**

If any tests fail due to UserAgent references, update them to use SystemAgents.

---

## Task 7: Final Verification

**Step 1: Search for any remaining references**

```bash
cd backend && grep -r "user_agents" src/ --include="*.py"
cd backend && grep -r "UserAgent" src/ --include="*.py"
```

Expected: No results (except in migration files)

**Step 2: Test new user registration**

Start the backend server and register a new user. Verify:
- Profile is created
- No user_agents records are created
- profiles.settings contains `agent_prompts: {}`

**Step 3: Commit final changes**

```bash
git add .
git commit -m "feat: complete agent architecture simplification"
```

---

## Success Criteria

- [ ] Migration runs successfully (upgrade and downgrade)
- [ ] No UserAgent model references in code
- [ ] SystemAgents model works correctly
- [ ] All tests pass
- [ ] Linting passes
- [ ] Type checking passes
- [ ] New user registration works without user_agents

## Notes

- Keep the design document updated if any changes are made during implementation
- Test migration thoroughly before deploying to production
- Backup database before running migration in production