运维与可观测性

8.1 日志与追踪

8.1.1 结构化日志

// 标准日志格式
{
  "timestamp": "2026-05-15T10:30:00Z",
  "trace_id": "abc123",
  "span_id": "span456",
  "parent_span_id": "span789",
  "agent_id": "planner-agent",
  "level": "INFO",
  "event": "tool_call",
  "data": {
    "tool_name": "search_database",
    "tool_args": {"query": "SELECT * FROM orders WHERE user_id = ?"},
    "execution_time_ms": 150,
    "token_used": 500
  }
}

8.1.2 分布式追踪

Trace: user_request_123
│
├─ [10:30:00.000] Orchestrator.receive_request
│   └─ [10:30:00.050] Orchestrator.plan
│       └─ [10:30:00.200] Orchestrator.delegate_to_worker
│
├─ [10:30:00.250] Worker_A.start
│   ├─ [10:30:00.260] Worker_A.tool_call: search_db
│   │   └─ [10:30:00.410] Worker_A.tool_response
│   └─ [10:30:00.450] Worker_A.complete
│
├─ [10:30:00.500] Worker_B.start
│   ├─ [10:30:00.510] Worker_B.tool_call: call_api
│   │   └─ [10:30:01.200] Worker_B.tool_response
│   └─ [10:30:01.250] Worker_B.complete
│
└─ [10:30:01.300] Orchestrator.aggregate_results
    └─ [10:30:01.500] Orchestrator.respond

8.2 监控告警

8.2.1 告警规则示例

# alerting_rules.yaml
alerts:
  - name: high_error_rate
    condition: error_rate > 0.05  # 5% 错误率
    duration: 5m
    severity: critical
    action:
      - notify_oncall
      - reduce_traffic

  - name: high_latency_p95
    condition: latency_p95 > 10000  # 10秒
    duration: 10m
    severity: warning
    action:
      - notify_team

  - name: token_cost_spike
    condition: token_cost_hourly > 2 * baseline_hourly
    duration: 1h
    severity: warning
    action:
      - notify_team
      - enable_cost_cache

  - name: agent_stuck
    condition: agent_execution_time > 300000  # 5分钟
    duration: immediate
    severity: critical
    action:
      - kill_agent
      - notify_oncall
      - save_state_for_debug

8.3 版本管理

8.3.1 Prompt 版本控制

prompts/
├── planner_agent/
│   ├── v1.0.0.md
│   ├── v1.1.0.md
│   ├── v1.2.0.md
│   └── current.md -> v1.2.0.md
├── executor_agent/
│   ├── v1.0.0.md
│   ├── v2.0.0.md  # 重大变更
│   └── current.md -> v2.0.0.md
└── CHANGELOG.md

8.3.2 灰度发布策略

┌─────────────────────────────────────────────────────────────┐
│  灰度发布流程                                                │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Stage 1: Canary (金丝雀)                                   │
│  ├── 流量: 1%                                               │
│  ├── 时长: 1 小时                                           │
│  ├── 指标: 错误率、延迟、用户反馈                            │
│  └── 回滚: 自动（错误率 > 2x baseline）                     │
│                                                             │
│  Stage 2: Early Adopters (早期采用者)                       │
│  ├── 流量: 10%                                              │
│  ├── 时长: 24 小时                                          │
│  ├── 指标: 全量指标                                         │
│  └── 回滚: 手动确认                                         │
│                                                             │
│  Stage 3: General Availability (全面可用)                   │
│  ├── 流量: 100%                                             │
│  ├── 持续监控                                               │
│  └── 回滚能力保留 7 天                                       │
│                                                             │
└─────────────────────────────────────────────────────────────┘

---