实用技巧与最佳实践

Tip 1：Use Session Prefix as Intent Rubric

问题：Vibe coding 无规格测试；用户意图未陈述、跨轮演变

解决方案：前 1-2 条用户消息作为 rubric（评分准则）

流程：

# 从用户开头轮次派生准则
opening = " ".join(session.user_messages[:2])
criteria = client.models.generate_content(
    model="gemini-3-pro",
    contents=f"Produce 3-5 acceptance criteria for: {opening}. Return JSON.",
).parsed["criteria"]

# 每轮 Agent 输出对比派生准则评分
score = client.models.generate_content(
    model="gemini-3-pro",
    contents=f"Does this output satisfy {criteria}? Score 1-5 with rationale."
             f"Output: {agent_response}",
).parsed

关键洞察：

This is the only practical way to evaluate dimension 1 (intent satisfaction) at scale.

— 这是大规模评估维度 1（意图满足）的唯一实用方法。

Tip 2：Judge the Rendered Artifact, Not the Code

问题：Vibe coding 用户判断输出而非 diff

解决方案：多模态模型查看渲染页面

捕获问题：

代码级评估错过：
• Layout broken on mobile ✓
• Contrast too low for accessibility ✓
• Button states wrong ✓

组合：
Multimodal judge → 捕获视觉和设计问题 ✓
Playwright assertions → 捕获破坏交互性 ✓

代码示例：

result = client.models.generate_content(
    model="gemini-3-pro",
    contents=[
        "Score this rendered web app against the spec on layout_match, "
        "styling, and interactive_correctness (1-5 each). Return JSON.",
        user_spec,
        types.Part.from_bytes(data=screenshot_bytes, mime_type="image/png"),
    ],
)

Tip 3：Evaluate Session Convergence, Not Turn-Level Accuracy

问题：Vibe-coding session 多轮构建

正确问题：

错误问题："was turn 4 correct?" ❌

正确问题："did the user converge on something they wanted?" ✓

成功定义：

• Few turns converge → 成功案例
• Sessions abandoned mid-flow → 最有信息失败（远超过轮级错误）

Cloud Trace 实现：

trace = trace_v2.TraceServiceClient().get_trace(
    name=f"projects/{project}/traces/{session_id}"
)

def session_outcome(trace):
    return {
        "converged": trace.last_turn.user_signal == "satisfied",
        "turns_to_converge": trace.user_correction_count,
        "abandoned": trace.last_user_action == "close",
        "cost_to_converge": trace.total_token_cost_usd,
    }

Tip 4：Mine User Corrections as Labeled Failure Data

价值：每条”不，不是那样”是标注失败示例

Vibe coding 特点：产生这些标注数据量巨大

方法：聚类 → Agent 系统性缺口可见 → 远快于构建合成失败基准

代码示例：

# 提取用户修正
corrections = [t.user_message for trace in traces for t in trace.turns
               if t.is_correction]

# 嵌入
emb = client.models.embed_content(
    model="text-embedding-005",
    contents=corrections,
)
vectors = [e.values for e in emb.embeddings]

# 聚类
clusters = KMeans(n_clusters=8).fit(vectors)
# clusters.labels_ → 下一迭代优先失败模式列表

Observability：Prerequisite（先决条件）

根本规则：要评估 Agent 内部推理，必须能看见它

Glass Box Evaluation 绝对先决条件：

无 observability → Agent 失败表现为不可解释单体事件 ❌

有 observability → 可追踪：
• Agent.reasoning
• Agent.tool_calls
• Agent.skill_invocations
• Agent.retrievals

技术实现：

OpenTelemetry spans：

agent.session span → 捕获整个任务持续时间
agent.think span → 记录行动前内部推理和提示循环
agent.tool span → 日志环境交互的具体参数和延迟

成本追踪：

Aggregating span attributes → 精确测量：

• Token consumption
• Inference latency
• Cost of self-repair loops

Dynamic Tail-Based Sampling：

问题：生产捕获 100% traces → 快速淹没存储预算 ❌

解决方案：Tail-based dynamic sampling
→ Collector 评估完成后完整 trace
→ 丢弃常规成功 ✓
→ 保留错误或过度自修复循环 traces ✓

---