AI Agent 系列 ⑦:Agent Engineering —— 评测、部署、监控、成本控制

2026/03/19 AI Agent 共 23607 字,约 68 分钟

系列文章: ① Agent 概览 · ② Tool Use · ③ RAG · ④ Memory · ⑤ Planning · ⑥ Multi-Agent · ⑦ Agent Engineering(本篇)


前六篇讲的是”怎么让 Agent 能用”。这篇讲的是”怎么让 Agent 在生产环境跑起来,还能跑好”。

两件事差距巨大。Demo 跑通很容易,生产环境要面对:成本爆炸、延迟抖动、奇怪的失败、用户的 prompt injection、无限循环……

这篇从工程视角出发,把 Agent 上生产要解决的问题一次性覆盖掉。


1. 先建评测体系,再动代码

改一个 Agent 的 prompt,不知道有没有变好?加了新工具,整体成功率有没有提升?没有评测体系,你是在盲飞。

1.1 核心指标

from dataclasses import dataclass, field
from typing import List, Dict, Optional, Any
import time


@dataclass
class AgentRunResult:
    """单次 Agent 运行的完整记录"""
    run_id: str
    goal: str

    # 结果
    success: bool
    output: str
    error: Optional[str] = None

    # 性能
    total_latency_ms: float = 0
    llm_calls: int = 0
    tool_calls: int = 0
    total_tokens: int = 0
    input_tokens: int = 0
    output_tokens: int = 0

    # 步骤详情
    steps: List[Dict] = field(default_factory=list)

    # 成本(美元)
    cost_usd: float = 0.0

    # 评分(0-1)
    quality_score: Optional[float] = None  # 自动或人工评分


@dataclass
class EvalCase:
    """评测用例"""
    id: str
    goal: str
    expected_contains: List[str] = field(default_factory=list)  # 输出必须包含
    expected_not_contains: List[str] = field(default_factory=list)  # 输出不能包含
    expected_tools: List[str] = field(default_factory=list)  # 必须调用的工具
    max_steps: int = 20
    max_cost_usd: float = 0.1
    tags: List[str] = field(default_factory=list)


class AgentEvaluator:
    def __init__(self, agent_factory):
        self.agent_factory = agent_factory  # Callable -> Agent instance
        self._results: List[Dict] = []

    async def run_suite(self, cases: List[EvalCase]) -> Dict:
        """运行完整评测集"""
        print(f"运行评测集: {len(cases)} 个用例")
        results = []

        for case in cases:
            result = await self._run_single(case)
            results.append(result)
            status = "✓" if result["passed"] else "✗"
            print(f"  {status} [{case.id}] {case.goal[:50]}")

        return self._aggregate(results)

    async def _run_single(self, case: EvalCase) -> Dict:
        agent = self.agent_factory()
        start = time.time()

        try:
            run_result: AgentRunResult = await agent.run(case.goal)
        except Exception as e:
            return {
                "case_id": case.id,
                "passed": False,
                "error": str(e),
                "checks": {},
            }

        latency = (time.time() - start) * 1000
        checks = self._check(case, run_result)
        passed = all(checks.values())

        return {
            "case_id": case.id,
            "passed": passed,
            "checks": checks,
            "latency_ms": latency,
            "tokens": run_result.total_tokens,
            "cost_usd": run_result.cost_usd,
            "steps": run_result.llm_calls + run_result.tool_calls,
            "output_preview": run_result.output[:200],
        }

    def _check(self, case: EvalCase, result: AgentRunResult) -> Dict[str, bool]:
        checks = {}

        # 成功检查
        checks["run_success"] = result.success

        # 输出内容检查
        output_lower = result.output.lower()
        for keyword in case.expected_contains:
            checks[f"contains:{keyword}"] = keyword.lower() in output_lower
        for keyword in case.expected_not_contains:
            checks[f"not_contains:{keyword}"] = keyword.lower() not in output_lower

        # 工具调用检查
        used_tools = {s.get("tool") for s in result.steps if s.get("tool")}
        for tool in case.expected_tools:
            checks[f"used_tool:{tool}"] = tool in used_tools

        # 成本检查
        checks["within_budget"] = result.cost_usd <= case.max_cost_usd

        # 步数检查
        steps = result.llm_calls + result.tool_calls
        checks["within_steps"] = steps <= case.max_steps

        return checks

    def _aggregate(self, results: List[Dict]) -> Dict:
        total = len(results)
        passed = sum(1 for r in results if r["passed"])
        failed_cases = [r for r in results if not r["passed"]]

        # 失败原因统计
        failure_reasons = {}
        for r in failed_cases:
            for check, ok in r.get("checks", {}).items():
                if not ok:
                    failure_reasons[check] = failure_reasons.get(check, 0) + 1

        successful = [r for r in results if r["passed"]]
        avg_latency = sum(r["latency_ms"] for r in successful) / len(successful) if successful else 0
        avg_cost = sum(r["cost_usd"] for r in results) / total if total else 0
        total_cost = sum(r["cost_usd"] for r in results)

        return {
            "total": total,
            "passed": passed,
            "failed": total - passed,
            "pass_rate": passed / total if total else 0,
            "avg_latency_ms": avg_latency,
            "avg_cost_usd": avg_cost,
            "total_cost_usd": total_cost,
            "failure_reasons": failure_reasons,
            "failed_cases": [r["case_id"] for r in failed_cases],
        }

1.2 LLM-as-Judge:自动质量评分

人工打分的规模上不去,用 LLM 做评判:

import anthropic
import json
import re


JUDGE_PROMPT = """你是一个严格的 AI Agent 质量评审员。

用户目标:
{goal}

Agent 输出:
{output}

评判标准:
1. 完整性(0-3分):是否完整回答了目标,有没有遗漏关键点?
2. 准确性(0-3分):内容是否准确,有无明显事实错误?
3. 可用性(0-2分):输出是否实际可用,格式是否合理?
4. 效率(0-2分):是否简洁,有无大量冗余废话?

以 JSON 格式输出:
scores,
  "total": 0-10,
  "strengths": ["优点1", "优点2"],
  "weaknesses": ["缺点1", "缺点2"],
  "one_line_summary": "一句话总结"
}}"""


class LLMJudge:
    def __init__(self, model: str = "claude-opus-4-6"):
        self.client = anthropic.Anthropic()
        self.model = model

    def judge(self, goal: str, output: str) -> Dict:
        prompt = JUDGE_PROMPT.format(goal=goal, output=output[:3000])
        response = self.client.messages.create(
            model=self.model,
            max_tokens=512,
            messages=[{"role": "user", "content": prompt}],
        )
        text = response.content[0].text
        try:
            match = re.search(r"\{[\s\S]+\}", text)
            result = json.loads(match.group(0)) if match else {}
            result["raw"] = text
            return result
        except Exception:
            return {"total": 5, "raw": text, "error": "解析失败"}

    def judge_batch(self, cases: List[Dict]) -> List[Dict]:
        """批量评判,返回评分列表"""
        results = []
        for case in cases:
            score = self.judge(case["goal"], case["output"])
            results.append({**case, "judge": score})
        return results

1.3 回归测试:防止改坏

每次改动 Agent 后,跑评测集对比:

class RegressionChecker:
    def __init__(self, baseline_path: str):
        self.baseline_path = baseline_path

    def save_baseline(self, results: Dict):
        import json
        with open(self.baseline_path, "w") as f:
            json.dump(results, f, indent=2)
        print(f"基线已保存: {self.baseline_path}")

    def compare(self, new_results: Dict) -> Dict:
        import json
        with open(self.baseline_path) as f:
            baseline = json.load(f)

        delta = {
            "pass_rate_delta": new_results["pass_rate"] - baseline["pass_rate"],
            "latency_delta_ms": new_results["avg_latency_ms"] - baseline["avg_latency_ms"],
            "cost_delta_usd": new_results["avg_cost_usd"] - baseline["avg_cost_usd"],
        }

        # 判断是否回归
        regressions = []
        if delta["pass_rate_delta"] < -0.05:
            regressions.append(f"通过率下降 {delta['pass_rate_delta']:.1%}")
        if delta["latency_delta_ms"] > 2000:
            regressions.append(f"延迟增加 {delta['latency_delta_ms']:.0f}ms")
        if delta["cost_delta_usd"] > 0.02:
            regressions.append(f"成本增加 ${delta['cost_delta_usd']:.4f}")

        return {
            "has_regression": bool(regressions),
            "regressions": regressions,
            "delta": delta,
            "baseline": baseline,
            "current": new_results,
        }

2. 成本控制

Agent 的成本主要来自 LLM token 消耗。一个没有优化的 Agent 跑生产,账单会很难看。

2.1 模型路由:按任务复杂度选模型

from enum import Enum


class TaskComplexity(Enum):
    SIMPLE = "simple"    # 分类、提取关键词、格式转换
    MEDIUM = "medium"    # 摘要、简单问答、代码补全
    COMPLEX = "complex"  # 多步推理、代码生成、复杂分析


MODEL_CONFIG = {
    TaskComplexity.SIMPLE: {
        "model": "claude-haiku-4-5-20251001",
        "max_tokens": 512,
        "cost_per_1k_input": 0.00025,
        "cost_per_1k_output": 0.00125,
    },
    TaskComplexity.MEDIUM: {
        "model": "claude-sonnet-4-6",
        "max_tokens": 2048,
        "cost_per_1k_input": 0.003,
        "cost_per_1k_output": 0.015,
    },
    TaskComplexity.COMPLEX: {
        "model": "claude-opus-4-6",
        "max_tokens": 8192,
        "cost_per_1k_input": 0.015,
        "cost_per_1k_output": 0.075,
    },
}


class ModelRouter:
    def __init__(self):
        self.client = anthropic.Anthropic()

    def route(self, task: str, context_length: int = 0) -> Dict:
        complexity = self._classify(task, context_length)
        config = MODEL_CONFIG[complexity]
        return config

    def _classify(self, task: str, context_length: int) -> TaskComplexity:
        # 简单规则分类(生产中可以用小模型分类)
        task_lower = task.lower()

        simple_keywords = ["提取", "分类", "是否", "格式化", "翻译", "总结一句话"]
        complex_keywords = ["推导", "分析", "设计", "优化", "对比", "为什么", "如何实现"]

        if any(kw in task_lower for kw in simple_keywords) and context_length < 1000:
            return TaskComplexity.SIMPLE
        if any(kw in task_lower for kw in complex_keywords) or context_length > 5000:
            return TaskComplexity.COMPLEX
        return TaskComplexity.MEDIUM

    def estimate_cost(self, input_tokens: int, output_tokens: int, complexity: TaskComplexity) -> float:
        cfg = MODEL_CONFIG[complexity]
        return (
            input_tokens / 1000 * cfg["cost_per_1k_input"] +
            output_tokens / 1000 * cfg["cost_per_1k_output"]
        )

2.2 Prompt 缓存

对于重复的系统提示(system prompt 通常几千 token),启用缓存可以省 90% 的输入 token 成本:

class CachedLLMClient:
    """
    利用 Anthropic prompt caching 降低重复 system prompt 的成本
    system prompt > 1024 tokens 时缓存效果最明显
    """

    def __init__(self):
        self.client = anthropic.Anthropic()

    def create_with_cache(
        self,
        system: str,
        messages: List[Dict],
        model: str = "claude-opus-4-6",
        max_tokens: int = 2048,
    ) -> Dict:
        response = self.client.messages.create(
            model=model,
            max_tokens=max_tokens,
            system=[
                {
                    "type": "text",
                    "text": system,
                    "cache_control": {"type": "ephemeral"},  # 标记为可缓存
                }
            ],
            messages=messages,
        )

        # 检查缓存命中情况
        usage = response.usage
        cache_hit = getattr(usage, "cache_read_input_tokens", 0)
        cache_miss = getattr(usage, "cache_creation_input_tokens", 0)

        return {
            "content": response.content[0].text,
            "usage": {
                "input_tokens": usage.input_tokens,
                "output_tokens": usage.output_tokens,
                "cache_hit_tokens": cache_hit,
                "cache_miss_tokens": cache_miss,
            },
            "cache_hit": cache_hit > 0,
        }

2.3 结果缓存:相同输入不重复调用

import hashlib
import json
import time
from typing import Optional


class LLMResultCache:
    """
    对 LLM 调用结果做 KV 缓存
    适合:工具结果处理、固定 prompt 的分类任务
    不适合:需要实时信息的查询
    """

    def __init__(self, ttl_seconds: int = 3600, max_size: int = 1000):
        self._cache: Dict[str, Dict] = {}
        self.ttl = ttl_seconds
        self.max_size = max_size
        self._hits = 0
        self._misses = 0

    def _key(self, model: str, system: str, user: str) -> str:
        content = f"{model}|{system}|{user}"
        return hashlib.sha256(content.encode()).hexdigest()[:16]

    def get(self, model: str, system: str, user: str) -> Optional[str]:
        key = self._key(model, system, user)
        entry = self._cache.get(key)
        if entry and time.time() - entry["ts"] < self.ttl:
            self._hits += 1
            return entry["value"]
        self._misses += 1
        return None

    def set(self, model: str, system: str, user: str, value: str):
        key = self._key(model, system, user)
        # LRU 淘汰
        if len(self._cache) >= self.max_size:
            oldest = min(self._cache, key=lambda k: self._cache[k]["ts"])
            del self._cache[oldest]
        self._cache[key] = {"value": value, "ts": time.time()}

    @property
    def hit_rate(self) -> float:
        total = self._hits + self._misses
        return self._hits / total if total else 0

    def stats(self) -> Dict:
        return {
            "size": len(self._cache),
            "hits": self._hits,
            "misses": self._misses,
            "hit_rate": f"{self.hit_rate:.1%}",
        }

2.4 成本预算守卫

class BudgetGuard:
    """给每个 Agent 运行设置成本上限,超出则中止"""

    def __init__(self, max_cost_usd: float, max_tokens: int = 100_000):
        self.max_cost = max_cost_usd
        self.max_tokens = max_tokens
        self._spent = 0.0
        self._tokens_used = 0

    def check_and_record(
        self,
        input_tokens: int,
        output_tokens: int,
        model: str,
    ) -> bool:
        """返回 False 表示超预算,应该中止"""
        cost = self._estimate_cost(input_tokens, output_tokens, model)
        self._spent += cost
        self._tokens_used += input_tokens + output_tokens

        if self._spent > self.max_cost:
            raise BudgetExceededError(
                f"成本超限: ${self._spent:.4f} > ${self.max_cost:.4f}"
            )
        if self._tokens_used > self.max_tokens:
            raise BudgetExceededError(
                f"Token 超限: {self._tokens_used} > {self.max_tokens}"
            )
        return True

    def _estimate_cost(self, input_tokens: int, output_tokens: int, model: str) -> float:
        # 按模型定价估算
        pricing = {
            "claude-opus-4-6": (0.015, 0.075),
            "claude-sonnet-4-6": (0.003, 0.015),
            "claude-haiku-4-5-20251001": (0.00025, 0.00125),
        }
        inp_price, out_price = pricing.get(model, (0.01, 0.03))
        return input_tokens / 1000 * inp_price + output_tokens / 1000 * out_price

    @property
    def summary(self) -> Dict:
        return {
            "spent_usd": round(self._spent, 6),
            "tokens_used": self._tokens_used,
            "budget_remaining_usd": round(self.max_cost - self._spent, 6),
            "budget_utilization": f"{self._spent / self.max_cost:.1%}",
        }


class BudgetExceededError(Exception):
    pass

3. 生产部署

3.1 流式输出

用户等 Agent 完整输出太慢,流式输出大幅改善体验:

import asyncio
from typing import AsyncGenerator


async def stream_agent_response(
    goal: str,
    agent,
) -> AsyncGenerator[str, None]:
    """
    流式输出 Agent 的思考和结果
    前端可以用 SSE 或 WebSocket 接收
    """
    client = anthropic.Anthropic()

    # 流式 LLM 调用
    with client.messages.stream(
        model="claude-opus-4-6",
        max_tokens=4096,
        system=agent.system_prompt,
        messages=[{"role": "user", "content": goal}],
    ) as stream:
        for text in stream.text_stream:
            yield text


# FastAPI 集成示例
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio

app = FastAPI()


@app.post("/agent/stream")
async def agent_stream(request: Dict):
    goal = request.get("goal", "")

    async def generate():
        async for chunk in stream_agent_response(goal, agent):
            # SSE 格式
            yield f"data: {json.dumps({'text': chunk})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        generate(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",  # 禁用 Nginx 缓冲
        },
    )

3.2 熔断器

工具调用或下游服务挂了,不要让 Agent 一直重试卡死:

from enum import Enum
import time


class CircuitState(Enum):
    CLOSED = "closed"      # 正常
    OPEN = "open"          # 熔断,拒绝请求
    HALF_OPEN = "half_open"  # 试探性恢复


class CircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        recovery_timeout: float = 60.0,
        success_threshold: int = 2,
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.success_threshold = success_threshold

        self._state = CircuitState.CLOSED
        self._failure_count = 0
        self._success_count = 0
        self._last_failure_time: Optional[float] = None

    async def call(self, fn, *args, **kwargs):
        if self._state == CircuitState.OPEN:
            if time.time() - self._last_failure_time > self.recovery_timeout:
                self._state = CircuitState.HALF_OPEN
                self._success_count = 0
            else:
                raise CircuitOpenError(f"熔断器开路,拒绝请求({self.recovery_timeout}s 后重试)")

        try:
            if asyncio.iscoroutinefunction(fn):
                result = await fn(*args, **kwargs)
            else:
                result = fn(*args, **kwargs)

            self._on_success()
            return result

        except Exception as e:
            self._on_failure()
            raise

    def _on_success(self):
        if self._state == CircuitState.HALF_OPEN:
            self._success_count += 1
            if self._success_count >= self.success_threshold:
                self._state = CircuitState.CLOSED
                self._failure_count = 0
        elif self._state == CircuitState.CLOSED:
            self._failure_count = max(0, self._failure_count - 1)

    def _on_failure(self):
        self._failure_count += 1
        self._last_failure_time = time.time()
        if self._failure_count >= self.failure_threshold:
            self._state = CircuitState.OPEN

    @property
    def state(self) -> CircuitState:
        return self._state


class CircuitOpenError(Exception):
    pass

3.3 降级策略

工具失败时,有备用方案:

class ToolWithFallback:
    """
    工具调用 + 降级:
    主工具失败 → 备用工具 → 纯 LLM 回答
    """

    def __init__(
        self,
        primary_tool: Callable,
        fallback_tool: Optional[Callable] = None,
        llm_fallback: bool = True,
    ):
        self.primary = primary_tool
        self.fallback = fallback_tool
        self.use_llm_fallback = llm_fallback
        self._client = anthropic.Anthropic()
        self._breaker = CircuitBreaker()

    async def call(self, task: str, **kwargs) -> Dict:
        # 尝试主工具
        try:
            result = await self._breaker.call(self.primary, task=task, **kwargs)
            return {"result": result, "source": "primary", "degraded": False}
        except (CircuitOpenError, Exception) as e:
            print(f"[Fallback] 主工具失败: {e}")

        # 尝试备用工具
        if self.fallback:
            try:
                result = await self.fallback(task=task, **kwargs)
                return {"result": result, "source": "fallback", "degraded": True}
            except Exception as e:
                print(f"[Fallback] 备用工具也失败: {e}")

        # LLM 兜底
        if self.use_llm_fallback:
            response = self._client.messages.create(
                model="claude-sonnet-4-6",
                max_tokens=1024,
                messages=[{
                    "role": "user",
                    "content": f"工具调用失败,请直接用你的知识回答:{task}"
                }],
            )
            return {
                "result": response.content[0].text,
                "source": "llm_fallback",
                "degraded": True,
                "warning": "工具不可用,结果可能不准确",
            }

        raise RuntimeError("所有降级策略均失败")

4. 监控与告警

4.1 指标收集

from collections import defaultdict, deque
import threading


class AgentMetrics:
    """
    Agent 运行指标收集
    生产环境接入 Prometheus / Datadog,这里用内存版示意
    """

    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self._lock = threading.Lock()

        # 滑动窗口内的数据点
        self._latencies: deque = deque()
        self._costs: deque = deque()
        self._successes: deque = deque()
        self._errors: Dict[str, int] = defaultdict(int)
        self._tool_calls: Dict[str, int] = defaultdict(int)
        self._token_usage: deque = deque()

    def record_run(self, result: AgentRunResult):
        now = time.time()
        with self._lock:
            self._latencies.append((now, result.total_latency_ms))
            self._costs.append((now, result.cost_usd))
            self._successes.append((now, 1 if result.success else 0))
            self._token_usage.append((now, result.total_tokens))

            if result.error:
                error_type = type(result.error).__name__ if not isinstance(result.error, str) else result.error.split(":")[0]
                self._errors[error_type] += 1

            for step in result.steps:
                if step.get("tool"):
                    self._tool_calls[step["tool"]] += 1

            self._cleanup(now)

    def _cleanup(self, now: float):
        cutoff = now - self.window
        for dq in [self._latencies, self._costs, self._successes, self._token_usage]:
            while dq and dq[0][0] < cutoff:
                dq.popleft()

    def snapshot(self) -> Dict:
        now = time.time()
        with self._lock:
            self._cleanup(now)
            latencies = [v for _, v in self._latencies]
            costs = [v for _, v in self._costs]
            successes = [v for _, v in self._successes]
            tokens = [v for _, v in self._token_usage]

        total = len(successes)
        if total == 0:
            return {"status": "no_data"}

        return {
            "window_seconds": self.window,
            "total_runs": total,
            "success_rate": sum(successes) / total,
            "p50_latency_ms": self._percentile(latencies, 50),
            "p95_latency_ms": self._percentile(latencies, 95),
            "p99_latency_ms": self._percentile(latencies, 99),
            "avg_cost_usd": sum(costs) / len(costs) if costs else 0,
            "total_cost_usd": sum(costs),
            "avg_tokens": sum(tokens) / len(tokens) if tokens else 0,
            "top_errors": dict(sorted(self._errors.items(), key=lambda x: -x[1])[:5]),
            "top_tools": dict(sorted(self._tool_calls.items(), key=lambda x: -x[1])[:5]),
        }

    @staticmethod
    def _percentile(data: List[float], p: int) -> float:
        if not data:
            return 0
        sorted_data = sorted(data)
        idx = int(len(sorted_data) * p / 100)
        return sorted_data[min(idx, len(sorted_data) - 1)]

4.2 告警规则

@dataclass
class AlertRule:
    name: str
    check: Callable[[Dict], bool]
    message: Callable[[Dict], str]
    severity: str = "warning"  # warning / critical
    cooldown_seconds: int = 300  # 同一告警的冷却时间


ALERT_RULES = [
    AlertRule(
        name="high_error_rate",
        check=lambda m: m.get("success_rate", 1) < 0.8,
        message=lambda m: f"Agent 成功率过低: {m['success_rate']:.1%}",
        severity="critical",
    ),
    AlertRule(
        name="high_latency",
        check=lambda m: m.get("p95_latency_ms", 0) > 30_000,
        message=lambda m: f"P95 延迟过高: {m['p95_latency_ms']:.0f}ms",
        severity="warning",
    ),
    AlertRule(
        name="cost_spike",
        check=lambda m: m.get("avg_cost_usd", 0) > 0.5,
        message=lambda m: f"平均成本过高: ${m['avg_cost_usd']:.3f}/次",
        severity="critical",
    ),
]


class AlertManager:
    def __init__(self, rules: List[AlertRule], notifier: Callable):
        self.rules = rules
        self.notifier = notifier  # 接入 PagerDuty / 企业微信 / Slack
        self._last_alert: Dict[str, float] = {}

    def check(self, metrics: Dict):
        now = time.time()
        for rule in self.rules:
            if not rule.check(metrics):
                continue

            # 冷却期内不重复告警
            last = self._last_alert.get(rule.name, 0)
            if now - last < rule.cooldown_seconds:
                continue

            self._last_alert[rule.name] = now
            alert_msg = rule.message(metrics)
            print(f"[ALERT/{rule.severity.upper()}] {alert_msg}")
            self.notifier(rule.severity, alert_msg, metrics)

4.3 异常检测:用 LLM 分析告警

这是个有意思的递归 —— 用 LLM Agent 来分析其他 Agent 的告警:

class LLMAlertAnalyzer:
    """用 LLM 分析 Agent 告警,给出根因和建议"""

    def __init__(self):
        self.client = anthropic.Anthropic()

    def analyze(
        self,
        alert: Dict,
        recent_errors: List[str],
        metrics_history: List[Dict],
    ) -> str:
        # 格式化上下文
        errors_text = "\n".join(f"- {e}" for e in recent_errors[-20:])
        metrics_text = json.dumps(metrics_history[-5:], indent=2, default=str)

        prompt = f"""你是一个 AI Agent 系统的运维专家。

当前告警:
{json.dumps(alert, indent=2)}

最近的错误日志(最新 20 条):
{errors_text}

最近 5 个时间窗口的指标趋势:
{metrics_text}

请分析:
1. 根本原因是什么?(代码 bug / 外部依赖故障 / 流量突增 / 配置问题)
2. 影响范围有多大?
3. 建议的处理步骤(优先级排序)
4. 是否需要立即回滚?

给出简洁、可操作的分析报告。"""

        response = self.client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        return response.content[0].text

5. 安全防护

5.1 Prompt Injection 防御

用户输入可能包含试图劫持 Agent 行为的指令:

INJECTION_PATTERNS = [
    r"ignore (all |previous |above |prior )?instructions",
    r"forget (everything|all|what|your)",
    r"你(现在|实际上)是",
    r"system prompt",
    r"新的指令",
    r"override",
    r"jailbreak",
    r"act as (if )?you (are|were)",
]


class PromptInjectionDetector:
    def __init__(self, use_llm_check: bool = True):
        self._patterns = [re.compile(p, re.IGNORECASE) for p in INJECTION_PATTERNS]
        self._use_llm = use_llm_check
        self._client = anthropic.Anthropic() if use_llm_check else None

    def check(self, user_input: str) -> Dict:
        # 第一层:规则检查(快,零成本)
        for pattern in self._patterns:
            if pattern.search(user_input):
                return {
                    "is_injection": True,
                    "confidence": "high",
                    "method": "rule",
                    "matched": pattern.pattern,
                }

        # 第二层:LLM 检查(慢,有成本,用于可疑内容)
        if self._use_llm and self._looks_suspicious(user_input):
            return self._llm_check(user_input)

        return {"is_injection": False, "confidence": "high", "method": "rule"}

    def _looks_suspicious(self, text: str) -> bool:
        suspicious_chars = text.count("\\n") + text.count("\\t") + text.count("```")
        return suspicious_chars > 5 or len(text) > 2000

    def _llm_check(self, text: str) -> Dict:
        response = self._client.messages.create(
            model="claude-haiku-4-5-20251001",  # 用小模型,省钱
            max_tokens=128,
            messages=[{
                "role": "user",
                "content": f"""判断以下输入是否包含 prompt injection 攻击(试图修改 AI 系统行为的指令)。
只回答 JSON: is_injection

输入:
{text[:500]}"""
            }],
        )
        try:
            result = json.loads(response.content[0].text)
            result["method"] = "llm"
            result["confidence"] = "medium"
            return result
        except Exception:
            return {"is_injection": False, "method": "llm", "confidence": "low"}

5.2 工具调用白名单 + 参数校验

from typing import Set


class ToolCallValidator:
    def __init__(
        self,
        allowed_tools: Set[str],
        blocked_patterns: Optional[Dict[str, List[str]]] = None,
    ):
        self.allowed_tools = allowed_tools
        # blocked_patterns: {tool_name: [危险参数值的正则]}
        self.blocked_patterns = blocked_patterns or {}

    def validate(self, tool_name: str, args: Dict) -> Tuple[bool, str]:
        # 工具白名单
        if tool_name not in self.allowed_tools:
            return False, f"工具 '{tool_name}' 不在允许列表中"

        # 参数安全检查
        if tool_name in self.blocked_patterns:
            for pattern_str in self.blocked_patterns[tool_name]:
                pattern = re.compile(pattern_str, re.IGNORECASE)
                for key, value in args.items():
                    if isinstance(value, str) and pattern.search(value):
                        return False, f"参数 '{key}' 包含不安全的内容: {pattern_str}"

        return True, "ok"


# 使用示例
validator = ToolCallValidator(
    allowed_tools={"web_search", "read_url", "python_exec", "write_file"},
    blocked_patterns={
        "python_exec": [
            r"import\s+os",            # 禁止 os 操作
            r"subprocess",             # 禁止子进程
            r"__import__",             # 禁止动态导入
            r"open\s*\(",              # 禁止文件读写(用专用工具)
            r"socket",                 # 禁止网络操作
        ],
        "write_file": [
            r"\.\./",                  # 禁止路径穿越
            r"^/etc",                  # 禁止系统目录
            r"^/root",
            r"\.ssh",
        ],
    },
)

5.3 输出过滤

class OutputFilter:
    """过滤 Agent 输出中的敏感信息"""

    PATTERNS = {
        "api_key": re.compile(r"(sk|pk|api[_-]?key)[_-]?[a-zA-Z0-9]{20,}", re.IGNORECASE),
        "password": re.compile(r"(password|passwd|pwd)\s*[:=]\s*\S+", re.IGNORECASE),
        "token": re.compile(r"(bearer|token)\s+[a-zA-Z0-9._-]{20,}", re.IGNORECASE),
        "credit_card": re.compile(r"\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b"),
        "phone": re.compile(r"1[3-9]\d{9}"),  # 中国手机号
    }

    def filter(self, text: str) -> Tuple[str, List[str]]:
        """返回 (过滤后的文本, 发现的敏感信息类型列表)"""
        found = []
        result = text

        for name, pattern in self.PATTERNS.items():
            if pattern.search(result):
                found.append(name)
                result = pattern.sub(f"[REDACTED:{name.upper()}]", result)

        return result, found

6. 常见生产事故复盘

6.1 无限循环

现象:Agent 在某个步骤卡住,不断重复调用同一个工具。

根因:工具返回空结果或错误,Agent 误以为需要再试一次,但 prompt 里没有明确的退出条件。

防御:

class LoopDetector:
    def __init__(self, max_repeats: int = 3, window: int = 10):
        self.max_repeats = max_repeats
        self.window = window
        self._history: deque = deque(maxlen=window)

    def check(self, action: str, args: Dict) -> bool:
        """返回 True 表示检测到循环,应该中止"""
        key = f"{action}:{json.dumps(args, sort_keys=True, default=str)}"
        self._history.append(key)

        # 统计最近 window 步内相同 action 的次数
        count = sum(1 for h in self._history if h == key)
        if count >= self.max_repeats:
            return True
        return False

6.2 Token 爆炸

现象:Agent 把所有中间结果都塞进上下文,最后一个 API 调用花了 $5。

根因:没有做结果压缩,前几步的工具输出(可能几十 KB)全部传给了下一步。

防御:在 Planning 篇介绍的 ResultCompressor 的基础上,加强制截断:

def truncate_for_context(text: str, max_chars: int = 3000, keep_head: int = 1500, keep_tail: int = 1000) -> str:
    """保留头尾,截断中间——头部通常是最重要的摘要,尾部是最新的信息"""
    if len(text) <= max_chars:
        return text
    middle_removed = len(text) - keep_head - keep_tail
    return (
        text[:keep_head] +
        f"\n\n... [省略 {middle_removed} 字符] ...\n\n" +
        text[-keep_tail:]
    )

6.3 工具调用参数幻觉

现象:LLM 生成了一个不存在的工具参数,或者把参数名写错了(file_path 写成了 filepath),导致工具报错。

根因:LLM 对工具 schema 记忆不稳定,生成时可能偏离。

防御:在工具层做 schema 校验,而不是直接把 LLM 输出当参数用:

from jsonschema import validate, ValidationError


class StrictToolCaller:
    def __init__(self, tools_schema: Dict[str, Dict]):
        self.schemas = tools_schema

    def call(self, tool_name: str, raw_args: Dict) -> Any:
        if tool_name not in self.schemas:
            raise ValueError(f"未知工具: {tool_name}")

        schema = self.schemas[tool_name].get("parameters", {})
        try:
            validate(instance=raw_args, schema=schema)
        except ValidationError as e:
            raise ValueError(f"工具参数校验失败: {e.message}")

        return self.schemas[tool_name]["fn"](**raw_args)

6.4 上下文污染

现象:Agent 在多轮对话中,把第一轮的信息错误地应用到了第五轮。

根因:Memory 系统召回了相关但不准确的历史记忆,干扰了当前任务。

防御:在 Memory 召回时,加时效权重(参考第 ④ 篇的 ForgettingCurve),并在 prompt 里明确区分”当前任务”和”历史参考”:

def build_memory_augmented_prompt(current_task: str, memories: List[Dict]) -> str:
    if not memories:
        return current_task

    memory_block = "\n".join(
        f"- [{m['type']}] {m['content'][:200]} (相关度: {m['score']:.2f})"
        for m in memories[:5]
    )

    return f"""当前任务(以此为准):
{current_task}

历史参考(仅供参考,如与当前任务冲突,以当前任务为准):
{memory_block}

请基于当前任务完成工作。"""

7. 上线 Checklist

在 Agent 上生产前,过一遍这个列表:

基础功能
  [ ] 评测集覆盖核心 use case,通过率 > 85%
  [ ] 错误处理:工具失败、LLM 超时、网络异常都有处理
  [ ] 输出格式稳定,不会因 LLM 随机性导致解析失败

性能
  [ ] P95 延迟在可接受范围(用户侧通常 < 30s)
  [ ] 有流式输出,用户不用等完整结果
  [ ] 熔断器配置正确,下游挂了不会拖垮整体

成本
  [ ] 每次请求的预期成本估算过
  [ ] BudgetGuard 配置了合理上限
  [ ] Prompt caching 对 system prompt > 1024 tokens 的场景已启用

安全
  [ ] Prompt injection 检测已启用
  [ ] 工具调用白名单已配置
  [ ] 代码执行在沙箱中
  [ ] 输出过滤已启用

监控
  [ ] 成功率、P95延迟、成本指标已接入监控
  [ ] 告警规则配置,关键指标异常时能触达
  [ ] 日志格式结构化,包含 run_id 方便排查

运维
  [ ] 支持快速回滚(prompt 版本化)
  [ ] 有 A/B 测试机制,可以灰度新版本
  [ ] 检查点机制,长任务中断可恢复

8. 系列总结

七篇文章覆盖了 AI Agent 从理论到工程的完整链路:

graph LR
    A[① 概览<br/>ReAct/MRKL/AutoGPT] --> B[② Tool Use<br/>Function Calling/沙箱]
    B --> C[③ RAG<br/>检索/重排/GraphRAG]
    C --> D[④ Memory<br/>向量存储/遗忘曲线]
    D --> E[⑤ Planning<br/>DAG执行/ReAct]
    E --> F[⑥ Multi-Agent<br/>消息总线/协调]
    F --> G[⑦ Engineering<br/>评测/成本/监控/安全]

每一篇都有完整可运行的代码。从单个 Agent 到 Multi-Agent 系统,从概念验证到生产部署,核心问题都覆盖到了。

真正难的部分,从来不是把 Agent 跑起来,而是把它跑稳、跑便宜、跑得可信。


AI Agent 系列完结。如有问题欢迎讨论。

文档信息

Search

    Table of Contents