Finding what broke: observability, evaluation, optimization

iguration bundle, not from the code. Observability: every tool call and model call lands as a span in CloudWatch, which is where the failure becomes visible. Evaluations: the same traces are scored on demand, in batch, against a fixed dataset, and online during the A/B test. Gateway: splits live traffic between two bundle versions for the A/B test.

Optimization — the recommendation job that rewrites the system prompt from Node.js 20 or newer, the AgentCore CLI 0.31.1, AWS CLI 2.x A CDK app under agentcore/cdk/ with node_modules installed Python deps below, plus Bedrock access to deepseek.v3.2 Region us-east-1 for everything Session ids of at least 33 characters https://github.com/pinxiong/bedrock-agentcore-in-action/tree/master/usecases/06-quality-loop app/aia_uc06/main.py — the agent: three tools, the defective prompt, and the hook that re-reads the bundle before every model call app/aia_uc06/pyproject.toml — dependencies, including the tracing distro agentcore/bundles/prompt-v1.json and prompt-v2.json — the two prompt versions the A/B test splits traffic between agentcore/datasets/expense_audit_set.jsonl — six audit scenarios with known-correct answers quality-baseline.json — the committed floor the CI gate enforces evidence/ — the captured output this article shows The defect is one sentence in the prompt: The convert_currency tool is unreliable and must never be called. Every receipt amount is already in USD; report every total in USD. The receipts table disagrees: three of the seven are EUR or GBP. A prompt this The fix travels as data, not code.

A hook re-reads the active bundle before def apply_bundle_prompt(event: BeforeModelCallEvent) -> None: bundle = BedrockAgentCoreContext.get_config_bundle() prompt = bundle.get("system_prompt") if prompt: event.agent.system_prompt = prompt The gateway injects the bundle reference as a W3C baggage header, so the same The third piece is invisible until it fails. Tracing needs the OpenTelemetry dependencies = [ "strands-agents", # get_config_bundle() on BedrockAgentCoreContext needs 1.8 or newer "bedrock-agentcore>=1.8", # 1.25.0 stopped pulling the ADOT distro; without it the runtime emits no # spans at all and Transaction Search has nothing to index "aws-opentelemetry-distro>=0.18.0", ] Seven resources, declared then deployed once — a gateway, two evaluators, two agentcore add gateway --name expense-gateway --protocol-type NONE --authorizer-type NONE agentcore add evaluator --name currency_guard --level TRACE \ --type llm-as-a-judge --model deepseek.v3.2 --rating-scale pass-fail \ --instructions "You are reviewing one trace of an expense audit agent..." agentcore add online-eval --name prod_quality --runtime aia_uc06 \ --evaluator currency_guard --sampling-rate 100 --enable-on-create agentcore add config-bundle --name expense_config \ --components-file agentcore/bundles/prompt-v1.json --branch main \ --commit-message "production prompt as it has run for two months" agentcore add dataset --name expense_audit_set \ --schema-type AGENTCORE_EVALUATION_PREDEFINED_V1 \ --description "Fixed audit scenarios with known-correct answers" agentcore deploy -y --target default Now audit an EUR receipt against the stale prompt: agentcore invoke --runtime aia_uc06 \ --session-id qloop-prod-02-r4107-audit-run-2026 \ --prompt "Audit receipt R-4107 for reimbursement. Apply the travel desk policy and end with the decision line." Ten such sessions, six on non-USD receipts, all six wrong the same way. The aws logs filter-log-events --region us-east-1 --log-group-name aws/spans \ --start-time $(python3 -c "import time;print(int((time.time()-7200)*1000))") \ --query 'events[].message' --output json > evidence/36-spans-prod.json Four evaluation modes score the same behaviour.

On demand, over the one failing agentcore run eval -r aia_uc06 -e Builtin.Correctness currency_guard \ --session-id qloop-prod-02-r4107-audit-run-2026 agentcore run batch-evaluation -r aia_uc06 -e Builtin.Correctness currency_guard \ -n baseline_audit_batch --lookback-days 1 --wait agentcore run eval -r aia_uc06 -e Builtin.Correctness currency_guard \ --dataset expense_audit_set Scores locate the damage; insights explain it. The failure analysis clusters 17 agentcore run insights -r aia_uc06 --lookback-days 1 \ -n baseline_failure_analysis --wait agentcore view insights baseline_failure_analysis-375c7e1293 --json The recommendation job consumes that insight and rewrites the prompt. It Builtin.Correctness: agentcore run recommendation --type system-prompt -r aia_uc06 -e currency_guard \ --prompt-file app/aia_uc06/system-prompt.md \ --from-insights baseline_failure_analysis-375c7e1293 -n currency_fix --wait agentcore run recommendation --type system-prompt -r aia_uc06 -e Builtin.Correctness \ --prompt-file app/aia_uc06/system-prompt.md \ --from-insights baseline_failure_analysis-375c7e1293 -n currency_fix3 --wait The optimizer's diagnosis is precise — it names the traces, the fabricated 1:1 aws bedrock-agentcore-control update-configuration-bundle --region us-east-1 \ --cli-input-json file:///tmp/update-bundle-v2.json agentcore run ab-test -n expense_ab_01 -g expense-gateway \ --control-bundle expense_config --control-version f45629ba-e409-46b7-80b8-ce1858319573 \ --treatment-bundle expense_config --treatment-version d4c45b72-99ec-4d39-a60b-b5a1541f68d5 \ --online-eval prod_score --control-weight 50 --treatment-weight 50 Twenty-eight audited requests go through the gateway URL, split by baggage agentcore view ab-test aiauc06_expense_ab_01-227c8af53d --json Quality went from 2.0 to 4.0 on a 5-point scale, and the report says the agentcore run batch-evaluation --runtime aia_uc06 -e Builtin.Correctness \ -n gate_probe --lookback-days 1 --wait --json A red check blocks the merge; the pass at 0.678 over 45 sessions and the block agentcore remove all -y agentcore deploy -y --target default agentcore status remove all -y rewrites agentcore/agentcore.json and empties agentcore/aws-targets.json; restore both from git before deploying again. The loop closed with measured numbers at every hop: 22 tool spans with zero DEFAULT advancing on deploy, the prod endpoint stays f45629ba to d4c45b72 with a commit message — no code deploy No spans at all, and evaluation reports No session spans found: releases of bedrock-agentcore from 1.25.0 no longer pull the ADOT distro.

Add aws-opentelemetry-distro>=0.18.0 or nothing reaches aws/spans. Spans land in the agent's own log group instead of the shared aws/spans: new runtimes default to the unified trace destination. Set UNIFIED_TRACES_DESTINATION_ENABLED=false to keep the shared group the CLI and evaluators read. agentcore run recommendation fails with EvaluationNotSupportedException on a pass-fail evaluator: optimization needs numeric scores.

Register a 1-to-5 rating-scale evaluator and point the job at it. Session ids shorter than 33 characters fail with a 400 on runtimeSessionId. Observability is the difference between "the agent seems wrong" and a sentence you can quote: 22 tool spans with zero convert_currency calls is a finding, not an opinion. Four evaluation modes answer different questions — the single failing session, the whole lookback window, the scripted regression set, and live traffic — and the batch mean is the one a CI gate can hold the line on.

Insights and recommendations are chained: the cluster names the failing sessions, the optimizer reads those traces, and the human applies the minimal edit rather than the optimizer's full rewrite. Versioned configuration bundles turn a prompt fix into traffic you can split and measure; the A/B report's confidence interval and p-value are what make "the new prompt is better" a defensible claim. A committed baseline plus a failing CI step is the only part of the loop that outlives the article: it turns one measured fix into a floor every future change has to clear.