GenAI Observability & Evaluation — Overview
Monitoring and evaluation are tested in AIP-C01 Domains 4 & 5 (23% combined). The exam tests FM-specific observability (token usage, latency, model drift), evaluation methods (automated metrics, LLM-as-Judge, human evaluation), and continuous testing patterns.
Latency (TTFT, TPS)
Throttling rate
Error rate
Cost per request
RAG retrieval time
Agent reasoning steps
Tool call duration
Service map
Guardrail decisions
Error details
Model invocation logs
Bedrock model invocation logging
LLM-as-Judge (Bedrock)
Human evaluation
A/B testing
Regression testing
Key Metrics for GenAI Applications
| Metric | What It Measures | Why It Matters |
|---|---|---|
| Time to First Token (TTFT) | Time from request to first output token | User-perceived responsiveness |
| Tokens Per Second (TPS) | Output generation speed | Streaming quality, throughput capacity |
| Input/Output Tokens | Token counts per request | Cost tracking, context window usage |
| Invocation Count | Number of FM API calls | Usage patterns, scaling needs |
| Guardrail Intervention Rate | % of requests blocked/modified by guardrails | Safety compliance, guardrail tuning |
| RAG Retrieval Relevance | How relevant are retrieved chunks (custom metric) | Knowledge base quality, chunking effectiveness |
Model Evaluation Methods
| Method | How | Best For |
|---|---|---|
| Automated Metrics | BLEU, ROUGE, BERTScore — compare output to reference | Summarization, translation (reference-based tasks) |
| LLM-as-Judge (Bedrock) | Use a powerful FM to evaluate another FM’s output (coherence, relevance, safety) | Open-ended generation, quality scoring at scale |
| Human Evaluation | Domain experts rate responses (SageMaker Ground Truth) | Final quality validation, subjective quality |
| A/B Testing | Route % of traffic to new model/prompt, compare metrics | Production validation of changes |
| Regression Testing | Run fixed test suite on every change, compare to baseline | Prevent quality degradation during updates |
Bedrock Model Invocation Logging
- Enable: Bedrock settings → Model invocation logging → S3 and/or CloudWatch Logs
- Captures: Full prompt, full response, model ID, token counts, latency, request metadata
- Use for: Debugging, compliance audit, prompt regression detection, cost attribution
- Privacy: Can contain PII — ensure encryption (KMS), access control, retention policies
X-Ray Tracing for GenAI
- End-to-end trace: User request → API Gateway → Lambda → Bedrock API → response
- Subsegments: RAG retrieval time, agent reasoning loops, tool call duration, guardrail processing
- Service map: Visualize dependencies between GenAI components
- Annotations: Tag traces with model ID, prompt template version, user segment for filtering
Exam Tips
| Exam | Key Points |
|---|---|
| AIP-C01 | “Evaluate FM quality at scale” → LLM-as-Judge (Bedrock evaluation). “Compare output to reference” → BLEU/ROUGE. “Track token costs” → CloudWatch metrics + Bedrock invocation logging. “Debug agent reasoning” → Bedrock agent trace + X-Ray. “Detect quality regression” → automated test suite + regression testing. “Monitor for drift” → custom CloudWatch metrics + alerting. |
AWS Certification Exam Practice Questions
Question 1:
A company needs to evaluate the quality of their GenAI chatbot’s responses at scale (1000+ responses daily). Human reviewers can only check 50 per day. Which approach provides comprehensive quality assessment?
- Only evaluate the 50 human-reviewed responses and extrapolate
- Use Bedrock model evaluation with LLM-as-Judge to automatically score all responses on coherence, relevance, and helpfulness
- Count the number of tokens in each response as a quality proxy
- Use BLEU score for all responses
Show Answer
Answer: B — LLM-as-Judge uses a powerful FM (like Claude) to evaluate another FM’s outputs against criteria you define (coherence, relevance, helpfulness, safety). It scales to all responses without human bottleneck. Correlates well with human judgment for most criteria. Use human review (A) for calibration and edge cases. BLEU (D) requires reference answers and doesn’t work for open-ended responses.
Question 2:
After updating a prompt template, a company notices their chatbot’s responses have degraded for certain query types. They need to detect such regressions automatically before deploying prompt changes. Which approach provides this?
- A/B test every prompt change with 50% of production traffic
- Maintain a regression test suite of queries with expected outputs, run automatically on every prompt change, alert if scores drop below threshold
- Monitor token output length — shorter responses indicate regression
- Wait for user complaints to identify issues
Show Answer
Answer: B — A regression test suite with curated query-response pairs catches quality drops before production. Run it in CI/CD pipeline: new prompt → run test suite → score with LLM-as-Judge or similarity metrics → gate deployment if scores drop. This is proactive (catches issues before users see them). A/B testing (A) is for production validation, not pre-deployment prevention. User complaints (D) are too late.
Question 3:
A development team needs to debug why their Bedrock Agent sometimes takes 30+ seconds to respond. The response eventually works but is too slow. Which observability tool shows where time is being spent?
- CloudWatch Logs Insights querying for slow requests
- AWS X-Ray tracing showing subsegments for each agent step (FM call, tool invocation, knowledge base retrieval)
- Bedrock console metrics dashboard
- CloudTrail showing API call sequence
Show Answer
Answer: B — X-Ray traces show the timing breakdown of each component: how long the FM reasoning took, how long each tool/action group Lambda ran, how long the knowledge base retrieval took. You can identify the bottleneck (e.g., one tool call taking 20s) and optimize specifically. CloudWatch metrics (C) show averages, not per-request breakdown. CloudTrail (D) shows API calls but not internal agent reasoning steps.
Related Posts
- GenAI Cost & Performance Optimization
- Agentic AI Architecture
- Responsible AI & Guardrails
- MLOps Pipeline Architecture
References
- Bedrock Model Evaluation — AWS Docs
- Model Invocation Logging — AWS Docs
- Monitor Bedrock with CloudWatch — AWS Blog
Frequently Asked Questions
What is LLM-as-Judge?
A technique where a powerful FM evaluates another FM’s outputs against defined criteria (relevance, coherence, safety, helpfulness). You provide the evaluation prompt (rubric), the original question, and the FM’s response. The judge FM scores it. Bedrock supports this natively in model evaluation. It scales to thousands of evaluations per hour without human reviewers.
Which metrics should I monitor for a GenAI chatbot?
Essential: (1) Latency — TTFT and total response time. (2) Token usage — input/output per request (cost tracking). (3) Error/throttle rate — availability. (4) Guardrail intervention rate — safety. (5) User satisfaction — thumbs up/down if available. (6) Retrieval relevance — for RAG applications. Create CloudWatch dashboard combining all six.