GenAI Cost & Performance Optimization — Overview
Cost and performance optimization is tested across AIP-C01 Domain 4 (12%) and appears in latency/streaming questions throughout the exam. The key decisions: model selection (capability vs cost), token optimization, caching strategies, provisioned throughput, and model routing patterns.
Model Routing — Cost-Performance Cascading
| Pattern | How | Savings |
|---|---|---|
| Tiered routing | Classify query complexity → route simple to cheap model, complex to expensive | 60-80% (most queries are simple) |
| Cascading | Try cheap model first → if confidence low, escalate to expensive model | 40-60% (only escalates when needed) |
| Bedrock Intelligent Routing | Bedrock automatically routes to optimal model based on query | Managed, no custom logic |
| Cross-Region Inference | Route to region with available capacity (avoids throttling) | Availability, not cost (same pricing) |
Token Optimization Techniques
- Prompt compression: Remove redundant instructions, use concise wording, abbreviate examples
- Context window pruning: Only include relevant conversation history (last N turns, or summarize older turns)
- Output limiting: Set max_tokens to expected response size (don’t let model ramble)
- Efficient few-shot: Use 2-3 diverse examples instead of 10+ similar ones
- Response format: Request JSON/structured output (shorter than prose for data extraction)
Caching Strategies for GenAI
| Cache Type | How | Best For |
|---|---|---|
| Prompt Caching (Bedrock) | Bedrock caches the prompt prefix processing. Subsequent calls with same prefix are cheaper/faster. | Same system prompt + different user queries (90%+ hit rate) |
| Semantic Caching | Hash query semantically → if similar query was asked before, return cached response | FAQ-style queries, repeated questions |
| API Response Caching | API Gateway or CloudFront caches entire response for identical requests | Deterministic queries (same input = same output) |
| RAG Result Caching | Cache retrieved chunks for common queries (ElastiCache) | Reduce vector DB queries, speed up retrieval |
Provisioned Throughput vs On-Demand
| Mode | Pricing | Best For |
|---|---|---|
| On-Demand | Pay per input/output token (no commitment) | Variable workloads, development, low volume |
| Provisioned Throughput | Reserved model units (hourly rate, 1-6 month term) | Predictable high volume, guaranteed capacity, custom models |
| Batch Inference | 50% cheaper than on-demand, async processing | Non-time-sensitive bulk processing (document summarization, classification) |
Latency Optimization
- Streaming responses: Return tokens as generated (InvokeModelWithResponseStream). User sees response building in real-time. Perceived latency drops dramatically.
- Smaller models: Haiku/Lite models respond 3-5x faster than large models. Use for simple tasks.
- Parallel requests: For multi-step tasks, run independent sub-tasks in parallel (Step Functions parallel state).
- Pre-computation: For predictable queries, pre-generate responses during off-peak and cache.
- Edge inference: For ultra-low latency, deploy smaller models on SageMaker endpoints closer to users.
Exam Tips
| Exam | Key Points |
|---|---|
| AIP-C01 | “Reduce FM cost” → model routing (cheap model first), token optimization, prompt caching. “Reduce latency” → streaming, smaller model, parallel requests. “Guaranteed throughput” → Provisioned Throughput. “Bulk processing cheaply” → Batch Inference. “Same system prompt repeated” → Prompt Caching. “Avoid redundant FM calls” → Semantic caching. “Handle traffic spikes” → Cross-Region Inference. |
AWS Certification Exam Practice Questions
Question 1:
A company’s GenAI chatbot costs $50,000/month using Claude Sonnet for all queries. Analysis shows 70% of queries are simple FAQ-type questions. How can they reduce costs while maintaining quality for complex queries?
- Switch all traffic to the cheapest model available
- Implement tiered model routing: classify queries → route simple to Haiku (cheap), complex to Sonnet (expensive)
- Cache all responses and never call the FM again
- Reduce the context window to 1000 tokens for all queries
Show Answer
Answer: B — Tiered routing sends 70% of simple queries to Haiku (10-20x cheaper than Sonnet) while complex queries still get Sonnet quality. Estimated savings: 60-70%. Use a lightweight classifier (or smaller model) to determine complexity. Switching everything to cheap (A) loses quality on complex queries. Caching everything (C) serves stale answers. Reducing context (D) hurts all queries equally.
Question 2:
An application sends the same 2000-token system prompt with every API call to Bedrock. The user query varies but the system prompt is identical across thousands of daily requests. Which feature reduces the cost of this repeated system prompt processing?
- Batch inference to process all requests at once
- Bedrock Prompt Caching — the system prompt is processed once and cached for subsequent requests
- Store the system prompt in DynamoDB instead of sending it each time
- Use a smaller model that processes system prompts faster
Show Answer
Answer: B — Bedrock Prompt Caching caches the processing of the prompt prefix (system prompt + any static context). Subsequent calls with the same prefix skip re-processing those tokens — paying only for the new (user query) tokens. For a 2000-token system prompt called 1000x/day, this significantly reduces both cost and latency. DynamoDB (C) stores text but still sends it to the model each time.
Question 3:
A company needs to process 100,000 product descriptions through an FM to generate marketing copy. The task is not time-sensitive — results needed within 24 hours. They want the lowest possible cost. Which inference mode should they use?
- On-demand inference with parallel Lambda invocations
- Provisioned Throughput with a 1-month commitment
- Batch Inference — submit all inputs as a batch job, results returned asynchronously at 50% discount
- Deploy a custom model on SageMaker spot instances
Show Answer
Answer: C — Batch Inference is designed for exactly this: large volume, non-time-sensitive processing. You submit inputs as a batch, Bedrock processes them asynchronously, results are delivered to S3. Pricing is 50% cheaper than on-demand. On-demand (A) works but costs 2x more. Provisioned (B) has monthly commitment, overkill for a one-time batch. This is the standard pattern for bulk content generation.
Question 4:
A real-time chatbot has a first-token-latency of 3 seconds (users wait 3s before seeing any response). Users report poor experience. The response quality is acceptable once it appears. How can the perceived latency be improved without changing the model?
- Increase the model’s max_tokens parameter
- Enable response streaming (InvokeModelWithResponseStream) — users see tokens as they’re generated
- Add CloudFront caching in front of the API
- Use provisioned throughput to reduce processing time
Show Answer
Answer: B — Streaming delivers tokens incrementally as they’re generated. Instead of waiting 3 seconds for the complete response, users see the first token in ~200-500ms and the response builds in real-time. Total generation time is the same, but perceived latency drops from 3s to <1s. This is the standard UX pattern for chatbots. CloudFront (C) can’t cache personalized responses. Provisioned throughput (D) helps with throughput, not necessarily first-token latency.
Question 5:
During peak hours, a GenAI application receives throttling errors (429) from Bedrock because the on-demand throughput limit is exceeded in the primary region. The application needs to maintain availability without over-provisioning. Which feature addresses this?
- Implement exponential backoff with jitter on retries
- Enable Bedrock Cross-Region Inference to automatically route requests to regions with available capacity
- Queue requests in SQS and process them one at a time
- Switch to a different model that has higher limits
Show Answer
Answer: B — Cross-Region Inference automatically routes requests to AWS regions with available model capacity when your primary region is at capacity. No code changes needed — Bedrock handles the routing transparently. This provides burst capacity beyond a single region’s limits without provisioned throughput costs. Retries (A) help but still hit the same region limit. SQS (C) adds latency. Cross-Region is the scalability answer for on-demand usage.
Related Posts
- Agentic AI Architecture
- RAG Advanced Patterns
- Bedrock vs SageMaker
- AWS Cost Optimization Architecture
References
- Provisioned Throughput — AWS Docs
- Prompt Caching — AWS Blog
- Batch Inference — AWS Docs
- Optimize GenAI Costs — AWS Blog
Frequently Asked Questions
When should I use Provisioned Throughput vs On-Demand?
On-Demand: Variable/unpredictable workloads, development, or low volume (<$10K/month). Provisioned Throughput: Predictable sustained usage, need guaranteed capacity (no throttling), or deploying custom/fine-tuned models. Break-even is typically when on-demand spend exceeds $5-10K/month consistently. Provisioned requires 1-6 month commitment.
What is the difference between Prompt Caching and Semantic Caching?
Prompt Caching (Bedrock feature): Caches the internal model processing of your prompt prefix. Same prompt prefix → faster/cheaper processing of new completions. Semantic Caching (you implement with ElastiCache): Caches entire Q&A pairs. If a similar question was asked before, return the cached answer without calling the FM at all. Prompt caching saves per-call cost; semantic caching eliminates calls entirely.
How does model routing reduce costs?
Most real-world traffic is bimodal: many simple queries (FAQ, greetings, lookups) and fewer complex queries (analysis, reasoning, creative). Route simple queries to cheap/fast models (Haiku: ~$0.25/M tokens) and complex queries to capable models (Sonnet: ~$3/M tokens). A simple classifier (or the cheap model’s confidence score) determines routing. Typical savings: 60-80% with no quality loss.