AWS MLOps Pipeline Architecture — Overview
MLOps (Machine Learning Operations) automates the lifecycle of ML models — from data preparation through training, evaluation, deployment, monitoring, and retraining. On AWS, SageMaker provides the end-to-end platform. This architecture is the core focus of MLA-C01 (Machine Learning Engineer) and appears in AIP-C01 for GenAI model customization workflows.
Glue / Processing
Feature Store
Data Wrangler
HyperPod
Built-in Algos
Custom Container
Bias (Clarify)
Explainability
Baseline Metrics
Version + Metadata
Approval Status
Lineage Tracking
Batch Transform
Serverless Inference
Shadow Testing
Data Drift
Quality Drift
Bias Drift
Pipeline Stages — Deep Dive
1. Data Preparation
| Component | Role | When to Use |
|---|---|---|
| S3 | Raw data lake — source of truth for all training data | Always — central data store |
| SageMaker Processing | Run data transformation jobs (Spark, scikit-learn, custom) | Feature engineering, data validation, post-processing |
| Feature Store | Centralized feature repository with online (real-time) and offline (batch) stores | Reusable features across models, consistent train/serve features |
| Data Wrangler | Visual data preparation with 300+ built-in transformations | Exploratory analysis, quick prototyping (exports to Processing jobs) |
2. Model Training
- Training Jobs — Managed compute: specify instance type, training script, hyperparameters, input/output S3 paths
- Built-in Algorithms — XGBoost, Linear Learner, K-Means, DeepAR, etc. — optimized for SageMaker with no custom code
- Custom Containers — Bring your own Docker image with any framework (PyTorch, TensorFlow, JAX)
- HyperPod — Managed distributed training clusters with automatic node recovery for large model training
- Automatic Model Tuning — Hyperparameter optimization (Bayesian, Random, Grid, Hyperband strategies)
- Experiments — Track every training run with parameters, metrics, and artifacts for comparison
3. Model Evaluation
- Model Quality — Accuracy, F1, AUC-ROC, RMSE against held-out test set
- SageMaker Clarify (Bias) — Pre-training bias (data imbalance) + post-training bias (disparate impact) metrics
- Clarify (Explainability) — SHAP values showing feature importance and individual prediction explanations
- Baseline creation — Establish performance/data statistics baseline for future drift monitoring
- Conditional approval — Pipeline step that auto-approves if metrics exceed threshold, otherwise requires manual review
4. Model Registry
- Model Groups — Logical collections of model versions (e.g., “fraud-detection-model”)
- Versioning — Each training run produces a versioned model package with metadata
- Approval workflow — Models start as “PendingApproval” → “Approved” (manual or automated) → eligible for deployment
- Lineage — Track which data, code, and parameters produced each model version
- Cross-account — Share model packages from ML account to production account via RAM or resource policies
5. Model Deployment
| Type | How It Works | Best For |
|---|---|---|
| Real-time Endpoint | Always-on instances serving predictions via HTTPS | Low-latency, synchronous inference (<100ms) |
| Serverless Inference | Auto-scales to zero, charges per inference | Intermittent/unpredictable traffic, cold start acceptable |
| Batch Transform | Process entire dataset in bulk, output to S3 | Nightly scoring, large-scale offline predictions |
| Async Inference | Queue requests, process async, scale to zero between | Large payloads, long processing time (minutes) |
| Shadow Testing | Route production traffic to new model in shadow (no impact) | Validate new model against production without risk |
6. Model Monitoring
- Data Quality Monitor — Detects drift in input feature distributions vs training baseline
- Model Quality Monitor — Detects degradation in prediction accuracy (requires ground truth labels)
- Bias Drift Monitor — Detects emerging bias in predictions over time
- Feature Attribution Drift — Detects changes in which features drive predictions (SHAP drift)
- Automated retraining — Monitor alert → EventBridge rule → triggers SageMaker Pipeline for retraining
SageMaker Pipelines — Orchestration
SageMaker Pipelines is the purpose-built ML workflow orchestrator:
- DAG-based — Define pipeline as directed acyclic graph of steps
- Step types: Processing, Training, Tuning, Transform, Register Model, Condition, Callback, Lambda, Fail
- Parameterized — Pipeline inputs (instance type, data path, threshold) configurable per run
- Caching — Steps with unchanged inputs/code skip execution, reusing previous outputs
- Integration — Trigger via EventBridge (on schedule, on S3 upload, on drift detection)
MLOps vs FMOps (GenAI Operations)
| Aspect | Traditional MLOps | FMOps / LLMOps |
|---|---|---|
| Model source | Train from scratch on your data | Pre-trained foundation model (fine-tune or prompt) |
| Customization | Feature engineering + algorithm selection + hyperparameter tuning | Prompt engineering + RAG + fine-tuning + model distillation |
| Evaluation | Accuracy, F1, AUC on test set | LLM-as-Judge, human evaluation, faithfulness, toxicity |
| Monitoring | Data drift, model quality drift | Response quality, hallucination rate, latency, cost per token |
| AWS Platform | SageMaker (full pipeline) | Bedrock (managed) + SageMaker (custom training) |
Exam Tips by Certification
| Exam | Focus Areas |
|---|---|
| MLA-C01 | SageMaker Pipelines steps, Feature Store (online vs offline), Model Monitor types, deployment strategies (A/B, shadow), Model Registry cross-account, Clarify bias metrics, instance selection for training vs inference |
| AIP-C01 | Fine-tuning vs RAG vs distillation decision, Bedrock model evaluation, FMOps differences, prompt management as version control, Guardrails as output monitoring |
AWS Certification Exam Practice Questions
Question 1:
A data science team trains multiple model versions weekly. They need to ensure only models meeting accuracy >0.95 and bias metric DI >0.8 are eligible for production deployment. How should they automate this in the pipeline?
- Manual review of each model’s metrics before deployment
- SageMaker Pipeline Condition step that checks metrics → auto-registers as “Approved” if passing, “Rejected” if not
- CloudWatch alarm on model accuracy with SNS notification
- Lambda function triggered by Model Registry events to check metrics
Show Answer
Answer: B – SageMaker Pipeline Condition steps evaluate expressions (e.g., accuracy > 0.95 AND bias_di > 0.8) and branch the pipeline accordingly. If conditions pass, the model is registered with “Approved” status. If not, it’s registered as “Rejected” or the pipeline fails. This is fully automated within the pipeline DAG.
Question 2:
An ML team notices their production model’s predictions are becoming less accurate over time, but the model hasn’t changed. What is the MOST likely cause and which monitoring capability would detect it?
- Model overfitting — detected by Model Quality Monitor
- Input data distribution drift — detected by Data Quality Monitor
- Concept drift — detected by Feature Attribution Drift Monitor
- Infrastructure issues — detected by CloudWatch endpoint metrics
Show Answer
Answer: B – When a model’s accuracy degrades without changes to the model itself, the most common cause is data drift — the real-world data has shifted away from what the model was trained on. Data Quality Monitor compares incoming feature distributions against the training baseline and alerts when statistical drift exceeds thresholds.
Question 3:
A company has separate AWS accounts for ML development and production. Their data scientists train models in the dev account and need to deploy approved models to the production account. What is the recommended approach?
- Copy model artifacts to production S3 and recreate endpoints manually
- Model Registry in dev account with cross-account resource policy, deployment pipeline in production account
- Use the same SageMaker endpoint across accounts via VPC peering
- Deploy from dev account directly to production account using admin credentials
Show Answer
Answer: B – The Model Registry supports cross-account access via resource policies. Data scientists register approved models in the dev account. A deployment pipeline in the production account (triggered by approval status change) pulls the model package and creates/updates the endpoint. This maintains separation of duties and least privilege.
Question 4:
A model serves predictions in real-time but traffic is highly variable — heavy during business hours, near-zero at night. The team wants to minimize costs while maintaining low latency during peak. Which deployment option fits?
- Real-time endpoint with a fixed instance count sized for peak
- Real-time endpoint with auto-scaling policy (target tracking on InvocationsPerInstance)
- Serverless Inference endpoint
- Batch Transform on a schedule
Show Answer
Answer: B – Auto-scaling on a real-time endpoint scales instances up during peak and down (to a minimum) during low traffic. This optimizes cost while maintaining low latency. Serverless would also work but has cold-start latency (~seconds) which may not meet low-latency requirements. Fixed sizing wastes money at night.
Question 5:
An ML engineer wants to ensure consistent features between model training (batch from S3) and real-time inference (API calls). Features are computed from raw data using complex transformations. Which SageMaker capability solves this?
- SageMaker Processing jobs for both batch and real-time
- SageMaker Feature Store with offline store (training) and online store (inference)
- Lambda function computing features at inference time using the same code as training
- Store pre-computed features in DynamoDB for both use cases
Show Answer
Answer: B – Feature Store provides dual storage: the offline store (S3-backed) serves training with historical features, while the online store (low-latency) serves inference with the latest feature values. Both stores are populated by the same ingestion pipeline, guaranteeing consistency between training and serving — eliminating the common “training-serving skew” problem.
Frequently Asked Questions
What is the difference between SageMaker Pipelines and Step Functions?
SageMaker Pipelines is purpose-built for ML workflows with native integration to training jobs, processing, model registry, and monitoring. Step Functions is a general-purpose workflow orchestrator. Use Pipelines for ML-specific DAGs; use Step Functions when orchestrating ML steps alongside non-ML services (API calls, human approvals, other AWS services).
When should I retrain my model?
Retrain when Model Monitor detects data drift or quality degradation beyond your threshold. Common triggers: weekly/monthly schedule, data drift alerts (statistical tests), accuracy drop below baseline, or new labeled data availability. Automate with EventBridge → Pipeline trigger.
What is shadow testing?
Shadow testing routes a copy of production traffic to a new model version without affecting users. The new model’s predictions are logged but not served to users. You compare shadow predictions against the production model’s results to validate the new version before switching traffic.