Cloud Composer

Cloud Composer is a managed workflow orchestration service that can be used to author, schedule, and monitor pipelines that span across clouds and on-premises data centers. Cloud Composer allows you to use Apache Airflow without the hassle of creating and managing complex Airflow infrastructure.

Google Cloud Composer – Managed Apache Airflow

~ Last updated on : ~ jayendrapatil

Google Cloud Composer – Managed Service for Apache Airflow

📢 Service Rebranding (2025): Google Cloud Composer is now officially called Managed Service for Apache Airflow. This name change reinforces Google Cloud’s commitment to the open-source ecosystem. The service functionality remains the same. Cloud Composer 3 became GA in March 2025 with support for Apache Airflow 3.
  • Cloud Composer (now Managed Service for Apache Airflow) is a fully managed workflow orchestration service, built on Apache Airflow, enabling workflow creation that spans across clouds and on-premises data centers.
  • Cloud Composer requires no installation or management overhead.
  • Cloud Composer integrates with Cloud Logging and Cloud Monitoring to provide a central place to view all Airflow service and workflow logs.
  • Cloud Composer supports Data Lineage integration for tracking data movement across pipelines.

Cloud Composer Versions

  • Cloud Composer 3 (Gen 3) – GA since March 2025
    • Simplified network setup with hidden infrastructure components
    • Supports Apache Airflow 2 and Airflow 3
    • Evergreen versioning – automatic infrastructure updates
    • Per-task CPU and memory control for fine-grained resource allocation
    • CeleryKubernetes Executor (hybrid of Celery and Kubernetes executors)
    • Environment cluster is NOT deployed into your project (Google manages it)
    • Simplified Private IP networking (can switch between Public/Private in existing environment)
    • Database retention policy support
    • DAG processors as a separate scalable component
  • Cloud Composer 2 (Gen 2)
    • Autopilot mode GKE cluster with automatic scaling
    • Supports Airflow 2
    • Uses Celery Executor
    • Note: Versions 2.0.x will reach EOL on September 15, 2026. Versions 2.1.x+ continue to be supported.
  • Cloud Composer 1 (Legacy Gen 1) – ⚠️ Post-maintenance mode since March 2024. EOL: September 15, 2026.
    • Manual environment scaling
    • Infrastructure deployed to your projects
    • No further updates, bugfixes, or security patches
    • Recommended: Migrate to Cloud Composer 3

Cloud Composer Components

  • Cloud Composer helps define a series of tasks as Workflows executed within an Environment
  • Workflows are created using DAGs or Directed Acyclic Graphs
  • A DAG is a collection of tasks that are scheduled and executed, organized in a way that reflects their relationships and dependencies.
  • DAGs are stored in Cloud Storage
  • Each Task can represent anything from ingestion, transform, filtering, monitoring, preparing, etc.
  • Environments are self-contained Airflow deployments that work with other Google Cloud services using connectors built into Airflow.
    • In Composer 1 & 2: based on Google Kubernetes Engine clusters deployed in your project
    • In Composer 3: cluster is hidden/managed by Google (not deployed to your project)
  • Cloud Composer environment components include: Web Server, Scheduler, Workers, Database, Cloud Storage bucket, and (in Composer 3) DAG Processors and Triggerers.

Cloud Composer 3 – Key Features

  • Simplified Networking – streamlined network configuration; can toggle between Public and Private IP on existing environments
  • Hidden Infrastructure – no GKE cluster visible in your project; Google manages security and infrastructure
  • Evergreen Versioning – environments receive infrastructure improvements automatically; you control Airflow version upgrades
  • Per-task Resource Control – configure CPU and memory at the individual task level
  • CeleryKubernetes Executor – combines benefits of Celery (fast scheduling) with Kubernetes (resource isolation)
  • Highly Resilient Environments – multi-zone deployment for high availability
  • Scheduled Snapshots – automated backup and recovery
  • Custom Environment Bucket – use your own Cloud Storage bucket
  • Composer Local Development CLI – test DAGs locally before deploying
  • Workforce Identity Federation – support for external identity providers
  • CMEK (Customer-Managed Encryption Keys) – encrypt environment data with your own keys
  • VPC Service Controls – supported for security perimeter enforcement

Apache Airflow 3 Support (GA in Cloud Composer 3)

  • Apache Airflow 3.0 became GA in April 2025; supported in Cloud Composer 3 (June 2025)
  • Cloud Composer was the first hyperscaler to offer Airflow 3.1 (November 2025)
  • Key Airflow 3 features supported in Cloud Composer 3:
    • DAG Versioning – track changes, manage versions, and rollback with confidence
    • Event-driven Scheduling – trigger DAGs based on external events (file arrivals, database changes) rather than time-based schedules only
    • New React-based UI – modernized user interface with improved usability
    • Scheduler-managed Backfills – reprocess historical data more simply and robustly
    • Assets – define and track data dependencies between DAGs
    • Inference Execution and Hyper-parameter Tuning – native ML/AI workflow support
    • airflowctl CLI tool – new command-line interface for environment management
  • Features not yet supported in Airflow 3 on Composer:
    • DAG Bundles (other than LocalDagBundle)
    • Edge Executor and tasks in other programming languages
    • In-place or snapshot-based upgrades from Airflow 2 to Airflow 3

Cloud Composer Use Cases

  • Multi-cloud and Hybrid Orchestration – orchestrate workflows across GCP, AWS, Azure, and on-premises
  • ETL/ELT Pipelines – coordinate data extraction, transformation, and loading
  • ML/AI Workflows (MLOps) – orchestrate model training, evaluation, and deployment pipelines
  • Data Warehouse Loading – schedule and manage BigQuery, Dataflow, and Dataproc jobs
  • Infrastructure Automation – trigger and manage cloud resource provisioning

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your company has a hybrid cloud initiative. You have a complex data pipeline that moves data between cloud provider services and leverages services from each of the cloud providers. Which cloud-native service should you use to orchestrate the entire pipeline?
    1. Cloud Dataflow
    2. Cloud Composer
    3. Cloud Dataprep
    4. Cloud Dataproc
  2. Your company is working on a multi-cloud initiative. The data processing pipelines require creating workflows that connect data, transfer data, processing, and using services across clouds. What cloud-native tool should be used for orchestration?
    1. Cloud Scheduler
    2. Cloud Dataflow
    3. Cloud Composer
    4. Cloud Dataproc
  3. Your team needs to orchestrate a machine learning pipeline that includes data preprocessing in Dataflow, model training on Vertex AI, and model deployment. The pipeline must support DAG versioning and event-driven triggers when new data arrives in Cloud Storage. Which solution meets these requirements?
    1. Use Cloud Scheduler with Pub/Sub triggers
    2. Use Workflows with Eventarc triggers
    3. Use Cloud Composer 3 with Apache Airflow 3 event-driven scheduling
    4. Use Cloud Functions chained with Cloud Tasks
  4. Your organization currently uses Cloud Composer 1 for orchestrating ETL pipelines. You receive a notification that the service will reach end of life. What is the recommended migration path?
    1. Migrate to Cloud Workflows
    2. Migrate to Cloud Scheduler with Cloud Functions
    3. Stay on Cloud Composer 1 with manual patches
    4. Migrate to Cloud Composer 3 (Managed Service for Apache Airflow Gen 3)
  5. You are designing a workflow orchestration system that requires fine-grained per-task resource allocation, automatic infrastructure management, and the ability to switch between public and private networking without recreating the environment. Which Cloud Composer version should you use?
    1. Cloud Composer 1
    2. Cloud Composer 2 with custom node pools
    3. Cloud Composer 3
    4. Self-managed Apache Airflow on GKE
  6. A data engineering team wants their Airflow environment to automatically receive security patches and infrastructure improvements without manual version upgrades. Which Cloud Composer feature addresses this requirement?
    1. Auto-upgrade node pools
    2. Continuous deployment pipelines
    3. Cloud Composer 3 Evergreen Versioning
    4. Cloud Composer 2 auto-scaling

References