Cheat Sheet

AWS Data Analytics Services Cheat Sheet

AWS Data Analytics Services
~ Last updated on : ~ jayendrapatil ~ 7 Comments

AWS Data Analytics Services Cheat Sheet

📋 Last Updated: June 2026. This post has been updated to reflect service renamings (Kinesis Data Firehose → Amazon Data Firehose, Kinesis Data Analytics → Amazon Managed Service for Apache Flink, Elasticsearch → OpenSearch Service, QuickSight → Quick Suite), deprecations (AWS Data Pipeline, Kinesis Data Analytics for SQL), and major new features (Zero-ETL integrations, MSK Express brokers, Glue 5.0, SageMaker Lakehouse).

AWS Data Analytics Services

Kinesis Data Streams – KDS

  • enables real-time processing of streaming data at a massive scale
  • provides ordering of records per shard
  • provides an ability to read and/or replay records in the same order
  • allows multiple applications to consume the same data
  • data is replicated across three data centers within a region
  • data is preserved for 24 hours, by default, and can be extended to 365 days
  • data inserted in Kinesis, it can’t be deleted (immutability) but only expires
  • streams can be scaled using multiple shards, based on the partition key
  • each shard provides the capacity of 1MB/sec data input and 2MB/sec data output with 1000 PUT requests per second
  • supports two capacity modes:
    • Provisioned mode – you manage the number of shards
    • On-demand mode – automatically scales to accommodate up to 10 GB/s write and 20 GB/s read throughput per stream
  • On-demand Advantage mode (launched Nov 2025) – enables on-demand streams to handle instant throughput increases via warm throughput capability, with up to 10GB/s or 10 million events/second, eliminating over-provisioning needs and offering 60%+ cost savings for consistent workloads
  • supports record sizes up to 10 MiB (increased from 1 MiB in Oct 2025)
  • supports up to 50 enhanced fan-out consumers per stream (increased from 20 in Nov 2025)
  • Kinesis vs SQS
    • real-time processing of streaming big data vs reliable, highly scalable hosted queue for storing messages
    • ordered records, as well as the ability to read and/or replay records in the same order vs no guarantee on data ordering (with the standard queues before the FIFO queue feature was released)
    • data storage up to 24 hours, extended to 365 days vs 1 minute to extended to 14 days but cleared if deleted by the consumer.
    • supports multiple consumers vs a single consumer at a time and requires multiple queues to deliver messages to multiple consumers.
  • Kinesis Producer
    • API
      • PutRecord and PutRecords are synchronous
      • PutRecords uses batching and increases throughput
      • might experience ProvisionedThroughputExceeded Exceptions, when sending more data. Use retries with backoff, resharding, or change partition key.
    • KPL
      • producer supports synchronous or asynchronous use cases
      • supports inbuilt batching and retry mechanism
      • ⚠️ KPL 0.x reached end-of-support on January 30, 2026. Migrate to KPL 1.x.
    • Kinesis Agent can help monitor log files and send them to KDS
    • supports third-party libraries like Spark, Flume, Kafka connect, etc.
  • Kinesis Consumers
    • Kinesis SDK
      • Records are polled by consumers from a shard
    • Kinesis Client Library (KCL)
      • Read records from Kinesis produced with the KPL (de-aggregation)
      • supports the checkpointing feature to keep track of the application’s state and resume progress using the DynamoDB table.
      • if application receives provisioned-throughput exceptions, increase the provisioned throughput for the DynamoDB table
      • ⚠️ KCL 1.x reached end-of-support on January 30, 2026. Migrate to KCL 2.x.
    • Kinesis Connector Library – can be replaced using Firehose or Lambda
    • Third-party libraries: Spark, Log4J Appenders, Flume, Kafka Connect…
    • Amazon Data Firehose, AWS Lambda
    • Kinesis Consumer Enhanced Fan-Out
      • supports Multiple Consumer applications for the same Stream
      • provides Low Latency ~70ms
      • Higher costs
      • now supports up to 50 consumers per stream
  • Kinesis Security
    • allows access/authorization control using IAM policies
    • supports Encryption in flight using HTTPS endpoints
    • supports data encryption at rest using either server-side encryption with KMS or using client-side encryption before pushing the data to data streams.
    • supports VPC Endpoints to access within VPC

Amazon Data Firehose

(Previously known as Amazon Kinesis Data Firehose, renamed February 2024)

  • data transfer solution for delivering near real-time streaming data to destinations such as S3, Redshift, OpenSearch Service, Splunk, Snowflake, and other 3rd-party analytics services.
  • is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration
  • is Near Real Time (min. 60 secs) as it buffers incoming streaming data to a certain size or for a certain period of time before delivering it
  • supports batching, compression, and encryption of the data before loading it, minimizing the amount of storage used at the destination and increasing security
  • supports data compression, minimizing the amount of storage used at the destination. It currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Redshift.
  • supports out of box data transformation as well as custom transformation using Lambda function to transform incoming source data and deliver the transformed data to destinations
  • uses at least once semantics for data delivery.
  • supports multiple producers as datasource, which include Kinesis data stream, KPL, Kinesis Agent, or the Data Firehose API using the AWS SDK, CloudWatch Logs, CloudWatch Events, or AWS IoT
  • does NOT support consumers like Spark and KCL
  • supports interface VPC endpoint to keep traffic between the VPC and Data Firehose from leaving the Amazon network.
  • Apache Iceberg Tables destination (launched 2024) – delivers streaming data directly into Apache Iceberg format tables in S3 and S3 Tables, supporting record routing to different Iceberg tables, CDC replication from databases, schema evolution, and ACID transactions.
  • Database CDC replication (Preview 2024) – supports continuous replication of database changes from MySQL and PostgreSQL directly into Apache Iceberg Tables in S3.

Kinesis Data Streams vs Amazon Data Firehose

Amazon Managed Service for Apache Flink

(Previously known as Amazon Kinesis Data Analytics, renamed August 2023)

⚠️ Kinesis Data Analytics for SQL was discontinued on January 27, 2026. Migrate to Amazon Managed Service for Apache Flink or Apache Flink Studio for real-time stream processing.

  • helps analyze streaming data, gain actionable insights, and respond to the business and customer needs in real time.
  • is a fully managed and serverless service for building and running real-time streaming applications using Apache Flink.
  • reduces the complexity of building, managing, and integrating streaming applications with other AWS services.
  • supports Apache Flink applications written in Java, Scala, Python, and SQL.
  • provides automatic scaling, high availability, and exactly-once processing semantics.
  • integrates with Kinesis Data Streams, Amazon MSK, and Amazon S3 as data sources and sinks.

Managed Streaming for Kafka – MSK

  • Managed Streaming for Kafka- MSK is an AWS streaming data service that manages Apache Kafka infrastructure and operations.
  • makes it easy for developers and DevOps managers to run Kafka applications and Kafka Connect connectors on AWS, without the need to become experts in operating Kafka.
  • operates, maintains, and scales Kafka clusters, provides enterprise-grade security features out of the box, and has built-in AWS integrations that accelerate development of streaming data applications.
  • always runs within a VPC managed by the MSK and is available to your own selected VPC, subnet, and security group when the cluster is setup.
  • IP addresses from the VPC are attached to the MSK resources through elastic network interfaces (ENIs), and all network traffic stays within the AWS network and is not accessible to the internet by default.
  • integrates with CloudWatch for monitoring, metrics, and logging.
  • MSK Serverless is a cluster type for MSK that makes it easy for you to run Apache Kafka clusters without having to manage compute and storage capacity.
  • MSK Express Brokers (GA November 2024) – a new broker type for MSK Provisioned designed to deliver:
    • up to 3x more throughput per broker (500 MBps ingress, 1000 MBps egress on m7g instances)
    • up to 20x faster scaling
    • 90% faster recovery from failures
    • up to 5x more partitions per broker
    • virtually unlimited storage with instant storage scaling
    • supports Intelligent Rebalancing for 180x faster operation performance
  • supports EBS server-side encryption using KMS to encrypt storage.
  • supports encryption in transit enabled via TLS for inter-broker communication.
  • For provisioned clusters, you have three options:
    • IAM Access Control for both AuthN/Z (recommended),
    • TLS certificate authentication (CA) for AuthN and access control lists for AuthZ
    • SASL/SCRAM for AuthN and access control lists for AuthZ.
  • For serverless clusters, IAM Access Control can be used for both authentication and authorization.

Redshift

  • Redshift is a fast, fully managed data warehouse
  • provides simple and cost-effective solutions to analyze all the data using standard SQL and the existing Business Intelligence (BI) tools.
  • manages the work needed to set up, operate, and scale a data warehouse, from provisioning the infrastructure capacity to automating ongoing administrative tasks such as backups, and patching.
  • automatically monitors your nodes and drives to help you recover from failures.
  • only supported Single-AZ deployments. However, now supports Multi-AZ deployments.
  • replicates all the data within the data warehouse cluster when it is loaded and also continuously backs up your data to S3.
  • attempts to maintain at least three copies of your data (the original and replica on the compute nodes and a backup in S3).
  • supports cross-region snapshot replication to another region for disaster recovery
  • Redshift supports four distribution styles; AUTO, EVEN, KEY, or ALL.
    • KEY distribution uses a single column as distribution key (DISTKEY) and helps place matching values on the same node slice
    • Even distribution distributes the rows across the slices in a round-robin fashion, regardless of the values in any particular column
    • ALL distribution replicates whole table in every compute node.
    • AUTO distribution lets Redshift assigns an optimal distribution style based on the size of the table data
  • Redshift supports Compound and Interleaved sort keys
    • Compound key
      • is made up of all of the columns listed in the sort key definition, in the order they are listed and is more efficient when query predicates use a prefix, or query’s filter applies conditions, such as filters and joins, which is a subset of the sort key columns in order.
    • Interleaved sort key
      • gives equal weight to each column in the sort key, so query predicates can use any subset of the columns that make up the sort key, in any order.
      • Not ideal for monotonically increasing attributes
  • Import/Export Data
    • UNLOAD helps copy data from Redshift table to S3
    • COPY command
      • helps copy data from S3 to Redshift
      • also supports EMR, DynamoDB, remote hosts using SSH
      • parallelized and efficient
      • can decrypt data as it is loaded from S3
      • DON’T use multiple concurrent COPY commands to load one table from multiple files as Redshift is forced to perform a serialized load, which is much slower.
      • supports data decryption when loading data, if data encrypted
      • supports decompressing data, if data is compressed.
    • Split the Load Data into Multiple Files
    • Load the data in sort key order to avoid needing to vacuum.
    • Use a Manifest File
      • provides Data consistency, to avoid S3 eventual consistency issues
      • helps specify different S3 locations in a more efficient way that with the use of S3 prefixes.
  • Zero-ETL Integrations (2024-2025)
    • enable near real-time analytics by connecting operational databases and applications to Redshift without building data pipelines
    • supports integrations from Aurora (MySQL/PostgreSQL), DynamoDB, RDS, and third-party applications (Salesforce, SAP, Zendesk)
    • works with both Redshift Serverless workgroups and provisioned clusters using RA3 instance types
    • includes SQL features: QUERY_ALL_STATES, TRUNCATECOLUMNS, and ACCEPTINVCHARS for zero-ETL data handling
    • integrates with Amazon SageMaker Lakehouse for unified analytics and AI/ML
  • Redshift Distribution Style determines how data is distributed across compute nodes and helps minimize the impact of the redistribution step by locating the data where it needs to be before the query is executed.
  • Redshift Enhanced VPC routing forces all COPY and UNLOAD traffic between the cluster and the data repositories through the VPC.
  • Workload management (WLM) enables users to flexibly manage priorities within workloads so that short, fast-running queries won’t get stuck in queues behind long-running queries.
  • Redshift Spectrum helps query and retrieve structured and semistructured data from files in S3 without having to load the data into Redshift tables.
    • Redshift Spectrum external tables are read-only. You can’t COPY or INSERT to an external table.
  • Federated Query feature allows querying and analyzing data across operational databases, data warehouses, and data lakes.
  • Short query acceleration (SQA) prioritizes selected short-running queries ahead of longer-running queries.
  • Redshift Serverless is a serverless option of Redshift that makes it more efficient to run and scale analytics in seconds without the need to set up and manage data warehouse infrastructure.

EMR

  • is a web service that utilizes a hosted Hadoop framework running on the web-scale infrastructure of EC2 and S3
  • launches all nodes for a given cluster in the same Availability Zone, which improves performance as it provides a higher data access rate.
  • seamlessly supports Reserved, On-Demand, and Spot Instances
  • consists of Master/Primary Node for management and Slave nodes, which consist of Core nodes holding data and providing compute and Task nodes for performing tasks only.
  • is fault tolerant for slave node failures and continues job execution if a slave node goes down
  • supports Persistent and Transient cluster types
    • Persistent EMR clusters continue to run after the data processing job is complete
    • Transient EMR clusters shut down when the job or the steps (series of jobs) are complete
  • supports EMRFS which allows S3 to be used as a durable HA data storage
  • EMR Serverless helps run big data frameworks such as Apache Spark and Apache Hive without configuring, managing, and scaling clusters.
    • now supports Spark Connect (2026) for interactive PySpark development from local environments, IDEs, and SageMaker Unified Studio Notebooks
    • eliminates local storage provisioning, reducing costs by up to 20%
  • Apache Spark 4.0 support (GA 2026) – includes VARIANT data type, state-management improvements, and Spark Connect, with EMR optimized runtime running workloads up to 4.5x faster than open-source Spark
  • EMR Studio is an IDE that helps data scientists and data engineers to develop, visualize, and debug data engineering and data science applications written in R, Python, Scala, and PySpark.
  • EMR Notebooks provide a managed environment, based on Jupyter Notebook, that helps prepare and visualize data, collaborate with peers, build applications, and perform interactive analysis using EMR clusters.

Glue

  • AWS Glue is a fully managed, ETL service that automates the time-consuming steps of data preparation for analytics.
  • is serverless and supports a pay-as-you-go model.
  • handles provisioning, configuration, and scaling of the resources required to run the ETL jobs on a fully managed, scale-out Apache Spark environment.
  • helps setup, orchestrate, and monitor complex data flows.
  • supports custom Scala or Python code and import custom libraries and Jar files into the AWS Glue ETL jobs to access data sources not natively supported by AWS Glue.
  • supports server side encryption for data at rest and SSL for data in motion.
  • provides development endpoints to edit, debug, and test the code it generates.
  • AWS Glue natively supports data stored in RDS, Redshift, DynamoDB, S3, MySQL, Oracle, Microsoft SQL Server, and PostgreSQL databases in the VPC running on EC2 and Data streams from MSK, Kinesis Data Streams, and Apache Kafka.
  • Glue ETL engine to Extract, Transform, and Load data that can automatically generate Scala or Python code.
  • AWS Glue 5.0/5.1 (2024-2026):
    • provides performance-optimized Apache Spark 3.5 runtime for batch and stream processing
    • native support for open table formats: Apache Iceberg, Delta Lake, and Apache Hudi
    • Spark-native fine-grained access control (FGAC) integration with AWS Lake Formation
    • faster job start times and automatic partition pruning
    • Glue 5.1 adds support for Apache Iceberg format version 3.0, deletion vectors, and row lineage tracking
    • new worker types: G.12X, G.16X general compute, and R.1X/R.2X/R.4X/R.8X memory-optimized
  • Glue Materialized Views (2025) – Apache Iceberg-based materialized views for transforming data and accelerating query performance
  • supports generative AI assistance for data integration tasks
  • Glue Data Catalog is a central repository and persistent metadata store to store structural and operational metadata for all the data assets.
  • Glue Crawlers scan various data stores to automatically infer schemas and partition structures to populate the Data Catalog with corresponding table definitions and statistics.
  • Glue Job Bookmark tracks data that has already been processed during a previous run of an ETL job by persisting state information from the job run.
  • AWS Glue Streaming ETL enables performing ETL operations on streaming data using continuously-running jobs.
  • Glue provides flexible scheduler that handles dependency resolution, job monitoring, and retries.
  • Glue Studio offers a graphical interface for authoring AWS Glue jobs to process data allowing you to define the flow of the data sources, transformations, and targets in the visual interface and generating Apache Spark code on your behalf.
  • Glue Data Quality helps reduces manual data quality effort by automatically measuring and monitoring the quality of data in data lakes and pipelines.
  • Glue DataBrew is a visual data preparation tool that makes it easy for data analysts and data scientists to prepare, visualize, clean, and normalize terabytes, and even petabytes of data directly from your data lake, data warehouses, and databases, including S3, Redshift, Aurora, and RDS.
  • ⚠️ AWS Glue for Ray will no longer accept new customers starting April 30, 2026. Existing customers can continue using the service. Explore Amazon EKS for similar capabilities.

Lake Formation

  • AWS Lake Formation helps create secure data lakes, making data available for wide-ranging analytics.
  • is an integrated data lake service that helps to discover, ingest, clean, catalog, transform, and secure data and make it available for analysis and ML.
  • automatically manages access to the registered data in S3 through services including AWS Glue, Athena, Redshift, QuickSight, and EMR using Zeppelin notebooks with Apache Spark to ensure compliance with your defined policies.
  • helps configure and manage your data lake without manually integrating multiple underlying AWS services.
  • uses a shared infrastructure with AWS Glue, including console controls, ETL code creation and job monitoring, blueprints to create workflows for data ingest, the same data catalog, and a serverless architecture.
  • can manage data ingestion through AWS Glue. Data is automatically classified, and relevant data definitions, schema, and metadata are stored in the central Glue Data Catalog. Once the data is in the S3 data lake, access policies, including table-and-column-level access controls can be defined, and encryption for data at rest enforced.
  • integrates with IAM so authenticated users and roles can be automatically mapped to data protection policies that are stored in the data catalog. The IAM integration also supports Microsoft Active Directory or LDAP to federate into IAM using SAML.
  • helps centralize data access policy controls. Users and roles can be defined to control access, down to the table and column level.
  • supports fine-grained access control (FGAC) including row-level and cell-level security, and tag-based access control (LF-Tags) for scalable permission management.
  • supports private endpoints in the VPC and records all activity in AWS CloudTrail for network isolation and auditability.
  • ⚠️ Lake Formation’s Governed Tables feature was deprecated in February 2025. Use Apache Iceberg tables with Lake Formation for transactional data lake capabilities.

Amazon Quick Suite (formerly QuickSight)

(Amazon QuickSight evolved to Amazon Quick Suite on October 9, 2025, expanding from a single BI product to a comprehensive analytics and AI platform)

  • is a cloud-powered business analytics service that integrates BI capabilities with AI agents for business insights, deep research, and automation in one unified experience.
  • delivers fast and responsive query performance by using a robust in-memory engine (SPICE).
    • “SPICE” stands for a Super-fast, Parallel, In-memory Calculation Engine
    • can also be configured to keep the data in SPICE up-to-date as the data in the underlying sources change.
    • automatically replicates data for high availability and enables Quick Suite to scale to support users to perform simultaneous fast interactive analysis across a wide variety of AWS data sources.
  • Amazon Q in QuickSight (GA April 2024) – generative BI capabilities powered by Amazon Bedrock:
    • multi-visual data Q&A for asking questions of data not in dashboards
    • executive summaries for quick trend and insight discovery
    • automated stories – documents and slides explaining data
    • natural language generation for pixel-perfect reports
    • available to all Enterprise Edition users without separate Q add-on
  • supports
    • Excel files and flat files like CSV, TSV, CLF, ELF
    • on-premises databases like PostgreSQL, SQL Server and MySQL
    • SaaS applications like Salesforce
    • and AWS data sources such as Redshift, RDS, Aurora, Athena, and S3
  • supports various functions to format and transform the data.
  • supports assorted visualizations that facilitate different analytical approaches:
    • Comparison and distribution – Bar charts (several assorted variants)
    • Changes over time – Line graphs, Area line charts
    • Correlation – Scatter plots, Heat maps
    • Aggregation – Pie graphs, Tree maps
    • Tabular – Pivot tables
  • Amazon Quick Sight (a capability within Quick Suite) now offers visual data preparation experience for advanced data transformations without code.

Data Pipeline

⚠️ AWS Data Pipeline – No Longer Available to New Customers (July 25, 2024)

AWS Data Pipeline is in maintenance mode and is no longer available to new customers. Console access was removed on April 30, 2023. Existing customers can continue to use the service via CLI and API.

Migration Options:

  • Amazon MWAA (Managed Workflows for Apache Airflow) – for complex workflow orchestration
  • AWS Step Functions – for serverless workflow orchestration
  • AWS Glue – for ETL-focused data movement pipelines
  • Amazon EventBridge – for event-driven scheduling
  • orchestration service that helps define data-driven workflows to automate and schedule regular data movement and data processing activities
  • integrates with on-premises and cloud-based storage systems
  • allows scheduling, retry, and failure logic for the workflows

Amazon OpenSearch Service

(Previously known as Amazon Elasticsearch Service, renamed September 8, 2021)

  • Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud.
  • OpenSearch Service provides
    • real-time, distributed search and analytics engine
    • ability to provision all the resources for OpenSearch cluster and launches the cluster
    • easy to use cluster scaling options. Scaling OpenSearch Service domain by adding or modifying instances, and storage volumes is an online operation that does not require any downtime.
    • provides self-healing clusters, which automatically detects and replaces failed nodes, reducing the overhead associated with self-managed infrastructures
    • domain snapshots to back up and restore domains and replicate domains across AZs
    • enhanced security with IAM, Network, Domain access policies, and fine-grained access control
    • storage volumes for the data using EBS volumes
    • ability to span cluster nodes across multiple AZs in the same region, known as zone awareness, for high availability and redundancy. OpenSearch Service automatically distributes the primary and replica shards across instances in different AZs.
    • dedicated master nodes to improve cluster stability
    • data visualization using OpenSearch Dashboards (formerly Kibana)
    • integration with CloudWatch for monitoring domain metrics
    • integration with CloudTrail for auditing configuration API calls to domains
    • integration with S3, Kinesis, and DynamoDB for loading streaming data
    • ability to handle structured and Unstructured data
    • supports encryption at rest through KMS, node-to-node encryption over TLS, and the ability to require clients to communicate with HTTPS
  • Amazon OpenSearch Serverless
    • automatically scales without managing infrastructure
    • NextGen architecture (2026) – decoupled compute from storage, provisions in seconds, scales to zero when idle, up to 20x faster autoscaling, and up to 60% lower cost than provisioned clusters
    • two collection architectures: Classic (original) and NextGen (default for new collections)
  • Vector Database Capabilities
    • stores vector embeddings from LLMs for semantic/similarity search
    • supports hybrid search combining vector, lexical, and agentic retrieval
    • GPU-accelerated vector indexes for billion-scale databases (2025)
    • auto-optimized vector indexes for search quality/speed/cost tradeoffs
    • integrates with Amazon Bedrock for RAG and agentic AI applications
  • Zero-ETL integrations – direct data access from other AWS services without pipeline management
  • Extended Support – Standard Support ends Nov 7, 2025 for legacy Elasticsearch versions up to 6.7, ES 7.1-7.8, and OpenSearch 1.0-1.2

Athena

  • Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats.
  • provides a simplified, flexible way to analyze petabytes of data in an S3 data lake and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python without loading the data.
  • is built on open-source Trino and Presto engines and Apache Spark frameworks, with no provisioning or configuration effort required.
  • is highly available and runs queries using compute resources across multiple facilities, automatically routing queries appropriately if a particular facility is unreachable
  • can process unstructured, semi-structured, and structured datasets.
  • integrates with QuickSight for visualizing the data or creating dashboards.
  • supports various standard data formats, including CSV, TSV, JSON, ORC, Avro, and Parquet.
  • supports compressed data in Snappy, Zlib, LZO, and GZIP formats. You can improve performance and reduce costs by compressing, partitioning, and using columnar formats.
  • can handle complex analysis, including large joins, window functions, and arrays
  • uses a managed Glue Data Catalog to store information and schemas about the databases and tables that you create for the data stored in S3
  • uses schema-on-read technology, which means that the table definitions are applied to the data in S3 when queries are being applied. There’s no data loading or transformation required. Table definitions and schema can be deleted without impacting the underlying data stored in S3.
  • supports fine-grained access control with AWS Lake Formation which allows for centrally managing permissions and access control for data catalog resources in the S3 data lake.
  • integrates with Amazon SageMaker Lakehouse for governed federated queries across data sources

Amazon SageMaker Lakehouse

(Launched at re:Invent 2024, GA March 2025)

  • part of the next generation of Amazon SageMaker – a unified platform for data, analytics, and AI
  • unifies data across S3 data lakes (including S3 Tables), Redshift data warehouses, and operational databases
  • supports zero-ETL integrations from Aurora, DynamoDB, RDS, and third-party applications (Salesforce, SAP, Zendesk) for near real-time data access
  • enables querying, analyzing, and joining data using Redshift, Athena, EMR, and AWS Glue
  • provides unified access through Amazon SageMaker Unified Studio – a single development experience for data engineers, data scientists, and analysts
  • supports Apache Iceberg open table format for interoperability
  • integrates with Lake Formation for fine-grained access control and governance

Google Cloud Data Analytics Services Cheat Sheet

~ Last updated on : ~ jayendrapatil

Cloud Pub/Sub

  • Pub/Sub is a fully managed, asynchronous messaging service designed to be highly reliable and scalable with latencies on the order of 100 ms
  • Pub/Sub offers at-least-once message delivery and best-effort ordering to existing subscribers
  • Pub/Sub also supports exactly-once delivery (GA since 2022) for pull subscriptions and StreamingPull API, ensuring messages are not redelivered after successful acknowledgment. Push and export subscriptions do not support exactly-once delivery.
  • Pub/Sub enables the creation of event producers and consumers, called publishers and subscribers.
  • Pub/Sub messages should be no greater than 10MB in size.
  • Messages can be received with pull or push delivery.
  • Messages published before a subscription is created will not be delivered to that subscription
  • Acknowledged messages are no longer available to subscribers and are deleted, by default. However, can be retained setting retention period.
  • Publishers can send messages with an ordering key and message ordering is set, Pub/Sub delivers the messages in order.
  • Pub/Sub support encryption at rest and encryption in transit.
  • Seek feature allows subscribers to alter the acknowledgment state of messages in bulk to replay or purge messages in bulk.
  • Supports BigQuery subscriptions to write messages directly to BigQuery tables without additional processing.
  • Supports Cloud Storage subscriptions to write messages to Cloud Storage buckets in Avro or Text format.
  • Message filtering allows subscribers to receive a subset of messages published to a topic using filter expressions.
  • Pub/Sub Lite is deprecated (EOL March 18, 2026). Migrate to standard Pub/Sub for cost-effective messaging.

BigQuery

  • BigQuery is a fully managed, durable, petabyte scale, serverless, highly scalable, and cost-effective multi-cloud data warehouse that has evolved into an AI data platform.
  • supports a standard SQL dialect. Legacy SQL is deprecated — effective June 1, 2026, BigQuery limits legacy SQL use for organizations that have not used it between Nov 2025–Jun 2026.
  • automatically replicates data and keeps a seven-day history of changes (time travel), allowing easy restoration and comparison of data from different times
  • supports federated data and can process external data sources in GCS for Parquet and ORC open-source file formats, transactional databases (Bigtable, Cloud SQL), or spreadsheets in Drive without moving the data.
  • BigLake provides a unified storage engine for data lakehouse workloads, supporting Apache Iceberg tables with fine-grained governance across multiple engines (Spark, Flink, Trino, BigQuery).
  • Data model consists of Datasets, tables
  • BigQuery performance can be improved using Partitioned tables and Clustered tables.
  • BigQuery encrypts all data at rest and supports encryption in transit.
  • BigQuery Data Transfer Service automates data movement into BigQuery on a scheduled, managed basis
  • BigQuery Editions (Standard, Enterprise, Enterprise Plus) provide different capability tiers with slot-based pricing, autoscaling reservations, and capacity commitments.
  • BigQuery Studio provides a unified workspace with SQL editor, notebooks (Colab Enterprise), and data canvas for end-to-end analytics workflows.
  • BigQuery ML (BQML) allows building and deploying ML models using SQL, including integration with Gemini and Vertex AI for generative AI tasks like text summarization, sentiment analysis, and embeddings.
  • Vector Search enables similarity search using embeddings directly in BigQuery, supporting RAG applications, semantic search, and KNN-based retrieval without needing external vector databases.
  • BI Engine provides in-memory caching and vectorized processing for sub-second query response times, accelerating dashboards and visualization tools.
  • Best Practices
    • Control projection, avoid select *
    • Estimate costs as queries are billed according to the number of bytes read and the cost can be estimated using --dry-run feature
    • Use the maximum bytes billed setting to limit query costs.
    • Use clustering and partitioning to reduce the amount of data scanned.
    • Avoid repeatedly transforming data via SQL queries. Materialize the query results in stages.
    • Use streaming inserts only if the data must be immediately available as streaming data is charged.
    • Prune partitioned queries, use the _PARTITIONTIME pseudo column to filter the partitions.
    • Denormalize data whenever possible using nested and repeated fields.
    • Avoid external data sources, if query performance is a top priority
    • Avoid using Javascript user-defined functions
    • Optimize Join patterns. Start with the largest table.
    • Use the expiration settings to remove unneeded tables and partitions
    • Keep the data in BigQuery to take advantage of the long-term storage cost benefits rather than exporting to other storage options.
    • Use BigQuery editions with autoscaling for predictable costs and optimal performance.

Bigtable

  • Bigtable is a fully managed, scalable, wide-column NoSQL database service with up to 99.999% availability.
  • ideal for applications that need very high throughput and scalability for key/value data, where each value is max. of 10 MB.
  • supports high read and write throughput at low latency and provides consistent sub-10ms latency – handles millions of requests/second
  • is a sparsely populated table that can scale to billions of rows and thousands of columns,
  • supports storage of terabytes or even petabytes of data
  • is not a relational database. It does not support joins or multi-row transactions.
  • Now supports GoogleSQL for querying data, including window functions for advanced analytic operations (GA 2026).
  • handles upgrades and restarts transparently, and it automatically maintains high data durability.
  • scales linearly in direct proportion to the number of nodes in the cluster
  • stores data in tables, which is composed of rows, each of which typically describes a single entity, and columns, which contain individual values for each row.
  • Each table has only one index, the row key. There are no secondary indices. Each row key must be unique.
  • Single-cluster Bigtable instances provide strong consistency.
  • Multi-cluster instances, by default, provide eventual consistency but can be configured to provide read-over-write consistency or strong consistency, depending on the workload and app profile settings
  • Bigtable Editions (Enterprise and Enterprise Plus) provide advanced features for performance, analytic query capabilities, and resource management (GA 2026).
  • Data Boost provides serverless compute for analytical queries without impacting operational workloads, eliminating the need for multiple data copies.
  • In-memory tier delivers hotspot resistance supporting up to 120,000 queries per second on a single row for ultra-low latency use cases.

Cloud Dataflow

  • Cloud Dataflow is a managed, serverless service for unified stream and batch data processing requirements
  • provides Horizontal autoscaling to automatically choose the appropriate number of worker instances required to run the job.
  • is based on Apache Beam, an open-source, unified model for defining both batch and streaming-data parallel-processing pipelines.
  • supports Windowing which enables grouping operations over unbounded collections by dividing the collection into windows of finite collections according to the timestamps of the individual elements.
  • supports drain feature to deploy incompatible updates
  • Runner v2 supports cross-language transforms, allowing use of Java transforms from Python pipelines and vice versa.
  • Dataflow Prime provides advanced features including Job Visualizer, Smart Recommendations, vertical autoscaling, and right-fitting for optimal resource utilization.
  • GPU and TPU support (TPU V5E, V5P, V6E) enables running high-volume, low-latency ML inference workloads directly within Dataflow jobs.
  • Global Compute enables enormous scaling by dynamically scheduling workloads across Google’s global infrastructure, automatically determining optimal locations based on data locality and resource availability.
  • Speculative Execution for batch pipelines mitigates the impact of slow-running tasks (stragglers) by launching redundant executions.
  • Scales to 4,000 workers per job and routinely processes petabytes of data.
  • Dataflow SQL was deprecated (July 31, 2024) and is no longer available in Google Cloud CLI as of January 31, 2025.

Managed Service for Apache Spark (formerly Cloud Dataproc)

  • Note: Cloud Dataproc has been rebranded to Managed Service for Apache Spark (2025), consolidating Dataproc on Compute Engine and Google Cloud Serverless for Apache Spark under a unified brand.
  • Managed Service for Apache Spark is a managed Spark and Hadoop service to take advantage of open-source data tools for batch processing, querying, streaming, and machine learning.
  • helps to create clusters quickly, manage them easily, and save money by turning clusters on and off as needed.
  • helps reduce time and money spent on administration and lets you focus on your jobs and your data.
  • has built-in integration with other GCP services, such as BigQuery, Cloud Storage, Bigtable, Cloud Logging, and Monitoring
  • support preemptible instances (now called Spot VMs) that have lower compute prices to reduce costs further.
  • also supports HBase, Flink, Hive WebHcat, Druid, Jupyter, Presto, Solr, Zeppelin, Ranger, Zookeeper, and much more.
  • supports connectors for BigQuery, Bigtable, Cloud Storage
  • can be configured for High Availability by specifying the number of master instances in the cluster
  • All nodes in a High Availability cluster reside in the same zone. If there is a failure that impacts all nodes in a zone, the failure will not be mitigated.
  • supports cluster scaling by increasing or decreasing the number of primary or secondary worker nodes (horizontal scaling)
  • supports Autoscaling that provides a mechanism for automating cluster resource management and enables cluster autoscaling.
  • supports initialization actions in executables or scripts that will run on all nodes in the cluster immediately after the cluster is set up
  • Serverless Spark allows submitting batch workloads without provisioning or managing clusters, with automatic scaling and resource management.
  • Supports Zero-Scale clusters for cost optimization when clusters are idle (2026).
  • Includes Lightning Engine for boosted Spark performance.

Cloud Dataprep

⚠️ SERVICE END OF SUPPORT

Cloud Dataprep by Trifacta reached End of Support in December 2025. Trifacta was acquired by Alteryx, and the service has transitioned to Alteryx Designer Cloud.

Migration Options:

  • Alteryx Designer Cloud — successor to Dataprep by Trifacta with enhanced AI/ML capabilities
  • Cloud Data Fusion — Google Cloud’s managed data integration service for visual ETL/ELT pipelines
  • Dataform — for SQL-based data transformations in BigQuery
  • BigQuery Data Preparations — built-in data preparation within BigQuery Studio
  • Cloud Dataprep by Trifacta was an intelligent data service for visually exploring, cleaning, and preparing structured and unstructured data for analysis, reporting, and machine learning.
  • was fully managed, serverless, and scaled on-demand with no infrastructure to deploy or manage
  • provided easy data preparation with clicks and no code.
  • automatically identified data anomalies & helped take fast corrective action
  • automatically detected schemas, data types, possible joins, and anomalies such as missing values, outliers, and duplicates
  • used Dataflow or BigQuery under the hood, enabling unstructured or structured datasets processing of any size with the ease of clicks, not code

Cloud Datalab

⚠️ SERVICE DEPRECATED

Cloud Datalab was deprecated on September 2, 2022.

Migration Options:

  • Vertex AI Workbench — managed notebook environment with JupyterLab, providing capabilities beyond Datalab with integrated ML workflows
  • Colab Enterprise — collaborative notebook environment integrated with BigQuery Studio
  • Cloud Datalab was a powerful interactive tool created to explore, analyze, transform and visualize data and build machine learning models using familiar languages, such as Python and SQL, interactively.
  • ran on Google Compute Engine and connected to multiple cloud services easily so you could focus on data science tasks.
  • was built on Jupyter (formerly IPython)
  • enabled analysis of the data on Google BigQuery, Cloud Machine Learning Engine, Google Compute Engine, and Google Cloud Storage using Python, SQL, and JavaScript (for BigQuery user-defined functions).

Dataplex / Knowledge Catalog

  • Dataplex (now Knowledge Catalog) is an AI-powered, unified data governance solution for managing, understanding, and governing data and AI assets across Google Cloud.
  • Provides centralized inventory to discover, manage, and govern data across BigQuery, Cloud Storage, Pub/Sub, and Spanner.
  • Supports data products — curated, ready-to-use packages of data assets, documentation, and governance controls assembled to solve specific business problems.
  • Offers automated data discovery, metadata management, data quality checks, data lineage, and semantic search.
  • Integrates with BigQuery Studio for unified governance across analytical workflows.

Related Posts

Google Cloud Services Cheat Sheet

~ Last updated on : ~ jayendrapatil

Google Cloud Certification Exam Cheat Sheet

Google Cloud Certification Exams cover a lot of topics and a wide range of services with minute details for features, patterns, anti-patterns, and their integration with other services. This blog post provides a quick summary of all the services and key points for a quick glance before you appear for the exam.

📋 2026 Exam Updates:
  • Google Cloud certifications are now delivered through Pearson VUE (transitioned from Kryterion in early 2026).
  • Exams are being updated to reflect Google Cloud Next ’26 announcements, including Gemini Enterprise Agent Platform and the updated data and analytics stack.
  • The Professional Machine Learning Engineer exam was significantly updated on June 1, 2026, now focusing heavily on the Gemini Enterprise Agent Platform.
  • A new Cloud Digital Leader beta exam is open through July 5, 2026.

Google Cloud Services

Google Cloud Service Categories (2026)

Google Cloud organizes its 100+ services into the following major categories:

  • Compute – App Engine, Compute Engine, Google Kubernetes Engine (GKE), Cloud Run, Batch, VMware Engine
  • Storage – Cloud Storage, Persistent Disk, Filestore, Cloud Managed Lustre, NetApp Volumes
  • Databases – AlloyDB, Cloud SQL, Spanner, Firestore, Bigtable, Memorystore
  • Networking – VPC, Cloud Load Balancing, Cloud CDN, Cloud DNS, Cloud Interconnect, Cloud Armor, Cloud NAT
  • Security & Identity – IAM, Security Command Center, Wiz (acquired March 2026), Cloud KMS, Sensitive Data Protection, Chrome Enterprise Premium
  • AI/ML & Generative AI – Gemini Enterprise Agent Platform (formerly Vertex AI), Gemini for Google Cloud, Gemini Code Assist, NotebookLM Enterprise
  • Data Analytics – BigQuery, Dataflow, Pub/Sub, Lakehouse (formerly BigLake), Knowledge Catalog (formerly Dataplex), Looker, Managed Service for Apache Spark (formerly Dataproc)
  • Serverless – Cloud Run, Cloud Run functions (formerly Cloud Functions), App Engine, Workflows, Eventarc
  • Developer Tools – Cloud Build, Artifact Registry, Cloud Deploy, Infrastructure Manager, Cloud Workstations, Gemini Code Assist
  • Operations – Cloud Logging, Cloud Monitoring, Cloud Trace, Error Reporting, Cloud Profiler
  • Management Tools – Cloud Shell, Infrastructure Manager, Recommender, Cloud Console
  • Migration – Migration Center, Database Migration Service, Migrate to Virtual Machines, Transfer Appliance
  • API Management – Apigee, API Gateway, Cloud Endpoints
  • Containers – GKE, Cloud Run, Cloud Build, Artifact Registry, Cloud Service Mesh

Key Service Renamings and Changes (2024-2026)

  • Vertex AI → Gemini Enterprise Agent Platform (announced at Cloud Next ’26, April 2026)
  • Cloud Functions → Cloud Run functions (merged into Cloud Run, August 2024)
  • Dataproc → Managed Service for Apache Spark
  • Cloud Composer → Managed Service for Apache Airflow
  • BigLake → Lakehouse
  • Dataplex Universal Catalog → Knowledge Catalog
  • Contact Center AI → Gemini Enterprise for Customer Experience (GECX)
  • Vertex AI Search → Agent Search on Gemini Enterprise Agent Platform
  • BeyondCorp Enterprise → Chrome Enterprise Premium
  • Container Registry → Artifact Registry (Container Registry shut down March 2025)

Google Cloud Marketplace

Note: Google Cloud Marketplace was previously known as Cloud Launcher (renamed in 2018).

  • Google Cloud Marketplace offers a universal catalog of solutions from Google and partner ecosystem for customers to easily discover, try, buy, and use.
    • Deploy production-grade solutions in a few clicks
    • Single bill for all Google Cloud and 3rd party services
    • Includes AI agents and agent tools marketplace (launched 2025)
    • Supports Gemini-powered natural language search to discover solutions
    • Private Marketplace for organizational governance (enhanced March 2025)
    • Variable revenue share model for partners (3% down to 1.5%)
⚠️ Deployment Manager Deprecated: Google Cloud Deployment Manager was discontinued on March 31, 2026. Solutions in Marketplace are now managed using Infrastructure Manager (Terraform-based) or standard Google Cloud tools. Migrate existing Deployment Manager configurations to Terraform via Infrastructure Manager.

Major New Services & Features (2024-2026)

Gemini Enterprise Agent Platform (formerly Vertex AI)

  • Unified platform to build, deploy, scale, govern, and optimize AI agents
  • All Vertex AI services are now delivered exclusively through the Agent Platform
  • Includes Model Garden, Agent Development Kit (ADK), Agent Governance Stack
  • Supports Agent-to-Agent (A2A) protocol for agent interoperability
  • Session memory and centralized governance for production agents

Gemini Enterprise

  • Unified agentic portfolio combining AI agents, Gemini models, and Google search
  • Connects to enterprise data sources for workflow automation
  • Formerly Agentspace (absorbed into Gemini Enterprise, October 2025)

Wiz (Acquired March 2026)

  • Google completed $32 billion acquisition of Wiz, a cloud and AI security platform
  • Wiz joined Google Cloud and maintains its brand
  • Provides cloud security posture management across multi-cloud environments

Infrastructure Manager

  • Fully-managed service for deploying infrastructure using Terraform
  • Official replacement for Cloud Deployment Manager
  • Supports Git repositories, Cloud Storage buckets, and local directories
  • Preview deployments to review changes before applying

Google Cloud Certifications (2026)

  • Cloud Digital Leader – foundational certification
  • Associate Cloud Engineer – hands-on cloud management
  • Professional Cloud Architect – design and plan cloud solutions
  • Professional Cloud Developer – develop and deploy applications
  • Professional Cloud Network Engineer – network architecture
  • Professional Cloud Security Engineer – security policies and controls
  • Professional Data Engineer – data processing and analytics
  • Professional Machine Learning Engineer – updated June 2026 for Gemini Enterprise Agent Platform
  • Professional Cloud DevOps Engineer – CI/CD and SRE practices
  • Professional Cloud Database Engineer – database solutions

References

Google Cloud Networking Services Cheat Sheet

~ Last updated on : ~ jayendrapatil

Virtual Private Cloud

  • Virtual Private Cloud (VPC) provides networking functionality for the cloud-based resources and services that is global, scalable, and flexible.
  • VPC networks are global resources, including the associated routes and firewall rules, and are not associated with any particular region or zone.
  • Subnets are regional resources and each subnet defines a range of IP addresses
  • IPv6 Support
    • VPC networks support dual-stack (IPv4 and IPv6) subnets in custom-mode VPC networks.
    • IPv6 functionality is available only in Premium Tier.
    • Supports both external (GUA – Globally Unique Addresses) and internal (ULA – Unique Local Addresses) IPv6 ranges.
    • VMs can have IPv4-only, dual-stack, or IPv6-only interfaces.
  • Cloud NGFW (Next Generation Firewall)
    • replaces legacy VPC firewall rules with network firewall policies.
    • protects workloads by applying controls at Layer 3, Layer 4, and Layer 7 of the network stack.
    • available in three tiers:
      • Essentials – rules based on IP ranges, ports, and protocols
      • Standard – adds FQDN objects, geo-location objects, and threat intelligence
      • Enterprise – adds Intrusion Detection and Prevention Service (IPS) powered by Palo Alto Networks, TLS inspection
    • Google recommends migrating from legacy VPC firewall rules to Cloud NGFW network firewall policies.
    • Network firewall policies can be attached to a single VPC or group of VPCs (unlike legacy VPC firewall rules which apply to a single VPC only).
  • Resources within a VPC network can communicate with one another by using internal IPv4 addresses, subject to applicable network firewall rules.
  • Private access options for services allow instances with internal IP addresses to communicate with Google APIs and services.
  • Private Service Connect (PSC)
    • allows consumers to access managed services privately from inside their VPC network.
    • allows managed service producers to host services in their own separate VPC networks and offer a private connection to consumers.
    • creates service endpoints in consumer VPCs that provide private connectivity and policy enforcement.
  • Shared VPC to keep a VPC network in a common host project shared with service projects. Authorized IAM members from other projects in the same organization can create resources that use subnets of the Shared VPC network
  • VPC Network Peering allows VPC networks to be connected with other VPC networks in different projects or organizations.
  • VPC networks can be securely connected in hybrid environments by using Cloud VPN or Cloud Interconnect.
  • Primary and Secondary IP address cannot overlap with the on-premises CIDR
  • VPC Flow Logs records a sample of network flows sent from and received by VM instances, including instances used as GKE nodes.

Cloud Load Balancing

  • Cloud Load Balancing is a fully distributed, software-defined managed load balancing service
  • distributes user traffic across multiple instances of the applications and reduces the risk of performance issues by spreading the load
  • provides health checking mechanisms that determine if backends, such as instance groups and zonal network endpoint groups (NEGs), are healthy and properly respond to traffic.
  • supports IPv6 clients with Application Load Balancers and proxy Network Load Balancers.
  • Note: Google Cloud has renamed load balancer types. HTTP(S) Load Balancing is now Application Load Balancer, TCP/UDP Load Balancing is now passthrough Network Load Balancer, and SSL Proxy/TCP Proxy are now proxy Network Load Balancer.
  • supports multiple Cloud Load Balancing types
    • Internal Application Load Balancer (formerly Internal HTTP(S) Load Balancing)
      • is a proxy-based, regional Layer 7 load balancer that enables running and scaling services behind an internal IP address.
      • also available as a cross-region internal Application Load Balancer for multi-region backends with automatic failover.
      • supports a regional backend service, which distributes HTTP and HTTPS requests to healthy backends (either instance groups containing CE VMs or NEGs containing GKE containers).
      • supports path based routing
      • preserves the Host header of the original client request and also appends two IP addresses (Client and LB) to the X-Forwarded-For header
      • supports a regional health check that periodically monitors the readiness of the backends.
      • has native support for the WebSocket protocol when using HTTP or HTTPS as the protocol to the backend
    • External Application Load Balancer (formerly External HTTP(S) Load Balancing)
      • is a global, proxy-based Layer 7 load balancer that enables running and scaling the services worldwide behind a single external IP address
      • distributes HTTP and HTTPS traffic to backends hosted on Compute Engine and GKE
      • offers global (cross-regional) and regional load balancing
      • supports content-based load balancing using URL maps
      • preserves the Host header of the original client request and also appends two IP addresses (Client and LB) to the X-Forwarded-For header
      • supports connection draining on backend services
      • has native support for the WebSocket protocol when using HTTP or HTTPS as the protocol to the backend
      • supports mutual TLS (mTLS) authentication for client certificate-based authentication.
    • Internal passthrough Network Load Balancer (formerly Internal TCP/UDP Load Balancing)
      • is a managed, internal, pass-through, regional Layer 4 load balancer that enables running and scaling services behind an internal IP address
      • distributes traffic among VM instances in the same region in a Virtual Private Cloud (VPC) network by using an internal IP address.
      • provides high-performance, pass-through Layer 4 load balancer for TCP or UDP traffic.
      • routes original connections directly from clients to the healthy backends, without any interruption.
      • does not terminate SSL traffic and SSL traffic can be terminated by the backends instead of by the load balancer
      • provides access through VPC Network Peering, Cloud VPN or Cloud Interconnect
      • supports health check that periodically monitors the readiness of the backends.
    • External passthrough Network Load Balancer (formerly External TCP/UDP Network Load Balancing)
      • is a managed, external, pass-through, regional Layer 4 load balancer that distributes TCP or UDP traffic originating from the internet to among VM instances in the same region
      • Load-balanced packets are received by backend VMs with their source IP unchanged.
      • Load-balanced connections are terminated by the backend VMs. Responses from the backend VMs go directly to the clients, not back through the load balancer (direct server return).
      • scope of a network load balancer is regional, not global. A network load balancer cannot span multiple regions. Within a single region, the load balancer services all zones.
      • supports connection tracking table and a configurable consistent hashing algorithm to determine how traffic is distributed to backend VMs.
      • supports additional protocols like UDP, ESP, GRE, ICMP, and ICMPv6.
    • External proxy Network Load Balancer (formerly External SSL Proxy and TCP Proxy Load Balancing)
      • is a reverse proxy load balancer that distributes SSL or TCP traffic coming from the internet to VM instances in the VPC network.
      • with SSL traffic, user SSL (TLS) connections are terminated at the load balancing layer, and then proxied to the closest available backend instances by using either SSL (recommended) or TCP.
      • supports global load balancing service with the Premium Tier and regional load balancing service with the Standard Tier
      • is intended for non-HTTP(S) traffic. For HTTP(S) traffic, use Application Load Balancers.
      • supports proxy protocol header to preserve the original source IP addresses of incoming connections to the load balancer

Cloud CDN

  • Cloud CDN is Google Cloud’s web acceleration solution that caches website and application content closer to the user.
  • uses Google’s global edge network to serve content closer to users, which accelerates the websites and applications.
  • works with the global external Application Load Balancer or the classic Application Load Balancer to deliver content to users
  • Cloud CDN content can be sourced from various types of backends
    • Instance groups
    • Zonal network endpoint groups (NEGs)
    • Serverless NEGs: One or more App Engine, Cloud Run, or Cloud Functions services
    • Internet NEGs, for endpoints that are outside of Google Cloud (also known as custom origins)
    • Buckets in Cloud Storage
  • supports content targeting (GA) — enables device characterization and geo-targeting for responsive websites, language customization, and currency settings.
  • Cloud CDN with Google Cloud Armor enforces security policies only for requests for dynamic content, cache misses, or other requests that are destined for the origin server. Cache hits are served even if the downstream Google Cloud Armor security policy would prevent that request from reaching the origin server.
  • recommends
    • using versioning instead of cache invalidation
    • using custom keys to improve cache hit ratio
    • cache static content
  • Media CDN
    • is Google Cloud’s media delivery solution, complementing Cloud CDN.
    • optimized for high-throughput egress workloads, such as streaming video and large file downloads.
    • uses YouTube’s infrastructure to bring video streams (VoD and live) and large file downloads closer to users.
    • supports Cloud Armor edge security policies for DDoS protection.

Cloud VPN

  • securely connects the peer network to the VPC network or two VPCs through an IPsec VPN connection.
  • encrypts the data as it travels over the internet.
  • only supports site-to-site IPsec VPN connectivity and not client-to-gateway scenarios. Cannot be used to route traffic to the public internet.
  • allows users to access private RFC1918 addresses on resources in the VPC from on-prem computers also using private RFC1918 addresses.
  • can be used with Private Google Access for on-premises hosts
  • HA VPN
    • provides a high-available and secure connection between the on-premises and the VPC network through an IPsec VPN connection in a single region
    • provides an SLA of 99.99% service availability, when configured with two interfaces and two external IP addresses.
    • supports IPv6 (dual-stack) tunnels for both inner and outer IP addresses.
    • supports customizable cipher options for VPN tunnels.
  • Classic VPN
    • provides a 99.9% SLA.
    • Classic VPN dynamic routing (BGP) was deprecated on August 1, 2025. HA VPN is now the only option for BGP connectivity in Cloud VPN. Existing tunnels continue to function but without an availability SLA. If deleted, they cannot be recreated.
    • Does not support IPv6 traffic.
  • HA VPN over Cloud Interconnect
    • allows encrypting traffic traversing Dedicated or Partner Interconnect connections.
    • deploys HA VPN tunnels over VLAN attachments for additional security and compliance.
    • Each HA VPN tunnel has a bandwidth of 3 Gbps.
  • supports up to 3Gbps per tunnel with a maximum of 8 tunnels
  • supports static as well as dynamic routing using Cloud Router
  • supports IKEv1 or IKEv2 using a shared secret

Cloud Interconnect

  • Cloud Interconnect provides options for extending the on-premises network to the VPC networks in Google Cloud.
  • Dedicated Interconnect (Dedicated connection)
    • provides a direct physical connection between the on-premises network and Google’s network
    • requires your network to physically meet Google’s network in a colocation facility with your own routing equipment
    • supports only dynamic routing
    • supports 10 Gbps, 100 Gbps, and 400 Gbps circuits.
  • Partner Interconnect (Use a service provider)
    • provides connectivity between the on-premises and VPC networks through a supported service provider.
    • supports bandwidth from 50 Mbps minimum to 50 Gbps maximum.
    • provides Layer 2 and Layer 3 connectivity
      • For Layer 2 connections, you must configure and establish a BGP session between the Cloud Routers and on-premises routers for each created VLAN attachment
      • For Layer 3 connections, the service provider establishes a BGP session between the Cloud Routers and their edge routers for each VLAN attachment.
  • Cross-Cloud Interconnect
    • provides dedicated, private connectivity between Google Cloud and another cloud provider (AWS, Azure, OCI, Alibaba Cloud).
    • offers 10 Gbps or 100 Gbps managed, encrypted links.
    • supports security options such as IPsec VPN or MACsec.
    • Partner Cross-Cloud Interconnect (for AWS) provides an on-demand method for establishing cross-cloud transport without manually setting up networking components.
  • Cross-Site Interconnect
    • is a transparent, on-demand, Layer 2 connectivity solution between on-premises network sites.
    • leverages Google’s global infrastructure for high-performance and high-bandwidth connectivity.
  • Single Interconnect connection does not offer redundancy or high availability and its recommended to
    • use 2 in the same metropolitan area (city) as the existing one, but in a different edge availability domain (metro availability zone).
    • use 4 with 2 connections in two different metropolitan areas (city), and each connection in a different edge availability domain (metro availability zone)
    • Cloud Routers are required one in each Google Cloud region
  • Cloud Interconnect does not encrypt the connection between your network and Google’s network. For additional security, use HA VPN over Cloud Interconnect or application-level encryption.

Cloud Router

  • is a fully distributed, managed service that provides dynamic routing and scales with the network traffic.
  • works with both legacy networks and VPC networks.
  • isn’t supported for Direct Peering or Carrier Peering connections.
  • helps dynamically exchange routes between the Google Cloud networks and the on-premises network.
  • peers with the on-premises VPN gateway or router to provide dynamic routing and exchanges topology information through BGP.
  • Google Cloud recommends creating two Cloud Routers in each region for a Cloud Interconnect for 99.99% availability.
  • supports following dynamic routing mode
    • Regional routing mode – provides visibility to resources only in the defined region.
    • Global routing mode – provides visibility to resources in all regions
  • is part of the Network Connectivity Center, which provides a hub-and-spoke model for managing connectivity across VPC networks, on-premises, and other clouds.

Network Connectivity Center

  • provides a hub-and-spoke model for managing network connectivity at scale.
  • enables site-to-site data transfer between on-premises locations through Google’s network.
  • supports VPC spokes, hybrid spokes (VPN/Interconnect), and router appliance spokes.
  • solves transitivity challenges through features like producer VPC spoke integration supporting Private Service Access (PSA) and Private Service Connect (PSC) propagation.
  • uses ECMP routing and BGP for route distribution between networks.

Cloud DNS

  • is a high-performance, resilient, reliable, low-latency, global DNS service that publishes the domain names to the global DNS in a cost-effective way.
  • With Shared VPC, Cloud DNS managed private zone, Cloud DNS peering zone, or Cloud DNS forwarding zone must be created in the host project
  • provides Private Zone which supports DNS services for a GCP project. VPCs in the same project can use the same name servers
  • supports DNS Forwarding for Private Zones, which overrides normal DNS resolution for the specified zones. Queries for the specified zones are forwarded to the listed forwarding targets.
  • supports DNS Peering, which allows sending requests for records that come from one zone’s namespace to another VPC network within GCP
  • supports DNS Outbound Policy, which forwards all DNS requests for a VPC network to the specified server targets. It disables internal DNS for the selected networks.
  • DNS Routing Policies
    • supports weighted round robin, geolocation, and failover routing policies.
    • can be configured with health checks for automatic failover.
    • supports internal passthrough Network Load Balancers and internal proxy Network Load Balancers as health checked targets.
  • Cloud DNS VPC Name Resolution Order
    • DNS Outbound Server Policy
    • DNS Forwarding Zone
    • DNS Peering
    • Compute Engine internal DNS
    • Public Zones
  • supports DNSSEC, a feature of DNS, that authenticates responses to domain name lookups and protects the domains from spoofing and cache poisoning attacks

Related Posts

Google Cloud Compute Services Cheat Sheet

~ Last updated on : ~ jayendrapatil

Google Cloud Compute Services

Google Cloud - Compute Services Options

Google Cloud provides a range of compute services to run workloads, from fully managed serverless platforms to infrastructure-level virtual machines. The key compute services include Compute Engine, GKE, App Engine, Cloud Run, and Cloud Run functions (formerly Cloud Functions).

Compute Engine

  • is a virtual machine (VM) hosted on Google’s infrastructure.
  • can run the public images for Google provided Linux and Windows Server as well as custom images created or imported from existing systems
  • availability policy determines how it behaves when there is a maintenance event
    • VM instance’s maintenance behavior onHostMaintenance, which determines whether the instance is live migrated MIGRATE (default) or stopped TERMINATE
    • Instance’s restart behavior automaticRestart which determines whether the instance automatically restarts (default) if it crashes or gets stopped
  • Live migration helps keep the VM instances running even when a host system event, such as a software or hardware update, occurs
  • Spot VMs (recommended replacement for Preemptible VMs) offer up to 60-91% discount compared to on-demand pricing
    • can be reclaimed by Compute Engine at any time when resources are needed
    • unlike Preemptible VMs, Spot VMs have no maximum runtime limit (no 24-hour expiration)
    • same pricing model as Preemptible VMs
    • ideal for fault-tolerant, batch, and stateless workloads
  • Preemptible VMs (legacy — Google recommends using Spot VMs instead)
    • can be created and run at a much lower price than normal instances
    • always stop after 24 hours of running
    • can be stopped at any time when Compute Engine needs resources
  • Shielded VM offers verifiable integrity of the Compute Engine VM instances, to confirm the instances haven’t been compromised by boot- or kernel-level malware or rootkits.
  • Confidential VMs keep sensitive code and data encrypted in memory during processing (encryption-in-use)
    • uses hardware-based memory encryption with AMD SEV, AMD SEV-SNP, or Intel TDX
    • supports GPU workloads with A3 machine types (NVIDIA H100) using Intel TDX
    • together with encryption-at-rest and encryption-in-transit, provides encryption at all times
  • Instance template is a resource used to create VM instances and managed instance groups (MIGs) with identical configuration
  • Instance group is a collection of virtual machine (VM) instances that can be managed as a single entity.
    • Managed instance groups (MIGs)
      • allows app creation with multiple identical VMs.
      • workloads can be made scalable and highly available by taking advantage of automated MIG services, including: autoscaling, autohealing, regional (multiple zone) deployment, and automatic updating
      • supports rolling update feature
      • works with load balancing services to distribute traffic across all of the instances in the group.
    • Unmanaged instance groups
      • allows load balance across a fleet of VMs that you manage yourself which may not be identical
  • Instance templates are global, while instance groups are regional.
  • Machine image stores all the configuration, data, metadata and permissions from one or more disks required to create a VM instance
  • Sole-tenancy provides dedicated hosting only for the project’s VM and provides added layer of hardware isolation
  • deletionProtection prevents accidental VM deletion esp. for VMs running critical workloads and need to be protected
  • provides Sustained Discounts, Committed discounts, free tier etc. in pricing

Machine Families (Updated 2025)

  • General-purpose — best price-performance for common workloads
    • C4 — powered by 5th/6th Gen Intel Xeon Scalable processors (Emerald Rapids/Granite Rapids) and Google Titanium
    • C4D — powered by 5th Gen AMD EPYC Turin processors and Titanium; up to 80% higher throughput per vCPU for web-serving workloads; supports confidential computing
    • C4A — powered by Google Axion (Arm-based) processors with Titanium SSD; up to 72 vCPUs
    • N4D — AMD Turin-based; up to 3.5x throughput for web-serving vs. previous-gen N2D
    • E2, N2, N2D, N1 — previous generation general-purpose options
  • Compute-optimized — highest per-core performance (C2, C2D, H3)
  • Memory-optimized — large in-memory workloads (M1, M2, M3, X4)
  • Accelerator-optimized — GPU/ML workloads (A2, A3, G2, G4)
    • A3 — NVIDIA H100 GPUs, ideal for AI/ML training and large model inference
    • G4 — NVIDIA RTX PRO 6000 GPUs for graphics and virtual workstations
  • Storage-optimized — high IOPS local storage (Z3)

Storage Options

  • Persistent Disk — network-attached block storage (Standard, Balanced, SSD, Extreme)
  • Hyperdisk — next-generation block storage with substantially higher IOPS and throughput
    • Hyperdisk Balanced — general workloads with independently configurable IOPS and throughput
    • Hyperdisk Throughput — optimized for sequential reads/writes
    • Hyperdisk Extreme — highest performance for databases
    • Hyperdisk Balanced High Availability — multi-zone replication for critical workloads
    • Supports Confidential mode for hardware-based encryption
  • Local SSD — physically attached high-performance local storage
  • Titanium SSD — Google’s custom SSD integrated with Titanium infrastructure (available on C4A)

App Engine

  • App Engine helps build highly scalable applications on a fully managed serverless platform
  • Each Cloud project can contain only a single App Engine application
  • App Engine is regional, which means the infrastructure that runs the apps is located in a specific region, and Google manages it so that it is available redundantly across all of the zones within that region
  • App Engine application location or region cannot be changed once created
  • App engine allows traffic management to an application version by migrating or splitting traffic.
    • Traffic Splitting (Canary) – distributes a percentage of traffic to versions of the application.
    • Traffic Migration – smoothly switches request routing
  • Supports Standard and Flexible environments
    • Standard environment
      • Application instances that run in a sandbox, using the runtime environment of a supported language only.
      • Sandbox restricts what the application can do
        • only allows the app to use a limited set of binary libraries
        • app cannot write to disk
        • limits the CPU and memory options available to the application
      • Sandbox does not support
        • SSH debugging
        • Background processes
        • Background threads (limited capability)
        • Using Cloud VPN
    • Flexible environment
      • Application instances run within Docker containers on Compute Engine virtual machines (VM).
      • As Flexible environment supports docker it can support custom runtime or source code written in other programming languages.
      • Allows selection of any Compute Engine machine type for instances so that the application has access to more memory and CPU.
  • min_idle_instances indicates the number of additional instances to be kept running and ready to serve traffic for this version.

App Engine Runtime Lifecycle (Updated 2025)

  • Legacy runtimes reached End of Support on January 30, 2024 — Python 2.7, Java 8, PHP 5.5, Go 1.11
  • Legacy runtimes deprecated on January 31, 2026 and will be decommissioned after that
  • Google recommends Cloud Run as the modern alternative for new serverless container workloads
  • Cloud Run offers more flexibility: custom containers, VPC connectivity, GPU support, and finer-grained scaling controls
  • App Engine migration center provides guidance for moving to Cloud Run

Google Kubernetes Engine (GKE)

  • is a managed Kubernetes service for deploying and managing containerized applications
  • provides a fully managed environment for deployment, management, and scaling of containerized apps using Google infrastructure
  • uses Compute Engine instances as nodes in a cluster

GKE Modes of Operation

  • Autopilot mode (default since 2023, recommended)
    • fully managed — GKE manages nodes, node pools, and cluster infrastructure
    • pay-per-pod model — charges based on requested vCPU, memory, and ephemeral storage
    • hardened security configuration enabled by default
    • automatic security patching of nodes
    • built-in best practices for security, reliability, performance, and scalability
    • In 2024, 30% of active GKE clusters used Autopilot mode
    • In 2025, Autopilot became available to all qualifying clusters (automatic migration)
    • supports custom compute classes for workload-specific hardware requirements
  • Standard mode
    • provides full control over node configuration and management
    • user is responsible for managing and configuring individual nodes
    • can run Autopilot-mode workloads within Standard clusters
    • multi-cluster management features (Fleets, Teams, Config Management, Policy Controller) now included at no additional cost (2025)

GKE Key Features

  • Node Pools — group of nodes with identical configuration within a cluster
    • supports autoscaling (automatically adjusts number of nodes)
    • supports auto-provisioning (GKE creates optimal node pools automatically)
    • surge upgrade strategy by default (maxSurge=1, maxUnavailable=0)
  • Cluster Autoscaler — automatically resizes number of nodes based on workload demands
  • Vertical Pod Autoscaler (VPA) — adjusts CPU/memory requests for containers
  • Horizontal Pod Autoscaler (HPA) — adjusts number of pod replicas
  • GKE Sandbox — provides an extra layer of security using gVisor for untrusted workloads
  • Confidential GKE Nodes — uses Confidential VMs for hardware-based memory encryption
  • Workload Identity Federation — recommended way to access Google Cloud services from GKE workloads
  • Binary Authorization — deploy-time security control to ensure only trusted containers are deployed

GKE AI/ML Features (2024-2026)

  • GKE Inference Gateway — optimized routing for AI model inference workloads
  • GKE Inference Quickstart — simplified deployment for AI inference serving
  • Custom Compute Classes — define hardware requirements for AI workloads (GPUs, TPUs)
  • GKE Agent Sandbox — secure execution environment for AI agents
  • Support for clusters up to 130,000 nodes for massive AI computation
  • GKE now powers AI workloads for all top 50 Google Cloud platform customers

GKE Commands

  • gcloud container clusters create — creates a new cluster (Autopilot by default)
  • gcloud container clusters create --mode=standard — creates a Standard mode cluster
  • gcloud container clusters resize --num-nodes — scales the cluster node count (--size is deprecated)
  • gcloud container node-pools create — creates a new node pool

Cloud Run

  • is a fully managed serverless platform for running containerized applications
  • automatically scales containers up and down, including scale to zero
  • supports any container that listens for HTTP requests or processes events
  • no infrastructure to manage — deploy with a single command (gcloud run deploy)
  • Cloud Run services — for serving HTTP requests (web apps, APIs)
  • Cloud Run jobs — for running batch tasks to completion (GA since 2023)
  • Cloud Run worker pools — for long-running background workers (2025)
  • GPU support — serverless GPU acceleration with NVIDIA L4 GPUs (GA 2025)
    • ideal for AI inference, image/video processing, and scientific computation
    • no cluster management required
  • pay-per-use pricing — charged only when container is handling requests or running jobs
  • supports both request-based and instance-based billing models
  • integrates with VPC for private networking and direct VPC egress

Cloud Run Functions (formerly Cloud Functions)

  • Renamed from Cloud Functions to Cloud Run functions in August 2024
  • is a Functions-as-a-Service (FaaS) offering powered by Cloud Run and Eventarc
  • provides a simple event-driven programming model for small code snippets
  • supports 90+ event sources via Eventarc triggers
  • deployed as Cloud Run services under the hood, providing same scalability and networking features
  • 1st gen — original version with limited event triggers (legacy)
  • 2nd gen (Cloud Run functions) — latest version with advanced control, longer timeouts, larger instances, and traffic splitting
  • no server management — write code and deploy, infrastructure is handled automatically
  • supports Node.js, Python, Go, Java, .NET, Ruby, and PHP

Compute Services Comparison

Feature Compute Engine GKE App Engine Cloud Run Cloud Run Functions
Type IaaS CaaS (Containers) PaaS Serverless containers FaaS
Use case Full VM control Container orchestration Web apps Stateless containers Event-driven functions
Scaling MIG autoscaling Pod/node autoscaling Automatic Automatic (to zero) Automatic (to zero)
Management User manages OS, runtime User manages containers Google manages runtime Google manages infra Google manages everything
Pricing Per VM (seconds) Per node or per pod Per instance hour Per request/instance Per invocation
GPU support Yes Yes No Yes (L4) No

Google Cloud Compute Services Cheat Sheet Questions

  1. A company needs to run a containerized application that automatically scales based on traffic and requires zero infrastructure management. Which service should they use?
    1. Compute Engine
    2. GKE Standard
    3. Cloud Run
    4. App Engine Flexible
    Show Answer

    Answer: c. Cloud Run – provides automatic scaling including scale-to-zero with no infrastructure management for containerized apps.

  2. A team wants to run Kubernetes workloads without managing node pools, patching, or capacity planning. Which GKE mode should they choose?
    1. GKE Standard
    2. GKE Autopilot
    3. GKE Enterprise
    4. GKE Sandbox
    Show Answer

    Answer: b. GKE Autopilot – fully managed mode where GKE handles all node management, patching, and capacity planning.

  3. Which Compute Engine VM type is recommended over Preemptible VMs and has no maximum runtime limit?
    1. Shielded VM
    2. Confidential VM
    3. Spot VM
    4. Sole-tenant VM
    Show Answer

    Answer: c. Spot VM – recommended replacement for Preemptible VMs with same pricing but no 24-hour time limit.

  4. What happened to Google Cloud Functions in August 2024?
    1. It was deprecated entirely
    2. It was renamed to Cloud Run functions
    3. It was merged into App Engine
    4. It moved to GKE
    Show Answer

    Answer: b. It was renamed to Cloud Run functions – Cloud Functions (2nd gen) is now Cloud Run functions, deployed as services on Cloud Run.

  5. An organization needs to deploy AI inference models on GKE with optimized routing. Which feature should they use?
    1. GKE Sandbox
    2. GKE Inference Gateway
    3. Cloud Run GPU
    4. Compute Engine A3
    Show Answer

    Answer: b. GKE Inference Gateway – provides optimized routing specifically for AI model inference workloads on GKE.

  6. Which storage option provides substantially higher IOPS and throughput than Persistent Disk with independently configurable performance?
    1. Local SSD
    2. Standard Persistent Disk
    3. Hyperdisk
    4. Filestore
    Show Answer

    Answer: c. Hyperdisk – next-generation block storage with independently configurable IOPS and throughput.

  7. A company wants to run serverless GPU workloads for AI inference without managing clusters. Which service is most appropriate?
    1. Compute Engine with GPU
    2. GKE with A3 nodes
    3. Cloud Run with GPU
    4. App Engine Flexible
    Show Answer

    Answer: c. Cloud Run with GPU – provides serverless GPU acceleration with NVIDIA L4 GPUs, no cluster management required.

  8. Which Compute Engine feature protects data in memory using hardware-based encryption?
    1. Shielded VM
    2. Confidential VM
    3. Sole-tenant node
    4. Live migration
    Show Answer

    Answer: b. Confidential VM – keeps data encrypted in memory during processing using AMD SEV, SEV-SNP, or Intel TDX.

Related Posts

Google Cloud Storage Services Cheat Sheet

~ Last updated on : ~ jayendrapatil ~ 2 Comments

Google Cloud Storage Options

  • Relational (SQL) – Cloud SQL, AlloyDB & Cloud Spanner
  • Non-Relational (NoSQL) – Firestore & Bigtable
  • Structured & Semi-structured – Cloud SQL, AlloyDB, Cloud Spanner, Firestore & Bigtable
  • Unstructured – Cloud Storage
  • Block Storage – Persistent Disk & Hyperdisk
  • File Storage – Filestore
  • In-Memory – Memorystore (Redis, Valkey)
  • Transactional (OLTP) – Cloud SQL, AlloyDB & Cloud Spanner
  • Analytical (OLAP) – Bigtable & BigQuery
  • Fully Managed (Serverless) – Cloud Spanner, Firestore, BigQuery, AlloyDB
  • Requires Provisioning – Cloud SQL, Bigtable
  • Global – Cloud Spanner
  • Regional – Cloud SQL, AlloyDB, Bigtable, Firestore

Google Cloud - Storage Options Decision Tree

Google Cloud Storage – GCS

  • provides service for storing unstructured data i.e. objects
  • consists of bucket and objects where an object is an immutable piece of data consisting of a file of any format stored in containers called buckets.
  • support different location types
    • regional
      • A region is a specific geographic place, such as London.
      • helps optimize latency and network bandwidth for data consumers, such as analytics pipelines, that are grouped in the same region.
    • dual-region
      • is a specific pair of regions, such as Finland and the Netherlands.
      • provides higher availability that comes with being geo-redundant.
    • multi-region
      • is a large geographic area, such as the United States, that contains two or more geographic places.
      • allows serving content to data consumers that are outside of the Google network and distributed across large geographic areas
      • provides higher availability that comes with being geo-redundant.
    • Objects stored in a multi-region or dual-region are geo-redundant i.e. data is stored redundantly in at least two separate geographic places separated by at least 100 miles.
  • Storage class affects the object’s availability and pricing model
    • Standard Storage is best for data that is frequently accessed (hot data) and/or stored for only brief periods of time.
    • Nearline Storage is a low-cost, highly durable storage service for storing infrequently accessed data (warm data)
    • Coldline Storage provides a very-low-cost, highly durable storage service for storing infrequently accessed data (cold data)
    • Archive Storage is the lowest-cost, highly durable storage service for data archiving, online backup, and disaster recovery. (coldest data)
  • Autoclass automatically transitions objects to appropriate storage classes based on access patterns, removing the need to manually manage lifecycle rules for cost optimization. Supports transitioning between Standard, Nearline, Coldline, and Archive classes.
  • Soft Delete (launched 2024) provides default bucket-level protection against accidental or malicious deletion by preserving recently deleted objects for a configurable retention period (7-90 days). Enabled by default with 7-day retention.
  • Object Versioning prevents accidental overwrites and deletion. It retains a noncurrent object version when the live object version gets replaced, overwritten or deleted
  • Object Lifecycle Management sets Time To Live (TTL) on an object and helps configure transition or expiration of the objects based on specified rules for e.g. SetStorageClass to change the storage class, delete to expire noncurrent or archived objects
  • Resumable uploads are the recommended method for uploading large files, because they don’t need to be restarted from the beginning if there is a network failure while the upload is underway.
  • Parallel composite uploads divides a file into up to 32 chunks, which are uploaded in parallel to temporary objects, the final object is recreated using the temporary objects, and the temporary objects are deleted
  • Requester Pays on the bucket that requires requester to include a billing project in their requests, thus billing the requester’s project.
  • supports upload and storage of any MIME type of data up to 5 TB in size.
  • Retention policy on a bucket ensures that all current and future objects in the bucket cannot be deleted or replaced until they reach the defined age
  • Retention policy locks will lock a retention policy on a bucket and prevents the policy from ever being removed or the retention period from ever being reduced (although it can be increased). Locking a retention policy is irreversible
  • Bucket Lock feature provides immutable storage on Cloud Storage
  • Object holds, when set on individual objects, prevents the object from being deleted or replaced, however allows metadata to be edited.
  • Signed URLs provide time-limited read or write access to an object through a generated URL.
  • Signed policy documents helps specify what can be uploaded to a bucket.
  • Cloud Storage supports encryption at rest and in transit as well
  • Cloud Storage supports both
    • Server-side encryption with support for Google managed, Customer managed and Customer supplied encryption keys
    • Client-side encryption: encryption that occurs before data is sent to Cloud Storage, encrypted at client side.
  • Cloud Storage operations are
    • strongly consistent for read after writes or deletes and listing
    • eventually consistent for granting access to or revoking access
  • Cloud Storage allows setting CORS configuration at the bucket level only

Cloud Storage Rapid (New – 2025)

  • Rapid Bucket (formerly Rapid Storage) is a high-performance zonal object storage offering designed for AI/ML and HPC workloads
    • Provides sub-millisecond random read and write latency
    • Up to 15 TB/s of aggregate throughput and 20 million QPS
    • Collocates data with AI accelerators (TPUs/GPUs) in the same physical zone
    • 5x faster checkpoint restores and 3.2x faster checkpoint writes
  • Rapid Cache (formerly Anywhere Cache) accelerates reads on-demand for workloads in existing multi-region buckets, providing up to 20 Tbps throughput
  • Storage Intelligence provides zero-configuration dashboards, aggregated activity views, and enhanced batch operations for streamlined data management

Cloud SQL

  • provides relational MySQL, PostgreSQL and SQL Server databases as a service
  • managed, however, needs to select and provision machines
  • supports automatic replication, managed backups, vertical scaling for read and write, Horizontal scaling (using read replicas)
  • provides High Availability configuration provides data redundancy and failover capability with minimal downtime, when a zone or instance becomes unavailable due to a zonal outage, or an instance corruption
  • HA standby instance does not increase scalability and cannot be used for read queries.
  • Read replicas help scale horizontally the use of data in a database without degrading performance
  • is regional – although it now supports cross region read replicas
  • supports data encryption at rest and in transit
  • supports Point-In-Time recovery with binary logging and backups
  • available in two editions (introduced 2023-2024):
    • Cloud SQL Enterprise – provides all core capabilities, suitable for applications with less stringent availability and performance requirements
    • Cloud SQL Enterprise Plus – provides enhanced performance (up to 3x faster reads, 2x better write latency), higher availability, advanced observability (Query Insights), and supports Performance-optimized and Memory-optimized machine families (up to 32 GiB RAM per vCPU)
  • supports Private Service Connect (PSC) for automated endpoint creation in VPCs
  • supports Managed Connection Pooling with IAM authentication
  • supports PostgreSQL 18, MySQL 8.4, and SQL Server 2022
  • Extended Support – starting Feb 2025, instances running EOL major versions are automatically enrolled in paid extended support

AlloyDB for PostgreSQL (New Service)

  • fully managed, PostgreSQL-compatible database service designed for demanding enterprise workloads
  • more than 4x faster for transactional workloads and up to 100x faster for analytical queries compared to standard PostgreSQL
  • provides 99.99% availability SLA including maintenance
  • 100% compatible with open-source PostgreSQL
  • key features:
    • Columnar Engine – built-in columnar engine for real-time analytical queries on transactional data
    • AI/ML Integration – built-in vector search, Vertex AI integration for gen AI applications
    • Adaptive Autovacuum and automatic memory management
    • Index Advisor – recommends indexes to improve query performance
  • AlloyDB Omni – downloadable version that can run anywhere (on-premises, other clouds, edge)
  • supports PostgreSQL 14, 15, 16, 17, and 18
  • is regional with cross-region replication for disaster recovery

Cloud Spanner

  • fully managed, globally distributed, strongly consistent relational database service
  • provides virtually unlimited horizontal scalability with 99.999% availability SLA
  • supports SQL (GoogleSQL and PostgreSQL interface)
  • now available in editions (introduced 2024):
    • Standard Edition – core relational database capabilities
    • Enterprise Edition – adds multi-model capabilities including Spanner Graph, full-text search, vector search, managed autoscaling, and incremental backups
    • Enterprise Plus Edition – highest performance and availability
  • Multi-model capabilities (2024-2025):
    • Spanner Graph – native graph support using industry-standard GQL (Graph Query Language), interoperable with SQL for querying relational and connected data in a single operation
    • Vector Search – built-in vector similarity search for gen AI applications (cosine, Euclidean, dot-product distance)
    • Full-text Search – integrated text search capabilities
    • Vertex AI Integration – direct integration for embedding generation and AI model invocation
  • Tiered Storage (GA) – store data across SSDs or HDDs to optimize costs
  • is Global – spans multiple regions with strong consistency

Firestore (formerly Datastore)

  • Cloud Datastore has been superseded by Firestore and is now available in two modes:
    • Firestore in Datastore mode – same data model as original Datastore but runs on the Firestore engine (built on Spanner); backward-compatible with existing Datastore applications
    • Firestore in Native mode – new data model with real-time listeners, offline support, and richer querying
  • serverless, fully managed NoSQL document database
  • scales automatically with strong consistency
  • Firestore in Datastore mode improvements:
    • Queries in transactions are no longer required to be ancestor queries
    • Transactions are no longer limited to 25 entity groups
  • Firestore Enterprise Edition (2026) supports Text Search and Geospatial Search
  • Query Engine with Pipelines (2025) – introduces 100+ new pipeline operations for complex queries directly within the database
  • supports vector search for gen AI applications

BigQuery

  • serverless, highly scalable enterprise data warehouse for analytics
  • user- or project-level custom query quota
  • supports dry-run which helps in pricing queries based on the amount of bytes read i.e. --dry_run flag in the bq command-line tool or dryRun parameter when submitting a query job using the API
  • Pricing models:
    • On-demand – pay per TiB of data processed (first 1 TiB/month free)
    • Capacity (Editions) – replaced legacy flat-rate pricing (July 2023). Available in Standard, Enterprise, and Enterprise Plus editions with autoscaling slots
    • Legacy flat-rate and Flex Slots are no longer available for new purchases; existing commitments migrate to Editions upon expiration
  • BigQuery ML – build and run ML models using SQL, including:
    • TimesFM – state-of-the-art pre-trained forecasting model from Google Research
    • Gemini and open-source LLM model integration
    • Row-wise inference functions for mixing gen AI with SQL
    • Contribution Analysis for explaining changes in metrics
  • AI/ML capabilities (2024-2025):
    • AI functions for processing unstructured data
    • MCP (Model Context Protocol) support for agent-building tools
    • Over 100x scalability gains for LLM inference
    • Structured data generation/extraction with LLMs
  • BigQuery Studio – unified interface for data analytics with improved resource search and explorer

Bigtable

  • fully managed, wide-column NoSQL database designed for large analytical and operational workloads
  • handles massive scale with consistent low-latency (single-digit millisecond)
  • ideal for time-series data, IoT, financial data, and ad-tech
  • now available in editions (GA April 2026):
    • Enterprise and Enterprise Plus editions with advanced analytics, performance, and resource management features
  • GoogleSQL support – query Bigtable using standard SQL with features like:
    • Window functions for advanced analytic operations (GA 2026)
    • Distributed counting for real-time dashboards
    • KNN similarity search
  • Data Boost – serverless compute for running analytical queries without impacting operational workloads
  • In-Memory Tier (2026) – supports up to 120,000 QPS on a single row with hotspot resistance
  • supports replication across multiple regions for high availability
  • is regional with multi-region replication options

Filestore

  • fully managed, high-performance NFS file storage service
  • provides shared file storage mountable by Compute Engine VMs, GKE nodes, and other Google Cloud compute
  • supports NFSv3 and NFSv4.1 protocols
  • available in multiple tiers:
    • Basic (HDD/SSD) – for file sharing, software development, web hosting
    • Zonal – high-performance tier with higher IOPS and throughput
    • Enterprise – multi-zone with 99.99% availability SLA for business-critical apps
  • supports integration with GKE via Filestore CSI driver

Memorystore

  • fully managed in-memory data store service for Redis, Memcached, and Valkey
  • Memorystore for Valkey (GA 2025) – open-source, Redis-compatible in-memory database
    • 99.99% availability SLA
    • Supports Valkey versions 7.2, 8.0, and 9.0
    • Features: Private Service Connect, multi-VPC access, cross-region replication, persistence
    • Zero-downtime scaling, instances up to 14.5 TB
    • Valkey 9.0 includes SIMD optimizations for improved throughput and latency
  • Memorystore for Redis – managed Redis with Basic and Standard tiers (Standard includes replication and automatic failover)
  • Memorystore for Redis Cluster – high-throughput with clustering support
  • Memorystore for Memcached – managed Memcached for caching
  • supports vector search capabilities for gen AI applications

Google Persistent Disk & Hyperdisk

  • Persistent Disk – durable block storage for Compute Engine VMs
    • Standard (pd-standard) – HDD-backed, suitable for sequential read/write workloads
    • Balanced (pd-balanced) – SSD-backed, balance of performance and cost
    • SSD (pd-ssd) – SSD-backed, high random IOPS
    • Available as zonal or regional (synchronous replication across 2 zones)
  • Hyperdisk (newer generation, recommended) – higher performance block storage leveraging Google’s Titanium offload technology
    • Hyperdisk Balanced – general-purpose with configurable IOPS and throughput
    • Hyperdisk Extreme – highest IOPS for demanding databases
    • Hyperdisk Throughput – high throughput for bandwidth-intensive workloads at cost similar to cold storage
    • Hyperdisk ML – optimized for serving ML models with high throughput reads
    • Hyperdisk Balanced High Availability – for GKE workloads requiring HA (GKE 1.33+)
  • Hyperdisk Storage Pools (2024) – provision IOPS and throughput in aggregate; dynamically allocated across volumes for better utilization
  • Backup Vaults (GA 2025) – support for standalone Persistent Disk and Hyperdisk backups with multi-region capability

Google Local SSD

  • physically attached storage providing very high IOPS and low latency
  • ephemeral – data does not persist beyond the life of the instance
  • ideal for caches, scratch disks, and temporary processing
  • Titanium Local SSD (2024-2025) – next-generation local storage available with newer machine series:
    • C3 machine series with -lssd machine types (e.g., c3-standard-88-lssd)
    • C4 machine series with Intel Xeon 6 – up to 35% lower access latency
    • C4A with Google Axion processors – up to 72 vCPUs, 576 GB memory, 6 TB local storage
    • Z3 storage-optimized – up to 3x disk throughput, 35% lower latency, 3-36 TiB per VM
  • not available with shared-core machine types
  • data may not be available during maintenance events on storage-optimized VMs

Related Posts

Google Cloud Identity Services Cheat Sheet

~ Last updated on : ~ jayendrapatil

Identity & Access Management – IAM

  • administrators authorize who can take what action on which resources
  • IAM Member (Principal) can be a Google Account (for end users), a service account (for apps and virtual machines), a Google group, a Google Workspace or Cloud Identity domain, or a workforce/workload identity that can access a resource.
  • IAM Role is a collection of permissions granted to authenticated members.
  • supports 3 kinds of roles
    • Basic roles (formerly called Primitive roles) – broad level of access (Owner, Editor, Viewer)
    • Predefined roles – finer-grained granular access control
    • Custom roles – tailored permissions when predefined roles don’t meet the needs.
  • Best practice is to use Predefined over basic roles
  • IAM Policy binds one or more members to a role.
  • IAM policy can be set at any level in the resource hierarchy: organization level, folder level, the project level, or the resource level.
  • IAM Policy inheritance is transitive and resources inherit the policies of all of their parent resources.
  • Effective policy for a resource is the union of the policy set on that resource and the policies inherited from higher up in the hierarchy.
  • Service account is a special kind of account used by an application or a virtual machine (VM) instance, not a person.
  • Access Scopes are the legacy method of specifying permissions for the instance for default service accounts
  • Best practice is to set the full cloud-platform access scope on the instance, then securely limit the service account’s access using IAM roles.
  • Delegate responsibility with groups (instead of individual users) and service accounts (for server-to-server interactions)

IAM Conditions

  • IAM Conditions allow granting resource access to identities (members) only if configured conditions are met.
  • Conditions are specified in role bindings of a resource’s IAM policy.
  • Conditions support attributes like date/time, resource type, resource name, IP address, and more.
  • Useful for temporary access, restricting access to specific resources, or limiting access based on device attributes.

IAM Deny Policies

  • IAM Deny Policies (GA) allow defining deny rules that prevent certain principals from using certain permissions, regardless of the roles they’re granted.
  • Deny policies act as guardrails and take precedence over allow policies.
  • Can be attached at the organization, folder, or project level.
  • Useful for centralizing management of administrative privileges and building defense in depth with Organization Policies.
  • Deny policies use the IAM v2 permission format (SERVICE_FQDN/RESOURCE.ACTION).

Workload Identity Federation

  • Workload Identity Federation allows external workloads (AWS, Azure, on-premises) to access Google Cloud resources without using service account keys.
  • Eliminates the maintenance and security burden associated with service account keys.
  • Uses identity pools and providers to map external identities to Google Cloud IAM.
  • Supports OIDC and SAML 2.0 protocols.
  • Recommended for GKE workloads — Workload Identity Federation for GKE enables pods to authenticate to Google Cloud services directly.

Privileged Access Manager (PAM)

  • Privileged Access Manager (PAM) is a Google Cloud native, managed solution for just-in-time temporary privilege elevation.
  • Enables on-demand, time-bound access to sensitive resources instead of always-on privileges.
  • Supports approval workflows — requesters can seek approval from designated approvers before access is granted.
  • Requesters can schedule grant requests up to seven days in advance (e.g., for maintenance windows or on-call shifts).
  • Provides audit logs to track who had access to what and when.
  • IAM recommender can remediate excessive permissions for Google groups by transitioning to PAM entitlements (Preview, 2026).

Principal Access Boundary Policies

  • Principal Access Boundary (PAB) Policies restrict which resources a principal is eligible to access.
  • By default, principals are eligible to access any Google Cloud resource (if they have the permission).
  • PAB policies define rules specifying which resources a set of principals can access — any resource not included is blocked.
  • Works alongside allow policies and deny policies for defense in depth.
  • Managed at the organization level with the Principal Access Boundary Admin role.

Agent Identity (GA – April 2026)

  • Agent Identity provides a first-class, strongly attested, cryptographic identity for AI agents — distinct from human identities or generic service accounts.
  • Based on the SPIFFE standard, tied to the lifecycle of the resource hosting the agent.
  • Enables agents to securely authenticate to MCP servers, cloud resources, endpoints, and other agents.
  • Supports acting on the agent’s own behalf or on behalf of an end user.
  • More secure than service accounts as the identity is per-agent and lifecycle-managed.
  • Agent Identity auth manager (Preview) helps authenticate agents to third-party services using OAuth or API keys.

Cloud Identity

  • Cloud Identity is an Identity as a Service (IDaaS) solution that helps centrally manage the users and groups.
  • configured to federate identities between Google and other identity providers, such as Active Directory and Microsoft Entra ID (formerly Azure Active Directory)
  • Cloud Identity and Google Workspace support Security Assertion Markup Language (SAML) 2.0 for single sign-on with authentication performed by an external identity provider (IdP)
  • With SAML, Cloud Identity or Google Workspace acts as a service provider that trusts the SAML IdP to verify a user’s identity on its behalf.
  • Google Cloud Directory Sync – GCDS implements the synchronization process between external IdP

Workforce Identity Federation

  • Workforce Identity Federation extends Google Cloud’s identity capabilities to support syncless, attribute-based single sign-on for human users.
  • Eliminates the need to synchronize user identities from an external IdP to Google-managed identities (no GCDS needed).
  • Supports multiple identity protocols: OpenID Connect (OIDC) and SAML 2.0.
  • Supports multiple IdPs per identity pool including Okta, Ping Identity, ADFS, and Microsoft Entra ID.
  • Over 95% of Google Cloud products now support Workforce Identity Federation.
  • Provides fine-grained, user-level access control without requiring users to have Google Accounts.

Cloud Billing

  • Google Cloud Billing defines billing accounts linked to Google Cloud Projects to determine who pays for a given set of Google Cloud resources.
  • To move the project to a different billing account, you must be a billing administrator and the project owner.
  • To link a project to a billing account, you must be a Billing Account Administrator or Billing Account User on the billing account OR Project Billing Manager on the project
  • Cloud Billing budgets can be created to monitor all of the Google Cloud charges in one place and configure alerts
  • supports BigQuery export with detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a specified BigQuery dataset
  • Google Cloud billing data is not added retroactively to BigQuery, so the data before export is enabled will not be visible.

FinOps Hub

  • FinOps Hub centralizes all cost optimization activities in one place, highlighting inefficiencies and providing actionable recommendations.
  • Uses Cloud Billing to retrieve cost data and various Google Cloud cost recommenders for optimization and utilization metrics.
  • FinOps Hub 2.0 (announced at Cloud Next 2025) focuses on bringing utilization insights on resources to the forefront to identify potential waste.
  • Enables collaboration between business professionals and development teams to drive cost optimization.

Cost Anomaly Detection (GA – 2025)

  • Cost Anomaly Detection uses AI to identify spending patterns based on historical and seasonal trends, forecasting expected daily spend per project.
  • Continuously monitors actual spend every hour and detects deviations.
  • Anomaly alerts are enabled by default for every customer across all projects.
  • Supports customizable thresholds for cost impact amount and percent of deviation.
  • Provides automated alerts/notifications and allows feedback to improve accuracy.

CUD Analysis (GA – June 2026)

  • CUD Analysis provides a unified interface to examine both spend-based and resource-based Committed Use Discounts (CUDs).
  • Helps understand savings, track how effectively commitments are used, and download data for offline analysis.
  • Supports the new spend-based CUD model with direct discounted pricing (replacing the previous credit-based model).

AI Cost Summary Agent (Preview – April 2026)

  • AI Cost Summary Agent analyzes AI costs and provides insights into AI-related spend.
  • Focuses on spending related to Gemini usage, including Gemini API and Vertex AI.
  • Available as a widget on the Billing Overview page.

Spend Caps (Private Preview – 2026)

  • Spend Caps allow administrators to set budget limits at the project level that are automatically enforced.
  • If a project reaches the limit, Google Cloud issues a warning and then pauses API traffic.
  • Addresses the limitation of budgets which only alert but don’t enforce spending limits.

Related Posts

Terraform Cheat Sheet

~ Last updated on : ~ jayendrapatil ~ 8 Comments

⚠️ Important Updates (June 2026)

License Change: As of Terraform 1.6 (October 2023), Terraform is licensed under the Business Source License (BSL 1.1), replacing the previous MPL 2.0 open-source license. The BSL restricts using Terraform to compete commercially with HashiCorp.

IBM Acquisition: IBM completed its $6.4 billion acquisition of HashiCorp in February 2025. Effective September 1, 2025, all HashiCorp business operations transitioned to IBM.

OpenTofu Fork: OpenTofu, a community fork under the Linux Foundation (CNCF Sandbox), remains under MPL 2.0 and is drop-in compatible with Terraform 1.5 configurations.

Terraform Cloud Rebranded: Effective April 22, 2024, Terraform Cloud is now HCP Terraform.

Certification: The Terraform Associate (003) exam was retired January 8, 2026. The current exam is Terraform Associate (004).

  • A provisioning declarative tool based on Infrastructure as Code paradigm, now licensed under BSL 1.1 (source-available, not open-source since v1.6)
  • Designed on immutable infrastructure principles
  • Written in Golang and uses own syntax – HCL (HashiCorp Configuration Language), but also supports JSON
  • Helps to evolve the infrastructure, safely and predictably
  • Applies Graph Theory to IaC and provides Automation, Versioning and Reusability
  • Terraform is a multipurpose composition tool:
    ○ Composes multiple tiers (SaaS/PaaS/IaaS)
    ○ A plugin-based architecture model
  • Terraform is not a cloud agnostic tool. It embraces all major Cloud Providers and provides common language to orchestrate the infrastructure resources
  • Terraform is not a configuration management tool and other tools like Chef, Ansible exist in the market.
  • Current stable version: Terraform 1.15.x (as of June 2026)

Terraform Architecture

Terraform Architecture

Terraform Providers (Plugins)

  • Provide abstraction above the upstream API and is responsible for understanding API interactions and exposing resources.
  • Invoke only upstream APIs for the basic CRUD operations
  • Providers are unaware of anything related to configuration loading, graph theory, etc.
  • Supports multiple provider instances using alias for e.g. multiple AWS providers with different regions
  • Can be integrated with any API using providers framework
  • Most providers configure a specific infrastructure platform (either cloud or self-hosted).
  • Can also offer local utilities for tasks like generating random numbers for unique resource names.
  • (New in 1.8) Providers can export provider-defined functions accessible via provider::<name>::function() syntax, enabling custom logic without brittle workarounds.

Terraform Provisioners

  • Run code locally or remotely on resource creation
    • local-exec executes code on the machine running Terraform
    • remote-exec
      • runs on the provisioned resource
      • supports ssh and winrm
    • requires inline list of commands
  • Should be used as a last resort — provisioners break Terraform’s declarative model by introducing imperative operations
  • Are defined within the resource block.
  • Support types – Create and Destroy
    • if creation time fails, resource is tainted if provisioning failed, by default. (next apply it will be re-created)
    • behavior can be overridden by setting the on_failure to continue, which means ignore and continue
    • for destroy, if it fails – resources are not removed
  • Note: Chef, Habitat, Puppet, and Salt Masterless provisioners were removed in Terraform v0.15.0. Only local-exec, remote-exec, and file provisioners remain.
  • Prefer alternatives: user_data, cloud-init, or configuration management tools (Ansible, Chef, etc.)

Terraform Workspaces

  • Helps manage multiple distinct sets of infrastructure resources or environments with the same code.
  • Just need to create needed workspace and use them, instead of creating a directory for each environment to manage
  • State files for each workspace are stored in the directory terraform.tfstate.d
  • terraform workspace new dev creates a new workspace and switches to it as well
  • terraform workspace select dev helps select workspace
  • terraform workspace list lists the workspaces and shows the current active one with *
  • Does not provide strong separation as it uses the same backend
  • Note: For stronger environment isolation at scale, consider Terraform Stacks (GA on HCP Terraform since September 2025), which provide explicit cross-environment orchestration with separate state per deployment.

Terraform Workflow

Terraform Workflow

init

  • Initializes a working directory containing Terraform configuration files.
  • Performs
    • Backend initialization, storage for Terraform state file.
    • Modules installation, downloaded from Terraform registry to local path
    • Provider(s) plugins installation, the plugins are downloaded in the sub-directory of the present working directory at the path of .terraform/plugins
  • Supports -upgrade to update all previously installed plugins to the newest version that complies with the configuration’s version constraints
  • Is safe to run multiple times, to bring the working directory up to date with changes in the configuration
  • Does not delete the existing configuration or state

validate

  • Validates syntactically for format and correctness.
  • Is used to validate/check the syntax of the Terraform files.
  • Verifies whether a configuration is syntactically valid and internally consistent, regardless of any provided variables or existing state.
  • A syntax check is done on all the terraform files in the directory, and will display an error if any of the files doesn’t validate.
  • (New in 1.15) terraform validate now also validates the backend block configuration.

plan

  • Creates an execution plan
  • Traverses each vertex and requests each provider using parallelism
  • Calculates the difference between the last-known state and the current state and presents this difference as the output of the terraform plan operation
  • Does not modify the infrastructure or state.
  • Allows a user to see which actions Terraform will perform prior to making any changes to reach the desired state
  • Will scan all *.tf files in the directory and create the plan
  • Will perform refresh for each resource and might hit rate limiting issues as it calls provider APIs
  • All resources refresh can be disabled or avoided using
    • -refresh=false or
    • -target=xxxx or
    • break resources into different directories.
  • Supports -out to save the plan
  • Supports -refresh-only mode to update state without making infrastructure changes (replacement for the deprecated terraform refresh command)

apply

  • Apply changes to reach the desired state.
  • Scans the current directory for the configuration and applies the changes appropriately.
  • Can be provided with an explicit plan, saved as out from terraform plan
  • If no explicit plan file is given on the command line, terraform apply will create a new plan automatically and prompt for approval to apply it
  • Will modify the infrastructure and the state.
  • If a resource successfully creates but fails during provisioning,
    • Terraform will error and mark the resource as “tainted”.
    • A resource that is tainted has been physically created, but can’t be considered safe to use since provisioning failed.
    • Terraform also does not automatically roll back and destroy the resource during the apply when the failure happens, because that would go against the execution plan.
  • Does not import any resource (use import blocks or terraform import CLI instead).
  • Supports -auto-approve to apply the changes without asking for a confirmation
  • Supports -target to apply a specific module
  • Supports -replace=ADDRESS to force recreation of a specific resource (replacement for the deprecated terraform taint command)
⚠️ Deprecated Command: refresh

  • Used to reconcile the state Terraform knows about (via its state file) with the real-world infrastructure
  • Does not modify infrastructure, but does modify the state file
  • Deprecated since Terraform 0.15.4 due to unsafe default behavior with misconfigured credentials
  • Use instead: terraform plan -refresh-only or terraform apply -refresh-only

destroy

  • Destroy the infrastructure and all resources
  • Modifies both state and infrastructure
  • terraform destroy -target can be used to destroy targeted resources
  • terraform plan -destroy allows creation of destroy plan

import

  • Helps import already-existing external resources, not managed by Terraform, into Terraform state and allow it to manage those resources
  • CLI usage: terraform import requires you to first write the resource definition in Terraform and then import the resource
  • (New in 1.6) Declarative import blocks: Import can now be defined directly in configuration files, enabling:
    • Bulk imports in a single plan/apply cycle
    • Version-controlled import definitions in Git
    • Plan preview before actual import
    • Auto-generation of resource configuration with terraform plan -generate-config-out=generated.tf
⚠️ Deprecated Command: taint

  • Previously marked a Terraform-managed resource as tainted, forcing it to be destroyed and recreated on the next apply.
  • Deprecated since Terraform v0.15.2
  • Use instead: terraform apply -replace=RESOURCE_ADDRESS
  • The -replace flag is safer because it integrates with the plan workflow, allowing you to preview the change before applying.

fmt

  • Format to lint the code into a standard format

console

  • Command provides an interactive console for evaluating expressions.

test (New in 1.6)

  • Native testing framework for validating Terraform modules
  • Uses .tftest.hcl files with run blocks containing commands (plan or apply), variables, and assertions
  • Tests run in parallel since Terraform 1.7
  • Eliminates the need for external testing tools like Terratest (Go) for basic module validation
  • (New in 1.15) Supports functions inside mock blocks for more flexible test scenarios

Terraform New Features (1.6 – 1.15)

  • Terraform 1.6 (Oct 2023): Declarative import blocks, native testing framework, first BSL release
  • Terraform 1.7: removed block (removes resource from state without destroying it), parallel test execution
  • Terraform 1.8: Provider-defined functions via provider::<name>::function()
  • Terraform 1.9: Ephemeral values — values that exist during plan/apply but are never written to state (ideal for secrets/tokens), enhanced module input validation with cross-variable references
  • Terraform 1.10: S3 native state locking (experimental), ephemeral resources
  • Terraform 1.11: S3 native state locking GA (use_lockfile = true), DynamoDB locking deprecated
  • Terraform 1.15 (April 2026):
    • Dynamic module sources — use variables (with const = true) in source and version fields
    • deprecated attribute on variable and output blocks for communicating breaking changes
    • convert() function for precise inline type conversion
    • type attribute for output blocks (like variables)
    • S3 backend supports aws login authentication
    • Windows ARM64 builds

Terraform Modules

  • Enables code reuse
  • Supports versioning to maintain compatibility
  • Stores code remotely
  • Enables easier testing (can use native terraform test since 1.6)
  • Enables encapsulation with all the separate resources under one configuration block
  • Modules can be nested inside other modules, allowing you to quickly spin up whole separate environments.
  • Can be referred using source attribute
  • (New in 1.15) Module source and version can reference variables declared with const = true
  • Supports Local and Remote modules
    • Local modules are stored alongside the Terraform configuration (in a separate directory, outside of each environment but in the same repository) with source path ./ or ../
    • Remote modules are stored externally in a separate repository, and supports versioning
  • Supports following backends/sources
    • Local paths
    • Terraform Registry
    • GitHub
    • Bitbucket
    • Generic Git, Mercurial repositories
    • HTTP URLs
    • S3 buckets
    • GCS buckets
  • Module requirements
    • Must be on GitHub and must be a public repo, if using public registry.
    • Must be named terraform-<PROVIDER>-<NAME>, where <NAME> reflects the type of infrastructure the module manages and <PROVIDER> is the main provider where it creates that infrastructure. for e.g. terraform-google-vault or terraform-aws-ec2-instance.
    • Must maintain x.y.z tags for releases to identify module versions. Release tag names must be a semantic version, which can optionally be prefixed with a v for example, v1.0.4 and 0.9.2. Tags that don’t look like version numbers are ignored.
    • Must maintain a Standard module structure, which allows the registry to inspect the module and generate documentation, track resource usage, parse submodules and examples, and more.

Terraform Read and Write Configuration

terraform_sample

  • Resources
    • resource are the most important element in the Terraform language that describes one or more infrastructure objects, such as compute instances etc
    • Resource type and local name together serve as an identifier for a given resource and must be unique within a module for e.g. aws_instance.local_name
  • Data Sources
    • data allow data to be fetched or computed for use elsewhere in Terraform configuration
    • Allows a Terraform configuration to make use of information defined outside of Terraform, or defined by another separate Terraform configuration
  • Variables
    • variable serve as parameters for a Terraform module and act like function arguments
    • Allows aspects of the module to be customized without altering the module’s own source code, and allowing modules to be shared between different configurations
    • Can be defined through multiple ways
      • Command line for e.g. -var="image_id=ami-abc123"
      • Variable definition files .tfvars or .tfvars.json. By default, Terraform automatically loads:
        • Files named exactly terraform.tfvars or terraform.tfvars.json.
        • Any files with names ending in .auto.tfvars or .auto.tfvars.json
        • File can also be passed with -var-file
      • Environment variables can be used to set variables using the format TF_VAR_name
    • Terraform loads variables in the following order, with later sources taking precedence over earlier ones:
      • Environment variables
      • terraform.tfvars file, if present.
      • terraform.tfvars.json file, if present.
      • Any *.auto.tfvars or *.auto.tfvars.json files, processed in lexical order of their filenames.
      • Any -var and -var-file options on the command line, in the order they are provided.
    • (New in 1.15) Variables support const = true attribute for compile-time constants usable in source/version, and deprecated = "message" attribute for communicating planned removals.
  • Local Values
    • locals assigns a name to an expression, allowing it to be used multiple times within a module without repeating it.
    • Are like a function’s temporary local variables.
    • Helps to avoid repeating the same values or expressions multiple times in a configuration.
  • Output
    • Are like function return values.
    • Output can be marked as containing sensitive material using the optional sensitive argument, which prevents Terraform from showing its value in the list of outputs. However, they are still stored in the state as plain text.
    • In a parent module, outputs of child modules are available in expressions as module.<MODULE NAME>.<OUTPUT NAME>.
    • (New in 1.15) Outputs now support a type attribute for type constraints, and deprecated = "message" for signaling breaking changes.
  • Ephemeral Values (New in 1.9)
    • Values that materialize during plan or apply but are never written to the state file
    • Ideal for secrets: dynamically generated passwords, tokens from Vault/AWS Secrets Manager, temporary credentials
    • Prevents sensitive data leakage through state files
  • Named Values
    • Is an expression that references the associated value for e.g. aws_instance.local_name, data.aws_ami.centos, var.instance_type etc.
    • Support Local named values for e.g count.index
  • Dependencies
    • Identifies implicit dependencies as Terraform automatically infers when one resource depends on another by studying the resource attributes used in interpolation expressions for e.g aws_eip on resource aws_instance
    • Explicit dependencies can be defined using depends_on where dependencies between resources are not visible to Terraform
  • Data Types
    • Supports primitive data types of
      • string, number and bool
      • Terraform language will automatically convert number and bool values to string values when needed
    • Supports complex data types of
      • list – a sequence of values identified by consecutive whole numbers starting with zero.
      • map – a collection of values where each is identified by a string label.
      • set – a collection of unique values that do not have any secondary identifiers or ordering.
    • Supports structural data types of
      • object – a collection of named attributes that each have their own type
      • tuple – a sequence of elements identified by consecutive whole numbers starting with zero, where each element has its own type.
  • Built-in Functions
    • Includes a number of built-in functions that can be called from within expressions to transform and combine values for e.g. min, max, file, concat, element, index, lookup etc.
    • Does not support user-defined functions
    • (New in 1.8) Provider-defined functions extend available functions via provider::<name>::function()
    • (New in 1.15) convert() function for precise inline type conversion
  • Dynamic Blocks
    • Acts much like a for expression, but produces nested blocks instead of a complex typed value. It iterates over a given complex value, and generates a nested block for each element of that complex value.
  • Terraform Comments
    • Supports three different syntaxes for comments:
      • #
      • //
      • /* and */

Terraform Backends

  • Determines how state is loaded and how an operation such as apply is executed
  • Are responsible for storing state and providing an API for optional state locking
  • Needs to be initialized
  • If switching the backend for the first time setup, Terraform provides a migration option
  • Helps
    • Collaboration and working as a team, with the state maintained remotely and state locking
    • Can provide enhanced security for sensitive data
    • Support remote operations
  • Supports local vs remote backends
    • Local (default) backend stores state in a local JSON file on disk
    • Remote backend stores state remotely like S3, OSS, GCS, Consul and support features like remote operation, state locking, encryption, versioning etc.
  • Supports partial configuration with remaining configuration arguments provided as part of the initialization process
  • Backend configuration doesn’t support interpolations.
  • GitHub is not a supported backend type in Terraform.

Terraform State Management

  • State helps keep track of the infrastructure Terraform manages
  • Stored locally in the terraform.tfstate
  • Recommended not to edit the state manually
  • Use terraform state command
    • mv – to move/rename modules
    • rm – to safely remove resource from the state. (destroy/retain like)
    • pull – to observe current remote state
    • list & show – to write/debug modules
  • (New in 1.7) The removed block provides a declarative alternative to terraform state rm, removing a resource from state without destroying it.

State Locking

  • Happens for all operations that could write state, if supported by backend
  • Prevents others from acquiring the lock & potentially corrupting the state
  • Backends which support state locking are
    • azurerm
    • HashiCorp Consul
    • Tencent Cloud Object Storage (COS)
    • Google Cloud Storage GCS
    • HTTP endpoints
    • Kubernetes Secret with locking done using a Lease resource
    • AliCloud Object Storage OSS with locking via TableStore
    • PostgreSQL
    • AWS S3 with native S3 locking via use_lockfile = true (GA since Terraform 1.11)
      ⚠️ Note: DynamoDB-based state locking for S3 backend is deprecated as of Terraform 1.11. The dynamodb_table argument and related DynamoDB arguments now emit deprecation warnings. Use use_lockfile = true for native S3 locking instead.
    • HCP Terraform (formerly Terraform Cloud) / Terraform Enterprise
  • Can be disabled for most commands with the -lock flag
  • Use force-unlock command to manually unlock the state if unlocking failed

State Security

  • Can contain sensitive data, depending on the resources in use for e.g passwords and keys
  • Using local state, data is stored in plain-text JSON files
  • Using remote state, state is held in memory when used by Terraform. It may be encrypted at rest, if supported by backend for e.g. S3, OSS
  • (New in 1.9) Ephemeral values are never written to state, providing additional security for secrets
  • Note: OpenTofu (the community fork) offers native state file encryption, which Terraform does not currently provide

Terraform Logging

  • Debugging can be controlled using TF_LOG, which can be configured for different levels TRACE, DEBUG, INFO, WARN or ERROR, with TRACE being the more verbose.
  • Logs path can be controlled with TF_LOG_PATH. TF_LOG needs to be specified.

HCP Terraform and Terraform Enterprise

📝 Rebranding Note: Effective April 22, 2024, Terraform Cloud has been renamed to HCP Terraform. The functionality remains the same. All references below use the current name.
  • HCP Terraform (formerly Terraform Cloud) provides Cloud Infrastructure Automation as a Service. It is offered as a multi-tenant SaaS platform and is designed to suit the needs of smaller teams and organizations. Its smaller plans default to one run at a time, which prevents users from executing multiple runs concurrently.
  • Terraform Enterprise is a private install for organizations who prefer to self-manage. It is designed to suit the needs of organizations with specific requirements for security, compliance and custom operations. Now an IBM product following the acquisition.
  • HCP Terraform provides features
    • Remote Terraform Execution – supports Remote Operations for Remote Terraform execution which helps provide consistency and visibility for critical provisioning operations.
    • Workspaces – organizes infrastructure with workspaces instead of directories. Each workspace contains everything necessary to manage a given collection of infrastructure, and Terraform uses that content whenever it executes in the context of that workspace.
    • Remote State Management – acts as a remote backend for the Terraform state. State storage is tied to workspaces, which helps keep state associated with the configuration that created it.
    • Version Control Integration – is designed to work directly with the version control system (VCS) provider.
    • Private Module Registry – provides a private and central library of versioned & validated modules to be used within the organization
    • Team based Permission System – can define groups of users that match the organization’s real-world teams and assign them only the permissions they need
    • Sentinel Policies – embeds the Sentinel policy-as-code framework, which lets you define and enforce granular policies for how the organization provisions infrastructure. Helps eliminate provisioned resources that don’t follow security, compliance, or operational policies.
    • Cost Estimation – can display an estimate of its total cost, as well as any change in cost caused by the proposed updates
    • Security – encrypts state at rest and protects it with TLS in transit.
    • Terraform Stacks (GA since Sept 2025) – enables deploying a single module across multiple environments with explicit cross-stack dependencies, coordinated orchestration, and per-deployment state management.
    • Drift Detection & Continuous Health Checks – automatically detects infrastructure drift and notifies teams
  • Terraform Enterprise features
    • Includes all the HCP Terraform features with
    • Audit – supports detailed audit logging and tracks the identity of the user requesting state and maintains a history of state changes.
    • SSO/SAML – SAML for SSO provides the ability to govern user access to your applications.
⚠️ Terraform Enterprise Deployment Change:

The Replicated Native Scheduler deployment option for Terraform Enterprise has reached end of life. The final Replicated release was in March 2025, with support ending April 1, 2026. After December 31, 2027, all Replicated installations will cease to function.

New deployment options: Docker (Docker Compose), Kubernetes, and Podman. These provide faster startups, reduced resource requirements, and improved security.

  • HCP Terraform currently supports following VCS Providers
    • GitHub.com
    • GitHub.com (OAuth)
    • GitHub Enterprise
    • GitLab.com
    • GitLab EE and CE
    • Bitbucket Cloud
    • Bitbucket Server
    • Azure DevOps Server
    • Azure DevOps Services
  • HCP Terraform uses a Resources Under Management (RUM) pricing model, calculating costs based on the number of resources connected to Terraform rather than by user count.
  • A Terraform Enterprise install that is provisioned on a network that does not have Internet access is generally known as an air-gapped install. These types of installs require you to pull updates, providers, etc. from external sources vs. being able to download them directly.

OpenTofu – The Community Fork

  • OpenTofu is a community fork of Terraform 1.5.x under the Linux Foundation, now a CNCF Sandbox project (April 2025)
  • Licensed under MPL 2.0 (truly open-source, guaranteed permanently)
  • Drop-in compatible with Terraform for most configurations; state files are interchangeable
  • Providers work identically between Terraform and OpenTofu
  • Key differentiator: Native state file encryption (not available in Terraform)
  • Does not have Stacks or ephemeral values (as of mid-2026)
  • Decision factors:
    • Do you have a contract with HashiCorp/IBM or use HCP?
    • Do you need Stacks or native state encryption?
    • What does internal policy say about BSL vs open-source licensing?

AWS Content Delivery – Cheat Sheet

~ Last updated on : ~ jayendrapatil

CloudFront

  • provides low latency and high data transfer speeds for distribution of static, dynamic web or streaming content to web users
  • delivers the content through a worldwide network of data centers called Edge Locations — over 600+ Points of Presence (PoPs) and 13 regional edge caches in 100+ cities across 50+ countries
  • supports Embedded POPs deployed directly within ISP/telco networks for highly scaled capacity during peak traffic events
  • keeps persistent connections with the origin servers so that the files can be fetched from the origin servers as quickly as possible.
  • dramatically reduces the number of network hops that users’ requests must pass through
  • supports multiple origin server options, like AWS hosted service for e.g. S3, EC2, ELB or an on premise server, which stores the original, definitive version of the objects
  • single distribution can have multiple origins and Path pattern in a cache behavior determines which requests are routed to the origin
  • supports Web distribution only (RTMP Streaming distribution was deprecated on Dec 31, 2020)
    • Web distribution supports static, dynamic web content, on demand using progressive download & HLS and live streaming video content
    • RTMP distribution was discontinued on December 31, 2020. Use HTTP-based streaming protocols (HLS, DASH) instead.
  • supports HTTP/1.0, HTTP/1.1, HTTP/2, and HTTP/3 (QUIC)
    • HTTP/3 uses QUIC, a UDP-based, stream-multiplexed, secure transport protocol that improves upon TCP and TLS
    • HTTP/2 and HTTP/3 can be enabled per distribution
  • supports gRPC delivery (launched Nov 2024) for lightweight, high-performance remote procedure calls over HTTP/2, ideal for microservices architectures
  • supports WebSocket connections automatically with any distribution, including through VPC origins (May 2026)
  • supports HTTPS using either
    • dedicated IP address, which is expensive as dedicated IP address is assigned to each CloudFront edge location
    • Server Name Indication (SNI), which is free but supported by modern browsers only with the domain name available in the request header
  • For E2E HTTPS connection,
    • Viewers -> CloudFront needs either self signed certificate, or certificate issued by CA or ACM
    • CloudFront -> Origin needs certificate issued by ACM for ELB and by CA for other origins
  • Security
    • Origin Access Control (OAC) is the recommended way to restrict S3 origin content to be accessible from CloudFront only
      • OAC supports SigV4, SSE-KMS, POST method in all regions, and granular policy configurations
      • OAC replaced the legacy Origin Access Identity (OAI) which is deprecated — new distributions since March 2026 can only use OAC
      • Migration from OAI to OAC is recommended for all existing distributions
    • VPC Origins (launched Nov 2024) allow CloudFront to point directly to ALBs, NLBs, or EC2 instances in private subnets, eliminating the need for public internet exposure
      • Supports cross-account VPC origins (Nov 2025)
      • CloudFront becomes the single entry point, enhancing security posture
    • supports Geo restriction (Geo-Blocking) to whitelist or blacklist countries that can access the content
    • Signed URLs
      • to restrict access to individual files, for e.g., an installation download for your application.
      • users using a client, for e.g. a custom HTTP client, that doesn’t support cookies
    • Signed Cookies
      • provide access to multiple restricted files, for e.g., video part files in HLS format or all of the files in the subscribers’ area of a website.
      • don’t want to change the current URLs
    • integrates with AWS WAF, a web application firewall that helps protect web applications from attacks by allowing rules configured based on IP addresses, HTTP headers, and custom URI strings
    • Security Dashboard (launched Nov 2023) provides a unified CDN and security experience with one-click WAF protections against common web threats (OWASP Top 10, IP reputation, scanners/probes)
    • integrates with AWS Shield Standard automatically for DDoS protection at no extra cost
  • supports GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE to get object & object headers, add, update, and delete objects
    • only caches responses to GET and HEAD requests and, optionally, OPTIONS requests
    • does not cache responses to PUT, POST, PATCH, DELETE request methods and these requests are proxied back to the origin
  • object removal from cache
    • would be removed upon expiry (TTL) from the cache, by default 24 hrs
    • can be invalidated explicitly, but has a cost associated, however might continue to see the old version until it expires from those caches
    • change object name, versioning, to serve different version
  • Cache Policies and Origin Request Policies
    • Cache Policies control what is included in the cache key (headers, cookies, query strings) and TTL settings
    • Origin Request Policies specify what information to forward to the origin (independent of cache key)
    • Managed policies are provided for common use cases (CachingOptimized, CachingDisabled, etc.)
    • Managed cache policies for web applications added July 2024
  • Response Headers Policies
    • Add HTTP headers (security, CORS, custom) to responses without modifying origin
    • Managed policies include security headers (Strict-Transport-Security, X-Frame-Options, Content-Security-Policy) and CORS configurations
  • supports adding or modifying custom headers before the request is sent to origin which can be used to
    • validate if user is accessing the content from CDN
    • identifying CDN from which the request was forwarded from, in case of multiple CloudFront distribution
    • for viewers not supporting CORS to return the Access-Control-Allow-Origin header for every request
  • supports Partial GET requests using range header to download object in smaller units improving the efficiency of partial downloads and recovery from partially failed transfers
  • supports compression to compress and serve compressed files when viewer requests include Accept-Encoding: gzip in the request header
  • supports different price class to include all regions, to include only least expensive regions and other regions to exclude most expensive regions
  • Flat-rate pricing plans (launched Nov 2025) combine CloudFront CDN, WAF, DDoS protection, bot management, Route 53 DNS, CloudWatch Logs, serverless edge compute, and S3 storage credits into a single monthly price with no overage charges
  • Origin Shield
    • additional centralized caching layer between regional edge caches and the origin
    • helps increase cache hit ratio by collapsing requests across regions into a single origin request per object
    • reduces origin load and operating costs, particularly beneficial for multi-CDN deployments
  • Edge Compute — CloudFront Functions and Lambda@Edge
    • CloudFront Functions — lightweight JavaScript functions running at edge locations for viewer request/response manipulation (URL rewrites, header manipulation, redirects, JWT validation)
      • Sub-millisecond startup, millions of requests/second
      • KeyValueStore (launched Nov 2023) — globally distributed, low-latency data store for CloudFront Functions enabling dynamic routing, feature flags, A/B testing, and tenant routing without code redeployment
    • Lambda@Edge — Node.js/Python functions running at regional edge caches for more complex processing (origin request/response events, network calls, larger compute)
  • Continuous Deployment
    • Test and validate configuration changes with a staging distribution using a percentage of live production traffic (up to 15%)
    • Supports header-based or weight-based traffic routing for safe blue/green deployments
    • Promote changes to primary distribution when validated
  • Access Logs
    • Standard logs (v2) — delivered via CloudWatch vended logs to S3, CloudWatch Logs, or Data Firehose (Nov 2024)
    • Real-time logs — delivered within seconds to Amazon Kinesis Data Streams for real-time monitoring
    • Legacy standard logs delivered to S3 with up to several minutes delay