GCP

Dataflow vs Dataproc – Streaming vs Batch

~ Last updated on : ~ jayendrapatil ~ 1 Comment

Google Cloud Dataflow vs Dataproc (Managed Service for Apache Spark)

📌 2025-2026 Update: Google Cloud has rebranded Dataproc to Managed Service for Apache Spark (formerly known as “Dataproc on Compute Engine” for cluster deployment and “Google Cloud Serverless for Apache Spark” for serverless deployment). The core functionality remains unchanged. Additionally, GCP AI Platform Training (Cloud ML Engine) has been superseded by Vertex AI, which is now part of the Gemini Enterprise Agent Platform.

Cloud Dataproc (Managed Service for Apache Spark)

  • Cloud Dataproc, now rebranded as Managed Service for Apache Spark, is a managed Spark and Hadoop service that lets you take advantage of open-source data tools for batch processing, querying, streaming, and machine learning.
  • Provides a Hadoop cluster on GCP with access to Hadoop-ecosystem tools (e.g., Apache Pig, Hive, and Spark); this has strong appeal if already familiar with Hadoop tools and have Hadoop jobs.
  • Ideal for Lift and Shift migration of existing Hadoop environment.
  • Offers two deployment modes:
    • Cluster deployment — managed Spark clusters on Compute Engine (you pay for cluster uptime)
    • Serverless deployment — Spark-jobs-as-a-service on fully managed Google Cloud infrastructure (you pay for job runtime only, zero cluster management)
  • Consider Dataproc (Managed Service for Apache Spark) when:
    • You have a substantial investment in Apache Spark or Hadoop on-premise and are considering moving to the cloud
    • You are looking at a Hybrid cloud and need portability across a private/multi-cloud environment
    • In the current environment, Spark is the primary machine learning tool and platform
    • The code depends on custom packages along with distributed computing needs
    • You need fine-grained cluster configuration, custom libraries, or specific Hadoop ecosystem tool versions

Key Features (2024-2026)

  • Lightning Engine — next-generation native C++ vectorized execution engine that delivers up to 4.9x faster performance than open-source Spark with zero code changes. Uses SIMD vectorization, intelligent caching, and optimized columnar shuffling. Available on the premium pricing tier.
  • Serverless Spark (Zero-Ops) — zero cluster management with intelligent autoscaling. Resources scale up and down automatically to match job needs, ensuring maximum performance and cost-efficiency without paying for idle time.
  • Enhanced Autoscaling — reduces cluster VM expenditures by up to 40% and cumulative job runtime by 10%.
  • BigQuery Integration — Serverless for Apache Spark is deeply integrated with the BigQuery unified data-to-AI platform, offering a unified developer experience in BigQuery Studio.
  • Vertex AI / Gemini Enterprise Agent Platform Integration — seamless interoperability for ML workflows.
  • Dataproc on GKE — run Spark workloads on Google Kubernetes Engine for containerized environments.

Cloud Dataflow

  • Google Cloud Dataflow is a fully managed, serverless service for unified stream and batch data processing at scale, based on the open-source Apache Beam SDK.
  • Scales to 4,000 workers per job and routinely processes petabytes of data with built-in autoscaling.
  • Ideal for new pipelines with minimal infrastructure management, event stream processing, and real-time analytics.
  • Consider Dataflow when:
    • Building new, greenfield data pipelines (no existing Spark/Hadoop code to migrate)
    • Requiring truly serverless, zero-ops data processing
    • Working with unified batch and streaming in a single pipeline
    • Using it as a pre-processing pipeline for ML models deployed in Vertex AI (Gemini Enterprise Agent Platform)
    • None of the above considerations made for Cloud Dataproc are relevant

Key Features (2024-2026)

  • Dataflow Prime — supports both horizontal autoscaling (more machines) and vertical autoscaling (larger machines) automatically for streaming and batch workloads.
  • AI/ML Integration (Dataflow ML) — RunInference API enables running ML models (PyTorch, TensorFlow, scikit-learn, ONNX, TensorRT) directly within Dataflow pipelines. MLTransform API for data preparation.
  • GPU Support — supports NVIDIA T4, L4, A100, H100, V100, and RTX Pro 6000 GPUs for ML inference workloads.
  • TPU Support — TPU V5E, V5P, and V6E for high-volume, low-latency ML inference at scale directly within Dataflow jobs.
  • Global Compute — dynamically schedules workloads across Google’s global infrastructure, automatically determining optimal location based on data locality and resource availability.
  • Streaming AI — build streaming AI with Gemini models and Gemma models, run remote inference, and streamline data processing.
⚠️ Deprecation Notice: Dataflow SQL was deprecated on July 31, 2024, and fully removed on January 31, 2025. Users should use BigQuery for SQL-based analytics instead.

Cloud Dataflow vs Dataproc — Key Differences

Criteria Dataflow Dataproc (Managed Service for Apache Spark)
Programming Model Apache Beam (Java, Python, Go) Apache Spark, Hadoop, Pig, Hive, Flink, Trino
Infrastructure Fully serverless, zero ops Managed clusters OR Serverless Spark
Cluster Management None (fully managed) User-managed (cluster) or None (serverless)
Best For New pipelines, streaming, minimal ops Existing Spark/Hadoop jobs, lift-and-shift
Streaming Native, unified batch + stream Spark Structured Streaming
ML/AI RunInference API, GPU/TPU support SparkML, Vertex AI integration
Scaling Auto (up to 4K workers), Global Compute Auto or manual, Lightning Engine (4.9x faster)
Code Changes Requires Apache Beam rewrite Minimal (run existing Spark/Hadoop code)

Cloud Dataflow vs Dataproc Decision Tree

Dataflow vs Dataproc

Dataflow vs Dataproc Table

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your company is forecasting a sharp increase in the number and size of Apache Spark and Hadoop jobs being run on your local data center. You want to utilize the cloud to help you scale this upcoming demand with the least amount of operations work and code change. Which product should you use?
    1. Google Cloud Dataflow
    2. Google Cloud Dataproc (Managed Service for Apache Spark)
    3. Google Compute Engine
    4. Google Kubernetes Engine
  2. A startup plans to use a data processing platform, which supports both batch and streaming applications. They would prefer to have a hands-off/serverless data processing platform to start with. Which GCP service is suited for them?
    1. Dataproc
    2. Dataprep
    3. Dataflow
    4. BigQuery
  3. A company has existing Apache Spark ML pipelines running on-premises and wants to migrate to Google Cloud with minimal code changes while achieving better performance. Which service should they use?
    1. Cloud Dataflow
    2. Managed Service for Apache Spark (Dataproc) with Lightning Engine
    3. BigQuery ML
    4. Vertex AI Training
  4. An organization needs to run real-time ML inference on streaming data with GPU acceleration in a fully serverless manner. Which GCP service best fits this requirement?
    1. Dataproc Serverless
    2. Google Cloud Dataflow with RunInference API
    3. Cloud Functions
    4. GKE with Spark Streaming
  5. A team wants to run Apache Spark batch jobs without managing any infrastructure and wants to pay only for the time their jobs are running. Which deployment option should they choose?
    1. Dataproc Cluster deployment
    2. Managed Service for Apache Spark – Serverless deployment
    3. Cloud Dataflow
    4. Compute Engine with Spark installed

Frequently Asked Questions

What is the difference between Dataflow and Dataproc?

Dataflow is a fully managed serverless stream/batch processing service based on Apache Beam. Dataproc (now Managed Service for Apache Spark) is a managed Hadoop/Spark cluster for lift-and-shift big data workloads.

When should I use Dataproc instead of Dataflow?

Use Dataproc when you have existing Spark/Hadoop code to migrate, need specific Hadoop ecosystem tools (Hive, Pig, Presto), or want fine-grained cluster control. Use Dataflow for new pipelines or unified stream+batch processing.

Is Dataflow serverless?

Yes, Dataflow is fully serverless — it automatically provisions, manages, and scales workers based on workload. You only pay for resources consumed during pipeline execution with no idle cluster costs.

Related Posts

References

BigQuery Data Transfer – 35+ Source Connectors

~ Last updated on : ~ jayendrapatil

Google Cloud BigQuery Data Transfer Service

  • BigQuery Data Transfer Service automates data movement into BigQuery on a scheduled, managed basis.
  • After a data transfer is configured, the BigQuery Data Transfer Service automatically loads data into BigQuery on a regular basis.
  • BigQuery Data Transfer Service can also initiate data backfills to recover from any outages or gaps.
  • BigQuery Data Transfer Service can only sink data to BigQuery and cannot be used to transfer data out of BigQuery.
  • BigQuery Data Transfer Service can be accessed using the Google Cloud console, the bq command-line tool, or the BigQuery Data Transfer Service API.
  • In addition to loading data into BigQuery, BigQuery Data Transfer Service is used for dataset copies and scheduled queries.

BigQuery Data Transfer Service Features

  • Simplicity – Eliminates the need for infrastructure management or complex coding.
  • Scalability – Handles massive data volumes and high numbers of concurrent users.
  • Security – Employs encryption, authentication, and authorization; supports FedRAMP High, CJIS compliance, EU Data Boundary, and Sovereign Controls.
  • Cost-effectiveness – Many first-party connectors (e.g., Google Ads, YouTube) are free. Third-party connectors use consumption-based pricing (billed by slot-hours).
  • Event-driven transfers (GA) – Transfers can trigger automatically when a new file arrives in a Cloud Storage bucket, enabling near-real-time data pipelines.
  • Transfer to Managed Iceberg tables – Supports loading data directly into Managed Iceberg tables in BigQuery.
  • Custom organization policies – Allow or deny specific operations on transfer configurations for compliance and security requirements.
  • Regional endpoints – Ensures requests are processed only if the resource exists in the specified location, supporting data residency requirements.

BigQuery Data Transfer Service Sources

  • BigQuery Data Transfer Service supports loading data from the following data sources:

SaaS Platforms

  • Salesforce
  • Salesforce Marketing Cloud (SFMC)
  • ServiceNow

Marketing Platforms

  • Facebook Ads
  • HubSpot (Preview)
  • Klaviyo (Preview)
  • Mailchimp (Preview)

Payment Platforms

  • PayPal (Preview)
  • Stripe (Preview)
  • Shopify (Preview)

Databases and Data Warehouses

  • Amazon Redshift
  • Apache Hive Metastore
  • Microsoft SQL Server (Preview)
  • MySQL
  • Oracle
  • PostgreSQL
  • Snowflake (Preview)
  • Teradata

Cloud Storage

  • Cloud Storage (supports event-driven transfers)
  • Amazon Simple Storage Service (Amazon S3)
  • Azure Blob Storage

Google Services

  • Campaign Manager
  • Comparison Shopping Service (CSS) Center (Preview)
  • Display & Video 360
  • Google Ads (supports custom reports via GAQL)
  • Google Ad Manager (supports incremental DT file updates)
  • Google Analytics 4 (GA4)
  • Google Merchant Center (Preview)
  • Search Ads 360 (supports Performance Max campaigns)
  • Google Play
  • YouTube Channel
  • YouTube Content Owner

Event-Driven Transfers

  • Event-driven transfers automatically load data based on event notifications from Cloud Storage.
  • When a new file arrives in a Cloud Storage bucket, the transfer triggers automatically.
  • After an event-driven transfer is triggered, the service waits up to 10 minutes before triggering the next run, regardless of additional events.
  • Recommended for incremental data ingestion that optimizes cost efficiency.
  • Event-driven transfers don’t support runtime parameters for source URI or data path.

Pricing

  • First-party Google connectors (Google Ads, YouTube, Campaign Manager, etc.) are provided at no cost.
  • Third-party SaaS and database connectors (Salesforce, Facebook Ads, Oracle, MySQL, etc.) use consumption-based pricing measured in slot-hours.
  • Connectors in Preview remain completely free of charge.
  • Standard BigQuery storage and query pricing applies once data is transferred.
  • Jobs triggered by DTS use reservation slots only if the project is assigned to a reservation with QUERY or PIPELINE job types.

Data Delivery SLO

  • The Data Delivery SLO applies only to automatically scheduled data transfers from sources within Google Cloud.
  • For transfers involving third-party or non-Google Cloud sources, service outages with those sources can impact performance, so the Data Delivery SLO does not apply.

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your company uses Google Analytics for tracking. You need to export the session and hit data from a Google Analytics 360 reporting view on a scheduled basis into BigQuery for analysis. How can the data be exported?
    1. Configure a scheduler in Google Analytics to convert the Google Analytics data to JSON format, then import directly into BigQuery using bq command line.
    2. Use gsutil to export the Google Analytics data to Cloud Storage, then import into BigQuery and schedule it using Cron.
    3. Import data to BigQuery directly from Google Analytics using Cron
    4. Use BigQuery Data Transfer Service to import the data from Google Analytics
  2. A company stores raw event files in Cloud Storage and needs near-real-time ingestion into BigQuery without managing schedulers. Which approach should they use?
    1. Set up a Cloud Scheduler job to trigger a Cloud Function every minute to load data.
    2. Use Pub/Sub to stream events directly into BigQuery.
    3. Configure an event-driven transfer in BigQuery Data Transfer Service that triggers automatically when new files arrive in Cloud Storage.
    4. Create a Dataflow streaming pipeline from Cloud Storage to BigQuery.
  3. Your organization needs to migrate data from a Salesforce CRM into BigQuery on a daily schedule. Which service provides a fully managed, code-free solution?
    1. Use Dataflow with a custom Salesforce connector.
    2. Export Salesforce data to CSV and load manually.
    3. Use BigQuery Data Transfer Service with the Salesforce connector.
    4. Use Cloud Data Fusion with a Salesforce plugin.
  4. Which of the following are valid data sources for BigQuery Data Transfer Service? (Choose 3)
    1. Amazon Redshift
    2. Oracle
    3. MongoDB
    4. Azure Blob Storage
    5. Google Cloud Bigtable

References

BigQuery Security – IAM, Encryption & Row-Level

~ Last updated on : ~ jayendrapatil

Google Cloud BigQuery Security

BigQuery Encryption

  • BigQuery automatically encrypts all data before it is written to disk.
  • Each BigQuery object’s data and metadata is encrypted using the Advanced Encryption Standard (AES).
  • By default, Google uses Google default encryption at rest and manages the key encryption keys used for data protection.
  • BigQuery supports Customer-Managed Encryption Keys (CMEK) using Cloud KMS, allowing you to control the encryption keys for datasets, tables, and query results.
    • CMEK can be set as a default for a dataset, ensuring any tables created in future use the specified CMEK.
    • BigQuery does NOT support Customer-Supplied Encryption Keys (CSEK).
  • BigQuery supports Column-level encryption with Cloud KMS using AEAD encryption SQL functions, providing a second layer of protection at the column level.
    • AEAD encryption functions allow you to create keysets for encrypting and decrypting individual values in a table.
    • Provides “double access control” — users need both BigQuery access and Cloud KMS key access to decrypt data.
    • Supports deterministic encryption and decryption interoperable with Sensitive Data Protection.
  • BigQuery uses TLS for data in transit encryption.
  • Sensitive Data Protection (formerly Cloud Data Loss Prevention / Cloud DLP) can be used to scan BigQuery tables and to protect sensitive data and meet compliance requirements.

BigQuery IAM Roles

  • BigQuery supports access control of datasets and tables using IAM.
  • Basic Roles (formerly called “Primitive Roles”)
    • Basic roles act at the project level.
    • By default, granting access to a project also grants access to datasets within it unless overridden.
    • Are not limited to BigQuery resources only.
    • Can separate data access permissions from job-running permissions.
    • Viewer
      • View all datasets
      • Run Jobs/Queries
      • View and update all jobs that they started
    • Editor
      • All Viewer access
      • Modify or delete all tables
      • Create new datasets
    • Owner
      • All Editor access
      • List, modify, or delete all datasets
      • View all jobs
  • Predefined Roles
    • dataViewer, dataEditor, and dataOwner roles
      • are similar to the basic roles except
        • can be assigned for individual datasets
        • don’t give users permission to run jobs or queries
    • user, jobUser roles
      • give users permission to run jobs or queries
      • A jobUser can only start jobs and cancel jobs, but cannot list datasets or tables
      • A user, on the other hand, can perform a variety of other tasks, such as listing or creating datasets
      • User or group granted the user role at the project level can create datasets and can run query jobs against tables in those datasets.
      • user role does not give permission to query data, view table data, or view table schema details for datasets the user did not create. Need to have the dataViewer role for the same.
    • bigquery.admin role — provides permissions to manage all resources within the project, manage all data, and cancel jobs from other users.

BigQuery Access Control Levels

  • BigQuery supports access controls at multiple levels:
    • Organization level — policies applied across all projects
    • Project level — IAM policies on the project resource
    • Dataset level — IAM policies on individual datasets
    • Table/View level — IAM policies on individual tables or views
    • Column level — policy tags for column-level access control
    • Row level — row access policies for row-level security
  • Access control can also be managed using IAM Conditions for attribute-based access control and Tags for tag-based access control.

Authorized Views

  • Authorized views help provide view access to a dataset.
  • Use authorized views to restrict access at a lower resource level such as the table, column, row, or cell.
  • An authorized view allows sharing query results with particular users and groups without giving them access to the underlying tables.
  • Authorized View’s SQL query can be used to restrict the columns (fields) the users are able to query.
  • Authorized views HAVE to be created in a separate dataset from the source dataset. As access controls can be assigned only at the dataset level, if the view is created in the same dataset as the source data, the users would have access to both the view and the data.
  • Authorized View creation process
    • Create a separate dataset to store the view.
    • Create the view in the new dataset.
    • Give the group read access to the dataset containing the view.
    • Authorize the view to access the source dataset.
    • Give the group bigquery.user role to run jobs, including query jobs within the project.
  • Project-level bigquery.user role does not give the users the ability to view or query table data in the dataset containing the tables queried by the view. They need READER access to the dataset containing the view.

Authorized Datasets

  • An authorized dataset lets you authorize all of the views in a specified dataset to access the data in a second dataset.
  • Simplifies management over individual authorized views — instead of authorizing each view separately, the entire dataset is authorized.
  • Any new views added to the authorized dataset automatically gain access to the source dataset.
  • Useful when you have many views that need access to the same source data.

Authorized Routines

  • Authorized routines let you share query results with specific users or groups without giving them access to the underlying tables that generated the results.
  • For UDFs and table functions, you can authorize the function to access source dataset resources on the caller’s behalf.
  • Authorized routines must be created in a separate dataset from the source data.
  • Useful for sharing data transformations while protecting raw data access.

Fine-Grained Access Control

  • BigQuery provides fine-grained access to sensitive columns using policy tags, or type-based classification of data.
  • Policy tags are managed in a hierarchical taxonomy in Dataplex Universal Catalog (formerly Data Catalog, which was deprecated Feb 2025 and discontinued Jan 30, 2026).
  • Using BigQuery column-level security, you can create policies that check, at query time, whether a user has proper access.
  • Users with the Data Catalog Fine-Grained Reader role on a policy tag can access unmasked column data.
  • Row-level security extends the principle of least privilege by enabling fine-grained access control to a subset of data in a BigQuery table, by means of row-level access policies.
    • Row access policies use a filter expression (e.g., FILTER USING (region = "US")).
    • One table can have multiple row-level access policies.
    • Row-level access policies can coexist with column-level security as well as dataset-level, table-level, and project-level access controls.

Dynamic Data Masking

  • BigQuery supports dynamic data masking at the column level, allowing you to selectively obscure column data for user groups while still allowing them access to the column.
  • Data masking is built on top of column-level access control and uses policy tags and data policies.
  • Unlike column-level access control alone, masked users don’t get “permission denied” — they see obscured data instead.
  • Existing queries automatically mask column data based on the roles the user has been granted.
  • Data Masking Rules available:
    • Nullify — returns NULL instead of the column value (highest security)
    • Default masking value — returns a type-appropriate default (e.g., “” for STRING, 0 for INTEGER)
    • Hash (SHA-256) — returns deterministic hash of the value (supports JOINs)
    • Random Hash — uses per-query random salt for stronger security (supports JOINs within same query only)
    • Email mask — replaces username with XXXXX (e.g., XXXXX@gmail.com)
    • First four characters — shows only first 4 chars, rest replaced with XXXXX
    • Last four characters — shows only last 4 chars, rest replaced with XXXXX
    • Date year mask — truncates dates to year only (e.g., 2030-07-17 → 2030-01-01)
    • Custom masking routine — applies a user-defined function (UDF) for custom masking logic
  • Key Roles:
    • BigQuery Masked Reader — can see masked (obscured) column data
    • Data Catalog Fine-Grained Reader — can see unmasked (original) column data
    • Users with neither role get permission denied
  • Up to nine data policies can be configured per policy tag.

Differential Privacy

  • BigQuery supports differential privacy, an anonymization technique that limits the personal information revealed by query outputs.
  • Allows statistical queries on datasets while preventing identification of individual records.
  • Uses a privacy budget (epsilon) to limit how much information can be extracted from the data.
  • Supports differential privacy analysis rules that can be applied to shared datasets, particularly in data clean rooms.
  • Can be extended to multi-cloud data sources and external differential privacy libraries.

VPC Service Controls for BigQuery

  • VPC Service Controls create a security perimeter around BigQuery resources to prevent data exfiltration.
  • Controls data export from BigQuery to Cloud Storage or other targets.
  • Prevents data from being copied to unauthorized resources outside the perimeter using service operations.
  • Restricts resource access to allowed IP addresses, identities, and trusted client devices.
  • BigQuery supports regional endpoints to ensure data stays within a specific region.

Data Governance with Dataplex Universal Catalog

  • BigQuery’s data governance capabilities are powered by Dataplex Universal Catalog (also known as Knowledge Catalog), which replaced Data Catalog (deprecated Feb 2025, discontinued Jan 30, 2026).
  • Provides unified, AI-powered data cataloging integrating discovery, security, and metastore capabilities.
  • Policy tags and policy tag taxonomies used for column-level access control in BigQuery are NOT deprecated — they continue to work with Knowledge Catalog.
  • Supports column-level lineage for BigQuery data at no extra cost.
  • Sensitive Data Protection (formerly Cloud DLP) integrates with Dataplex for automated discovery and classification of sensitive data across BigQuery datasets.

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. You have multiple Data Analysts who work with the dataset hosted in BigQuery within the same project. As a BigQuery Administrator, you are required to grant the data analyst only the privilege to create jobs/queries and the ability to cancel self-submitted jobs. Which role should assign to the user?
    1. User
    2. Jobuser
    3. Owner
    4. Viewer
  2. Your analytics system executes queries against a BigQuery dataset. The SQL query is executed in batch and passes the contents of a SQL file to the BigQuery CLI. Then it redirects the BigQuery CLI output to another process. However, you are getting a permission error from the BigQuery CLI when the queries are executed. You want to resolve the issue. What should you do?
    1. Grant the service account BigQuery Data Viewer and BigQuery Job User roles.
    2. Grant the service account BigQuery Data Editor and BigQuery Data Viewer roles.
    3. Create a view in BigQuery from the SQL query and SELECT * from the view in the CLI.
    4. Create a new dataset in BigQuery, and copy the source table to the new dataset. Query the new dataset and table from the CLI.
  3. You are responsible for the security and access control to a BigQuery dataset hosted within a project. Multiple users from multiple teams need to have access to the different tables within the dataset. How can access be controlled?
    1. Create Authorized views for tables in a separate project and grant access to the teams
    2. Create Authorized views for tables in the same project and grant access to the teams
    3. Create Materialized views for tables in a separate project and grant access to the teams
    4. Create Materialized views for tables in the same project and grant access to the teams
  4. Your organization stores sensitive PII data in BigQuery. The security team wants different user groups to see different levels of data: the accounting team should see full SSN values, the analytics team should see hashed values for joining purposes, and general users should not see the data at all. What BigQuery feature should you use?
    1. Row-level security policies
    2. Authorized views with different SQL queries per team
    3. Dynamic data masking with policy tags and multiple data policies
    4. Column-level encryption with CMEK
  5. You need to prevent BigQuery data from being copied to unauthorized projects outside your organization while still allowing legitimate cross-project queries within your organization. What should you implement?
    1. Row-level access policies
    2. Column-level security with policy tags
    3. Authorized datasets
    4. VPC Service Controls with a service perimeter
  6. Your company wants to share a BigQuery dataset with a partner organization for analytics while ensuring that individual records cannot be identified. The privacy team requires mathematical guarantees of privacy protection. Which feature should you use?
    1. Dynamic data masking with SHA-256 hash
    2. Column-level security with authorized views
    3. Differential privacy with analysis rules in a data clean room
    4. Row-level security to filter out sensitive records

References

Google Cloud Dataproc – Managed Spark & Hadoop

~ Last updated on : ~ jayendrapatil

🔄 SERVICE REBRANDED — Now “Managed Service for Apache Spark”

Google Cloud Dataproc has been renamed to “Managed Service for Apache Spark” (2026).

The rebrand unifies the former “Dataproc on Compute Engine” (cluster deployment) and “Google Cloud Serverless for Apache Spark” (serverless deployment) under a single umbrella. The gcloud dataproc CLI commands and console URLs remain functional, but documentation now uses the new name.

This post uses the original “Dataproc” name for continuity, as the service functionality remains identical.

Google Cloud Dataproc (Managed Service for Apache Spark)

  • Cloud Dataproc is a managed Spark and Hadoop service that lets you take advantage of open-source data tools for batch processing, querying, streaming, and machine learning.
  • Dataproc automation helps to create clusters quickly, manage them easily, and save money by turning clusters on and off as needed.
  • Dataproc helps reduce time and money spent on administration and lets you focus on your jobs and your data.
  • Dataproc clusters are quick to start, scale, and shutdown, with each of these operations taking 90 seconds or less, on average.
  • Dataproc has built-in integration with other GCP services, such as BigQuery, Cloud Storage, Bigtable, Cloud Logging, and Monitoring.
  • Dataproc clusters support Spot VMs (previously called preemptible instances) that have lower compute prices to reduce costs further.
  • Dataproc supports connectors for BigQuery, Bigtable, Cloud Storage, and Cloud Spanner.
  • Dataproc supports Anaconda, HBase, Flink, Hive WebHCat, Druid, Jupyter, Presto, Trino, Solr, Zeppelin, Ranger, Zookeeper, Delta Lake, Iceberg, Hudi, and much more as optional components.
  • Dataproc offers two deployment modes:
    • Cluster Deployment (Dataproc on Compute Engine) — Spark-clusters-as-a-service; you manage infrastructure configuration and pay for cluster uptime.
    • Serverless Deployment (Serverless for Apache Spark) — Spark-jobs-as-a-service; fully managed Google Cloud infrastructure with pay-per-job-runtime billing.

Dataproc Serverless for Apache Spark

  • Dataproc Serverless lets you run Spark workloads without provisioning or managing a cluster.
  • Serverless supports two workload types:
    • Batch Workloads — Submit PySpark, Spark SQL, SparkR, or Spark (Java/Scala) batch jobs. Resources are auto-scaled and charges apply only during execution.
    • Interactive Sessions — Write and run code in Jupyter notebooks or BigQuery Studio notebooks via Spark Connect.
  • Serverless uses Dynamic Resource Allocation for autoscaling (not YARN-based).
  • Supports scheduling via Cloud Composer (Airflow) or Cloud Scheduler.
  • Offers Standard and Premium tiers:
    • Standard Tier — Core batch execution with autoscaling.
    • Premium Tier — Adds Lightning Engine, Native Query Execution, interactive sessions, and Gemini-powered autotuning.
  • Supports custom container images, GPUs, and VPC Service Controls.

Dataproc Lightning Engine

  • Lightning Engine is a next-generation performance layer that accelerates Spark workloads up to 4.9x faster than open-source Apache Spark with zero code changes.
  • Available for both cluster and serverless deployments.
  • Key components:
    • Native Query Execution (NQE) — A C++ vectorized execution engine built on Velox and Apache Gluten that bypasses JVM bottlenecks.
    • Intelligent Caching — Automatically caches frequently accessed data for faster reads.
    • Optimized Columnar Shuffling — Reduces shuffle overhead for large joins and aggregations.
  • Enabled by specifying --engine=lightning during cluster creation or selecting the Premium tier for serverless workloads.
  • Does not require any application code changes to existing Spark jobs.

Dataproc Cluster High Availability

  • Dataproc cluster can be configured for High Availability by specifying the number of master instances in the cluster.
  • Dataproc supports the following cluster configurations:
    • Single Node Cluster — 1 master, 0 Workers (default, non-HA)
      • Provides one node for both master and worker.
      • If the master fails, in-flight jobs will fail and need to be retried, and HDFS will be inaccessible until the single NameNode fully recovers on reboot.
    • Standard Cluster — 1 master, N Workers (default for multi-node)
      • Standard configuration with separate master and worker nodes.
    • High Availability Cluster — 3 masters, N Workers (Hadoop HA)
      • HDFS High Availability and YARN High Availability are configured to allow uninterrupted YARN and HDFS operations despite any single-node failures/reboots.
  • All nodes in a High Availability cluster reside in the same zone. If there is a failure that impacts all nodes in a zone, the failure will not be mitigated.

Dataproc Cluster Scaling

  • Dataproc cluster can be adjusted to scale by increasing or decreasing the number of primary or secondary worker nodes (horizontal scaling).
  • Dataproc cluster can be scaled at any time, even when jobs are running on the cluster.
  • Machine type of an existing cluster (vertical scaling) cannot be changed. To vertically scale, create a cluster using a supported machine type, then migrate jobs to the new cluster.
  • Dataproc cluster can help scale:
    • to increase the number of workers to make a job run faster
    • to decrease the number of workers to save money
    • to increase the number of nodes to expand available Hadoop Distributed Filesystem (HDFS) storage

Dataproc Cluster Autoscaling

  • Dataproc Autoscaling provides a mechanism for automating cluster resource management and enables cluster autoscaling.
  • An Autoscaling Policy is a reusable configuration that describes how clusters using the autoscaling policy should scale.
  • It defines scaling boundaries, frequency, and aggressiveness to provide fine-grained control over cluster resources throughout cluster lifetime.
  • Recent enhancements to Dataproc autoscaling have shown to decrease cluster VM expenditures by up to 40% and reduce cumulative job runtime by 10%.
  • Autoscaling is recommended for:
    • clusters that store data in external services, such as Cloud Storage
    • clusters that process many jobs
    • scaling up single-job clusters
  • Autoscaling is not recommended with/for:
    • HDFS: Autoscaling is not intended for scaling on-cluster HDFS.
    • YARN Node Labels: Autoscaling does not support YARN Node Labels. YARN incorrectly reports cluster metrics when node labels are used.
    • Spark Structured Streaming: Autoscaling does not support Spark Structured Streaming.
    • Idle Clusters: Autoscaling is not recommended for the purpose of scaling a cluster down to minimum size when the cluster is idle. Use Scheduled Stop or delete idle clusters instead.
  • Dataproc also supports Autotuning (Premium tier), which uses Gemini AI to automatically tune Spark properties, optimize memory allocation, and prevent OOM errors based on historical job patterns.

Dataproc Zero-Scale Clusters

  • Zero-scale clusters use only secondary workers (Spot VMs) that can be scaled down to zero when no processing is active.
  • Unlike standard clusters that require at least two primary workers, zero-scale clusters leave only the master node online to preserve metadata.
  • Ideal for development and testing environments where you want to eliminate idle compute costs.
  • Workers automatically scale up when jobs are submitted and scale back to zero when idle.

Dataproc Cluster Lifecycle Management

  • Scheduled Deletion — Automatically delete a cluster after a specified idle period, at a specified future time, or after a specified duration from creation.
  • Scheduled Stop — Automatically stop (not delete) a cluster after a specified idle period or at a future time. Preserves cluster configuration for easy restart.
  • Cluster Rotation — Recreate clusters at regular intervals for security patching and freshness.
  • Start/Stop — Manually stop and restart clusters to save costs without losing configuration.

Dataproc Workers

  • Primary workers are standard Compute Engine VMs.
  • Secondary workers can be used to scale compute with the below characteristics:
    • Processing only
      • Secondary workers do not store data.
      • Can only function as processing nodes.
      • Useful to scale compute without scaling storage.
    • No secondary-worker-only clusters (except zero-scale clusters)
      • Standard clusters must have primary workers.
      • Dataproc adds two primary workers by default if none are specified.
    • VM Types for Secondary Workers
      • Spot VMs (recommended) — Latest version of preemptible VMs with no maximum runtime limit. Can be reclaimed at any time.
      • Preemptible VMs (legacy) — Limited to 24-hour runtime. Spot VMs are recommended instead.
      • Non-preemptible VMs — Standard pricing, not subject to reclamation.
    • Persistent disk size
      • Created, by default, with the smaller of 100GB or the primary worker boot disk size.
      • This disk space is used for local caching of data and is not available through HDFS.
    • Asynchronous Creation
      • Dataproc manages secondary workers using Managed Instance Groups (MIGs), which create VMs asynchronously as soon as they can be provisioned.
  • Flexible VMs (GA 2026) — Define up to ten ranked machine types for worker nodes. Dataproc dynamically scans the entire region to fulfill capacity requests, improving resilience against localized shortages.

Dataproc Driver Node Groups

  • Driver node groups provide dedicated nodes for running Spark drivers, separating them from executors running on worker nodes.
  • Recommended for shared clusters running many concurrent jobs to prevent driver resource contention.
  • Increase master node resources before using driver node groups to avoid limitations.

Dataproc Initialization Actions

  • Dataproc supports initialization actions in executables or scripts that will run on all nodes in the cluster immediately after the cluster is set up.
  • Initialization actions often set up job dependencies, such as installing Python packages, so that jobs can be submitted to the cluster without having to install dependencies when the jobs are run.

Dataproc Cloud Storage Connector

  • Dataproc Cloud Storage connector helps Dataproc use Google Cloud Storage as the persistent store instead of HDFS.
  • Cloud Storage connector helps separate the storage from the cluster lifecycle and allows the cluster to be shut down when not processing data.
  • Cloud Storage connector benefits:
    • Direct data access — Store the data in Cloud Storage and access it directly. You do not need to transfer it into HDFS first.
    • HDFS compatibility — Can easily access your data in Cloud Storage using the gs:// prefix instead of hdfs://.
    • Interoperability — Storing data in Cloud Storage enables seamless interoperability between Spark, Hadoop, and Google services.
    • Data accessibility — Data is accessible even after shutting down the cluster, unlike HDFS.
    • High data availability — Data stored in Cloud Storage is highly available and globally replicated without a loss of performance.
    • No storage management overhead — Unlike HDFS, Cloud Storage requires no routine maintenance, such as checking the file system, or upgrading or rolling back to a previous version of the file system.

Dataproc Open Table Format Support

  • Dataproc supports modern open table formats as optional cluster components:
    • Apache Iceberg — Supports creating and querying Iceberg tables with metadata in Dataproc Metastore or BigLake Metastore.
    • Delta Lake — Supports reading and writing Delta tables on Cloud Storage.
    • Apache Hudi — Supports Hudi’s Copy-on-Write and Merge-on-Read table types.
  • Integration with Google Cloud Lakehouse enables read/write interoperability between Managed Service for Apache Spark and BigQuery using a unified metadata layer.

Dataproc on GKE

  • Dataproc on GKE allows running Spark and other data processing workloads on a Google Kubernetes Engine (GKE) cluster.
  • Provides Kubernetes-native resource management, scaling, and multi-tenancy for Spark workloads.
  • Useful for organizations that have standardized on Kubernetes and want unified infrastructure management.
  • Supports custom container images and executor pod scheduling on specific node pools.

Cloud Dataproc vs Dataflow

Refer blog post @ Cloud Dataproc vs Dataflow

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your company is forecasting a sharp increase in the number and size of Apache Spark and Hadoop jobs being run on your local data center. You want to utilize the cloud to help you scale this upcoming demand with the least amount of operations work and code change. Which product should you use?
    1. Google Cloud Dataflow
    2. Google Cloud Dataproc
    3. Google Compute Engine
    4. Google Kubernetes Engine
  2. Your company is migrating to the Google cloud and looking for HBase alternative. Current solution uses a lot of custom code using the observer coprocessor. You are required to find the best alternative for migration while using managed services, if possible?
    1. Dataflow
    2. HBase on Dataproc
    3. Bigtable
    4. BigQuery
  3. A data engineering team runs hundreds of short-lived Spark ETL jobs daily. They want to minimize infrastructure management and only pay for actual job execution time. Which deployment option is most appropriate?
    1. Dataproc cluster with autoscaling
    2. Dataproc Serverless (Managed Service for Apache Spark — Serverless)
    3. Dataproc on GKE
    4. Dataproc with scheduled deletion
  4. Your organization wants to accelerate existing Spark SQL workloads on Dataproc by up to 4.9x without modifying application code. Which feature should you enable?
    1. Dataproc Autoscaling
    2. Dataproc Enhanced Flexibility Mode
    3. Lightning Engine
    4. Dataproc Premium Machine Types
  5. A team wants a persistent Dataproc development environment that automatically stops incurring worker costs when no jobs are running, while preserving cluster metadata. Which feature should they use?
    1. Scheduled Deletion
    2. Standard Autoscaling
    3. Single Node Cluster
    4. Zero-Scale Cluster
  6. Your Dataproc cluster frequently fails to scale due to temporary capacity constraints in the selected zone. What feature would improve resource obtainability? (Choose TWO)
    1. Flexible VMs with ranked machine type preferences
    2. Auto Zone Placement
    3. Increasing the autoscaling cooldown period
    4. Using only preemptible VMs
  7. Which of the following are NOT recommended use cases for Dataproc Autoscaling? (Choose TWO)
    1. Clusters that store data in Cloud Storage
    2. Clusters running Spark Structured Streaming
    3. Scaling on-cluster HDFS storage
    4. Clusters processing many batch jobs

See also: Google Cloud Data Services Cheat Sheet

References

Google Cloud Dataflow – Stream & Batch Processing

~ Last updated on : ~ jayendrapatil

Google Cloud Dataflow

  • Google Cloud Dataflow is a fully managed, serverless service for unified stream and batch data processing at enterprise scale.
  • Dataflow provides Horizontal autoscaling to automatically choose the appropriate number of worker instances required to run the job.
  • Dataflow is based on Apache Beam, an open-source, unified model for defining both batch and streaming-data parallel-processing pipelines.
  • Dataflow scales to 4,000 workers per job and routinely processes petabytes of data.
  • Dataflow provides exactly-once processing by default for streaming pipelines, with an at-least-once mode available for lower latency when duplicates are tolerable.
  • Dataflow supports Java, Python, and Go SDKs, as well as multi-language pipelines.

Dataflow Prime

  • Dataflow Prime is an enhanced execution mode that provides advanced autoscaling and resource optimization features.
  • Starting August 4, 2025, Google-managed template jobs run on Dataflow Prime by default.
  • Vertical Autoscaling – Automatically scales up or down the memory available to workers to fit the requirements of the job, preventing out-of-memory (OOM) errors and maximizing pipeline efficiency.
  • Right Fitting – Uses Apache Beam resource hints to customize worker resources for specific pipeline steps, providing flexibility and potential cost savings.
  • Dynamic Thread Scaling – Adjusts the number of parallel tasks (bundles) each worker runs, complementing horizontal autoscaling.
  • Vertical Autoscaling works alongside Horizontal Autoscaling to scale resources dynamically for both batch and streaming pipelines.

Dataflow (Apache Beam) Programming Model

Data Processing Model

Pipelines

  • A pipeline encapsulates the entire series of computations involved in reading input data, transforming that data, and writing output data.
  • The input source and output sink can be the same or of different types, allowing data conversion from one format to another.
  • Apache Beam programs start by constructing a Pipeline object and then using that object as the basis for creating the pipeline’s datasets.
  • Each pipeline represents a single, repeatable job.

PCollection

  • A PCollection represents a potentially distributed, multi-element dataset that acts as the pipeline’s data.
  • Apache Beam transforms use PCollection objects as inputs and outputs for each step in your pipeline.

Transforms

  • A transform represents a processing operation that transforms data.
  • A transform takes one or more PCollections as input, performs a specified operation on each element in that collection, and produces one or more PCollections as output.
  • A transform can perform nearly any kind of processing operation like
    • performing mathematical computations,
    • data conversion from one format to another,
    • grouping data together,
    • reading and writing data,
    • filtering data to output only the required elements, or
    • combining data elements into single values.

ParDo

  • ParDo is the core parallel processing operation invoking a user-specified function on each of the elements of the input PCollection.
  • ParDo collects the zero or more output elements into an output PCollection.
  • ParDo processes elements independently and in parallel, if possible.

Pipeline I/O

  • Apache Beam I/O connectors help read data into the pipeline and write output data from the pipeline.
  • An I/O connector consists of a source and a sink.
  • All Apache Beam sources and sinks are transforms that let the pipeline work with data from several different data storage formats.
  • Managed I/O (introduced 2024) – Dataflow automatically upgrades managed I/O connectors in your pipeline, providing security fixes, performance improvements, and bug fixes without requiring code changes. Managed I/O also supports rolling upgrades for streaming jobs.

Aggregation

  • Aggregation is the process of computing some value from multiple input elements.
  • The primary computational pattern for aggregation is to
    • group all elements with a common key and window.
    • combine each group of elements using an associative and commutative operation.

User-defined functions (UDFs)

  • User-defined functions allow executing user-defined code as a way of configuring the transform.
  • For ParDo, user-defined code specifies the operation to apply to every element, and for Combine, it specifies how values should be combined.
  • A pipeline might contain UDFs written in a different language than the language of the runner.
  • A pipeline might also contain UDFs written in multiple languages.

Runner

  • Runners are the software that accepts a pipeline and executes it.
  • Dataflow Runner v2 is the current recommended runner for Dataflow jobs. It supports multi-language pipelines (Java transforms from Python and vice versa), custom containers, and is required for GPU/TPU workloads.

Event time

  • Time a data event occurs, determined by the timestamp on the data element itself.
  • This contrasts with the time the actual data element gets processed at any stage in the pipeline.

Windowing

  • Windowing enables grouping operations over unbounded collections by dividing the collection into windows of finite collections according to the timestamps of the individual elements.
  • A windowing function tells the runner how to assign elements to an initial window, and how to merge windows of grouped elements.

Tumbling Windows (Fixed Windows)

  • A tumbling window represents a consistent, disjoint time interval, for e.g. every 1 min, in the data stream.

An image that shows tumbling windows, 30 seconds in duration

Hopping Windows (Sliding Windows)

  • A hopping window represents a consistent time interval in the data stream for e.g., a hopping window can start every 30 seconds and capture 1 min of data and the window. The frequency with which hopping windows begin is called the period.
  • Hopping windows can overlap, whereas tumbling windows are disjoint.
  • Hopping windows are ideal to take running averages of data

An image that shows hopping windows with 1 minute window duration and 30 second window period

Session windows

  • A session window contains elements within a gap duration of another element for e.g., session windows can divide a data stream representing user mouse activity. This data stream might have long periods of idle time interspersed with many clicks. A session window can contain the data generated by the clicks.
  • The gap duration is an interval between new data in a data stream.
  • If data arrives after the gap duration, the data is assigned to a new window
  • Session windowing assigns different windows to each data key.
  • Tumbling and hopping windows contain all elements in the specified time interval, regardless of data keys.

An image that shows session windows with a minimum gap duration

Watermarks

  • A Watermark is a threshold that indicates when Dataflow expects all of the data in a window to have arrived.
  • Watermark is tracked and its a system’s notion of when all data in a certain window can be expected to have arrived in the pipeline
  • If new data arrives with a timestamp that’s in the window but older than the watermark, the data is considered late data.
  • Dataflow tracks watermarks because of the following:
    • Data is not guaranteed to arrive in time order or at predictable intervals.
    • Data events are not guaranteed to appear in pipelines in the same order that they were generated.

Trigger

  • Triggers determine when to emit aggregated results as data arrives.
  • For bounded data, results are emitted after all of the input has been processed.
  • For unbounded data, results are emitted when the watermark passes the end of the window, indicating that the system believes all input data for that window has been processed.

Dataflow Streaming Engine

  • Streaming Engine offloads the window state storage from worker VMs to a back-end service, enabling better autoscaling and reducing resource consumption on worker VMs.
  • Streaming Engine is recommended for all streaming jobs and is required for in-flight job updates.
  • Exactly-once processing – Default mode that ensures results are accurate with no duplicate records in the output.
  • At-least-once mode – Available for use cases that can tolerate duplicate records; reduces cost and improves latency by skipping deduplication overhead.
  • Streaming Engine provides responsive autoscaling that can be tuned using an autoscaling hint value (0.3 for minimal latency, 0.7 for minimal cost).

Dataflow Templates

  • Dataflow templates allow you to package a pipeline for deployment without requiring a development environment.
  • Templates separate pipeline design from deployment — a developer creates a template, and others can deploy it later.
  • Flex Templates (recommended for new templates)
    • Package the pipeline as a Docker image in Artifact Registry, along with a template specification file in Cloud Storage.
    • Support any source or sink I/O and allow parameterized customization.
    • Support launching from the Console, CLI, or REST API.
  • Classic Templates – Older template format; Flex Templates are recommended for new development.
  • Google provides pre-built templates for common scenarios (e.g., Pub/Sub to BigQuery, Cloud Storage to BigQuery).

Dataflow ML and AI Integration

  • Dataflow supports machine learning inference directly within pipelines using the RunInference transform from Apache Beam.
  • RunInference provides intelligent model memory management, automatic model refresh, batching for throughput optimization, and support for multiple ML frameworks (TensorFlow, PyTorch, scikit-learn, and custom models).
  • GPU Support – Dataflow supports GPUs (including NVIDIA RTX Pro 6000) for ML inference workloads. Requires Runner v2 and custom Docker containers.
  • TPU Support (launched 2025) – Supports TPU V5E, V5P, and V6E for high-volume, low-latency ML inference at scale. Features include TPU-aware autoscaling and duty-cycle policy enforcement.
  • MLTransform – Provides ready-to-use patterns for data preprocessing and feature engineering for ML pipelines.
  • Vertex AI Integration – Supports remote inference with Vertex AI models, Gemini models, and Gemma models.
  • Global Compute – Dynamically schedules workloads across Google’s global infrastructure, automatically determining optimal location based on data locality and resource availability.

Dataflow Pipeline Operations

  • Cancelling a job
    • causes the Dataflow service to stop the job immediately.
    • might lose in-flight data
  • Draining a job
    • supports graceful stop
    • prevents data loss
    • is useful to deploy incompatible changes
    • allows the job to clear the existing queue before stopping
    • supports only streaming jobs and does not support batch pipelines
  • Updating a streaming job
    • In-flight job update – For streaming jobs using Streaming Engine, you can update min/max worker counts without stopping the job or changing the job ID.
    • Job replacement – Submit updated pipeline code; Dataflow performs a compatibility check before swapping the job (new job ID, same job name).
  • Snapshots
    • Save the state of a streaming pipeline, allowing you to start a new version without losing state.
    • Useful for backup and recovery, testing, and rolling back updates to streaming pipelines.

Dataflow Security

  • Dataflow provides data-in-transit encryption.
    • All communication with Google Cloud sources and sinks is encrypted and is carried over HTTPS.
    • All inter-worker communication occurs over a private network and is subject to the project’s permissions and firewall rules.
  • Supports Customer-Managed Encryption Keys (CMEK) for data at rest.
  • Supports VPC Service Controls to restrict data exfiltration.
  • Supports Private Google Access and private IPs for workers to run without public IP addresses.

Dataflow SQL (Deprecated)

⚠️ Dataflow SQL has been deprecated.

  • As of July 31, 2024, Dataflow SQL is no longer accessible in the Google Cloud Console.
  • As of January 31, 2025, Dataflow SQL cannot be used in the Google Cloud CLI.

Alternatives: Use BigQuery for SQL-based analytics or Apache Beam SQL transform for SQL within pipelines.

Cloud Dataflow vs Dataproc

Refer blog post @ Cloud Dataflow vs Dataproc

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. A startup plans to use a data processing platform, which supports both batch and streaming applications. They would prefer to have a hands-off/serverless data processing platform to start with. Which GCP service is suited for them?
    1. Dataproc
    2. Dataprep
    3. Dataflow
    4. BigQuery
  2. A company is running a streaming Dataflow pipeline that processes real-time events. They need to minimize costs but can tolerate occasional duplicate records in the output. What should they do?
    1. Use Dataflow Prime with vertical autoscaling
    2. Enable at-least-once streaming mode
    3. Use Classic templates instead of Flex templates
    4. Disable Streaming Engine
  3. An ML team wants to run inference using a large language model directly within their Dataflow streaming pipeline. Which feature should they use?
    1. Cloud Functions trigger
    2. BigQuery ML
    3. RunInference transform with GPU/TPU support
    4. Vertex AI batch prediction
  4. A data engineer’s streaming Dataflow job is experiencing out-of-memory errors. They want an automated solution without manually tuning worker configurations. What should they use?
    1. Increase the machine type for all workers
    2. Set a higher max-num-workers value
    3. Enable Dataflow Prime with Vertical Autoscaling
    4. Use session windows to reduce data volume
  5. A company needs to update a running streaming pipeline with new transformation logic without losing the pipeline’s in-flight state. What approach should they use?
    1. Cancel the job and start a new one
    2. Drain the job and resubmit
    3. Use the pipeline update feature to replace the running job
    4. Use Dataflow snapshots and start a new job from the snapshot
  6. A team wants to deploy Dataflow pipelines without requiring a development environment on the deployment machine. What should they use?
    1. Cloud Composer DAGs
    2. Cloud Run Jobs
    3. Dataflow Flex Templates
    4. Cloud Build triggers
  7. Which of the following is NOT a valid window type in Apache Beam / Dataflow?
    1. Fixed (Tumbling) windows
    2. Sliding (Hopping) windows
    3. Session windows
    4. Rotating windows

See also: Google Cloud Data Services Cheat Sheet

References

Cloud Spanner Cheat Sheet – Features, Pricing & Use Cases

~ Last updated on : ~ jayendrapatil

Google Cloud Spanner

🆕 Major Updates (2024-2026)

  • Spanner Editions (2024): New tier-based pricing – Standard, Enterprise, and Enterprise Plus editions replacing legacy pricing.
  • Multi-Model Database (2024-2025): Spanner Graph, Full-Text Search, and Vector Search capabilities added.
  • Spanner Omni (2026 Preview): Run Spanner on-premises, across clouds, or on a laptop.
  • Columnar Engine (2025): Real-time analytics on operational data without impacting transactional workloads.
  • Managed Autoscaler (2025 GA): Automatic scaling with independent read-only replica scaling.
  • Tiered Storage (2025 GA): SSD and HDD storage tiers within same instance (~80% cost reduction for cold data).
  • Cloud Spanner is a fully managed, mission-critical relational database service
  • Cloud Spanner provides a scalable online transaction processing (OLTP) database with high availability and strong consistency at a global scale.
  • Cloud Spanner provides traditional relational semantics like schemas, ACID transactions and SQL interface
  • Cloud Spanner provides Automatic, Synchronous replication within and across regions for high availability (99.999%)
  • Spanner now handles over 6 billion queries per second at peak and more than 17 exabytes of data
  • Cloud Spanner benefits
    • OLTP (Online Transactional Processing)
    • Global scale
    • Relational data model
    • ACID/Strong or External consistency
    • Low latency
    • Fully managed and highly available
    • Automatic replication
    • Multi-model support – relational, graph, key-value, vector, and full-text search in a single database
    • AI-ready – built-in vector search, ML predictions, and Vertex AI integrations

Spanner Editions

  • Spanner offers editions, a tier-based pricing model introduced in 2024 that provides different capabilities at different price points.
  • Standard Edition
    • Comprehensive suite of established capabilities (all GA features prior to September 2024)
    • Regional instance configurations only
    • 99.99% availability SLA
    • Relational (GoogleSQL, PostgreSQL) and Key-value support
    • BigQuery federation, Data Boost, Reverse ETL
    • Standard backups, 7-day PITR, Scheduled backups
  • Enterprise Edition
    • Builds on Standard edition with multi-model capabilities
    • Regional instance configurations with optional custom read-only replicas
    • 99.99% availability SLA
    • Adds Spanner Graph, Full-text search, Vector Search (KNN & ANN)
    • Managed autoscaler, Tiered storage, Locality groups
    • Columnar engine for analytics
    • Incremental backups
  • Enterprise Plus Edition
    • Designed for most demanding workloads
    • Multi-region and dual-region instance configurations
    • Up to 99.999% availability SLA
    • All Enterprise features plus geo-partitioning
  • Committed Use Discounts (CUDs): 20% for 1-year, 40% for 3-year commitments across all editions
  • Free trial instance available (defaults to Enterprise edition on upgrade)

Cloud Spanner Architecture

Cloud Spanner ArchitectureInstance

  • Cloud Spanner Instance determines the location and the allocation of resources
  • Instance creation includes two important choices
    • Instance configuration
      • determines the geographic placement i.e. location and replication of the databases
      • Location can be regional, dual-region, or multi-regional
      • cannot be changed once selected during the creation (but instances can be moved using the Move Instance feature)
    • Compute capacity (nodes or processing units)
      • determines the amount of the instance’s serving and storage resources
      • measured in processing units (PUs) or nodes, with 1000 PUs = 1 node
      • minimum of 100 PUs for granular instances, scaling in batches of 100 PUs
      • can be updated manually or via managed autoscaler
  • Cloud Spanner distributes an instance across zones of one or more regions to provide high performance and high availability
  • Cloud Spanner instances have:
    • At least three read-write replicas of the database each in a different zone
    • Each zone is a separate isolation fault domain
    • Paxos distributed consensus protocol used for writes/transaction commits
    • Synchronous replication of writes to all zones across all regions
    • Database is available even if one zone fails (99.999% availability SLA for multi-region and 99.99% availability SLA for regional)

Regional vs Dual-Region vs Multi-Regional

  • Regional Configuration
    • Cloud Spanner maintains 3 read-write replicas, each within a different Google Cloud zone in that region.
    • Each read-write replica contains a full copy of the operational database that is able to serve read-write and read-only requests.
    • Cloud Spanner uses replicas in different zones so that if a single-zone failure occurs, the database remains available.
    • Every Cloud Spanner mutation requires a write quorum that’s composed of a majority of voting replicas. Write quorums are formed from two out of the three replicas in regional configurations.
    • Provides 99.99% availability
    • Available in Standard, Enterprise, and Enterprise Plus editions
  • Dual-Region Configuration (New – 2024)
    • Provides five 9s (99.999%) of availability with only two regions
    • Helps meet local residency requirements in geographies with only two regions
    • Zero recovery-point objective (RPO) guarantees
    • Available only in Enterprise Plus edition
  • Multi-Regional Configuration
    • Multi-region configurations allow replicating the database’s data not just in multiple zones, but in multiple zones across multiple regions
    • Additional replicas enable reading data with low latency from multiple locations close to or within the regions in the configuration.
    • As the quorum (read-write) replicas are spread across more than one region, additional network latency is incurred when these replicas communicate with each other to vote on writes.
    • Multi-region configurations enable the application to achieve faster reads in more places at the cost of a small increase in write latency.
    • Provides 99.999% availability
    • Multi-regional makes use of Paxos-based replication, TrueTime and leader election, to provide global consistency and higher availability
    • Available only in Enterprise Plus edition

Cloud Spanner - Regional vs Multi-Regional Configurations

Geo-Partitioning (New – 2024)

  • Partition table data at the row-level across the globe
  • Serves data closer to users for reduced application latency
  • Provides data residency benefits – store sensitive data within specific geographic jurisdictions
  • Optimizes costs by placing data in appropriate regions
  • Available only in Enterprise Plus edition

Replication

  • Cloud Spanner automatically gets replication at the byte level from the underlying distributed filesystem.
  • Cloud Spanner also performs data replication to provide global availability and geographic locality, with fail-over between replicas being transparent to the client.
  • Cloud Spanner creates multiple copies, or “replicas,” of the rows, then stores these replicas in different geographic areas.
  • Cloud Spanner uses a synchronous, Paxos distributed consensus protocol, in which voting replicas take a vote on every write request to ensure transactions are available in sufficient replicas before being committed.
  • Globally synchronous replication gives the ability to read the most up-to-date data from any Cloud Spanner read-write or read-only replica.
  • Cloud Spanner creates replicas of each database split
  • A split holds a range of contiguous rows, where the rows are ordered by the primary key.
  • All of the data in a split is physically stored together in the replica, and Cloud Spanner serves each replica out of an independent failure zone.
  • A set of splits is stored and replicated using Paxos.
  • Within each Paxos replica set, one replica is elected to act as the leader.
  • Leader replicas are responsible for handling writes, while any read-write or read-only replica can serve a read request without communicating with the leader (though if a strong read is requested, the leader will typically be consulted to ensure that the read-only replica has received all recent mutations)
  • Cloud Spanner automatically reshards data into splits and automatically migrates data across machines (even across datacenters) to balance load, and in response to failures.
  • Spanner’s sharding considers the parent child relationships in interleaved tables and related data is migrated together to preserve query performance
  • Read Leases (New – 2025): Read leases can improve read latency for strongly consistent data in multi-region configurations by trading off some write performance for common read-mostly workloads

Cloud Spanner Data Model

  • A Cloud Spanner Instance can contain one or more databases
  • A Cloud Spanner database can contain one or more tables
  • Tables look like relational database tables in that they are structured with rows, columns, and values, and they contain primary keys
  • Every table must have a primary key, and that primary key can be composed of zero or more columns of that table
  • Supports both GoogleSQL and PostgreSQL dialects
  • Parent-child relationships in Cloud Spanner
    • Table Interleaving
      • Table interleaving is a good choice for many parent-child relationships where the child table’s primary key includes the parent table’s primary key columns
      • Child rows are colocated with the parent rows significantly improving the performance
      • Primary key column(s) of the parent table must be the prefix of the primary key of the child table
    • Foreign Keys
      • Foreign keys are similar to traditional databases.
      • They are not limited to primary key columns, and tables can have multiple foreign key relationships, both as a parent in some relationships and a child in others.
      • The foreign key relationship does not guarantee data co-location
  • Cloud Spanner automatically creates an index for each table’s primary key
  • Secondary indexes can be created for other columns
  • Named Schemas (New – 2025): Organize and encapsulate database objects using named schema support

Multi-Model Capabilities (New – 2024/2025)

  • Spanner evolved into a multi-model database supporting multiple data models within a single instance without data movement
  • Spanner Graph
    • Native graph database capabilities using GQL (Graph Query Language)
    • Supports schemaless data for iterative development
    • Build graphs on named schema objects and SQL views
    • Graph algorithms support for connected data analysis
    • Interoperable with SQL – combine graph and relational queries
    • Available in Enterprise and Enterprise Plus editions
  • Full-Text Search
    • Built-in full-text search with tokenization and search indexes
    • Enhanced query mode with automatic synonym matching and spell correction
    • JSON indexing for accelerated queries over JSON data
    • Fuzzy search, faceted search, and substring search support
    • Available in Enterprise and Enterprise Plus editions
  • Vector Search
    • Exact K-Nearest Neighbors (KNN) search
    • Approximate Nearest Neighbors (ANN) search using Google’s ScaNN algorithm (GA in 2025)
    • Scales to more than 10 billion vectors
    • Combine vector searches with SQL and graph GQL queries for RAG applications
    • Available in Enterprise and Enterprise Plus editions
  • Key-Value (Cassandra Interface)
    • Cassandra interface allows using familiar CQL tools and syntax
    • Lift and shift Cassandra applications with virtually no changes
    • Available in all editions

Spanner AI Integration (New – 2024/2025)

  • Vector Embeddings: Generate and store vector embeddings directly in Spanner with Vertex AI integration
  • ML.PREDICT: Natural language queries using ML predictions via SQL
  • MCP Toolbox for Databases: Build agentic AI applications with Spanner
  • Agent Development Kit (ADK): Integration with Google’s ADK for Spanner
  • Vertex RAG Engine: Use Spanner as a RAG-managed database for data indexing and retrieval
  • LangChain & LlamaIndex: Framework integrations for LLM-powered applications
  • Conversational Data Agents: Build data agents with conversational analytics
  • Key AI use cases: product search and recommendations, fraud detection, identity resolution, autonomous network operations

Cloud Spanner Scaling

  • Increase the compute capacity of the instance to scale up the server and storage resources in the instance.
  • Each node (1000 PUs) provides up to 4TB of data storage (increased from 2TB in 2022)
  • Nodes provide additional compute resources to increase throughput
  • Increasing compute capacity does not increase the replica count but gives each replica more CPU and RAM, which increases the replica’s throughput (that is, more reads and writes per second can occur).
  • Start with as little as 100 processing units for granular instances (as low as $65/month)
  • Managed Autoscaler (GA – 2025)
    • Automatically scales compute capacity based on workload levels
    • Define minimum and maximum compute capacity limits
    • Supports independent scaling of read-only replicas from read-write replicas
    • Available in Enterprise and Enterprise Plus editions
  • Manual Split Points (New – 2025): Pre-split data to handle anticipated traffic spikes (e.g., flash sales, game launches)
  • Tiered Storage (GA – 2025)
    • Store data across SSD and HDD within the same instance
    • Older data automatically moves to lower cost HDD storage (~80% cheaper)
    • No API changes required to access data on different tiers
    • Available in Enterprise and Enterprise Plus editions

Spanner Analytics & BigQuery Integration (New – 2024/2025)

  • Columnar Engine (Preview – 2025)
    • Analyze vast amounts of operational data in real-time
    • Maintains global consistency, high availability, and strong transactional guarantees
    • Does not impact transactional workloads
    • Available in Enterprise and Enterprise Plus editions
  • BigQuery Integration
    • External Datasets (GA): Query live Spanner data directly in BigQuery with zero ETL
    • Materialized Views: Ultra-fast reporting using pre-computed query results on Spanner data
    • Reverse ETL: Export/stream computed insights from BigQuery to Spanner
    • Apache Iceberg Support: Join live Spanner data with Iceberg tables in BigQuery
    • Data Boost: Independent compute resources for analytics queries without impacting operational workloads

Cloud Spanner Backup & PITR

  • Cloud Spanner Backup and Restore helps create backups of Cloud Spanner databases on demand, and restore them to provide protection against operator and application errors that result in logical data corruption.
  • Backups are highly available, encrypted, and can be retained for up to a year from the time they are created.
  • Cloud Spanner point-in-time recovery (PITR) provides protection against accidental deletion or writes.
  • PITR works by letting you configure a database’s version_retention_period to retain all versions of data and schema, from a minimum of 1 hour up to a maximum of 7 days.
  • Scheduled Backups (New – 2024): Configure automatic backup schedules for databases
  • Incremental Backups (New – 2024): Only back up changes since last backup, reducing cost and time (Enterprise/Enterprise Plus editions)
  • Default Backup Schedules (New – 2025): Automatic recovery baseline from the moment a database is created
  • Schema Object Drop Protection (New – 2025): Safeguard against accidental deletion of critical tables, indexes, and columns

Performance & Isolation (New – 2025)

  • Repeatable Read Isolation (Preview): Reduces locking overhead for workloads with low read-write contention
  • Query Optimizer v8: Automated enhancements optimizing join strategies and index usage
  • Index Advisor: Analyzes query patterns and proactively suggests new indexes or identifies unused ones
  • Schema Recommendations (Preview): Scans schema design for anti-patterns like hotspot-prone primary keys
  • Leader-Aware Routing: Routes requests to the nearest leader replica for reduced latency
  • Multiplexed Sessions: Enabled by default in major SDKs for improved throughput and resource utilization
  • Spanner CLI: Bundled with gcloud for running SQL, managing sessions, and automating scripts

Spanner Omni (Preview – 2026)

  • Downloadable version of Spanner that runs in your own environment
  • Deploy on-premises data centers, across public clouds, or on a developer laptop
  • Brings Spanner’s unlimited scalability, high availability, strong consistency, and security anywhere
  • Supports connected, hybrid, multicloud, multiregion, and air-gapped configurations
  • Scales from a single machine to clusters of thousands of servers
  • Full multi-model capabilities available in Spanner Omni

Cloud Spanner Best Practices

  • Design a schema that prevents hotspots and other performance issues.
  • For optimal write latency, place compute resources for write-heavy workloads within or close to the default leader region.
  • For optimal read performance outside of the default leader region, use staleness of at least 15 seconds.
  • To avoid single-region dependency for the workloads, place critical compute resources in at least two regions.
  • Provision enough compute capacity to keep high priority total CPU utilization under
    • 65% in each region for regional configuration
    • 45% in each region for multi-regional configuration
  • Use the managed autoscaler to automatically adjust capacity based on workload patterns
  • Use tiered storage for cost optimization of historical/cold data
  • Use manual split points to pre-split data before anticipated traffic spikes
  • Leverage the Index Advisor and Schema Recommendations for performance optimization

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your customer has implemented a solution that uses Cloud Spanner and notices some read latency-related performance issues on one table. This table is accessed only by their users using a primary key. The table schema is shown below. You want to resolve the issue. What should you do?
    1. Remove the profile_picture field from the table.
    2. Add a secondary index on the person_id column.
    3. Change the primary key to not have monotonically increasing values.
    4. Create a secondary index using the following Data Definition Language (DDL) CREATE INDEX person_id_ix ON Persons (
      person_id, firstname, lastname ) STORING ( profile_picture )
  2. You are building an application that stores relational data from users. Users across the globe will use this application. Your CTO is concerned about the scaling requirements because the size of the user base is unknown. You need to implement a database solution that can scale with your user growth with minimum configuration changes. Which storage solution should you use?
    1. Cloud SQL
    2. Cloud Spanner
    3. Cloud Firestore
    4. Cloud Datastore
  3. A financial organization wishes to develop a global application to store transactions happening from different part of the world. The storage system must provide low latency transaction support and horizontal scaling. Which GCP service is appropriate for this use case?
    1. Bigtable
    2. Datastore
    3. Cloud Storage
    4. Cloud Spanner
  4. A company needs to run similarity searches on product embeddings for their recommendation engine while maintaining strong transactional consistency across multiple regions. Which Spanner capability should they use?
    1. Full-text search with search indexes
    2. Vector search with ANN using ScaNN algorithm
    3. Spanner Graph with GQL queries
    4. BigQuery federation with Data Boost
  5. Your organization wants to use Cloud Spanner with 99.999% availability SLA and geo-partitioning to meet data residency requirements. Which Spanner edition should you select?
    1. Standard edition
    2. Enterprise edition
    3. Enterprise Plus edition
    4. Any edition with multi-region configuration
  6. You need to migrate an existing Apache Cassandra application to Cloud Spanner with minimal application code changes. What should you use?
    1. Spanner Migration Tool (SMT)
    2. Dataflow with Spanner connector
    3. Spanner Cassandra interface with CQL compatibility
    4. Manual schema translation and data import
  7. Your team wants to analyze real-time operational data in Spanner without impacting transactional workloads. Which feature should you enable?
    1. Data Boost
    2. BigQuery external datasets
    3. Spanner columnar engine
    4. Change streams with Dataflow
  8. A retail company expects a flash sale that will cause a 10x traffic spike on their Spanner database. What feature can help prepare for this anticipated load?
    1. Managed autoscaler with higher maximum limits
    2. Manual split points to pre-split data for anticipated traffic
    3. Adding more processing units ahead of time
    4. Enabling read leases for all tables

References

Google Cloud Data Analytics Services Cheat Sheet

~ Last updated on : ~ jayendrapatil

Cloud Pub/Sub

  • Pub/Sub is a fully managed, asynchronous messaging service designed to be highly reliable and scalable with latencies on the order of 100 ms
  • Pub/Sub offers at-least-once message delivery and best-effort ordering to existing subscribers
  • Pub/Sub also supports exactly-once delivery (GA since 2022) for pull subscriptions and StreamingPull API, ensuring messages are not redelivered after successful acknowledgment. Push and export subscriptions do not support exactly-once delivery.
  • Pub/Sub enables the creation of event producers and consumers, called publishers and subscribers.
  • Pub/Sub messages should be no greater than 10MB in size.
  • Messages can be received with pull or push delivery.
  • Messages published before a subscription is created will not be delivered to that subscription
  • Acknowledged messages are no longer available to subscribers and are deleted, by default. However, can be retained setting retention period.
  • Publishers can send messages with an ordering key and message ordering is set, Pub/Sub delivers the messages in order.
  • Pub/Sub support encryption at rest and encryption in transit.
  • Seek feature allows subscribers to alter the acknowledgment state of messages in bulk to replay or purge messages in bulk.
  • Supports BigQuery subscriptions to write messages directly to BigQuery tables without additional processing.
  • Supports Cloud Storage subscriptions to write messages to Cloud Storage buckets in Avro or Text format.
  • Message filtering allows subscribers to receive a subset of messages published to a topic using filter expressions.
  • Pub/Sub Lite is deprecated (EOL March 18, 2026). Migrate to standard Pub/Sub for cost-effective messaging.

BigQuery

  • BigQuery is a fully managed, durable, petabyte scale, serverless, highly scalable, and cost-effective multi-cloud data warehouse that has evolved into an AI data platform.
  • supports a standard SQL dialect. Legacy SQL is deprecated — effective June 1, 2026, BigQuery limits legacy SQL use for organizations that have not used it between Nov 2025–Jun 2026.
  • automatically replicates data and keeps a seven-day history of changes (time travel), allowing easy restoration and comparison of data from different times
  • supports federated data and can process external data sources in GCS for Parquet and ORC open-source file formats, transactional databases (Bigtable, Cloud SQL), or spreadsheets in Drive without moving the data.
  • BigLake provides a unified storage engine for data lakehouse workloads, supporting Apache Iceberg tables with fine-grained governance across multiple engines (Spark, Flink, Trino, BigQuery).
  • Data model consists of Datasets, tables
  • BigQuery performance can be improved using Partitioned tables and Clustered tables.
  • BigQuery encrypts all data at rest and supports encryption in transit.
  • BigQuery Data Transfer Service automates data movement into BigQuery on a scheduled, managed basis
  • BigQuery Editions (Standard, Enterprise, Enterprise Plus) provide different capability tiers with slot-based pricing, autoscaling reservations, and capacity commitments.
  • BigQuery Studio provides a unified workspace with SQL editor, notebooks (Colab Enterprise), and data canvas for end-to-end analytics workflows.
  • BigQuery ML (BQML) allows building and deploying ML models using SQL, including integration with Gemini and Vertex AI for generative AI tasks like text summarization, sentiment analysis, and embeddings.
  • Vector Search enables similarity search using embeddings directly in BigQuery, supporting RAG applications, semantic search, and KNN-based retrieval without needing external vector databases.
  • BI Engine provides in-memory caching and vectorized processing for sub-second query response times, accelerating dashboards and visualization tools.
  • Best Practices
    • Control projection, avoid select *
    • Estimate costs as queries are billed according to the number of bytes read and the cost can be estimated using --dry-run feature
    • Use the maximum bytes billed setting to limit query costs.
    • Use clustering and partitioning to reduce the amount of data scanned.
    • Avoid repeatedly transforming data via SQL queries. Materialize the query results in stages.
    • Use streaming inserts only if the data must be immediately available as streaming data is charged.
    • Prune partitioned queries, use the _PARTITIONTIME pseudo column to filter the partitions.
    • Denormalize data whenever possible using nested and repeated fields.
    • Avoid external data sources, if query performance is a top priority
    • Avoid using Javascript user-defined functions
    • Optimize Join patterns. Start with the largest table.
    • Use the expiration settings to remove unneeded tables and partitions
    • Keep the data in BigQuery to take advantage of the long-term storage cost benefits rather than exporting to other storage options.
    • Use BigQuery editions with autoscaling for predictable costs and optimal performance.

Bigtable

  • Bigtable is a fully managed, scalable, wide-column NoSQL database service with up to 99.999% availability.
  • ideal for applications that need very high throughput and scalability for key/value data, where each value is max. of 10 MB.
  • supports high read and write throughput at low latency and provides consistent sub-10ms latency – handles millions of requests/second
  • is a sparsely populated table that can scale to billions of rows and thousands of columns,
  • supports storage of terabytes or even petabytes of data
  • is not a relational database. It does not support joins or multi-row transactions.
  • Now supports GoogleSQL for querying data, including window functions for advanced analytic operations (GA 2026).
  • handles upgrades and restarts transparently, and it automatically maintains high data durability.
  • scales linearly in direct proportion to the number of nodes in the cluster
  • stores data in tables, which is composed of rows, each of which typically describes a single entity, and columns, which contain individual values for each row.
  • Each table has only one index, the row key. There are no secondary indices. Each row key must be unique.
  • Single-cluster Bigtable instances provide strong consistency.
  • Multi-cluster instances, by default, provide eventual consistency but can be configured to provide read-over-write consistency or strong consistency, depending on the workload and app profile settings
  • Bigtable Editions (Enterprise and Enterprise Plus) provide advanced features for performance, analytic query capabilities, and resource management (GA 2026).
  • Data Boost provides serverless compute for analytical queries without impacting operational workloads, eliminating the need for multiple data copies.
  • In-memory tier delivers hotspot resistance supporting up to 120,000 queries per second on a single row for ultra-low latency use cases.

Cloud Dataflow

  • Cloud Dataflow is a managed, serverless service for unified stream and batch data processing requirements
  • provides Horizontal autoscaling to automatically choose the appropriate number of worker instances required to run the job.
  • is based on Apache Beam, an open-source, unified model for defining both batch and streaming-data parallel-processing pipelines.
  • supports Windowing which enables grouping operations over unbounded collections by dividing the collection into windows of finite collections according to the timestamps of the individual elements.
  • supports drain feature to deploy incompatible updates
  • Runner v2 supports cross-language transforms, allowing use of Java transforms from Python pipelines and vice versa.
  • Dataflow Prime provides advanced features including Job Visualizer, Smart Recommendations, vertical autoscaling, and right-fitting for optimal resource utilization.
  • GPU and TPU support (TPU V5E, V5P, V6E) enables running high-volume, low-latency ML inference workloads directly within Dataflow jobs.
  • Global Compute enables enormous scaling by dynamically scheduling workloads across Google’s global infrastructure, automatically determining optimal locations based on data locality and resource availability.
  • Speculative Execution for batch pipelines mitigates the impact of slow-running tasks (stragglers) by launching redundant executions.
  • Scales to 4,000 workers per job and routinely processes petabytes of data.
  • Dataflow SQL was deprecated (July 31, 2024) and is no longer available in Google Cloud CLI as of January 31, 2025.

Managed Service for Apache Spark (formerly Cloud Dataproc)

  • Note: Cloud Dataproc has been rebranded to Managed Service for Apache Spark (2025), consolidating Dataproc on Compute Engine and Google Cloud Serverless for Apache Spark under a unified brand.
  • Managed Service for Apache Spark is a managed Spark and Hadoop service to take advantage of open-source data tools for batch processing, querying, streaming, and machine learning.
  • helps to create clusters quickly, manage them easily, and save money by turning clusters on and off as needed.
  • helps reduce time and money spent on administration and lets you focus on your jobs and your data.
  • has built-in integration with other GCP services, such as BigQuery, Cloud Storage, Bigtable, Cloud Logging, and Monitoring
  • support preemptible instances (now called Spot VMs) that have lower compute prices to reduce costs further.
  • also supports HBase, Flink, Hive WebHcat, Druid, Jupyter, Presto, Solr, Zeppelin, Ranger, Zookeeper, and much more.
  • supports connectors for BigQuery, Bigtable, Cloud Storage
  • can be configured for High Availability by specifying the number of master instances in the cluster
  • All nodes in a High Availability cluster reside in the same zone. If there is a failure that impacts all nodes in a zone, the failure will not be mitigated.
  • supports cluster scaling by increasing or decreasing the number of primary or secondary worker nodes (horizontal scaling)
  • supports Autoscaling that provides a mechanism for automating cluster resource management and enables cluster autoscaling.
  • supports initialization actions in executables or scripts that will run on all nodes in the cluster immediately after the cluster is set up
  • Serverless Spark allows submitting batch workloads without provisioning or managing clusters, with automatic scaling and resource management.
  • Supports Zero-Scale clusters for cost optimization when clusters are idle (2026).
  • Includes Lightning Engine for boosted Spark performance.

Cloud Dataprep

⚠️ SERVICE END OF SUPPORT

Cloud Dataprep by Trifacta reached End of Support in December 2025. Trifacta was acquired by Alteryx, and the service has transitioned to Alteryx Designer Cloud.

Migration Options:

  • Alteryx Designer Cloud — successor to Dataprep by Trifacta with enhanced AI/ML capabilities
  • Cloud Data Fusion — Google Cloud’s managed data integration service for visual ETL/ELT pipelines
  • Dataform — for SQL-based data transformations in BigQuery
  • BigQuery Data Preparations — built-in data preparation within BigQuery Studio
  • Cloud Dataprep by Trifacta was an intelligent data service for visually exploring, cleaning, and preparing structured and unstructured data for analysis, reporting, and machine learning.
  • was fully managed, serverless, and scaled on-demand with no infrastructure to deploy or manage
  • provided easy data preparation with clicks and no code.
  • automatically identified data anomalies & helped take fast corrective action
  • automatically detected schemas, data types, possible joins, and anomalies such as missing values, outliers, and duplicates
  • used Dataflow or BigQuery under the hood, enabling unstructured or structured datasets processing of any size with the ease of clicks, not code

Cloud Datalab

⚠️ SERVICE DEPRECATED

Cloud Datalab was deprecated on September 2, 2022.

Migration Options:

  • Vertex AI Workbench — managed notebook environment with JupyterLab, providing capabilities beyond Datalab with integrated ML workflows
  • Colab Enterprise — collaborative notebook environment integrated with BigQuery Studio
  • Cloud Datalab was a powerful interactive tool created to explore, analyze, transform and visualize data and build machine learning models using familiar languages, such as Python and SQL, interactively.
  • ran on Google Compute Engine and connected to multiple cloud services easily so you could focus on data science tasks.
  • was built on Jupyter (formerly IPython)
  • enabled analysis of the data on Google BigQuery, Cloud Machine Learning Engine, Google Compute Engine, and Google Cloud Storage using Python, SQL, and JavaScript (for BigQuery user-defined functions).

Dataplex / Knowledge Catalog

  • Dataplex (now Knowledge Catalog) is an AI-powered, unified data governance solution for managing, understanding, and governing data and AI assets across Google Cloud.
  • Provides centralized inventory to discover, manage, and govern data across BigQuery, Cloud Storage, Pub/Sub, and Spanner.
  • Supports data products — curated, ready-to-use packages of data assets, documentation, and governance controls assembled to solve specific business problems.
  • Offers automated data discovery, metadata management, data quality checks, data lineage, and semantic search.
  • Integrates with BigQuery Studio for unified governance across analytical workflows.

Related Posts

Google Cloud Firestore – Serverless Document Database

~ Last updated on : ~ jayendrapatil

Google Cloud Firestore

  • Google Cloud Firestore provides a fully managed, scalable, and serverless document database.
  • Firestore stores the data in the form of documents and collections
  • Firestore provides horizontal autoscaling, strong consistency with support for ACID transactions
  • Firestore database can be regional or multi-regional
  • Firestore multi-region instances provide five-nines (99.999%) availability SLA and regional instances with four-nines (99.99%) availability SLA
  • Firestore supports multiple databases per project, enabling isolation of production/testing environments, customer data separation, and data regionalization.

Firestore Editions (2024+)

  • Firestore is now available in two editions: Enterprise and Standard.
  • Enterprise Edition
    • Provides the most advanced Firestore capabilities with an advanced query engine featuring over 180 stages and operators
    • Supports aggregations, arithmetic, arrays, sets, type conversions, and joining data (relational-style joins through correlated subqueries)
    • Indexes are optional — queries can run with or without indexes
    • Supports advanced index types: unique, dense, and sparse indexes
    • Up to 5x improved performance over Standard edition, especially at tail latencies
    • Can handle bursty network traffic at a rate up to 8x higher than Standard edition
    • Supports Firestore with MongoDB compatibility API (16 MiB document size limit with MongoDB compatibility)
    • SSD-based storage
    • Text search and Geospatial search (Preview)
    • Pricing based on tranches of bytes read/written, storage consumed, and network egress
    • Committed Use Discounts: 20% for 1 year; 40% for 3 years
  • Standard Edition
    • Provides core Firestore capabilities with a standard query engine
    • All queries require covered indexes
    • Automatic, basic indexing on all document fields
    • 1 MiB document size limit
    • Hybrid storage (SSD & HDD)
    • Pricing based on documents read/written, storage consumed, and network egress
    • Committed Use Discounts: 20% for 1 year; 40% for 3 years
  • Both editions support real-time synchronization, offline queries, Firebase SDKs, CMEK, scheduled backups, PITR, and Cloud Monitoring
  • You can create both Enterprise and Standard edition databases in the same project
  • Data is compatible between editions; migration from Standard to Enterprise is supported

Data Model

  • Firestore is schemaless
  • Document & Collections
    • Unit of storage is the document in Firestore
    • Each document contains a set of key-value pairs
    • stores the data in documents organized into collections.
    • is optimized for storing large collections of small documents.
    • supports a variety of data types for values: boolean, number, string, geo point, binary blob, and timestamp.
    • Documents can contain subcollections, arrays, or nested objects, which can include primitive fields like strings or complex objects like lists.
    • Documents within a collection are unique and can be identified using your own keys, such as user IDs, or Firestore generated random IDs.
    • Document size limit is 1 MiB (Standard edition and Native mode) or 16 MiB (Enterprise edition with MongoDB compatibility)
  • Indexes
    • Firestore guarantees high query performance by using indexes for all queries (Standard edition requires indexes; Enterprise edition makes them optional).
    • Standard Edition supports two types of indexes:
      • Single-field
        • automatically maintains single-field indexes for each field in a document and each subfield in a map.
        • Single-field index exemption can be used to exempt a field from automatic indexing settings
        • Single-field index exemption for a map field is inherited by the map’s subfields
      • Composite
        • A composite index stores a sorted mapping of all the documents in a collection, based on an ordered list of fields to index.
        • does not automatically create composite indexes but helps identify fields based on the query pattern
    • Enterprise Edition supports fully customizable indexing with advanced index types including unique, dense, and sparse indexes

Query Engine & Pipeline Operations (Enterprise Edition)

  • Firestore Enterprise edition features an advanced query engine that introduces Pipeline operations.
  • Pipeline operations provide a new query interface with over 180 stages and operators.
  • Key capabilities include:
    • Aggregations (min, max, sum, avg, count, array_agg, first, last)
    • Arithmetic and type conversion operations
    • Array and set operations
    • String functions (substring, regex_match)
    • Relational-style joins through correlated subqueries
    • Explicit “stage” ordering for complex query composition
    • No index requirement — indexes are fully optional for pipeline queries
  • Pipeline operations are available only in Firestore Enterprise edition (Preview stage as of 2025).
  • Standard edition uses Core operations with basic comparisons and matches, requiring covered indexes.

Data Contention

  • Data Contention occurs when two or more operations compete to control the same document.
  • Mobile/Web SDKs
    • uses optimistic concurrency controls to resolve data contention
    • resolves data contention by delaying or failing one of the operations
    • client libraries automatically retry transactions that fail due to data contention. After a finite number of retries, the transaction operation fails and returns an error message
  • Server Client Libraries
    • use pessimistic concurrency controls to resolve data contention.
    • Pessimistic transactions use database locks to prevent other operations from modifying data.
    • Transactions place locks on the documents they read. A transaction’s lock on a document blocks other transactions, batched writes, and non-transactional writes from changing that document.
    • A transaction releases its document locks at commit time. It also releases its locks if it times out or fails for any reason.

Firestore Security

  • Firestore automatically encrypts all data before it is written to disk using Google-owned and Google-managed encryption keys.
  • Customer-Managed Encryption Keys (CMEK) — allows you to manage your own encryption keys via Cloud KMS for compliance and regulatory requirements. Available for both Enterprise and Standard editions.
  • Server-side encryption can be used in combination with client-side encryption, where data is encrypted by the client as well as server i.e double encryption
  • Firestore uses Transport Layer Security (TLS) to protect the data as it travels over the Internet during read and write operations.
  • Supports VPC Service Controls for network-level security isolation.
  • Firestore Security Rules — for mobile and web clients, provides declarative security rules to control access at the document and field level.

Vector Search

  • Firestore supports vector embeddings for performing K-Nearest Neighbor (KNN) similarity search directly on Firestore data.
  • Enables AI-powered experiences such as semantic search, recommendation engines, and RAG (Retrieval-Augmented Generation) pipelines.
  • Firestore does not generate embeddings — use services like Vertex AI to create vector values (e.g., text embeddings) and store them back in Firestore documents.
  • Supports multiple distance measures for similarity search.
  • Eliminates the need to copy data to a separate vector search solution, maintaining operational simplicity.

Firestore with MongoDB Compatibility

  • Available as part of Firestore Enterprise edition.
  • Provides a MongoDB-compatible API allowing use of existing MongoDB application code, drivers, tools, and the open-source MongoDB ecosystem integrations.
  • Key capabilities:
    • MongoDB wire-compatible API on Firestore’s serverless database service
    • Pay-per-use serverless pricing model with no up-front commitments
    • Document size limit increased to 16 MiB
    • Automatic scaling without capacity planning
    • Full-text search and expressive queries
  • Supports migration from MongoDB to Firestore using Datastream connection profiles.
  • Free tier: 50,000 reads, 40,000 writes, and 1 GB storage free per day.

Generative AI & Agentic AI Integration

  • Firestore provides native integrations for building AI-powered applications:
    • LangChain Integration — official LangChain packages for using Firestore as a Vector Store, Document Loader, Document Saver, and Chat Memory (available in Python, Go, Java, and JavaScript).
    • MCP (Model Context Protocol) Server — Firestore remote MCP server allows AI agents and tools (Gemini CLI, Claude, Cursor, VS Code Copilot, etc.) to interact with Firestore documents directly.
    • MCP Toolbox for Databases — open-source MCP server enabling gen AI agents to connect to enterprise data in Firestore.
    • Vector Search — enables RAG pipelines by storing and querying vector embeddings directly in Firestore.
  • Enables use cases: personalized recommendations, question answering, document search & synthesis, customer service automation, and AI chatbots.

Data Protection & Disaster Recovery

  • Scheduled Backups — create backup schedules to automatically protect data. Supported in both Enterprise and Standard editions.
  • Point-in-Time Recovery (PITR) — restore data to any point in time within the past 7 days. PITR data is retained for 7 days in the PITR window. Does not affect read/write performance.
  • In-Place Restore — perform restores directly on an existing database.
  • TTL (Time-to-Live) Policies — designate a field as the expiration time for documents to automatically clean up obsolete data. Data is typically deleted within 24 hours after expiration, helping reduce storage costs.

Event-Driven Architecture

  • Eventarc Integration (GA) — create event-driven architectures triggered by Firestore document changes.
    • Supports both Native mode and Datastore mode
    • Register multiple Cloud Functions in different regions against a multi-regional database for increased reliability
    • Auth Context extension for CloudEvents
    • Trigger types: document created, updated, deleted, written
  • BigQuery Integration — replicate Firestore data to BigQuery for analytics.
  • Dataflow Connector — process Firestore data in bulk with Apache Beam/Dataflow.

Firestore Native vs Datastore Mode

Firestore in Native mode

  • Strongly consistent storage layer
  • Collection and document data model
  • Real-time updates
  • Mobile and Web client libraries
  • Firestore is backward compatible with Datastore, but the new data model, real-time updates, and mobile and web client library features are not.
  • Native mode can automatically scale to millions of concurrent clients.
  • Native mode is recommended for Mobile and Web apps
  • Available in both Enterprise and Standard editions

Firestore in Datastore mode

  • Datastore mode is fully supported and recommended for applications with a dependency on the Datastore API.
  • Native mode and Datastore mode share an underlying storage layer with the same availability, consistency, and scaling capabilities.
  • Datastore mode uses Datastore system behavior but accesses Firestore’s storage layer, removing the following Datastore limitations:
    • No more eventual consistency. Is a strongly consistent database
    • No more entity group limits on writes per second. Writes to an entity group are no longer limited to 1 per second. Transactions are no longer limited to 25 entity groups.
    • Transactions can be as complex as you want to design them.
    • No more cross-entity group transaction limits. Transactions can span documents and be as complex as your app requires. Queries in transactions are no longer required to be ancestor queries.
  • Datastore mode disables Firestore features that are not compatible with Datastore:
    • accepts only Datastore API requests and denies Firestore API requests.
    • uses Datastore indexes instead of Firestore indexes.
    • do not support Firestore client libraries, but only Datastore client libraries
    • do not support Firestore real-time capabilities
  • Datastore mode can automatically scale to millions of writes per second.
  • Datastore mode is available only in the Standard edition.
  • Note: Google has transparently migrated all original Datastore databases (stored in Megastore) to Firestore databases (stored in Spanner) as the underlying storage layer.

Firestore Native Mode vs Datastore Mode

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your existing application keeps user state information in a single MySQL database. This state information is very user-specific and depends heavily on how long a user has been using an application. The MySQL database is causing challenges to maintain and enhance the schema for various users. Which storage option should you choose?
    1. Cloud SQL
    2. Cloud Storage
    3. Cloud Spanner
    4. Cloud Firestore
  2. A company needs to build a recommendation engine that performs similarity search on product embeddings stored alongside product documents. They want a serverless solution that avoids data synchronization between systems. Which Firestore feature should they use?
    1. Composite indexes with range filters
    2. Vector search with vector embeddings
    3. Pipeline operations with joins
    4. Full-text search with geospatial queries
  3. A team is migrating an existing MongoDB application to Google Cloud. They want to reuse their existing MongoDB drivers and application code without major rewrites while benefiting from serverless scaling. Which option should they choose?
    1. Firestore in Datastore mode
    2. Firestore Standard edition in Native mode
    3. Firestore Enterprise edition with MongoDB compatibility
    4. Cloud SQL for PostgreSQL with MongoDB-compatible extension
  4. An organization requires complex analytical queries with joins, aggregations, and arithmetic operations on their Firestore data without creating indexes. Which Firestore capability should they use?
    1. Core operations with composite indexes
    2. BigQuery integration for analytics
    3. Pipeline operations in Enterprise edition
    4. Dataflow connector with Apache Beam
  5. A company needs to automatically delete user session documents after 30 days to reduce storage costs. Which Firestore feature addresses this requirement?
    1. Scheduled Cloud Functions to delete expired documents
    2. Firestore Security Rules with time-based conditions
    3. TTL (Time-to-Live) policies
    4. Point-in-time recovery with data expiration

References

Google Cloud – EHR Healthcare Case Study

~ Last updated on : ~ jayendrapatil

Google Cloud – EHR Healthcare Case Study

EHR Healthcare is a leading provider of electronic health record software to the medical industry. EHR Healthcare provides its software as a service to multi-national medical offices, hospitals, and insurance providers.

📋 Exam Relevance (2025-2026): EHR Healthcare remains one of the four active case studies on the Google Cloud Professional Cloud Architect (PCA) exam. The case study focuses on healthcare workload migration, HIPAA compliance, hybrid connectivity, container orchestration, and centralized observability. Key service updates include GKE Autopilot (recommended for containerized workloads), Cloud Deployment Manager deprecation (March 2026, replaced by Terraform/Infrastructure Manager), and GKE Enterprise (formerly Anthos) for multi-cluster management.

Executive statement

Our on-premises strategy has worked for years but has required a major investment of time and money in training our team on distinctly different systems, managing similar but separate environments, and responding to outages. Many of these outages have been a result of misconfigured systems, inadequate capacity to manage spikes in traffic, and inconsistent monitoring practices. We want to use Google Cloud to leverage a scalable, resilient platform that can span multiple environments seamlessly and provide a consistent and stable user experience that positions us for future growth.

EHR Healthcare wants to move to Google Cloud to expand, build scalable and highly available applications. It also wants to leverage automation and IaaC to provide consistency across environments and reduce provisioning errors.

Solution Concept

Due to rapid changes in the healthcare and insurance industry, EHR Healthcare’s business has been growing exponentially year over year. They need to be able to scale their environment, adapt their disaster recovery plan, and roll out new continuous deployment capabilities to update their software at a fast pace. Google Cloud has been chosen to replace its current colocation facilities.

EHR wants to build a scalable, Highly Available, Disaster Recovery setup and introduce Continuous Integration and Deployment in their setup.

Existing Technical Environment

EHR’s software is currently hosted in multiple colocation facilities. The lease on one of the data centers is about to expire.
Customer-facing applications are web-based, and many have recently been containerized to run on a group of Kubernetes clusters. Data is stored in a mixture of relational and NoSQL databases (MySQL, MS SQL Server, Redis, and MongoDB).
EHR is hosting several legacy file- and API-based integrations with insurance providers on-premises. These systems are scheduled to be replaced over the next several years. There is no plan to upgrade or move these systems at the current time.
Users are managed via Microsoft Active Directory. Monitoring is currently being done via various open-source tools. Alerts are sent via email and are often ignored.

  • As the lease of one of the data centers is about to expire, time is critical
  • Some web applications are containerized and have SQL and NoSQL databases and can be moved
  • Some of the systems are legacy and would be replaced and need not be migrated
  • Team has multiple monitoring tools and might need consolidation

Business requirements

On-board new insurance providers as quickly as possible.

Provide a minimum 99.9% availability for all customer-facing systems.

  • Availability can be increased by hosting applications across multiple zones and using managed services which span multiple AZs
  • GKE regional clusters with Autopilot mode provide built-in high availability across zones with automated node management

Provide centralized visibility and proactive action on system performance and usage.

  • Google Cloud Observability (Cloud Monitoring + Cloud Logging + Cloud Trace) provides a unified platform for centralized visibility, alerting, and proactive action
  • Application Monitoring (launched 2025) automatically labels and correlates telemetry for registered applications
  • Cloud Logging can be used for log monitoring, analysis, and alerting

Increase ability to provide insights into healthcare trends.

  • Data can be pushed and analyzed using BigQuery (now a unified AI-ready data platform) and insights visualized using Data Studio
  • BigQuery ML enables building predictive ML models directly within BigQuery for healthcare trend analysis

Reduce latency to all customers.

  • Performance can be improved using Global Load Balancer to expose the applications
  • Applications can also be hosted across regions for low latency access
  • Cloud CDN or Media CDN can further reduce latency for static content delivery

Maintain regulatory compliance.

  • Regulatory compliance (HIPAA, HITRUST) can be maintained using data localization, data retention policies, as well as security measures
  • Assured Workloads helps configure compliant environments with guardrails
  • VPC Service Controls provides data exfiltration protection for sensitive healthcare data
  • Cloud Healthcare API provides HIPAA-compliant FHIR, HL7v2, and DICOM data management

Decrease infrastructure administration costs.

  • Infrastructure administration costs can be reduced using automation with Terraform or Infrastructure Manager (Google’s managed Terraform service)
  • Note: Cloud Deployment Manager reached End of Life on March 31, 2026. Use Terraform or Infrastructure Manager for all new IaC deployments

Make predictions and generate reports on industry trends based on provider data.

  • Data can be pushed and analyzed using BigQuery with built-in ML capabilities (BigQuery ML) for predictions
  • Vertex AI can be used for advanced ML model training on healthcare data

Technical requirements

Maintain legacy interfaces to insurance providers with connectivity to both on-premises systems and cloud providers.

  • Hybrid connectivity can be established using Cloud VPN, Dedicated Interconnect, or Partner Interconnect
  • Cross-Cloud Interconnect enables private connectivity to other cloud providers (AWS, Azure) if needed
  • Network Connectivity Center provides centralized hub-and-spoke network management for multi-cloud and hybrid environments

Provide a consistent way to manage customer-facing applications that are container-based.

  • Container-based applications can be deployed on GKE (Autopilot mode recommended for reduced operational overhead) or Cloud Run for serverless containers
  • GKE Enterprise (formerly Anthos) enables consistent multi-cluster management across environments with fleet-level policies
  • Cloud Deploy provides managed continuous delivery pipelines for GKE and Cloud Run

Provide a secure and high-performance connection between on-premises systems and Google Cloud.

  • Cloud VPN (HA VPN), Dedicated Interconnect, or Partner Interconnect connections can be established between on-premises and Google Cloud
  • Dedicated Interconnect provides 10/100 Gbps connections with 99.99% SLA when configured with recommended topology

Provide consistent logging, log retention, monitoring, and alerting capabilities.

  • Cloud Monitoring and Cloud Logging (part of Google Cloud Observability) provide a single pane of glass for monitoring, logging, and alerting
  • Managed Service for Prometheus enables Kubernetes-native monitoring with PromQL
  • OpenTelemetry Protocol (OTLP) support (2025) enables standardized telemetry collection

Maintain and manage multiple container-based environments.

  • Use Terraform or Infrastructure Manager to provide consistent implementations across environments
  • GKE Enterprise fleet management enables centralized policy and configuration across multiple GKE clusters
  • Cloud Service Mesh (replaced Anthos Service Mesh, GA June 2024) provides traffic management, security, and observability across microservices

Dynamically scale and provision new environments.

  • GKE Autopilot automatically provisions and scales nodes based on workload demands (recommended mode for new clusters)
  • GKE Standard clusters can use Cluster Autoscaler and HPA/VPA for deployments
  • Cloud Run provides automatic scaling from zero to thousands of instances

Create interfaces to ingest and process data from new providers.

  • Cloud Healthcare API provides FHIR R4, HL7v2, and DICOM interfaces for healthcare data ingestion and exchange
  • Pub/Sub enables real-time streaming data ingestion from multiple providers
  • Dataflow provides serverless stream and batch data processing pipelines

Key Google Cloud Services for EHR Healthcare

Requirement Google Cloud Service
Container Orchestration GKE (Autopilot mode), Cloud Run
Multi-cluster Management GKE Enterprise (fleet management)
Hybrid Connectivity Cloud VPN (HA), Dedicated/Partner Interconnect
Infrastructure as Code Terraform, Infrastructure Manager
Observability Cloud Monitoring, Cloud Logging, Cloud Trace
Data Analytics BigQuery, Data Studio, BigQuery ML
Healthcare Data Cloud Healthcare API (FHIR, HL7v2, DICOM)
Service Mesh Cloud Service Mesh
CI/CD Cloud Build, Cloud Deploy
Compliance Assured Workloads, VPC Service Controls
Relational Databases Cloud SQL (MySQL, SQL Server), AlloyDB
NoSQL Databases Memorystore (Redis), MongoDB Atlas on GCP
Identity Management Cloud Identity, Google Cloud Directory Sync (for AD)

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. For this question, refer to the EHR Healthcare case study. In the past, configuration errors put IP addresses on backend servers that should not have been accessible from the internet. You need to ensure that no one can put external IP addresses on backend Compute Engine instances and that external IP addresses can only be configured on the front end Compute Engine instances. What should you do?
    1. Create an organizational policy with a constraint to allow external IP addresses on the front end Compute Engine instances
    2. Revoke the compute.networkadmin role from all users in the project with front end instances
    3. Create an Identity and Access Management (IAM) policy that maps the IT staff to the compute.networkadmin role for the organization
    4. Create a custom Identity and Access Management (IAM) role named GCE_FRONTEND with the compute.addresses.create permission

    Explanation: The constraints/compute.vmExternalIpAccess organization policy constraint controls which VM instances can have external IP addresses. By configuring this constraint at the organization or project level and specifying only the front-end instances, you prevent backend instances from being assigned external IPs while allowing them on designated front-end instances.

  2. For this question, refer to the EHR Healthcare case study. EHR Healthcare needs to ensure consistent environments across development, staging, and production while reducing provisioning errors. Their infrastructure team currently uses manual processes. What should you recommend?
    1. Use Cloud Console to manually configure all environments and document the steps
    2. Create custom scripts using gcloud CLI to provision resources
    3. Use Terraform with Infrastructure Manager to define infrastructure as code and automate deployments across all environments
    4. Use Cloud Deployment Manager templates to provision all environments

    Explanation: Terraform with Infrastructure Manager (Google’s managed Terraform service) provides a declarative, version-controlled approach to infrastructure provisioning. Cloud Deployment Manager was deprecated and reached End of Life on March 31, 2026. Infrastructure Manager enables plan/apply workflows with built-in IAM, logging, and governance.

  3. For this question, refer to the EHR Healthcare case study. EHR Healthcare wants centralized visibility and proactive alerting for their containerized applications running on GKE. They want to consolidate their current fragmented monitoring tools. What should you recommend?
    1. Deploy Prometheus and Grafana on each GKE cluster independently
    2. Use Google Cloud Observability with Cloud Monitoring, Cloud Logging, and Managed Service for Prometheus for unified observability across all clusters
    3. Use a third-party monitoring solution deployed in a separate project
    4. Configure email alerts from each application individually

    Explanation: Google Cloud Observability (Cloud Monitoring + Cloud Logging + Cloud Trace + Managed Service for Prometheus) provides a unified, fully managed observability platform. Managed Service for Prometheus allows teams familiar with Prometheus to continue using PromQL while benefiting from global, managed storage and querying.

  4. For this question, refer to the EHR Healthcare case study. EHR Healthcare needs to manage multiple containerized environments consistently while ensuring security best practices. They want to reduce the operational overhead of managing Kubernetes clusters. What is the recommended approach?
    1. Deploy all workloads on Compute Engine instances with Docker
    2. Use GKE Standard mode with manual node pool management in each environment
    3. Use GKE Autopilot mode with GKE Enterprise fleet management for centralized policy and configuration across environments
    4. Deploy each application as a separate Cloud Run service without orchestration

    Explanation: GKE Autopilot mode provides a fully managed Kubernetes experience where Google manages nodes, scaling, and security hardening. GKE Enterprise fleet management enables centralized governance with fleet-level policies, Config Sync for GitOps, and Policy Controller for guardrails across multiple clusters and environments.

  5. For this question, refer to the EHR Healthcare case study. EHR Healthcare needs to maintain connectivity between their legacy on-premises insurance provider integrations and their new Google Cloud environment with high bandwidth and low latency. What should you recommend?
    1. Set up a standard VPN tunnel between on-premises and Google Cloud
    2. Use Dedicated Interconnect with redundant connections for high-bandwidth, low-latency, SLA-backed connectivity
    3. Use public internet with SSL/TLS encryption for all communications
    4. Migrate all legacy systems to Google Cloud immediately

    Explanation: Dedicated Interconnect provides 10/100 Gbps private connections between on-premises networks and Google Cloud with up to 99.99% SLA when configured with recommended redundant topology. The case study specifies legacy systems cannot be migrated immediately and require high-performance connectivity.

EHR Healthcare References

Google Cloud BigQuery – Data Warehouse & Analytics

~ Last updated on : ~ jayendrapatil

Google Cloud BigQuery

  • Google Cloud BigQuery is a fully managed, peta-byte scale, serverless, highly scalable, and cost-effective multi-cloud data-to-AI platform.
  • BigQuery supports GoogleSQL (standard SQL dialect that is ANSI:2011 compliant), which reduces the need for code rewrites.
  • BigQuery transparently and automatically provides highly durable, replicated storage in multiple locations and high availability with no extra charge and no additional setup.
  • BigQuery supports federated data and can process external data sources in GCS for Parquet and ORC open-source file formats, transactional databases (Bigtable, Cloud SQL, Spanner, AlloyDB), or spreadsheets in Drive without moving the data.
  • BigQuery automatically replicates data and keeps a seven-day history of changes, allowing easy restoration and comparison of data from different times (time travel).
  • BigQuery Data Transfer Service automatically transfers data from external data sources, like Google Marketing Platform, Google Ads, YouTube, external sources like S3 or Teradata, and partner SaaS applications to BigQuery on a scheduled and fully managed basis.
  • BigQuery provides a REST API for easy programmatic access and application integration. Client libraries are available in Java, Python, Node.js, C#, Go, Ruby, and PHP.
  • BigQuery provides three editions — Standard, Enterprise, and Enterprise Plus — each offering different feature sets and pricing tiers for capacity-based compute.
  • BigQuery supports both on-demand pricing (pay per TiB processed) and capacity-based pricing (pay per slot-hour via editions).

⚠️ Legacy SQL Deprecation Notice

Legacy SQL is being phased out. For organizations and projects that don’t use legacy SQL between November 1, 2025, and June 1, 2026, legacy SQL becomes unavailable after the evaluation period ends. Migrate all queries to GoogleSQL (Standard SQL).

BigQuery Key Features (2024-2026 Updates)

  • Gemini in BigQuery — AI-powered SQL code generation, explanation, Python code generation, data canvas, data insights, and partitioning/clustering recommendations (GA since 2024).
  • BigQuery Data Canvas — Visual, natural-language-driven workspace for data discovery, preparation, querying, and visualization all within BigQuery.
  • Vector Search — Native vector search with embedding generation, vector indexes, and semantic search capabilities for RAG and AI applications (GA since 2024).
  • BigQuery Graph — Graph analytics solution for modeling, analyzing, and visualizing massive-scale relationships using graph queries (Preview 2025).
  • Continuous Queries — Real-time, always-on queries that process streaming data as it arrives, enabling true real-time analytics (Enterprise/Enterprise Plus editions).
  • Apache Iceberg Managed Tables — Fully managed tables using open Iceberg format stored in customer-owned buckets, with interoperability across open-source engines.
  • BigQuery Studio — Unified workspace for SQL, Python notebooks, Spark, and data pipelines in a single interface.
  • Change Data Capture (CDC) — Native CDC ingestion via Storage Write API for streaming row-level changes (inserts, updates, deletes) directly.
  • MCP Server — Model Context Protocol server for connecting LLMs and AI agents directly to BigQuery.
  • Multimodal Data Analysis — Analyze images, audio, video, and documents directly using SQL with Gemini integration.

BigQuery Resources

BigQuery Resources

Datasets

  • Datasets are the top-level containers used to organize and control access to the BigQuery tables and views.
  • Datasets frequently map to schemas in standard relational databases and data warehouses.
  • Datasets are scoped to the Cloud project.
  • A dataset is bound to a location and can be defined as
    • Regional: A specific geographic place, such as London.
    • Multi-regional: A large geographic area, such as the United States, that contains two or more geographic places.
  • Dataset location can be set only at the time of its creation.
  • A query can contain tables or views from different datasets in the same location.
  • Dataset names must be unique for each project.
  • BigQuery supports cross-region dataset replication for disaster recovery and data residency requirements.
  • BigQuery supports dataset data retention (time travel) allowing access to data at any point within the configured window.

Tables

  • BigQuery tables are row-column structures that hold the data.
  • A BigQuery table contains individual records organized in rows. Each record is composed of columns (also called fields).
  • Every table is defined by a schema that describes the column names, data types, and other information.
  • BigQuery has the following types of tables:
    • Native tables: Tables backed by native BigQuery storage.
    • External tables: Tables backed by storage external to BigQuery.
    • BigLake tables: Tables backed by external storage with fine-grained access control and metadata caching.
    • Apache Iceberg managed tables: Fully managed tables using open Iceberg format in customer-owned buckets.
    • Views: Virtual tables defined by a SQL query.
  • Schema of a table can either be defined during creation or specified in the query job or load job that first populates it with data.
  • Schema auto-detection is also supported when data is loaded from BigQuery or an external data source. BigQuery makes a best-effort attempt to automatically infer the schema for CSV and JSON files.
  • Columns datatype cannot be changed once defined (except for supported type widening conversions in Iceberg tables).
  • BigQuery supports table clones (lightweight, writable copies) and table snapshots (point-in-time read-only copies) for efficient data management.
  • BigQuery supports primary and foreign keys for query optimization (used by the optimizer for better join planning).

Partitioned Tables

  • A partitioned table is a special table that is divided into segments, called partitions, that make it easier to manage and query your data.
  • By dividing a large table into smaller partitions, query performance and costs can be controlled by reducing the number of bytes read by a query.
  • BigQuery tables can be partitioned by:
    • Time-unit column: Tables are partitioned based on a TIMESTAMP, DATE, or DATETIME column in the table.
    • Ingestion time: Tables are partitioned based on the timestamp when BigQuery ingests the data.
    • Integer range: Tables are partitioned based on an integer column.
  • If a query filters on the value of the partitioning column, BigQuery can scan the partitions that match the filter and skip the remaining partitions. This process is called pruning.
  • BigQuery provides partition and cluster recommendations based on query patterns to help optimize table layouts automatically.

Clustered Tables

  • With Clustered tables, the table data is automatically organized based on the contents of one or more columns in the table’s schema.
  • Columns specified are used to colocate the data.
  • Clustering can be performed on multiple columns, where the order of the columns is important as it determines the sort order of the data.
  • Clustering can improve query performance for specific filter queries or ones that aggregate data as BigQuery uses the sorted blocks to eliminate scans of unnecessary data.
  • Clustering does not provide cost guarantees before running the query.
  • Partitioning can be used with clustering where data is first partitioned and then data in each partition is clustered by the clustering columns. When the table is queried, partitioning sets an upper bound of the query cost based on partition pruning.
  • BigQuery supports automatic clustering for Iceberg managed tables, which optimizes file layout without manual configuration.

Views

  • A View is a virtual table defined by a SQL query.
  • View query results contain data only from the tables and fields specified in the query that defines the view.
  • Views are read-only and do not support DML queries.
  • Dataset that contains the view and the dataset that contains the tables referenced by the view must be in the same location.
  • View does not support BigQuery job that exports data.
  • View does not support JSON API to retrieve data from the view.
  • Standard SQL and legacy SQL queries cannot be mixed.
  • Legacy SQL view cannot be automatically updated to standard SQL syntax.
  • No user-defined functions allowed.
  • No wildcard table references allowed.
  • BigQuery supports authorized views to share query results with particular users/groups without giving access to underlying tables.

Materialized Views

  • Materialized views are precomputed views that periodically cache the results of a query for increased performance and efficiency.
  • BigQuery leverages pre-computed results from materialized views and whenever possible reads only delta changes from the base table to compute up-to-date results.
  • Materialized views can be queried directly or can be used by the BigQuery optimizer to process queries to the base tables (smart tuning).
  • Materialized views queries are generally faster and consume fewer resources than queries that retrieve the same data only from the base table.
  • Materialized views can significantly improve the performance of workloads that have the characteristic of common and repeated queries.
  • BigQuery supports automatic refresh and manual refresh of materialized views.
  • BigQuery provides materialized view recommendations based on query history to suggest views that would improve performance.
  • Creating and refreshing materialized views requires Enterprise edition or higher (Standard edition can only query existing materialized views directly).

Jobs

  • Jobs are actions that BigQuery runs on your behalf to load data, export data, query data, or copy data.
  • Jobs are not linked to the same project that the data is stored in. However, the location where the job can execute is linked to the dataset location.

External Data Sources

  • An external data source (federated data source) is a data source that can be queried directly even though the data is not stored in BigQuery.
  • Instead of loading or streaming the data, a table can be created that references the external data source.
  • BigQuery offers support for querying data directly from:
    • Cloud Bigtable
    • Cloud Storage
    • Google Drive
    • Cloud SQL
    • Spanner
    • AlloyDB
    • Amazon S3 (via BigQuery Omni)
    • Azure Blob Storage (via BigQuery Omni)
  • Supported formats are:
    • Avro
    • CSV
    • JSON (newline delimited only)
    • ORC
    • Parquet
    • Apache Iceberg
  • External data sources use cases
    • Loading and cleaning the data in one pass by querying the data from an external data source and writing the cleaned result into BigQuery storage.
    • Having a small amount of frequently changing data that needs to be joined with other tables. As an external data source, the frequently changing data does not need to be reloaded every time it is updated.
  • Permanent vs Temporary external tables
    • The external data sources can be queried in BigQuery by using a permanent table or a temporary table.
    • Permanent Table
      • is a table that is created in a dataset and is linked to the external data source.
      • access controls can be used to share the table with others who also have access to the underlying external data source.
    • Temporary Table
      • you submit a command that includes a query and creates a non-permanent table linked to the external data source.
      • no table is created in the BigQuery datasets.
      • cannot be shared with others.
      • Querying an external data source using a temporary table is useful for one-time, ad-hoc queries over external data, or for extract, transform, and load (ETL) processes.
  • Limitations
    • does not guarantee data consistency for external data sources.
    • query performance for external data sources may not be as high as querying data in a native BigQuery table.
    • cannot reference an external data source in a wildcard table query.
    • support table partitioning or clustering in limited ways.
    • results are not cached and would be charged for each query execution.
  • Metadata Caching — BigQuery supports metadata caching for external tables to improve query planning performance and reduce latency.

BigQuery Editions

  • BigQuery provides three editions for capacity-based pricing, each with different feature sets:
    • Standard — Autoscaling only (max 1,600 slots per reservation), no capacity commitments, 99.9% SLO. No access to BigQuery ML, continuous queries, BigQuery Graph, or fine-grained security controls.
    • Enterprise — Autoscaling + Baseline slots, advanced workload management, BigQuery ML, continuous queries, BigQuery Omni, VPC Service Controls, CMEK, column/row-level security, 99.99% SLO. Supports 1-year (20% discount) and 3-year (40% discount) commitments.
    • Enterprise Plus — All Enterprise features plus managed disaster recovery, Assured Workloads compliance controls, 99.99% SLO.
  • BigQuery also offers on-demand pricing at $6.25 per TiB processed (first 1 TiB/month free), which includes BigQuery ML, CMEK, and fine-grained security.
  • Editions are a property of reservations (compute), not storage. Datasets and tables are unaffected by edition choice.
  • To change an edition, you must delete and recreate the reservation.
  • Slots autoscaling automatically scales capacity to accommodate workload demands without pre-provisioning.
  • BigQuery fluid scaling removes the 1-minute minimum duration for slot usage.

BigLake

  • BigLake is a storage engine that unifies data warehouses and data lakes by extending BigQuery’s fine-grained security and governance to data stored in Cloud Storage, Amazon S3, and Azure Blob Storage.
  • BigLake tables provide:
    • Fine-grained access control (column-level and row-level security) on external data.
    • Metadata caching for improved query performance on external data.
    • Support for open formats (Parquet, ORC, Avro, JSON, CSV, Iceberg).
    • Consistent governance across multi-cloud storage.
  • Apache Iceberg Managed Tables (formerly BigLake tables for Apache Iceberg) provide:
    • Fully managed experience similar to native BigQuery tables but with data stored in customer-owned buckets.
    • DML support, streaming via Storage Write API, schema evolution.
    • Automatic storage optimization (adaptive file sizing, automatic clustering, garbage collection).
    • Iceberg V2 snapshot export for interoperability with Spark, Flink, Presto, and other engines.
    • Time travel, column-level security, and data masking.

BigQuery Omni

  • BigQuery Omni is a multi-cloud analytics solution that lets you query data across Google Cloud, AWS, and Azure without moving data.
  • Uses BigLake tables to access data stored in Amazon S3 and Azure Blob Storage.
  • Supports cross-cloud joins — query data across different clouds in a single SQL statement.
  • Supports cross-cloud transfer — move data between clouds for consolidation.
  • Available in Enterprise edition with on-demand pricing.
  • Compute runs locally in the cloud where data resides (data never leaves the cloud it’s stored in).

BigQuery AI and ML

  • BigQuery ML allows building and deploying ML models directly within BigQuery using SQL, supporting classification, regression, clustering, forecasting, recommendation, and anomaly detection.
  • Gemini Integration — BigQuery integrates with Gemini models via Vertex AI for:
    • Text generation and summarization using ML.GENERATE_TEXT
    • Embedding generation using ML.GENERATE_EMBEDDING
    • Image, audio, and video analysis
    • Document processing and OCR
    • Sentiment analysis and text understanding
  • Generative AI Functions — SQL functions like AI.GENERATE for calling AI models directly from queries.
  • Remote Models — Connect to models hosted on Vertex AI, Cloud AI services, or custom endpoints.
  • Gemini in BigQuery (Assistive AI) — AI-powered assistance for:
    • SQL code generation and explanation
    • Python code generation in notebooks
    • Data insights and recommendations
    • Partitioning and clustering recommendations
  • Requires Enterprise edition or higher (not available in Standard edition).

BigQuery Vector Search

  • Vector Search enables semantic search and similarity matching directly in BigQuery using vector embeddings.
  • Supports generating embeddings using Vertex AI models, then searching with the VECTOR_SEARCH function.
  • Vector Indexes optimize search performance for large-scale embedding datasets (requires Enterprise edition or higher for index acceleration).
  • Use cases include: RAG (Retrieval-Augmented Generation), product recommendations, anomaly detection, multi-modal search, and semantic analysis.
  • Supports automated embedding generation that automatically generates and maintains embeddings as data changes.
  • Eliminates the need for specialized vector databases for many use cases.

BigQuery Graph

  • BigQuery Graph (Preview 2025) provides graph analytics capabilities for modeling, analyzing, and visualizing relationships in data.
  • Enables multi-hop traversals, path finding, and relationship analysis that would require complex nested JOINs in traditional SQL.
  • Use cases: fraud detection, social network analysis, supply chain mapping, knowledge graphs.
  • Includes a visual graph modeler for schema design and graph visualization tools.
  • Available in Enterprise and Enterprise Plus editions only.

BigQuery Continuous Queries

  • Continuous queries are always-on queries that process data in real-time as it arrives via streaming.
  • Enables true real-time analytics without the latency of batch processing.
  • Supports stream-to-stream joins and window aggregations.
  • Can export results to Pub/Sub, Bigtable, or other BigQuery tables for downstream consumption.
  • Available in Enterprise and Enterprise Plus editions only.
  • Uses dedicated CONTINUOUS assignment type in reservations.

BigQuery Security

Refer blog post @ BigQuery Security

BigQuery Best Practices

  • Cost Control
    • Query only the needed columns and avoid select * as BigQuery does a full scan of every column in the table.
    • Don’t run queries to explore or preview table data. Use preview option.
    • Before running queries, preview them to estimate costs. Queries are billed according to the number of bytes read and the cost can be estimated using --dry-run feature.
    • Use the maximum bytes billed setting to limit query costs.
    • Use clustering and partitioning to reduce the amount of data scanned.
    • For non-clustered tables, do not use a LIMIT clause as a method of cost control. Applying a LIMIT clause to a query does not affect the amount of data read, but shows limited results only. With a clustered table, a LIMIT clause can reduce the number of bytes scanned.
    • Partition the tables by date which helps query relevant subsets of data which improves performance and reduces costs.
    • Materialize the query results in stages. Break the query into stages where each stage materializes the query results by writing them to a destination table.
    • Use streaming inserts only if the data must be immediately available as streaming data is charged.
    • Consider editions with capacity-based pricing for predictable workloads to reduce costs vs. on-demand.
    • Use commitment plans (1-year at 20% or 3-year at 40% discount) for sustained workloads.
  • Query Performance
    • Control projection, Query only the needed columns. Avoid SELECT *.
    • Prune partitioned queries, use the _PARTITIONTIME pseudo column to filter the partitions.
    • Denormalize data whenever possible using nested and repeated fields.
    • Avoid external data sources, if query performance is a top priority.
    • Avoid repeatedly transforming data via SQL queries, use materialized views instead.
    • Avoid using Javascript user-defined functions.
    • Optimize Join patterns. Start with the largest table.
    • Use BI Engine for sub-second query response on dashboards and BI tools (Enterprise edition+).
    • Use search indexes for full-text search acceleration on large text columns.
    • Use vector indexes to accelerate vector search on embedding columns.
    • Leverage history-based optimizations where BigQuery automatically optimizes repeated query patterns.
  • Optimizing Storage
    • Use the expiration settings to remove unneeded tables and partitions.
    • Keep the data in BigQuery to take advantage of the long-term storage cost benefits rather than exporting to other storage options.
    • Long-term storage pricing automatically applies to data not modified for 90 consecutive days.
    • Use physical storage billing (compressed bytes) instead of logical storage billing for better cost efficiency on compressed data.

BigQuery Data Transfer Service

Refer GCP blog post @ Google Cloud BigQuery Data Transfer Service

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated.
  • Open to further feedback, discussion and correction.
  1. A user wishes to generate reports on petabyte-scale data using Business Intelligence (BI) tools. Which storage option provides integration with BI tools and supports OLAP workloads up to petabyte-scale?
    1. Bigtable
    2. Cloud Datastore
    3. Cloud Storage
    4. BigQuery
  2. Your company uses Google Analytics for tracking. You need to export the session and hit data from a Google Analytics 360 reporting view on a scheduled basis into BigQuery for analysis. How can the data be exported?
    1. Configure a scheduler in Google Analytics to convert the Google Analytics data to JSON format, then import directly into BigQuery using bq command line.
    2. Use gsutil to export the Google Analytics data to Cloud Storage, then import into BigQuery and schedule it using Cron.
    3. Import data to BigQuery directly from Google Analytics using Cron.
    4. Use BigQuery Data Transfer Service to import the data from Google Analytics.
  3. A company wants to run machine learning models on their BigQuery data without moving it to a separate ML platform. Which feature should they use?
    1. Export data to Vertex AI Workbench
    2. BigQuery ML
    3. Dataflow ML
    4. Cloud Composer with TensorFlow
  4. Your organization needs to query data stored in Amazon S3 from BigQuery without copying the data to Google Cloud. Which feature enables this?
    1. BigQuery Data Transfer Service
    2. Cloud Storage Transfer Service
    3. BigQuery Omni
    4. Storage Transfer Service
  5. A data team needs to perform real-time analytics on streaming data arriving in BigQuery. They need results to update continuously as new data arrives. Which BigQuery feature should they use?
    1. Scheduled queries with 1-minute interval
    2. Materialized views with automatic refresh
    3. Continuous queries
    4. Streaming buffer queries
  6. You need to store data in an open format that can be accessed by both BigQuery and Apache Spark, while maintaining BigQuery’s fully managed experience. Which table type should you use?
    1. External tables on Cloud Storage
    2. BigQuery native tables
    3. Apache Iceberg managed tables (BigLake)
    4. Bigtable federated tables
  7. A company wants to build a semantic search application using product descriptions stored in BigQuery. They need to find products based on meaning rather than exact keyword matches. Which BigQuery feature should they use?
    1. SEARCH function with search indexes
    2. LIKE operator with wildcard patterns
    3. Full-text search with text analyzers
    4. Vector Search with ML.GENERATE_EMBEDDING
  8. Which BigQuery edition is required to use BigQuery ML and continuous queries?
    1. Standard edition
    2. Enterprise edition or higher
    3. Enterprise Plus edition only
    4. On-demand pricing only

References