Google Cloud BigQuery Data Transfer Service

BigQuery Data Transfer Service automates data movement into BigQuery on a scheduled, managed basis.

After a data transfer is configured, the BigQuery Data Transfer Service automatically loads data into BigQuery on a regular basis.
BigQuery Data Transfer Service can also initiate data backfills to recover from any outages or gaps.

BigQuery Data Transfer Service can only sink data to BigQuery and cannot be used to transfer data out of BigQuery.
BigQuery Data Transfer Service can be accessed using the Google Cloud console, the bq command-line tool, or the BigQuery Data Transfer Service API.
In addition to loading data into BigQuery, BigQuery Data Transfer Service is used for dataset copies and scheduled queries.

BigQuery Data Transfer Service Features

Simplicity – Eliminates the need for infrastructure management or complex coding.
Scalability – Handles massive data volumes and high numbers of concurrent users.
Security – Employs encryption, authentication, and authorization; supports FedRAMP High, CJIS compliance, EU Data Boundary, and Sovereign Controls.

Cost-effectiveness – Many first-party connectors (e.g., Google Ads, YouTube) are free. Third-party connectors use consumption-based pricing (billed by slot-hours).
Event-driven transfers (GA) – Transfers can trigger automatically when a new file arrives in a Cloud Storage bucket, enabling near-real-time data pipelines.
Transfer to Managed Iceberg tables – Supports loading data directly into Managed Iceberg tables in BigQuery.

Custom organization policies – Allow or deny specific operations on transfer configurations for compliance and security requirements.
Regional endpoints – Ensures requests are processed only if the resource exists in the specified location, supporting data residency requirements.

BigQuery Data Transfer Service Sources

BigQuery Data Transfer Service supports loading data from the following data sources:

SaaS Platforms

Salesforce
Salesforce Marketing Cloud (SFMC)
ServiceNow

Marketing Platforms

Facebook Ads
HubSpot (Preview)
Klaviyo (Preview)
Mailchimp (Preview)

Payment Platforms

PayPal (Preview)
Stripe (Preview)
Shopify (Preview)

Databases and Data Warehouses

Amazon Redshift
Apache Hive Metastore
Microsoft SQL Server (Preview)
MySQL
Oracle
PostgreSQL
Snowflake (Preview)
Teradata

Cloud Storage

Cloud Storage (supports event-driven transfers)
Amazon Simple Storage Service (Amazon S3)
Azure Blob Storage

Google Services

Campaign Manager
Comparison Shopping Service (CSS) Center (Preview)
Display & Video 360
Google Ads (supports custom reports via GAQL)
Google Ad Manager (supports incremental DT file updates)
Google Analytics 4 (GA4)
Google Merchant Center (Preview)
Search Ads 360 (supports Performance Max campaigns)
Google Play
YouTube Channel
YouTube Content Owner

Event-Driven Transfers

Event-driven transfers automatically load data based on event notifications from Cloud Storage.
When a new file arrives in a Cloud Storage bucket, the transfer triggers automatically.
After an event-driven transfer is triggered, the service waits up to 10 minutes before triggering the next run, regardless of additional events.

Recommended for incremental data ingestion that optimizes cost efficiency.
Event-driven transfers don’t support runtime parameters for source URI or data path.

Pricing

First-party Google connectors (Google Ads, YouTube, Campaign Manager, etc.) are provided at no cost.
Third-party SaaS and database connectors (Salesforce, Facebook Ads, Oracle, MySQL, etc.) use consumption-based pricing measured in slot-hours.

Connectors in Preview remain completely free of charge.
Standard BigQuery storage and query pricing applies once data is transferred.
Jobs triggered by DTS use reservation slots only if the project is assigned to a reservation with QUERY or PIPELINE job types.

Data Delivery SLO

The Data Delivery SLO applies only to automatically scheduled data transfers from sources within Google Cloud.

For transfers involving third-party or non-Google Cloud sources, service outages with those sources can impact performance, so the Data Delivery SLO does not apply.

GCP Certification Exam Practice Questions

Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).

GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.

GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated

Open to further feedback, discussion and correction.

Your company uses Google Analytics for tracking. You need to export the session and hit data from a Google Analytics 360 reporting view on a scheduled basis into BigQuery for analysis. How can the data be exported?
1. Configure a scheduler in Google Analytics to convert the Google Analytics data to JSON format, then import directly into BigQuery using bq command line.
2. Use gsutil to export the Google Analytics data to Cloud Storage, then import into BigQuery and schedule it using Cron.
3. Import data to BigQuery directly from Google Analytics using Cron
4. Use BigQuery Data Transfer Service to import the data from Google Analytics

A company stores raw event files in Cloud Storage and needs near-real-time ingestion into BigQuery without managing schedulers. Which approach should they use?
1. Set up a Cloud Scheduler job to trigger a Cloud Function every minute to load data.
2. Use Pub/Sub to stream events directly into BigQuery.
3. Configure an event-driven transfer in BigQuery Data Transfer Service that triggers automatically when new files arrive in Cloud Storage.
4. Create a Dataflow streaming pipeline from Cloud Storage to BigQuery.
Your organization needs to migrate data from a Salesforce CRM into BigQuery on a daily schedule. Which service provides a fully managed, code-free solution?
1. Use Dataflow with a custom Salesforce connector.
2. Export Salesforce data to CSV and load manually.
3. Use BigQuery Data Transfer Service with the Salesforce connector.
4. Use Cloud Data Fusion with a Salesforce plugin.
Which of the following are valid data sources for BigQuery Data Transfer Service? (Choose 3)
1. Amazon Redshift
2. Oracle
3. MongoDB
4. Azure Blob Storage
5. Google Cloud Bigtable