Google Cloud Pub/Sub

• Pub/Sub is a fully managed, asynchronous messaging service designed to be highly reliable and scalable.
• Pub/Sub service allows applications to exchange messages reliably, quickly, and asynchronously
• Pub/Sub allows services to communicate asynchronously, with latencies on the order of 100 milliseconds.
• Pub/Sub enables the creation of event producers and consumers, called publishers and subscribers.
• Publishers communicate with subscribers asynchronously by broadcasting events, rather than by synchronous remote procedure calls.
• Pub/Sub offers at-least-once message delivery and best-effort ordering to existing subscribers
• Pub/Sub also supports exactly-once delivery for pull subscriptions, ensuring messages are not redelivered after a successful acknowledgment within the same region.
• Pub/Sub accepts a maximum of 1,000 messages in a batch, and the size of a batch can not exceed 10 megabytes.
• Pub/Sub serves as a versatile entry point to ingest streaming data into Google Cloud’s ecosystem and is integrated with products like BigQuery, Cloud Storage, Dataflow, and more.

Pub/Sub Core Concepts

• Topic: A named resource to which messages are sent by publishers.
• Publisher: An application that creates and sends messages to a topic(s).
• Subscriber: An application with a subscription to a topic(s) to receive messages from it.
• Subscription: A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application.
• Message: The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.
• Message attribute: A key-value pair that a publisher can define for a message.
• Acknowledgment (or “ack”): A signal sent by a subscriber to Pub/Sub after it has received a message successfully. Acked messages are removed from the subscription’s message queue.
• Schema: A schema is a format that messages must follow, creating a contract between publisher and subscriber that Pub/Sub will enforce
• Push and pull: The two message delivery methods. A subscriber receives messages either by Pub/Sub pushing them to the subscriber’s chosen endpoint or by the subscriber pulling them from the service.

Pub/Sub Topic Types

• Pub/Sub supports two kinds of topics: Standard topics and Import topics.
• Standard Topic
  • A standard topic receives messages from publishers through Pub/Sub client libraries or REST/gRPC APIs.
• Import Topics (New – 2024)
  • Import topics provide a fully managed, no-code way to ingest streaming data from external sources directly into Pub/Sub.
  • Supported sources include:
    • AWS Kinesis Data Streams — Ingest streaming data from AWS Kinesis without custom connectors
    • Cloud Storage — Ingest batch data from GCS buckets for batch-to-streaming use cases or replaying archived data
    • Azure Event Hubs — Ingest from Azure messaging infrastructure
    • Amazon MSK (Managed Streaming for Apache Kafka) — Cross-cloud Kafka ingestion
    • Confluent Cloud — Ingest from Confluent Kafka clusters
  • Once data flows into an import topic, you can create any subscription type (Pull, Push, BigQuery, or Cloud Storage) to route data to downstream sinks.

Pub/Sub Subscription Types

• Pull Subscription
  • The subscriber application initiates requests to the Pub/Sub server to retrieve messages.
  • If unspecified, Pub/Sub subscriptions use pull delivery.
• Push Subscription
  • Pub/Sub initiates requests to the subscriber application to deliver messages.
  • The push endpoint must be a publicly accessible HTTPS address.
• BigQuery Subscription (Export Subscription)
  • Writes messages directly from Pub/Sub to a BigQuery table without needing Dataflow or custom middleware.
  • Supports BigQuery tables for Apache Iceberg for high-throughput ingestion stored in Parquet format.
• Cloud Storage Subscription (Export Subscription)
  • Writes messages directly from Pub/Sub to Cloud Storage buckets in Text or Avro format.
• Bigtable Subscription (Export Subscription – New 2025)
  • Writes messages directly from Pub/Sub into Cloud Bigtable tables.
• Messages published before a subscription is created will not be delivered to that subscription

Pub/Sub Subscription Properties

• Acknowledgment deadline
  • Message not acknowledged before the deadline is sent again.
  • Default acknowledgment deadline is 10 secs. with a min of 10 secs and max of 600 secs (10 mins).
• Message retention duration
  • Message retention duration specifies how long Pub/Sub retains messages after publication.
  • Acknowledged messages are no longer available to subscribers and are deleted, by default
  • After the message retention duration, Pub/Sub might discard the message, regardless of its acknowledgment state.
  • Default message retention duration is 7 days with a min of 10 mins and max of 31 days
  • Retaining unacknowledged messages for more than 24 hours incurs additional storage charges.
• Dead-letter topics
  • If a subscriber can’t acknowledge a message, Pub/Sub can forward the message to a dead-letter topic.
  • With a dead-letter topic, message ordering can’t be enabled
  • With a dead-letter topic, the maximum number of delivery attempts can be specified.
  • Default is 5 delivery attempts; with a min-max of 5-100
• Expiration period
  • Subscriptions expire without any subscriber activity such as open connections, active pulls, or successful pushes
  • Subscription deletion clock restarts, if subscriber activity is detected or subscription properties are updated
  • Default expiration period is 31 days with a min of 1 day; can be set to “never expire”
• Retry policy
  • If the acknowledgment deadline expires or a subscriber responds with a negative acknowledgment, Pub/Sub can send the message again using exponential backoff.
  • If the retry policy isn’t set, Pub/Sub resends the message as soon as the acknowledgment deadline expires or a subscriber responds with a negative acknowledgment (Retry immediately).
  • Maximum and minimum backoff values can be configured, with a maximum backoff of 600 seconds.
• Message ordering
  • If publishers send messages with an ordering key, are in the same region and message ordering is set, Pub/Sub delivers the messages in order.
  • If not set, Pub/Sub doesn’t deliver messages in order, including messages with ordering keys.
  • When using ordered delivery, acknowledgments for later messages are not processed until acknowledgments for earlier messages are processed.
• Filter
  • Filter is a string with a filtering expression where the subscription only delivers the messages that match the filter.
  • Pub/Sub service automatically acknowledges the messages that don’t match the filter.
  • Messages can be filtered using their attributes.
  • Filters cannot be changed or removed after they are applied.
  • Filtered (auto-acknowledged) messages don’t incur egress fees but do incur message delivery fees.
• Exactly-once delivery
  • Pub/Sub supports exactly-once delivery for pull subscriptions (including StreamingPull).
  • Push and export subscriptions don’t support exactly-once delivery.
  • Exactly-once guarantee only applies when subscribers connect to the service in the same region.
  • Provides stronger guarantees that messages are not redelivered before the acknowledgment deadline passes.

Pub/Sub Single Message Transforms (SMTs)

• Single Message Transforms (SMTs) allow lightweight modifications to message attributes and data directly within Pub/Sub without additional services. (GA June 2025)
• SMTs can be applied to a topic, a subscription, or both independently.
• Up to five transforms can be added per topic or subscription.
• JavaScript User-Defined Functions (UDFs)
  • Perform lightweight modifications to message data and attributes using JavaScript code snippets.
  • Use cases include: data format conversion, field casting, adding composite fields, data masking/redaction of PII, and enhanced filtering on message data (not just attributes).
  • If a Topic SMT is configured, the message is transformed and persisted in its transformed state.
  • If a Subscription SMT is configured, the message is transformed before delivery to the subscriber.
• AI Inference SMT
  • Sends message data to a Vertex AI endpoint for inference and enriches/transforms the message with the response.
• SMTs eliminate the need for maintaining extra services (Dataflow, Cloud Run) for simple, lightweight transformations.

Pub/Sub Seek Feature

• Acknowledged messages are no longer available to subscribers and are deleted
• Subscriber clients must process every message in a subscription even if only a subset is needed.
• Seek feature extends subscriber functionality by allowing you to alter the acknowledgment state of messages in bulk
• Timestamp Seeking
  • With Seek feature, you can replay previously acknowledged messages or purge messages in bulk
  • Seeking to a time marks every message received by Pub/Sub before the time as acknowledged, and all messages received after the time as unacknowledged.
  • Seeking to a time in the future allows you to purge messages.
  • Seeking to a time in the past allows replay and reprocess previously acknowledged messages
  • Timestamp seeking approach is imprecise as
    • Possible clock skew among Pub/Sub servers.
    • Pub/Sub has to work with the arrival time of the publish request rather than when an event occurred in the source system.
• Snapshot Seeking
  • State of one subscription can be copied to another by using seek in combination with a Snapshot.
  • Once a snapshot is created, it retains:
    • All messages that were unacknowledged in the source subscription at the time of the snapshot’s creation.
    • Any messages published to the topic thereafter.
  • The maximum possible lifetime of a snapshot is seven days.

Pub/Sub Locations

• Pub/Sub servers run in all GCP regions around the world, which helps offer fast, global data access while giving users control over where messages are stored
• Cloud Pub/Sub offers global data access in that publisher and subscriber clients are not aware of the location of the servers to which they connect or how those services route the data.
• Pub/Sub’s load balancing mechanisms direct publisher traffic to the nearest GCP data center where data storage is allowed, as defined in the Resource Location Restriction
• Publishers in multiple regions may publish messages to a single topic with low latency. Any individual message is stored in a single region. However, a topic may have messages stored in many regions.
• Subscriber client requesting messages published to this topic connects to the nearest server which aggregates data from all messages published to the topic for delivery to the client.
• Message Storage Policy
  • Message Storage Policy helps ensure that messages published to a topic are never persisted outside a set of specified Google Cloud regions, regardless of where the publish requests originate.
  • Pub/Sub chooses the nearest allowed region, when multiple regions are allowed by the policy

Pub/Sub Security

• Pub/Sub encrypts messages with Google-managed keys, by default.
• Pub/Sub also supports Customer-Managed Encryption Keys (CMEK) using Cloud KMS, giving control over key protection level, location, rotation schedule, usage permissions, and cryptographic boundaries.
• Every message is encrypted at the following states and layers:
  • At rest
    • Hardware layer
    • Infrastructure layer
    • Application layer
      • Pub/Sub individually encrypts incoming messages as soon as the message is received
  • In transit
• Pub/Sub does not encrypt message attributes at the application layer.
• Message attributes are still encrypted at the hardware and infrastructure layers.

Pub/Sub Observability

• OpenTelemetry Tracing (New – 2024)
  • Provides a detailed distributed trace of the message lifecycle, from publish to receive and process.
  • Helps identify bottlenecks, misconfigurations, and failures in Pub/Sub applications.
  • Allows tracing of client library operations including batching, lease management, and flow control.
  • Integrates with Cloud Trace for analysis.
• Cloud Monitoring Integration
  • Monitor topics and subscriptions with built-in metrics.
  • Labels can be used to organize and filter monitoring data.

Pub/Sub Integrations

• Dataflow — Natively integrated for stream processing pipelines.
• BigQuery Engine for Apache Flink — Serverless Flink with native Pub/Sub integration (Preview 2024).
• Apache Flink Connector — GA connector for existing Flink deployments to read from/write to Pub/Sub.
• BigQuery Continuous Queries — Process real-time data in BigQuery and export results to a Pub/Sub topic (reverse ETL).
• Analytics Hub — Share Pub/Sub topics as data products for real-time streaming data sharing across organizations (Preview 2024).
• Cloud Run / Cloud Functions — Event-driven compute triggered by Pub/Sub messages.

Common use cases

• Ingestion user interaction and server events
• Real-time event distribution
• Replicating data among databases
• Parallel processing and workflows
• Data streaming from IoT devices
• Refreshing distributed caches
• Load balancing for reliability
• Cross-cloud data ingestion (AWS, Azure)
• Streaming data sharing and monetization

Pub/Sub Lite (Deprecated)

GCP Certification Exam Practice Questions

• Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
• GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
• GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
• Open to further feedback, discussion and correction.

Questions on New Features:

1. A company wants to ingest streaming data from AWS Kinesis Data Streams into BigQuery without maintaining custom connectors. What is the simplest approach using Pub/Sub?
   1. Use a Dataflow pipeline to read from Kinesis and write to Pub/Sub
   2. Use a Pub/Sub import topic configured for AWS Kinesis Data Streams, then create a BigQuery subscription
   3. Use Cloud Functions to poll Kinesis and publish to Pub/Sub
   4. Use the Kafka Connect Pub/Sub connector

   Answer: b. Import topics provide no-code ingestion from Kinesis, and BigQuery subscriptions write directly to BigQuery.
2. Which Pub/Sub delivery guarantee requires subscribers to connect to the service in the same region?
   1. At-least-once delivery
   2. At-most-once delivery
   3. Exactly-once delivery
   4. Best-effort delivery

   Answer: c. Exactly-once delivery guarantee only applies when subscribers connect in the same region.

3. A team needs to redact PII from messages before they are delivered to subscribers, without deploying additional infrastructure. Which Pub/Sub feature should they use?
   1. Subscription filters
   2. Dead-letter topics
   3. Single Message Transforms with JavaScript UDFs
   4. Schema validation

   Answer: c. SMTs with JavaScript UDFs can perform data masking and redaction directly within Pub/Sub.

4. Which subscription types does Pub/Sub’s exactly-once delivery support? (Choose TWO)
   1. Pull subscriptions
   2. Push subscriptions
   3. BigQuery subscriptions
   4. StreamingPull subscriptions
   5. Cloud Storage subscriptions

   Answer: a, d. Exactly-once delivery is supported for pull subscriptions including StreamingPull. Push and export subscriptions do not support it.
5. An organization using Pub/Sub Lite needs to migrate before the service is discontinued. What are the recommended migration targets? (Choose TWO)
   1. Cloud Tasks
   2. Standard Pub/Sub
   3. Cloud Scheduler
   4. Google Cloud Managed Service for Apache Kafka
   5. Eventarc

   Answer: b, d. Google recommends migrating Pub/Sub Lite workloads to standard Pub/Sub or Managed Service for Apache Kafka.

6. Where can Single Message Transforms (SMTs) be applied in Pub/Sub? (Choose TWO)
   1. At the project level
   2. On a topic
   3. On a subscription
   4. At the message attribute level only
   5. On the dead-letter topic

   Answer: b, c. SMTs can be applied independently to a topic, a subscription, or both.

References

• Google Cloud Pub/Sub Overview
• Choose a Subscription Type
• Exactly-Once Delivery
• Single Message Transforms Overview
• Import Topics – AWS Kinesis
• OpenTelemetry Tracing
• Choose Pub/Sub or Managed Service for Apache Kafka