Google Cloud Spanner

Google Cloud Spanner

  • Cloud Spanner is a fully managed, mission-critical relational database service
  • Cloud Spanner provides a scalable online transaction processing (OLTP) database with high availability and strong consistency at a global scale.
  • Cloud Spanner provides traditional relational semantics like schemas, ACID transactions and SQL interface
  • Cloud Spanner provides Automatic, Synchronous replication within and across regions for high availability (99.999%)
  • Cloud Spanner benefits
    • OLTP (Online Transactional Processing)
    • Global scale
    • Relational data model
    • ACID/Strong or External consistency
    • Low latency
    • Fully managed and highly available
    • Automatic replication

Cloud Spanner Architecture

Cloud Spanner ArchitectureInstance

  • Cloud Spanner Instance determines the location and the allocation of resources
  • Instance creation includes two important choices
    • Instance configuration
      • determines the geographic placement i.e. location and replication of the databases
      • Location can be regional or multi-regional
      • cannot be changed once selected during the creation
    • Node count
      • determines the amount of the instance’s serving and storage resources
      • can be updated
  • Cloud Spanner distributes an instance across zones of one or more regions to provide high performance and high availability
  • Cloud Spanner instances have:
    • At least three read-write replicas of the database each in a different zone
    • Each zone is a separate isolation fault domain
    • Paxos distributed consensus protocol used for writes/transaction commits
    • Synchronous replication of writes to all zones across all regions
    • Database is available even if one zone fails (99.999% availability SLA for multi-region and 99.99% availability SLA for regional)

Regional vs Multi-Regional

  • Regional Configuration
    • Cloud Spanner maintains 3 read-write replicas, each within a different Google Cloud zone in that region.
    • Each read-write replica contains a full copy of the operational database that is able to serve read-write and read-only requests.
    • Cloud Spanner uses replicas in different zones so that if a single-zone failure occurs, the database remains available.
    • Every Cloud Spanner mutation requires a write quorum that’s composed of a majority of voting replicas. Write quorums are formed from two out of the three replicas in regional configurations.
    • Provides 99.99% availability
  • Multi-Regional Configuration
    • Multi-region configurations allow replicating the database’s data not just in multiple zones, but in multiple zones across multiple regions
    • Additional replicas enable reading data with low latency from multiple locations close to or within the regions in the configuration.
    • As the quorum (read-write) replicas are spread across more than one region, additional network latency is incurred when these replicas communicate with each other to vote on writes.
    • Multi-region configurations enable the application to achieve faster reads in more places at the cost of a small increase in write latency.
    • Provides 99.999% availability
    • Multi-regional makes use of, paxos based replication, TrueTime and leader election, to provide global consistency and higher availability

Cloud Spanner - Regional vs Multi-Regional Configurations

Replication

  • Cloud Spanner automatically gets replication at the byte level from the underlying distributed filesystem.
  • Cloud Spanner also performs data replication to provide global availability and geographic locality, with fail-over between replicas being transparent to the client.
  • Cloud Spanner creates multiple copies, or “replicas,” of the rows, then stores these replicas in different geographic areas.
  • Cloud Spanner uses a synchronous, Paxos distributed consensus protocol, in which voting replicas take a vote on every write request to ensure transactions are available in sufficient replicas before being committed.
  • Globally synchronous replication gives the ability to read the most up-to-date data from any Cloud Spanner read-write or read-only replica.
  • Cloud Spanner creates replicas of each database split
  • A split holds a range of contiguous rows, where the rows are ordered by the primary key.
  • All of the data in a split is physically stored together in the replica, and Cloud Spanner serves each replica out of an independent failure zone.
  • A set of splits is stored and replicated using Paxos.
  • Within each Paxos replica set, one replica is elected to act as the leader.
  • Leader replicas are responsible for handling writes, while any read-write or read-only replica can serve a read request without communicating with the leader (though if a strong read is requested, the leader will typically be consulted to ensure that the read-only replica has received all recent mutations)
  • Cloud Spanner automatically reshards data into splits and automatically migrates data across machines (even across datacenters) to balance load, and in response to failures.
  • Spanner’s sharding considers the parent child relationships in interleaved tables and related data is migrated together to preserve query performance

Cloud Spanner Data Model

  • A Cloud Spanner Instance can contain one or more databases
  • A Cloud Spanner database can contain one or more tables
  •  Tables look like relational database tables in that they are structured with rows, columns, and values, and they contain primary keys
  • Every table must have a primary key, and that primary key can be composed of zero or more columns of that table
  • Parent-child relationships in Cloud Spanner
    • Table Interleaving
      • Table interleaving is a good choice for many parent-child relationships where the child table’s primary key includes the parent table’s primary key columns
      • Child rows are colocated with the parent rows significantly improving the performance
      • Primary key column(s) of the parent table must be the prefix of the primary key of the child table
    • Foreign Keys
      • Foreign keys are similar to traditional databases.
      • They are not limited to primary key columns, and tables can have multiple foreign key relationships, both as a parent in some relationships and a child in others.
      • The foreign key relationship does not guarantee data co-location
  • Cloud Spanner automatically creates an index for each table’s primary key
  • Secondary indexes can be created for other columns

Cloud Spanner Scaling

  • Increase the compute capacity of the instance to scale up the server and storage resources in the instance.
  • Each node allows for an additional 2TB of data storage
  • Nodes provide additional compute resources to increase throughput
  • Increasing compute capacity does not increase the replica count but gives each replica more CPU and RAM, which increases the replica’s throughput (that is, more reads and writes per second can occur).

Cloud Spanner Backup & PITR

  • Cloud Spanner Backup and Restore helps create backups of Cloud Spanner databases on demand, and restore them to provide protection against operator and application errors that result in logical data corruption.
  • Backups are highly available, encrypted, and can be retained for up to a year from the time they are created.
  • Cloud Spanner point-in-time recovery (PITR) provides protection against accidental deletion or writes.
  • PITR works by letting you configure a database’s version_retention_period to retain all versions of data and schema, from a minimum of 1 hour up to a maximum of 7 days.

Cloud Spanner Best Practices

  • Design a schema that prevents hotspots and other performance issues.
  • For optimal write latency, place compute resources for write-heavy workloads within or close to the default leader region.
  • For optimal read performance outside of the default leader region, use staleness of at least 15 seconds.
  • To avoid single-region dependency for the workloads, place critical compute resources in at least two regions.
  • Provision enough compute capacity to keep high priority total CPU utilization under
    • 65% in each region for regional configuration
    • 45% in each region for multi-regional configuration

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your customer has implemented a solution that uses Cloud Spanner and notices some read latency-related performance issues on one table. This table is accessed only by their users using a primary key. The table schema is shown below. You want to resolve the issue. What should you do?
    1. Remove the profile_picture field from the table.
    2. Add a secondary index on the person_id column.
    3. Change the primary key to not have monotonically increasing values.
    4. Create a secondary index using the following Data Definition Language (DDL) CREATE INDEX person_id_ix ON Persons (
      person_id, firstname, lastname ) STORING ( profile_picture )
  2. You are building an application that stores relational data from users. Users across the globe will use this application. Your CTO is concerned about the scaling requirements because the size of the user base is unknown. You need to implement a database solution that can scale with your user growth with minimum configuration changes. Which storage solution should you use?
    1. Cloud SQL
    2. Cloud Spanner
    3. Cloud Firestore
    4. Cloud Datastore
  3. A financial organization wishes to develop a global application to store transactions happening from different part of the world. The storage system must provide low latency transaction support and horizontal scaling. Which GCP service is appropriate for this use case?
    1. Bigtable
      B Datastore
      C Cloud Storage
      D Cloud Spanner

References

Google_Cloud_Spanner

Google Cloud Storage Options

GCP Storage Options

GCP provides various storage options and the selection can be based on

  • Structured vs Unstructured
  • Relational (SQL) vs Non-Relational (NoSQL)
  • Transactional (OLTP) vs Analytical (OLAP)
  • Fully Managed vs Requires Provisioning
  • Global vs Regional
  • Horizontal vs Vertical scaling

Cloud Firestore

  • Cloud Firestore is a fully managed, highly scalable, serverless, non-relational NoSQL document database
  • fully managed with no-ops and no planned downtime and no need to provision database instances (vs Bigtable)
  • uses a distributed architecture to automatically manage scaling.
  • queries scale with the size of the result set, not the size of the data set
  • supports ACID Atomic transactionsall or nothing (vs Bigtable)
  • provides High availability of reads and writesruns in Google data centers, which use redundancy to minimize impact from points of failure.
  • provides massive scalability with high performanceuses a distributed architecture to automatically manage scaling.
  • scales from zero to terabytes with flexible storage and querying of data
  • provides SQL-like query language
  • supports strong consistency
  • supports data encryption at rest and in transit
  • provides terabytes of capacity with a maximum unit size of 1 MB per entity (vs Bigtable)
  • Consider using Cloud Firestore if you need to store semi-structured objects, or if require support for transactions and SQL-like queries.

Cloud Bigtable

  • Bigtable provides a scalable, fully managed, non-relational NoSQL wide-column analytical big data database service suitable for both low-latency single-point lookups and precalculated analytics.
  • supports large quantities (>1 TB) of semi-structured or structured data (vs Datastore)
  • supports high throughput or rapidly changing data (vs BigQuery)
  • managed, but needs provisioning of nodes and can be expensive (vs Datastore and BigQuery)
  • does not support transactions or strong relational semantics (vs Datastore)
  • does not support SQL queries (vs BigQuery and Datastore)
  • Not Transactional and does not support ACID
  • provides eventual consistency
  • ideal for time-series or natural semantic ordering data
  • can run asynchronous batch or real-time processing on the data
  • can run machine learning algorithms on the data
  • provides petabytes of capacity with a maximum unit size of 10 MB per cell and 100 MB per row.
  • Usage Patterns
    • Low-latency read/write access
    • High-throughput data processing
    • Time series support
  • Anti Patterns
    • Not an ideal storage option for future analysis – Use BigQuery instead
    • Not an ideal storage option for transactional data – Use relational database or Datastore
  • Common Use cases
    • IoT, finance, adtech
    • Personalization, recommendations
    • Monitoring
    • Geospatial datasets
    • Graphs
  • Consider using Cloud Bigtable, if you need high-performance datastore to perform analytics on a large number of structured objects

Cloud Storage

  • Cloud Storage provides durable and highly available object storage.
  • fully managed, simple administration, cost-effective, and scalable service that does not require capacity management
  • supports unstructured data storage like binary or raw objects
  • provides high performance, internet-scale
  • supports data encryption at rest and in transit
  • Consider using Cloud Storage, if you need to store immutable blobs larger than 10 MB, such as large images or movies. This storage service provides petabytes of capacity with a maximum unit size of 5 TB per object.
  • Usage Patterns
    • Images, pictures, and videos
    • Objects and blobs
    • Unstructured data
    • Long term storage for archival or compliance
  • Anti Patterns
  • Common Use cases
    • Storing and streaming multimedia
    • Storage for custom data analytics pipelines
    • Archive, backup, and disaster recovery

Cloud SQL

  • provides fully managed, relational SQL databases
  • offers MySQL, PostgreSQL, MSSQL databases as a service
  • manages OS & Software installation, patches and updates, backups and configuring replications, failover however needs to select and provision machines (vs Cloud Spanner)
  • single region only – although it now supports cross-region read replicas (vs Cloud Spanner)
  • Scaling
    • provides vertical scalability (Max. storage of 10TB)
    • storage can be increased without incurring any downtime
    • provides an option to increase the storage automatically
    • storage CANNOT be decreased
    • supports Horizontal scaling for read-only using read replicas (vs Cloud Spanner)
    • performance is linked to the disk size
  • Security
    • data is encrypted when stored in database tables, temporary files, and backups.
    • external connections can be encrypted by using SSL, or by using the Cloud SQL Proxy.
  • High Availability
    • fault-tolerance across zones can be achieved by configuring the instance for high availability by adding a failover replica
    • failover is automatic
    • can be created from primary instance only
    • replication from the primary instance to failover replica is semi-synchronous.
    • failover replica must be in the same region as the primary instance, but in a different zone
    • only one instance for every primary instance allowed
    • supports managed backups and backups are created on primary instance only
    • supports automatic replication
  • Backups
    • Automated backups can be configured and are stored for 7 days
    • Manual backups (snapshots) can be created and are not deleted automatically
  • Point-in-time recovery
    • requires binary logging enabled.
    • every update to the database is written to an independent log, which involves a small reduction in write performance.
    • performance of the read operations is unaffected by binary logging, regardless of the size of the binary log files.
  • Usage Patterns
    • direct lift and shift for MySQL, PostgreSQL, MSSQL database only
    • relational database service with strong consistency
    • OLTP workloads
  • Anti Patterns
    • need data storage more than 10TB, use Cloud Spanner
    • need global availability with low latency, use Cloud Spanner
    • not a direct replacement for Oracle use installation on GCE
  • Common Use cases
    • Websites, blogs, and content management systems (CMS)
    • Business intelligence (BI) applications
    • ERP, CRM, and eCommerce applications
    • Geospatial applications
  • Consider using Cloud SQL for full relational SQL support for OTLP and lift and shift of MySQL, PostgreSQL databases

Cloud Spanner

  • Cloud Spanner provides fully managed, relational SQL databases with joins and secondary indexes
  • provides cross-region, global, horizontal scalability, and availability
  • supports strong consistency, including strongly consistent secondary indexes
  • provides high availability through synchronous and built-in data replication.
  • provides strong global consistency
  • supports database sizes exceeding ~2 TB (vs Cloud SQL)
  • does not provide direct lift and shift for relational databases (vs Cloud SQL)
  • expensive as compared to Cloud SQL
  • Consider using Cloud Spanner for full relational SQL support, with horizontal scalability spanning petabytes for OTLP

BigQuery

  • provides fully managed, no-ops,  OLAP, enterprise data warehouse (EDW) with SQL and fast ad-hoc queries.
  • provides high capacity, data warehousing analytics solution
  • ideal for big data exploration and processing
  • not ideal for operational or transactional databases
  • provides SQL interface
  • A scalable, fully managed
  • Usage Patterns
    • OLAP workloads up to petabyte-scale
    • Big data exploration and processing
    • Reporting via business intelligence (BI) tools
  • Anti Patterns
    • Not an ideal storage option for transactional data or OLTP – Use Cloud SQL or Cloud Spanner instead
    • Low-latency read/write access – Use Bigtable instead
  • Common Use cases
    • Analytical reporting on large data
    • Data science and advanced analyses
    • Big data processing using SQL

Memorystore

  • provides scalable, secure, and highly available in-memory service for Redis and Memcached.
  • fully managed as provisioning, replication, failover, and patching are all automated, which drastically reduces the time spent doing DevOps.
  • provides 100% compatibility with open source Redis and Memcached
  • is protected from the internet using VPC networks and private IP and comes with IAM integration
  • Usage Patterns
    • Lift and shift migration of applications
    • Low latency data caching and retrieval
  • Anti Patterns
    • Relational or NoSQL database
    • Analytics solution
  • Common Use cases
    • User session management

GCP Storage Options Decision Tree

GCP Storage Options Decision Tree

GCP Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • GCP services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • GCP exam questions are not updated to keep up the pace with GCP updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Your application is hosted across multiple regions and consists of both relational database data and static images. Your database has over 10 TB of data. You want to use a single storage repository for each data type across all regions. Which two products would you choose for this task? (Choose two)
    1. Cloud Bigtable
    2. Cloud Spanner
    3. Cloud SQL
    4. Cloud Storage
  2. You are building an application that stores relational data from users. Users across the globe will use this application. Your CTO is concerned about the scaling requirements because the size of the user base is unknown. You need to implement a database solution that can scale with your user growth with minimum configuration changes. Which storage solution should you use?
    1. Cloud SQL
    2. Cloud Spanner
    3. Cloud Firestore
    4. Cloud Datastore
  3. Your company processes high volumes of IoT data that are time-stamped. The total data volume can be several petabytes. The data needs to be written and changed at a high speed. You want to use the most performant storage option for your data. Which product should you use?
    1. Cloud Datastore
    2. Cloud Storage
    3. Cloud Bigtable
    4. BigQuery
  4. Your App Engine application needs to store stateful data in a proper storage service. Your data is non-relational database data. You do not expect the database size to grow beyond 10 GB and you need to have the ability to scale down to zero to avoid unnecessary costs. Which storage service should you use?
    1. Cloud Bigtable
    2. Cloud Dataproc
    3. Cloud SQL
    4. Cloud Datastore
  5. A financial organization wishes to develop a global application to store transactions happening from different part of the world. The storage system must provide low latency transaction support and horizontal scaling. Which GCP service is appropriate for this use case?
    1. Bigtable
    2. Datastore
    3. Cloud Storage
    4. Cloud Spanner
  6. You work for a mid-sized enterprise that needs to move its operational system transaction data from an on-premises database to GCP. The database is about 20 TB in size. Which database should you choose?
    1. Cloud SQL
    2. Cloud Bigtable
    3. Cloud Spanner
    4. Cloud Datastore