https://www.udemy.com/course/postgresql-replication-high-availability-ha-and-scalability

Solutions for Scaling Postgres with Master-Slave Replication, PgBouncer, PgPool II, HAProxy, Partitioning, Sharding

What you'll learn: 

Assess your scaling needs

How to scale reads using Replication and Load-Balancing

Which is the best Replication solution for a certain use case

How to manage database connections with PgBouncer connection pooler

How to make use of multiple PostgreSQL instances in the cloud (Google Cloud)

How to achieve High-Availability

How to perform Automatic Failover using PgPool II

How to scale writes using Partitioning and Sharding

Requirements:

You need access to a Windows/Mac/Linux PC with 10GB of free disk space

Basic familiarity with database objects such as tables and indexes is expected

Some familiarity with Linux will be helpful

Description:

PostgreSQL is one of the most powerful and easy-to-use database management systems. It has strong support from the community and is being actively developed with a new release every year. 

PostgreSQL supports the most advanced features included in SQL standards. It also provides NoSQL capabilities and very rich data types and extensions. All of this makes PostgreSQL a very attractive solution in software systems. 

In this course, we discussed the problem of building scalable solutions based on PostgreSQL utilizing the resources of several servers. There is a natural limitation for such systems—basically, there is always a compromise between performance, reliability, and consistency. It's possible to improve one aspect, but others will suffer. In this course, we'll see how to find the best match for our use-cases so that we know eactly which aspects need scaling, and avoid the common trade-offs of distributed systems.

Scaling PostgreSQL is a journey. You should come out of this course more prepared to assess your scaling needs and understand how to scale reads and how to scale writes.

Each of this solution presented in this course will improve some aspect of the scalability topic, but each of them will add some complexity, and maybe some limitation or constraint.

We have to ask the right questions to get the system requirements, and this why we dedicated an entire lecture, so that we examine what questions we have to put ourself, before starting the Scaling Journey.

After this course, we should come out more prepared and understand how to scale reads.

We have several options for replication, depending on wether we favor performance or flexibility.

Replication can be used as a backup or a standby solution that would take over in case the main server crashes. 

Replication can also be used to improve the performance of a software system by making it possible to distribute the load on several database servers.

Then, if we have one sort of replication in place, we could ask ourself if we want to allow several computers to serve the same data.

To achieve this, we should have a mechanism to distribute the requests. We’ll see here two of the most popular options available.

Next, if the number of database connections is great, then we’ll probably want to use a connection pooler. Again, we’ll cover two options here.

We’ll also see, how to scale writes, and how to make your traffic growth more predictable by adding queuing to your architecture.

Then, we’ll check partitioning for those cases when we have to deal with big tables.

Also, we’ll check sharding to scale writes, and all the complex decisions that come with it.

Finally, we’ll see shortly the multi-master solution, which is a relatively new concept that seems to be promising.

If our goal is to achieve only High availability, or the ability to continue working even in the situation where one part of the cluster fails, we can check out only those solutions.

The pre-requirements for HA is to put in place a replication strategy.

Then, we can use tools to allow a second server to take over quickly, if the primary server fails.

Introduction to Scaling PostgreSQL

Why scale PostgreSQL?

What is Vertical Scaling?

What is Horizontal Scaling?

Read Versus Write Bound Workloads

Why Statistics are essential?

How to enable and make us of Statistics? (Hands-on)

How to scale Postgres for Reads?

How replication helps to scale out?

What are the Load-Balancers?

How to scale Postgres for Writes?

How to make use of Queues?

How could Partitioning and Sharding help in scaling out?

What is the Multi-Master solution about?

Understanding the Limitations of Scaling out PostgreSQL

CAP Theorem Explained 

PostgreSQL vs. Cassandra

Use case: CA Systems 

Use case: AP Systems

How to use Streaming Replication?

What is Streaming Replication? 

Asynchronous vs. Synchronous Replication 

How to Initialise Primary Database? (Hands-on)

How to Configuring the Primary for Replication?  (Hands-on)

How to Configuring the Replica Instance?  (Hands-on)

Testing Replication Setup  (Hands-on)

How to use Logical Replication?

What is Logical Replication in Postgres? 

Step by step Logical Replication setup 

How to setup the servers for Logical Replication? (Hands-on)

How to make a selective Copy of the Data? (Hands-on)

How to Create the Publication? (Hands-on)

How to Create the Subscription? (Hands-on)

Postgres Limitations of Logical Replication 

How to Monitoring Logical Replication? (Hands-on)

Best use-cases for using Logical Replication

How to make use of PgBouncer?

What is PgBouncer? 

Fundamental concepts of connection pooling 

How to build a PgBouncer Setup? (Hands-on)

How to install and configure PgBouncer? (Hands-on)

How to create a basic configuration file for PgBouncer? (Hands-on)

How to connect to PgBouncer? (Hands-on)

Explaining Advanced Settings for Performance 

Which are the available Pool Modes?

Executing a benchmark with PgBouncer (Hands-on)

How to scale PostgreSQL in Google Cloud?

Introduction 

Key Components on Google Cloud

Key Characteristics of the Architecture 

How to create PostgreSQL Instances on Google Cloud?  (Hands-on)

How to create a Google Cloud Engine (GCE) for HAProxy? (Hands-on)

How to configure HAProxy for Load-Balancing? (Hands-on)

Testing Load-Balancing

How to make use of PostgreSQL Partitioning?

What is Partitioning? 

Which Tables Need Partitioning? 

How should the Tables be Partitioned?  

Declarative vs. Inheritance Partitioning 

How to create a Partitioned Table? (Hands-on)

Partitioning Methods

How to Shard PostgreSQL?

What is Sharding?

Pain-Points of Sharding?

What is Second Level Sharding?

What is good Sharding?

How to query across multiple Shards?

How to setup High Availability (HA) on PostgreSQL?

Why High Availability? 

Steps to achieve High Availability 

Essential Questions to ask before setting-up High Availability

Log-Shipping Replication 

Streaming Replication and Logical Replication 

Cascading Replication 

Synchronous vs. Asynchronous Replication 

Automatic Failover and Always-on Strategy 

Simple HA Solution Example 

Better HA Solution Example

How to make use of PgPool II?

What is PgPool II?

Pgpool-II Features 

How to Configure Pgpool-II with Streaming Replication? (Hands-on)

How to setup Streaming Replication? (Hands-on)

How to Configuring Pgpool-II for Load Balancing ? (Hands-on)

Testing load-balancing & read/write separation (Hands-on)

How to Configure Pgpool for PostgreSQL High-Availability? (Hands-on)

How to Configure PostgreSQL Primary Server? (Hands-on)

How to Configure Pgpool-II Server? (Hands-on)

How to Configure PostgreSQL Replica Server? (Hands-on)

Testing The Failover (Hands-on)

How to restore failed nodes? (Hands-on)

Who this course is for:Software Engineers interested in designing Scalable and HA solutions on top of PostgreSQLDatabase AdministratorsEveryone interested in building better PostgreSQL applications

Who this course is for:

Software Engineers interested in designing Scalable and HA solutions on top of PostgreSQL

Database Administrators

Everyone interested in building better PostgreSQL applications