I would like to first surface with a fun fact that during my time as a undergraduate, I took an algorithms class under a guy named Seth Gilbert.  This was during my first year in college and I was rather naive. But it turns out, he was the person that proved formally the CAP theorem. I only found this out when i took my distributed systems class (which i really enjoyed) and it really is a surreal experience to reflect on the opportunities i received. The CAP theorem is so influential to this day such that it governs system design for all software engineers, and tell us that tradeoff has always and will always be the heart of computer science. Now, **the blog**.

Lets describe what is CAP Theorem. CAP is an acronym that stands for Consistency, Availability and Partition Tolerance.  The theorem tells us that out of the 3 conditions, we can only choose 2. 

**Consistency (C):** Every read request receives the **most recent write** or an error. All working nodes in the distributed system show the **exact same data** at the same time.

**Availability (A):** Every non-failing node returns a **non-error response** to every single request. 

**Partition Tolerance (P):** The system continues to operate despite an **arbitrary number of messages** being dropped or delayed by the network. 

Fun fact, I was once interviewing for a Database role, and the interviewer asked me on the spot, to show it, as such I found that it can very easily be explained with a simple drawing and a mental model about what the theorem actually is.  Let us demonstrate this clearly through our drawings below.

**A server setup that is distributed (has partition tolerance) but not consistent**

![AP](/uploads/20260912-235615.png "AP")

In this case, each box represents the same server in a distributed system. In our example, a server has a network partition from the rest of the server and as such, no updates is ever received on the creation of the profile of "Sean". This leads to a problem where Bob is unable to retrieve "Sean" profile. Thus, in this case as a counter example, the system can show different data on different servers.

**Has consistency and partiton tolerance, but no availability**

![CP](/uploads/20260913-000113.png "AP")

In this example, we then try to make it consistent.  That is every server in the system has to show the same data, which requires the network partition to be removed and our request succeed. However, since there is a partition, we cannot show Bob his request, and thus we need to return an error, violating the definition of availability. 

**Has Consistency and Availability (But not partition tolerance)**

![CA](/uploads/20260913-000830.png "CA")

In this last example, the only way we can have consistency and availability is to have a single server.  We can show that we do not have any partition tolerance since if any message like "Get profile of Sean" is lost, then the system will never know, and will never return a response.  However, the server will always be able to show a consistent view of the data, since it receives the data first hand. It also satisfies availability since if it is non-failing, it will respond.

As such, we have informally demostrated the CAP theorem tradeoff in a distributed system. We will now move on to distributed transactions.

**What is 2PL**

2PL stands for 2 Phase Locking and is one of, if not the most famous method that ensures conflict serializability in a ACID database. A brief idea to touch on this, is you have 2 phases as the name suggest

1. **Growing Phase**: The transaction starts to acquire all the lock it requires , no locks can be released once acquired
2. **Shrinking Phase**: Once the transaction operations are carried out, all locks are released, no more locks are acquired

Think of it using this analogy. You have 3 artwork (A, B, C), and in your first operation, you need to paint 2 (B, C). So in order, to ensure no other person alters your artwork, you take it with you and lock it in a room. If you were to leave it,  say you only lock B, there is a chance your friend, "Bob" might use his paint brush to amend it. Hence, the reason for the rule of no locks can be released! This analogy shows how to effectively isolate the needed rows, so that no 2 concurrent writes occur.

**What is a transaction**

A transaction is a single logical unit of work that bundles one or more  operations (like reads, writes, updates, or deletes) into an all-or-nothing execution. 

**Core Motivation**

In E-commerce, there are many operations that occur as a result of a customer's order. There is `payment,  accounting, inventory.`Since there is so many services, in order to scale and hit tens or hundred of millions of users, there might be a need to horizontally scale (create multiple servers) in order to handle the load such that our services are micro-services.

To demonstrate a need for a transaction, here is an example: A bag has a stock of one  and  is being checkout. We have a orchestrator service to coordinate between various microservicea sudden flood hits the warehouse and the item is damaged.  The order is no longer valid, therefore everything else should be cancelled and reversed - All or nothing.

Now, the naive idea here is to use **2PL**  and treat the whole flow like a transaction such that, it either fully completes or fails. This is done by a coordinating orchestrator between services.

![](/uploads/20260913-003936.png)

The checkout event triggers each of the different service, and continues to proceed. The payment service, inventory service and accounting service downstream services will have to lock data until the transaction either fails or is complete to prevent any other transactions can modify them and each service durably records its operation.

In the event, that any single service fails and chooses to abort, the coordinator must thus abort the whole transaction

![](/uploads/20260913-004324.png)

**Problem**: each service might take different times, for example if inventory service requires 5sec versus payment service which requires 1sec, then the whole transaction must wait for 5 seconds. Further more, what if the user wants to check out another piece again? and clicks checkout on the same item before transaction finishes for the same exact item. The user will have to wait , and it is a pretty bad user experience.

2PL is a blocking protocol, and assumes that all system must be healthy at the same time. A server going down, can also mean the transaction might go on indefinitely and in a big tech company, services are handled by different team and possibly different infrastructure

**A compromise: SAGA pattern**

Instead of a all-or-nothing, we guarantee a looser form, eventual consistency. That is, eventually, the system will become consistent, there is no gaurantee how long it will take though. Break the work into independent local transaction for each service. If work is committed, we will run a "**compensating action**", which are business level undo that undos all effects of the commit.

**Two patterns: Choreography vs Orchestration**

**Choreography: Pub-Sub architecture**

![](/uploads/20260913-005106.png)

In choreography, we allow services to listen to events, using a message broker and react accordingly. In this case , **Payment service** might start processing and then pushes a **CARD CHARGED** , the **Inventory service** listens and sees the event, and starts to process with the item details, and pushes **INVENTORY RESERVED.** A failing service publishes a failing event, upstream services will run compensating actions to reverse their own actions

![](/uploads/20260913-005430.png)

The problem as hello interview mentions, is when the steps to a transaction has increasing steps, like 2 or 3 steps is fine, but 5 or 6, figuring out where the transaction is becomes difficult. The point of failure, and the number of compensating actions carried out becomes harder to see.  Additionally, compensating actions might fail. Thus in order to debug, you need to dig through multiple logs across services. We assume there is no central server or orchestrator, just a queue like kafka here to publish events.

**Orchestration**: The alternative

Get an orchestrator that hands the steps accordingly to the right services. Only when it is successful, does the orchestration call the next service. This solves the problem whereby you know exactly the point of failure, and you can run compensating action for the affected service only.

E.g Temporal / AWS Step functions

![](/uploads/20260913-005829.png)

During a crash, the orchestrator is durable, when it restarts, it will **read its own logs from its database** and pick up where it left off. There are no locks on any rows, and no transactions are ever blocked. Now, other services using **payment service, notification service** can continue to function, like in-house dashboard etc.

**Complexity:** The compensating action is the trade off we made, because for payment system, it could be that once you trigger that action, it becomes visible to the user, and has some repurcussion, since it is outside your control.  For example, **refunds** which are visible to the user, and might result in banks deducting, or holding that money as well as sending push notifications. In this case, you have to deem if it is acceptable.

Additionally, there are actions that cannot be easily undone, for example a trigger of an external api which starts a flow, while you can send an additional request to cancel it, there is no guarantee.

Lastly, compensation actions can also fail, now you need to retry compensating actions. For example, refunds. Actions now, need to be **idempotent**. There is a need for reliability for your error cases now , adding complexity to it as well.

**Duo write problems** - within service

There needs to be 2 things that are often done concurrently, which is writing the event and also to your own database to commit.  This needs to be atomic, because if your DB update works but ur update to your message broker fails, then the next event in SAGA will not be triggered. Similarly, event updates can succeed while DB fail, so your flow should fail, but further services are triggered. Hence there is a need for this to be **atomic**

![](/uploads/20260913-014604.png)

**Transactional Outbox**

Instead of 2 separate operations, to make it atomic, use the Database Transaction to help us! Updatethe event into the transaction commit onto a outbox table, use a background process to watch the outbox table and publish the event to message table. Can use Change Data Capture (CDC) that tails database transaction logs, or pull outbox table on a regular interval. 

**Decision Framework**

If you can design your service boundary such that your data that transact together are stored together in the same database, always use it because it is **ALWAYS easier and more recommended** than handling distributed transactions. If you can't then use SAGA  for distributed transactions. For whether to choose orchestration or choreography, it depends on the scale. 

For simple services, choreography is easier to set up and when there is  no need for centralized visibility.  For more complex service, orchestration might be better. 

Key learnings: Do not overengineer,  ultimately it is about trade offs , we should not pick anything just because its an industrial standard. Ultimately, it is the use case, that defines our decision, and often also the estimated 5 - 10 years growth ahead (foresight). 

Lastly, personally from me, the idea of horizontally scaling, microservices has been the way to scale for any large size tech firms. But talk to most distributed systems expert, and you will find out that the best way to build a system is TO NOT MAKE IT DISTRIBUTED IN THE FIRST PLACE!  Scale vertically then horizontally, **consensus** after all is not a trivial problem featuring **Leslie Lamport!** 

Do you really need to horizontally scale: [https://openai.com/index/scaling-postgresql/](https://openai.com/index/scaling-postgresql/)

For this blog post, no AI was used, everything was typed by me.

Sincerely,

Sean