ACID Transactions Are Overrated

For many years, the canonical example why we need database transactions has been banking. If you move $100, you don’t really want the money be subtracted from the first account, but never be added to the second because of some problem in between. You want both the subtraction and the addition to happen, or neither.

Sounds good so far. Just apparently that is not how banks work in the real world, and they certainly use enough database systems that have ACID transactions. The Economist (July 24, 2010, “Computer says no”) quotes a former executive of the Royal Bank of Scotland saying: “The reality was you could never be certain that anything was correct.” Continuing: “Reported numbers fot the bank’s exposure were regularly billions of dollars adrift of reality.” The article offers an explanation: “banks tend to operate lots of different databases producing conflicting numbers.” HSBC is quoted: 55 separate systems for core banking, 24 for credit cards, and 41 for internet banking.

According to traditional transaction wisdom, if a customer makes an internet transaction to pay off his credit card, it should be a single transaction: start transaction, take money from checking account, put it into credit card account, commit. But transactions generally cannot span systems. Because the system responsible for internet banking is separate in this real-world example from core banking and from credit card systems, no such single ACID transaction is possible. Given the numbers above, it looks very much like those money transfers that actually can follow the canonical ACID transaction pattern constitute only a very small fraction of all transactions (like transferring from checking to savings.)

If I look at my own banking, the vast majority of my banking activity isn’t even within the same bank, but with other banks: bills to pay usually have to be paid at other banks. No cross-bank ACID database transactions that I’ve ever heard of.

So banking software necessarily has to have functionality that prevents that money is deducted but never arrives, all without depending on database transactions. If we have to have this functionality anyway, why then are transactions “indispensable” as some people still want to make us believe?

This pattern can be generalized: the more distributed and decentralized a system is, the less likely it is that we can use transactions that span the entire system. That is certainly true for the banking system, apparently also true for systems inside banks, and in many other places. ACID transactions were invented for the mainframe, the world’s most centralized computing construct. But computing is not “one mainframe” any more I’m afraid as it was in the sixties.

Instead of trotting out transactions as the answer, what we need for NoSQL databases is the ability to get the same benefits in a distributed system (“nothing is lost”) without relying on transactions. That’s where all of our efforts should be.

In the InfoGrid graph database, we have a weaker form of transactions for individual MeshBases, but then synchronize with the rest of the world by passing XPRISO messages. I’d be surprised if the eventual “transactional” architecture for large-scale distributed and decentralized systems looked very different.

Comments:

  1. I worked on banking systems for a huge multinational bank for several years, though admittedly this was ten year ago. We certainly used database transactions for all internal bank transactions. (That sentence reminds me how much fun it wasn’t when talking about transactions to the non-technical managers).

    For transactions between banks having a strong audit trail was more important.

  2. This is basically bullshit.
    Of course transactions do not span systems – that would require XA and this is hard.
    But the usual case is that you decrease account balance and write transaction record according to that. Those 2 operation MUST be in the same transaction. And this is just a simplified example. In reality, it could be 10 SQL statements that you need to do for a banking operation and it would be a disaster is some of them fail but some persisted.

  3. @Banker: I fail to understand your point if you have one. And you fail to understand mine it appears.

  4. BeAShinigami says on August 5th, 2010 at 12:04 am:

    NoSql offer really interesting question and new solutions and also technical discussions, dense of meanings…
    You’re article is really interesting… and it’s a reality that distributed systems are changing… but not every where…
    In Italy for example there’s a lot of mainframe and a lot of web services but few really distributed systems for banks… The need for safeguards and to have someone to contact it’s always a big need and if your name isn’t IBM or Oracle or “Big-Someone”, you will not arrive to offer or install alternative solutions…
    Maybe, it’s also ’cause economic world crisis…
    However…
    i considere this article a starting point!
    We need answers! We need examples of how to reduce the transactional systems of old logic and persistence methods, in environments that are bigger/different than monothematic entity (twitter, facebook, so on…) or “front-end-cache-oriented” sites…
    Waiting for news!
    Thanks all for Your Advanced Works! :-D

  5. Leonardo says on August 5th, 2010 at 1:40 am:

    This is an interesting point. But in my understanding, a credit card payment don’t need to whitdraw money immediately, because the bank give “credit” to you, therefore it needs a different kind of check i.e. the limit of your card, the validity etc.
    In this scenario a transaction does not make sense, apart when your account is debited with the sum of all your transaction.
    Infact for all the other “debit” payment, I think that your account at YourbankDB must be at least in “synchronized” state. Don’t know if we can call this acid transaction, but I can see no option other than locking the db, be sure the operation succeed or not and unlocking.

    I agree that transaction may not be indispensable on daily basis. But there will be a time when someone will present you the bill and whitdraw money. So, in my opinion the core must be ACID, and those few core operations are much more indispensable and important than many of other bank transactions.

  6. For transactions between banks having a strong audit trail was more important.

    @BlackWasp, it is interesting to note that ACID transactions are based on logging (ie, a form of audit trail). So in other words, all transactions are based on logging of some form.

  7. ACID transactions make some things simple. If your system contains 95% database-internal transactions and only 5% transactions that involve a second database, then you can use ACID to simply 95% of the work significantly. ACID is great when failure is not an option – but as soon as you distribute data on multiple databases, failure will always be an option.

    That said, I agree that non-ACID solutions can be great, and will be used much more in the future – but mainly because the added overhead of dropping ACID, is paid by other kinds of improvements.

Leave a comment:

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>