InfoGrid 2.9.5 Released

It’s been a while since the last release, but why change if users are been generally happy with it ;-) … Here’s what’s new:


  • Native support for currency values (e.g. $1.23 USD)
  • Even simpler JSON generation
  • StringDataTypes can now carry a regular expression, which is checked before assignment (e.g. makes it impossible to assign a String that isn’t to a property that must contain an e-mail address)
  • Support for cascading delete
  • InfoGrid compilation is now reported to work on Windows
  • Multi-tenancy support for LID
  • More flexible handling of HTTP status codes in Probe framework (e.g. what should happen upon a redirect)
  • Extensions to custom JEE tags and HttpShell to make JSP programming with InfoGrid even simpler, e.g. simple tags to create Javascript-based overlay forms
  • Improved error reporting
  • Additional tests
  • Lots of small usability improvements and bug fixes

Here are the gory details:


  • ModuleAdvertisementSerializer to generate relative paths to JARs
  • more types of MeshObjectSelector
  • added getLocalId to NetMeshObjectIdentifier
  • added guessFromExternalForm to IdentifierFactory
  • use “” not null for localId of home object
  • Augmented set of pre-defined BlobDataTypes
  • Improved BlobDataType to check for MIME compatibility with regexes
  • added CurrencyValue and CurrencyDataType to represent money as a primitive type
  • added convenience method to MeshObject: determine whether the relationship to a neighbor MeshObject is blessed with a certain RoleType
  • Moved OnDemandTransaction from HttpShell into utils
  • implemented cascading delete support
  • added TimeStampValue.earlierOf and TimeStampValue.laterOf for convenience
  • Special value NOW for TimeStampValue parsing
  • regex can be be specified for StringDataTypes
  • added AbstractDelegatingLocalizedRuntimeException for symmetry and completeness
  • better error reporting for TransactionActionExceptions
  • Explicitly set latin1 character set in MySQL store to avoid key size error when the default character set is multi-byte.
  • TransactionAction.execute() now does not take a Transaction parameter any more, but provides it as a member variable
  • check raw id even when guessing MeshObjectIdentifier
  • when there’s a ParseException during parsing of store content, don’t return null but throw NoSuchElementException
  • Added MeshObjectHelper for simpler JSP programming with TypedMeshObjectFacades
  • Carry specify exceptions keyed by failed MeshObjectIdentifiers in MeshObjectAccessException, instead of swallowing all exceptions other than the first
  • MeshObjectsNotFoundException is not a subclass of MeshObjectAccessException any more
  • Correct singular/plural error messages for MeshObjectAccessException and MeshObjectsNotFoundException
  • NetMeshObjectAccessException now carries redirect NetMeshObjectAccessSpecifications
  • NetMeshObjectAccessSpecification is now an Identifier
  • Migrated sweeping functionality from the various MeshBase implementations into Sweeper. The policy functionality of Sweeper became SweepPolicy.
  • Better pingpong debugging support
  • created MeshObjectSet.hasSameContent
  • upgrade to warn if MySQL database cannot be detected/created
  • Id column in MySQL needs to be case-sensitive
  • don’t report exception in rollback that isn’t an error
  • default expiration for UnnecessaryReplicasSweepPolicy
  • Allow the sweep of a single MeshObject
  • MeshObject.delete and sub-methods have no business to throw NotPermittedException
  • Non-semantic “kill” of a ShadowMeshBase
  • Kill button in ShadowAwareAllMeshBasesViewlet
  • No need for public method initiateCeaseCommunications on Proxy
  • Don’t try to talk to Proxies when NetMeshBase is dead
  • Tolerate some IsDeadExceptions when dead (duh)
  • Use default mime type for BlobValue when none is given when parsing StringRepresentation
  • Avoid that IsDeadException kills meaningful error message
  • support typefield XML attribute for StringDataTypes as well
  • added StringDataType for e-mail
  • fixed automatic merge problem
  • allow MeshObject.equals( TypedMeshObjectFacade ) for simplicity
  • use get_XXX pattern in TypedMeshObjectFacade for non-Properties/non-pseudo-Properties to avoid name clashes with code-generated setters
  • use get_XXX pattern in TypedMeshObjectFacade for non-Properties/non-pseudo-Properties to avoid name clashes with code-generated setters
  • renamed Rfc3339Util to DateTimeUtil and expanded to W3C timestamps
  • DefaultNetMeshBaseIdentifierFactory now uses pluggable “Scheme”s to support identifiers of various protocols, including http, https, file, xri, acct
  • renamed UnknownProtocolParseException to UnknownSchemeParseException
  • added ability to specify Maps in Resource files
  • When parsing NetMeshBaseIdentifiers, turn each String into canonical form as well, e.g. hTTp: and http:
  • added toColloquialExternalForm() to Identifier; more sane
  • removed IdentifierStringifier.toColloquialIdentifier()
  • use guessFromExternalForm in more places for more tolerance (e.g. when reading from Store to be more tolerant of Scheme changes)
  • be more tolerant of Exceptions when restarting MeshBases; try best-effort
  • Better logging of start / die of InfoGridWebApps
  • when resetting PingPong, only increment token by 1
  • more defensive programming if Storage layer is corrupt
  • refactor message sending framework to always deliver all received messages together instead of singly
  • merge two XprisoMessages before processing if possible
  • added ByPropertyValueSelector and PropertyComparisonOperator
  • NotRelatedException not thrown any more when testing whether two MeshObjects are related by a given RoleType; was counter-intuitive and pedantic
  • StringRepresentationParameters must always be given.
  • Make it easier to generate MeshObjectIdentifiers with different lengths
  • upgraded Postgresql JDBC for 64bit support
  • Netbeans 7 support
  • Make sure Threadpool is always cleanly shut down. Now Tomcat won’t hang occasionally after undeploy.
  • support custom regex violation error messages for PropertyTypes that specify regular expressions
  • refactored L10Map into L10PropertyValueMap and L10StringMap, moved to utils package
  • more HTML class declarations in BlobValue HTML
  • added invert method to most MeshObjectSelectors
  • added MeshObjectSet.subset convenience method
  • Don’t include dead MeshObjects in MeshObjectSets
  • factored out ExternalCommand so invoking an external process becomes simpler
  • default MeshObjectIdentifier to contain _ instead of ~


  • AccountManager needs a site parameter for determining account to support multi-tenancy
  • generate login event even if only the session gets renewed
  • XrdsProbe to cache the XRDS file
  • added content and content type as parameters to XmlProbe interface; enables XrdsProbe to cache the XRDS file
  • added getGroupNames to LidAccount
  • check group membership by group identifier as well as group name
  • also check group membership by groups set as request attribute
  • factored out some symbolic OpenID parameter names
  • fix when associations expire
  • Better integrated AccessManager and ThreadIdentityManager
  • root may do anything in AclBasedAccessManager
  • Split AclBasedSecurity into two: an ACL-based AccessManager implementation that does not rely on Guards, and an AccessManager implementation that evaluates Guards but is independent on a particular model (such as the AclBasedSecurity model)
  • Eliminated org.infogrid.model.AclBasedSecurity, and instead introduced, which also includes a model
  • DelegatingAccessManager now supports delegating to multiple other AccessManagers, all of which must agree
  • added ThreadIdentityManager.suExec
  • When there isn’t a ProtectionDomain, it’s free for all in AclBasedAccessManager
  • moved org.infogrid.model.SecurityTest into the attic; currently not used
  • added expiring credentials to AuthenticationStatus
  • expired OpenID associations throw OpenIdAssociationExpiredException
  • OpenID dumb mode fixes
  • Added getCarriedExpiredCredentialTypes to LidClientAuthenticationStatus
  • added org.infogrid.lid.model.account/Account_LastLoggedIn which can come in handy
  • Removed Account Status value CREATED — not particularly useful and would have needed update logic
  • credential values are now separate from LidAccount, e.g. the StoreLidPasswordCredentialType knows where to look for the password
  • don’t need LidCredentialType.isRemote any more
  • allow password change from StoreLidPasswordCredentialType
  • removed realm from LID; never quite used
  • better support for multi-tenancy / virtual hosts in LID: carry siteIdentifier around in more places
  • distinguish one-time-tokens from other types of credentials in LID
  • more methods for setting, checking, resetting password credentials
  • support that MeshObject can own itself in AclUtils
  • A little helper utility to generate secret keys


  • Code-generate an entry for the SubjectArea itself, too
  • exactly one SubjectArea element is expected in each model element in the model XML
  • Added HTTP status code and location properties to WebResource
  • minor rename of SubjectArea user-visible name


  • Handle HTTP redirects better when encountered by Probes; existing behavior should be unchanged however
  • Do not throw FactoryException as a direct cause of NetMeshObjectAccessException but its cause instead
  • Introduced ProbeException.HttpRedirectResponse
  • Added NetMeshBaseRedirectException, so ProbeException.HttpRedirectResponse does not need to be moved up to module
  • Introduced HttpMappingPolicy that enables different policies for mapping HTTP non-OK status codes into InfoGrid; default remains the same behavior
  • Change in API to set up ProbeManagers etc.: ProbeDirectory passed to ProbeManager instead of ShadowMeshBaseFactory, then ShadowMeshBaseFactory is notified where ProbeManager can be found
  • ProbeException.HttpErrorResponse not needed any more
  • Support HTTP redirects that give relative URLs as Location, against the HTTP spec (I’m looking at you, Google)
  • HttpMappingPolicy has moved from ProbeManager to ProbeDirectory
  • Fixes issue where wrong HttpMappingPolicy was set upon restore of Shadow from disk
  • Throw exception if ProbeManager is configured without a ProbeDirectory
  • added basic XRD support to InfoGrid
  • added acct: as one of the basic protocols supported by NetMeshBaseIdentifier
  • added Webfinger support
  • Don’t throw ProbeException.DontHaveNonXmlStreamProbe if the found content is of length zero


  • Renamed NotIfViewletState to NotIfViewletStateTag for consistency
  • throw correct error message if no subject
  • fix status in AbstractSetIterateTag
  • colloquial=true by default
  • by default, sort alphabetically in TreeIterateTag
  • fix display of <HOME> for home object broken in last checkin
  • added potentiallyShorten to WebContextAwareMeshObjectIdentifierStringifier
  • instrument to track HTTP client-side calls
  • trim strings first to deal with sloppy JSP writing
  • added request property to JeeMeshObjectsToView
  • change default app server to Tomcat6
  • pass SaneUrl through to ViewletFactory, so it can access request attributes
  • created tag library for AclBasedSecurity Subject Area
  • added TextStructuredResponseSection.containsContent
  • replaced ViewletAlternativesTag.js with more general-purpose ToggleCssClass.js
  • support redirect to newly created object in HttpShell
  • Created BracketTags, which allow conditional generation of, say, <ul> and </ul> tags depending on whether or not the content tag has any non-whitespace content.
  • factored out AbstractSaneRequest.urlWithoutMatchingArguments for easier reusability
  • trim entered identifiers before trying to resolve them in HttpShellFilter
  • Better error reporting for HttpShell
  • IncludeViewletTag more robust if path not specified
  • Removed the List<Throwable> in request attribute for collecting processing exceptions; now abstracted into new interface ProblemReporter
  • Allow null identifiers for create access verb in HttpShell again
  • hyperlink on UnsafePostException’s message
  • TitleTag to use a separate section in the StructuredResponse
  • default mechanism for TitleTag with Viewlet name and app name, can be overridden with TitleTag
  • <tmpl:title> tag prints <title> tags itself
  • fixed userVisibleName on Viewlet
  • JeeViewlets’ default POST URL used to sometimes leave out reached MeshObjects, which would list all found-by-traversal MeshObjects after post, which was very confusing. Removed traversal spec in case there are no reached MeshObjects
  • Pass subject’s userVisibleString into default Viewlet title construction.
  • Created InvalidViewletActionException.
  • enable logging of when InfoGridWebApps start and stop
  • added HttpShellHandler to HttpShell
  • rollback Transaction if there is an exception during execution by HttpShell
  • support for radiobox-driven “which MeshObject should I be related to” relate/unrelate from the HttpShell
  • Added sweep and accessLocally to HttpShell HTML
  • Created ProxyTag in analogy to emitting other objects like MeshObjects etc.
  • Sweep support for the HttpShell
  • Added limit parameter to SetIterateTags in analogy to SQL limit
  • Added MeshObjectSetIterateBeyondLimitTag that gets invoked when a MeshObjectSet iteration stops due to a limit parameter so the JSP can display a “more” link or something like that
  • display trailing slash on URLs even if isColloquial; if not, user cannot distinguish URLs with and without trailing slashes
  • added IfNullTag and NotIfNull tag
  • allow to set properties via the HttpShell at the same time the MeshObject is only being created
  • working on allowing to set properties via the HttpShell at the same time the MeshObject is only being created
  • added allowNull attribute to PropertyTag to remove “remove” label for optional properties when so needed
  • fix ignore=”true” bug in MeshObjectTag
  • implemented a “JSP subroutine” set of custom tags
  • allow TypedMeshObjectFacade in place of any MeshObject for JSP custom tags; makes life simpler
  • use ‘block’ instead of ‘inline’ for create label in PropertyTags, so CSS can be used to move label vertically
  • added the just-completed Transactions to the invocation of HttpShellHandler, so implementations have a chance to understand what happened
  • added deleted MeshObjects to variable map passed to HttpShellHandler
  • tag to easily report a problem into the error log
  • tags to safeguard JSPs against inadvertent invocation by the wrong user or with the group user groups
  • support AccountStatus APPLIEDFOR and REFUSED
  • change default HTML title if Viewlet reports error; that way we don’t accidentially leak information if there is an access control problem
  • Re-implemented CSRF remedy: now match cookie and form instead of issuing form tokens
  • use lowercase cookie names
  • Added NotIfErrorsTag and NotIfInfoMessagesTag
  • Use StringRepresentation framework to format the label String for null PropertyValues
  • augmented HttpShellHandler to have invocable method before, after, at the beginning and at the end of Transactions
  • MeshType may be given directly in PropertyIterateTag
  • Change where <HOME> special identifier is defined. Stringifying MeshObjectIdentifier for HomeObject now produces “”, while stringifying MeshObject produces <HOME>
  • HttpShell now recognizes “not set” value because “” is now a valid value for a MeshObjectIdentifier
  • added JspoXXXTags
  • removed now unnecessary project
  • carry incoming request in DefaultJeeNetMeshObjectsToView
  • Allow both bean and identifiers in AbstractRelatedTag and subclasses
  • always use multi-part encoding for Jspo forms so file uploads are always possible
  • make JSP tags more consistent that use MeshObjects: can now specify MeshObject directly, MeshObjectIdentifier (direct and as String), or name of bean
  • added MeshObjectBlessedByTag
  • allow optional parameters on Jspo and Jspf tags
  • Support $1.23 in addition to 1.23 USD for CurrencyValue
  • support different capitalizations for MeshObjectBlessedByTag
  • SingleMemberTraversalTag for convenience
  • additional keywords for HttpShell
  • allow HttpShell to override/specify direction of a redirect after shell execution
  • additional callback afterAccess for HttpShell
  • allow immediate activation of called JspoTag through property on call
  • Added Json viewlet. New viewlet that will write a direct acyclic object graph in Json to level n.
  • added custom tag that makes it easy to iterate over the intersection of two traversal sets
  • allow MeshObjectBlessedByTag to write to a variable instead of to stdout
  • rollback transaction when HttpShellHandler throws HttpShellException
  • added MissingArgumentException that turns out to be generally useful
  • Created custom tags to help matching URL arguments easily
  • fixed incorrect processing of RuntimeException in HttpShell (particularly from HttpShellHandlers)
  • DataType would sometimes invalidly truncate HTML output with maxLength specified. Hope this implementation is better.
  • slight refactoring of JeeViewlet-internal API to make it easier to remove URL parameters from Viewlet’s default Post URL


  • Show number of MeshObjects in the MeshBase in AllMeshObjectsViewlet
  • round borders for webkit-based browsers too
  • NetMeshWorld has no business depending on AclBasedSecurity at this time
  • introduced property AnetMeshBase.ALLOW_NON_LOCAL_MESHOBJECT_CREATION to allow non-local NetMeshObject creation — useful for test instrumentation, for example
  • created ModuleInventoryViewlet
  • added app resource file to MeshWorld and NetMeshWorld
  • Added Sweeping button into NetMeshWorld.
  • Added ability to filter MeshBase content in AllMeshObjectsViewlet
  • Enable AllMeshObjectsViewlet to view empty sets.
  • Added properties file for Log4jConfigurationViewlet
  • add direct link to PropertySheetViewlet from AllMeshObjectsViewlet
  • Added filtering by home proxy to AllNetMeshObjectsViewlet
  • NetMeshWorld uses AllNetMeshObjectsViewlet, not AllMeshObjectsViewlet now
  • added “requires Javascript” via <noscript> to templates in example and test apps
  • Sort EntityTypes alphabetically in AllMeshObjectsViewlet

Database Impedance Mismatches

Alex Popescu argues that document databases have an impedance mismatch with object-based software. He compares the situation to relational databases:

“One of the most often mentioned issues reported by software engineers working with relational databases from object-oriented languages is the object-relational impedance mismatch. Document databases adopters are saying that one benefit of document stores is that there is no impedance mismatch between the object and document worlds.

“I don’t think this is entirely true.”

Given that databases and in-memory representations of objects serve different purposes, I don’t think we’ll ever completely manage to do without mappers of some kind. (Otherwise we’d all be using Java Object Serialization as our database.)

However, to me at least it appears that of all database technologies, graph databases like InfoGrid have the smallest impedance mismatch: a graph of objects is fundamentally closer to what is happening in memory of an object-based application than any other representation (e.g. document-based / hierarchical / relational / key-value etc.)

What’s the biggest obstacle to GraphDB adoption?

The recent workshop on Graph Databases in Barcelona sparked an interesting debate among vendors of graph databases about how to accelerate graph database adoption.

I don’t think that debate has been resolved yet. Perhaps this post will help a bit.

Some opinions that I heard were:

  • there are significant differences between the various graph database products on the market (e.g. graphs, property graphs, properties on edges or not, hypergraphs etc.). Unless they are more similar, customers will fear lock-in and not buy any of them. Here is a variation:
  • relational databases only took off once there was agreement on the SQL standard among vendors. We need to cooperate to create a similar language, otherwise graph database adoption will not take off either.
  • most potential users of graph databases have never heard of them. How could they use any if they don’t know they exist?
  • even if possible users know of graph databases, they do not know what the use cases are because use cases and success stories have not broadly been documented.

What do you think the biggest obstacles are for graph database adoption?

Generating a Hierarchical HTML Outline

Graph databases like InfoGrid are much better at traversing recursive data structures than relational databases. That is probably an established fact by now and not controversial.

But what does this mean in practice?

Here is an example just how simple it can be to generate a hierarchical HTML outline from a graph using InfoGrid’s custom tags:

        <mesh:meshObject meshObjectName="current"/>

The result is a set of nested HTML definition lists, which are a good match for an outline in HTML.

By way of explanation of this example, the outer tag is a big loop, which iterates over all elements in the outline. Every time we walk down the tree, the <tree:down> tag fires and emits the HTML list start element contained there. Every time we walk up the tree, the corresponding <tree:up> tag triggers and closes the list. “Subject” is the name of the start node as JSP bean, and the xpath-like expression is used to describe which branches to traverse in the graph, because it would just as simple traversing the opposite direction, or sideways.

I don’t think it gets much simpler ;-) Imagine the pain in SQL, particularly if the tree is deep …

Web Apps and InfoGrid

Graph databases are great for web apps: instead of somehow having to map tables and rows to URLs, developers can simply 1-to-1 map objects in the GraphDB to URLs.

Unlike most other GraphDBs, InfoGrid has extensive libraries to make the creation of web pages from dynamic graphs straightforward.

Here’s an example. Let’s say on your web page, you want to show a list of nodes related to some node ‘x’. For example, a social media friends list, or list of purchases, or bookmarks etc. Here is the JSP code:

<set:traverseIterate startObjectName="x" traversalSpecification="*" loopVar="y">
  <set:iterateNoContent><p>Sorry, no neighbors.</p></set:iterateNoContent>
      <mesh:meshObjectLink meshObjectName="y">
        <mesh:meshObject meshObjectName="y" />

If you look closely, you see that not only does it print the neighbor nodes in the graph, but also adds a hyperlink to them, so they can clicked on and examined. And if there are no neighbors, a message is printed saying so.

What if I don’t want any kind of neighbor, but only those related in some fashion, say business colleagues? Well, depends a bit on your model, but it’s usually as simple as replacing the first line of JSP above with something like this:

<set:traverseIterate startObjectName="x"

Notice that there is no other code that needs to be written, no object-relational mapping, no SQL syntax to get right, no handler code or other clumsy magic. Of course, you can have may of these statements on the same page, nested etc.

A great way of developing rich web pages really quickly.

The Bless Relationships API

InfoGrid supports types on both nodes and edges, aka MeshObjects and Relationships. They are called EntityTypes and RelationshipsTypes, respectively.

Assume we have two related MeshObjects, each blessed with a (hypothetical) type Person. Like this:

MeshObject obj1 = ...;
MeshObject obj2 = ...;
obj1.bless( PERSON );
obj2.bless( PERSON );
obj1.relate( obj2 );

Now if we want to express that the second Person is the daughter of the first Person, we cannot do this:

obj1.blessRelationship( PERSON_PARENT_PERSON, obj2 );

because we don’t know, from this API, whether obj1 or obj2 is the daughter or the parent.

Instead, in InfoGrid, we have to explicitly specify the direction, and we do this by specifying the “end” of the relationship type instead of the relationship type for the bless operation, like this:

obj1.blessRelationship( PERSON_PARENT_PERSON.getSource(), obj2 );

Hopefully, the documentation of the parent relationship type in the model indicates the semantics of the source and the destination of each relationship type. These “ends” are called RoleTypes, by the way.

One nice side effect of this approach to the API is that it becomes quite straightforward to extend it if InfoGrid ever decided to support ternary or N-ary relationship types: Just pick a RoleType beyond a source or destination RoleType.

If you bless relationships from the HttpShell — the simplest way of manipulating the graph from a web application — you thus need to specify the identifier of the RoleType, not of the RelationshipType. By convention, the source RoleType of a RelationshipType with identifier ID is ID-S, and ID-D for the destination.

One More Word on GraphDBs and Schemas

myNoSQL picked up our recent post on why having a schema is A Good Thing. In the comments to his post, various other graph database vendors speak up. They are far more ambiguous on the matter than we are …

For now, I think of Brian Akers’s take on this subject as the final word. With the comment “I know where everything is… don’t touch”, he shows this picture:

[very messy office]

If you can afford code and data like this, be my (schema-less) guest. Personally, I can’t.

’nuff said.

Required vs. Optional Property Values

InfoGrid distinguishes between properties that must have a non-null value, and properties that may or or may not be null.

When creating an InfoGrid model, a developer has to specify which by using the <isoptional/> tag in the model file.


By way of parallel, consider the following piece of Java code:

class Foo {
    private int max1 = 10;
    private Integer max2 = 20;

    public void doSomething() {
        for( int i=0 ; i<max1 ; ++i ) {
        for( int i=0 ; i<max2 ; ++i ) {

Spot the problem? max2 of course might be null, which means our code will throw an exception in the innocent-looking second for loop. To get the code right, we will have to protect that section with an if-then-else section that checks for null first.

Of course, such a protection is often the right thing to do. But in this example, a “max” should hardly ever be null, so using an “int” as a data type like for max1 (which can’t be null) is much better than using an “Integer” like for max2 (which may be null).

It’s the same thing for properties in InfoGrid models. Some properties simply should never be null. For example, consider a time stamp indicating when a MeshObject was created. Given that the MeshObject was created, the time stamp must exist, and therefore a null value makes no sense. In which case the property would be specified as “mandatory”. On the contrary, a time stamp when a MeshObject is likely to become obsolete is very likely optional: we might not know that time (yet), or it might never become obsolete, so null values are fine.

If InfoGrid did not distinguish between required and optional values, application code would be littered with unnecessary tests for null values. (or failing that, unexpected NullPointerExceptions.) We think being specific is better when creating the model; higher-quality and less cluttered application code is the reward.

Also check out the following related posts:

ACID Transactions Are Overrated

For many years, the canonical example why we need database transactions has been banking. If you move $100, you don’t really want the money be subtracted from the first account, but never be added to the second because of some problem in between. You want both the subtraction and the addition to happen, or neither.

Sounds good so far. Just apparently that is not how banks work in the real world, and they certainly use enough database systems that have ACID transactions. The Economist (July 24, 2010, “Computer says no”) quotes a former executive of the Royal Bank of Scotland saying: “The reality was you could never be certain that anything was correct.” Continuing: “Reported numbers fot the bank’s exposure were regularly billions of dollars adrift of reality.” The article offers an explanation: “banks tend to operate lots of different databases producing conflicting numbers.” HSBC is quoted: 55 separate systems for core banking, 24 for credit cards, and 41 for internet banking.

According to traditional transaction wisdom, if a customer makes an internet transaction to pay off his credit card, it should be a single transaction: start transaction, take money from checking account, put it into credit card account, commit. But transactions generally cannot span systems. Because the system responsible for internet banking is separate in this real-world example from core banking and from credit card systems, no such single ACID transaction is possible. Given the numbers above, it looks very much like those money transfers that actually can follow the canonical ACID transaction pattern constitute only a very small fraction of all transactions (like transferring from checking to savings.)

If I look at my own banking, the vast majority of my banking activity isn’t even within the same bank, but with other banks: bills to pay usually have to be paid at other banks. No cross-bank ACID database transactions that I’ve ever heard of.

So banking software necessarily has to have functionality that prevents that money is deducted but never arrives, all without depending on database transactions. If we have to have this functionality anyway, why then are transactions “indispensable” as some people still want to make us believe?

This pattern can be generalized: the more distributed and decentralized a system is, the less likely it is that we can use transactions that span the entire system. That is certainly true for the banking system, apparently also true for systems inside banks, and in many other places. ACID transactions were invented for the mainframe, the world’s most centralized computing construct. But computing is not “one mainframe” any more I’m afraid as it was in the sixties.

Instead of trotting out transactions as the answer, what we need for NoSQL databases is the ability to get the same benefits in a distributed system (“nothing is lost”) without relying on transactions. That’s where all of our efforts should be.

In the InfoGrid graph database, we have a weaker form of transactions for individual MeshBases, but then synchronize with the rest of the world by passing XPRISO messages. I’d be surprised if the eventual “transactional” architecture for large-scale distributed and decentralized systems looked very different.

InfiniteGraph Implementation of FirstStep

InfiniteGraph, the currently youngest member of the GraphDB party, has now also implemented our FirstStep example. Todd Stavish’s code is here. It joins implementations of the same example from InfoGrid, Neo4j, Sones, and Filament.

[Update: see Todd's comment below. Apparently there is checking in the second step.] On cursory examination, I’m surprised that InfiniteGraph allows you (requires you?) to create edges without source and destination nodes. Only after the creation of the edge does one assign the nodes to the edge. I’m unclear why this is an advantage for any particular scenario; however, I would think there’s a clear disadvantage as an application developer, because I now have to check for null pointers that I wouldn’t have to in most other graph databases (InfoGrid included).

Hope somebody more neutral than me will perform an API comparison using this example some day.