This is the first of a series of posts documenting the creation of an open-source web app called MyHealth to manage personal health and wellness data, based on the InfoGrid Web Graph Database.

This scratches a personal itch, and I figure I might as well chronicle the progress of this little side project so developers new to InfoGrid get a taste.

I don’t think I’m the only geek who’d like to manage their health, wellness and fitness information better and keep it under their own control, so perhaps you might want to help out. There seems to be little or no open-source software for personal health information management that was written in this century, or so, and it will be cool to do it based on cutting-edge NoSQL graph database technology.

I’ll check today’s code every day and post on this blog. As it is a side project, I don’t expect to do more than an hour of work each day, and likely less. You are very welcome to contribute if you like, just send a note to the InfoGrid mailing list.

So here is day 1:

1. Let’s get the InfoGrid code:

mkdir -p ~/svn/svn.infogrid.org/infogrid
svn checkout http://svn.infogrid.org/infogrid/trunk

2. Make sure our setup is right and InfoGrid builds

cd ~/svn/svn.infogrid.org/infogrid/trunk
ig-tools/build.sh -a

This takes a while, so off to writing some of this blog post.

Oops, test failed because I forgot to start Tomcat. So let’s run NetBeans, which I’ll use as my IDE, and run Tomcat from there. Then re-run the last of the user interface tests that depend on Tomcat:

ig-tools/build.sh -run ig-ui

3. We’ll going to put MyHealth right into the InfoGrid SVN:

mkdir ig-app-myhealth
mkdir ig-app-myhealth/modules
mkdir ig-app-myhealth/app

I learned this past week that this directory structure is best described as an “InfoGrid-ism” (Hi Ben!) i.e. a convention that makes using InfoGrid easier. The modules subdirectory will contain the modules from which MyHealth is assembled, other than the InfoGrid modules, and the app directory will contain the web app project.

4. Use the brand-new project creation scripts:

cd ig-app-myhealth/modules
../../ig-tools/create-model-module.pl --name org.infogrid.model.MyHealth \
    --infogridpath ../..

This created a ModelModule, aka a schema for the graph database. Which also can be opened up as a project in NetBeans.

Then we also create a web project by cloning the MeshWorld example app:

cd ../app
../../ig-tools/clone-meshworld.pl --name org.infogrid.myhealth.www \
    --infogridpath ../..

5. Open both of those projects in NetBeans to make sure everything is right and can be built.

Select “Clean and Build” first on the model project, then on the web project. Works fine. We haven’t told those project about each other, but we will later.

6. Setting up a MySQL database.

In MySql, create a database and set some privileges:

mysql -u root -p
mysql> create database myhealth;
mysql> grant all privileges on myhealth.* to 'myhealth'@'localhost' identified by 'secret';
mysql> flush privileges;

In the org.infogrid.myhealth.www project in NetBeans, edit the context.xml file to read as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Context path="/org.infogrid.myhealth.www" cookies="false">
  <Resource auth="Container"
            type="javax.sql.DataSource"
            driverClassName="com.mysql.jdbc.Driver"
            name="jdbc/org.infogrid.myhealth.www"
            url="jdbc:mysql://localhost:3306/myhealth"
            username="myhealth"
            password="secret"
            maxActive="20"
            maxIdle="10"
            maxWait="-1" />
</Context>

To stay on the safe side, rebuild the web project.

6. Assigning an App Server and Run

In NetBeans, open the properties for project org.infogrid.myhealth.www, select the Run tab, and pick a local Tomcat installation as the assigned app server.

Running the web project. It runs! It is totally and utterly useless so far, and has no health content or health-relevant schema of any kind. But it runs, and we’ll keep the MyHealth app running from now on regardless how many changes we make and features we write.

7. Check-in so you guys can check out what has been created so far:

cd ../..
svn add -N ig-app-myhealth ig-app-myhealth/modules ig-app-myhealth/app
ig-tools/add-project-to-svn.pl ig-app-myhealth-modules/org.infogrid.model.MyHealth
ig-tools/add-project-to-svn.pl ig-app-myhealth-app/org.infogrid.myhealth.www

(These Perl scripts are just wrappers around svn to to check the right files and directories of NetBeans projects into SVN)

Done for today. We have a working app even if it does not do anything useful yet.

And, as JP is fond to say, “so to bed”.

It’s been a while since the last release, but why change if users are been generally happy with it … Here’s what’s new:

Highlights:

• Native support for currency values (e.g. $1.23 USD)
• Even simpler JSON generation
• StringDataTypes can now carry a regular expression, which is checked before assignment (e.g. makes it impossible to assign a String that isn’t to a property that must contain an e-mail address)
• Support for cascading delete
• InfoGrid compilation is now reported to work on Windows
• Multi-tenancy support for LID
• More flexible handling of HTTP status codes in Probe framework (e.g. what should happen upon a redirect)
• Extensions to custom JEE tags and HttpShell to make JSP programming with InfoGrid even simpler, e.g. simple tags to create Javascript-based overlay forms
• Improved error reporting
• Additional tests
• Lots of small usability improvements and bug fixes

Here are the gory details:

Core:

• ModuleAdvertisementSerializer to generate relative paths to JARs
• more types of MeshObjectSelector
• added getLocalId to NetMeshObjectIdentifier
• added guessFromExternalForm to IdentifierFactory
• use “” not null for localId of home object
• Augmented set of pre-defined BlobDataTypes
• Improved BlobDataType to check for MIME compatibility with regexes
• added CurrencyValue and CurrencyDataType to represent money as a primitive type
• added convenience method to MeshObject: determine whether the relationship to a neighbor MeshObject is blessed with a certain RoleType
• Moved OnDemandTransaction from HttpShell into utils
• implemented cascading delete support
• added TimeStampValue.earlierOf and TimeStampValue.laterOf for convenience
• Special value NOW for TimeStampValue parsing
• regex can be be specified for StringDataTypes
• added AbstractDelegatingLocalizedRuntimeException for symmetry and completeness
• better error reporting for TransactionActionExceptions
• Explicitly set latin1 character set in MySQL store to avoid key size error when the default character set is multi-byte.
• TransactionAction.execute() now does not take a Transaction parameter any more, but provides it as a member variable
• check raw id even when guessing MeshObjectIdentifier
• when there’s a ParseException during parsing of store content, don’t return null but throw NoSuchElementException
• Added MeshObjectHelper for simpler JSP programming with TypedMeshObjectFacades
• Carry specify exceptions keyed by failed MeshObjectIdentifiers in MeshObjectAccessException, instead of swallowing all exceptions other than the first
• MeshObjectsNotFoundException is not a subclass of MeshObjectAccessException any more
• Correct singular/plural error messages for MeshObjectAccessException and MeshObjectsNotFoundException
• NetMeshObjectAccessException now carries redirect NetMeshObjectAccessSpecifications
• NetMeshObjectAccessSpecification is now an Identifier
• Migrated sweeping functionality from the various MeshBase implementations into Sweeper. The policy functionality of Sweeper became SweepPolicy.
• Better pingpong debugging support
• created MeshObjectSet.hasSameContent
• upgrade to warn if MySQL database cannot be detected/created
• Id column in MySQL needs to be case-sensitive
• don’t report exception in rollback that isn’t an error
• default expiration for UnnecessaryReplicasSweepPolicy
• Allow the sweep of a single MeshObject
• MeshObject.delete and sub-methods have no business to throw NotPermittedException
• Non-semantic “kill” of a ShadowMeshBase
• Kill button in ShadowAwareAllMeshBasesViewlet
• No need for public method initiateCeaseCommunications on Proxy
• Don’t try to talk to Proxies when NetMeshBase is dead
• Tolerate some IsDeadExceptions when dead (duh)
• Use default mime type for BlobValue when none is given when parsing StringRepresentation
• Avoid that IsDeadException kills meaningful error message
• support typefield XML attribute for StringDataTypes as well
• added StringDataType for e-mail
• fixed automatic merge problem
• allow MeshObject.equals( TypedMeshObjectFacade ) for simplicity
• use get_XXX pattern in TypedMeshObjectFacade for non-Properties/non-pseudo-Properties to avoid name clashes with code-generated setters
• use get_XXX pattern in TypedMeshObjectFacade for non-Properties/non-pseudo-Properties to avoid name clashes with code-generated setters
• renamed Rfc3339Util to DateTimeUtil and expanded to W3C timestamps
• DefaultNetMeshBaseIdentifierFactory now uses pluggable “Scheme”s to support identifiers of various protocols, including http, https, file, xri, acct
• renamed UnknownProtocolParseException to UnknownSchemeParseException
• added ability to specify Maps in Resource files
• When parsing NetMeshBaseIdentifiers, turn each String into canonical form as well, e.g. hTTp: and http:
• added toColloquialExternalForm() to Identifier; more sane
• removed IdentifierStringifier.toColloquialIdentifier()
• use guessFromExternalForm in more places for more tolerance (e.g. when reading from Store to be more tolerant of Scheme changes)
• be more tolerant of Exceptions when restarting MeshBases; try best-effort
• Better logging of start / die of InfoGridWebApps
• when resetting PingPong, only increment token by 1
• more defensive programming if Storage layer is corrupt
• refactor message sending framework to always deliver all received messages together instead of singly
• merge two XprisoMessages before processing if possible
• added ByPropertyValueSelector and PropertyComparisonOperator
• NotRelatedException not thrown any more when testing whether two MeshObjects are related by a given RoleType; was counter-intuitive and pedantic
• StringRepresentationParameters must always be given.
• Make it easier to generate MeshObjectIdentifiers with different lengths
• upgraded Postgresql JDBC for 64bit support
• Netbeans 7 support
• Make sure Threadpool is always cleanly shut down. Now Tomcat won’t hang occasionally after undeploy.
• support custom regex violation error messages for PropertyTypes that specify regular expressions
• refactored L10Map into L10PropertyValueMap and L10StringMap, moved to utils package
• more HTML class declarations in BlobValue HTML
• added invert method to most MeshObjectSelectors
• added MeshObjectSet.subset convenience method
• Don’t include dead MeshObjects in MeshObjectSets
• factored out ExternalCommand so invoking an external process becomes simpler
• default MeshObjectIdentifier to contain _ instead of ~

Identity-related:

• AccountManager needs a site parameter for determining account to support multi-tenancy
• generate login event even if only the session gets renewed
• XrdsProbe to cache the XRDS file
• added content and content type as parameters to XmlProbe interface; enables XrdsProbe to cache the XRDS file
• added getGroupNames to LidAccount
• check group membership by group identifier as well as group name
• also check group membership by groups set as request attribute
• factored out some symbolic OpenID parameter names
• fix when associations expire
• Better integrated AccessManager and ThreadIdentityManager
• root may do anything in AclBasedAccessManager
• Split AclBasedSecurity into two: an ACL-based AccessManager implementation that does not rely on Guards, and an AccessManager implementation that evaluates Guards but is independent on a particular model (such as the AclBasedSecurity model)
• Eliminated org.infogrid.model.AclBasedSecurity, and instead introduced org.infogrid.meshbase.security.aclbased, which also includes a model
• DelegatingAccessManager now supports delegating to multiple other AccessManagers, all of which must agree
• added ThreadIdentityManager.suExec
• When there isn’t a ProtectionDomain, it’s free for all in AclBasedAccessManager
• moved org.infogrid.model.SecurityTest into the attic; currently not used
• added expiring credentials to AuthenticationStatus
• expired OpenID associations throw OpenIdAssociationExpiredException
• OpenID dumb mode fixes
• Added getCarriedExpiredCredentialTypes to LidClientAuthenticationStatus
• added org.infogrid.lid.model.account/Account_LastLoggedIn which can come in handy
• Removed Account Status value CREATED — not particularly useful and would have needed update logic
• credential values are now separate from LidAccount, e.g. the StoreLidPasswordCredentialType knows where to look for the password
• don’t need LidCredentialType.isRemote any more
• allow password change from StoreLidPasswordCredentialType
• removed realm from LID; never quite used
• better support for multi-tenancy / virtual hosts in LID: carry siteIdentifier around in more places
• distinguish one-time-tokens from other types of credentials in LID
• more methods for setting, checking, resetting password credentials
• support that MeshObject can own itself in AclUtils
• A little helper utility to generate secret keys

Model-related:

• Code-generate an entry for the SubjectArea itself, too
• exactly one SubjectArea element is expected in each model element in the model XML
• Added HTTP status code and location properties to WebResource
• minor rename of SubjectArea user-visible name

Probe-related:

• Handle HTTP redirects better when encountered by Probes; existing behavior should be unchanged however
• Do not throw FactoryException as a direct cause of NetMeshObjectAccessException but its cause instead
• Introduced ProbeException.HttpRedirectResponse
• Added NetMeshBaseRedirectException, so ProbeException.HttpRedirectResponse does not need to be moved up to module org.infogrid.kernel.net
• Introduced HttpMappingPolicy that enables different policies for mapping HTTP non-OK status codes into InfoGrid; default remains the same behavior
• Change in API to set up ProbeManagers etc.: ProbeDirectory passed to ProbeManager instead of ShadowMeshBaseFactory, then ShadowMeshBaseFactory is notified where ProbeManager can be found
• ProbeException.HttpErrorResponse not needed any more
• Support HTTP redirects that give relative URLs as Location, against the HTTP spec (I’m looking at you, Google)
• HttpMappingPolicy has moved from ProbeManager to ProbeDirectory
• Fixes issue where wrong HttpMappingPolicy was set upon restore of Shadow from disk
• Throw exception if ProbeManager is configured without a ProbeDirectory
• added basic XRD support to InfoGrid
• added acct: as one of the basic protocols supported by NetMeshBaseIdentifier
• added Webfinger support
• Don’t throw ProbeException.DontHaveNonXmlStreamProbe if the found content is of length zero

Viewlet/GUI-related:

• Renamed NotIfViewletState to NotIfViewletStateTag for consistency
• throw correct error message if no subject
• fix status in AbstractSetIterateTag
• colloquial=true by default
• by default, sort alphabetically in TreeIterateTag
• fix display of <HOME> for home object broken in last checkin
• added potentiallyShorten to WebContextAwareMeshObjectIdentifierStringifier
• instrument to track HTTP client-side calls
• trim strings first to deal with sloppy JSP writing
• added request property to JeeMeshObjectsToView
• change default app server to Tomcat6
• pass SaneUrl through to ViewletFactory, so it can access request attributes
• created org.infogrid.jee.security.aclbased tag library for AclBasedSecurity Subject Area
• added TextStructuredResponseSection.containsContent
• replaced ViewletAlternativesTag.js with more general-purpose ToggleCssClass.js
• support redirect to newly created object in HttpShell
• Created BracketTags, which allow conditional generation of, say, <ul> and </ul> tags depending on whether or not the content tag has any non-whitespace content.
• factored out AbstractSaneRequest.urlWithoutMatchingArguments for easier reusability
• trim entered identifiers before trying to resolve them in HttpShellFilter
• Better error reporting for HttpShell
• IncludeViewletTag more robust if path not specified
• Removed the List<Throwable> in request attribute for collecting processing exceptions; now abstracted into new interface ProblemReporter
• Allow null identifiers for create access verb in HttpShell again
• hyperlink on UnsafePostException’s message
• TitleTag to use a separate section in the StructuredResponse
• default mechanism for TitleTag with Viewlet name and app name, can be overridden with TitleTag
• <tmpl:title> tag prints <title> tags itself
• fixed userVisibleName on Viewlet
• JeeViewlets’ default POST URL used to sometimes leave out reached MeshObjects, which would list all found-by-traversal MeshObjects after post, which was very confusing. Removed traversal spec in case there are no reached MeshObjects
• Pass subject’s userVisibleString into default Viewlet title construction.
• Created InvalidViewletActionException.
• enable logging of when InfoGridWebApps start and stop
• added HttpShellHandler to HttpShell
• rollback Transaction if there is an exception during execution by HttpShell
• support for radiobox-driven “which MeshObject should I be related to” relate/unrelate from the HttpShell
• Added sweep and accessLocally to HttpShell HTML
• Created ProxyTag in analogy to emitting other objects like MeshObjects etc.
• Sweep support for the HttpShell
• Added limit parameter to SetIterateTags in analogy to SQL limit
• Added MeshObjectSetIterateBeyondLimitTag that gets invoked when a MeshObjectSet iteration stops due to a limit parameter so the JSP can display a “more” link or something like that
• display trailing slash on URLs even if isColloquial; if not, user cannot distinguish URLs with and without trailing slashes
• added IfNullTag and NotIfNull tag
• allow to set properties via the HttpShell at the same time the MeshObject is only being created
• working on allowing to set properties via the HttpShell at the same time the MeshObject is only being created
• added allowNull attribute to PropertyTag to remove “remove” label for optional properties when so needed
• fix ignore=”true” bug in MeshObjectTag
• implemented a “JSP subroutine” set of custom tags
• allow TypedMeshObjectFacade in place of any MeshObject for JSP custom tags; makes life simpler
• use ‘block’ instead of ‘inline’ for create label in PropertyTags, so CSS can be used to move label vertically
• added the just-completed Transactions to the invocation of HttpShellHandler, so implementations have a chance to understand what happened
• added deleted MeshObjects to variable map passed to HttpShellHandler
• tag to easily report a problem into the error log
• tags to safeguard JSPs against inadvertent invocation by the wrong user or with the group user groups
• support AccountStatus APPLIEDFOR and REFUSED
• change default HTML title if Viewlet reports error; that way we don’t accidentially leak information if there is an access control problem
• Re-implemented CSRF remedy: now match cookie and form instead of issuing form tokens
• use lowercase cookie names
• Added NotIfErrorsTag and NotIfInfoMessagesTag
• Use StringRepresentation framework to format the label String for null PropertyValues
• augmented HttpShellHandler to have invocable method before, after, at the beginning and at the end of Transactions
• MeshType may be given directly in PropertyIterateTag
• Change where <HOME> special identifier is defined. Stringifying MeshObjectIdentifier for HomeObject now produces “”, while stringifying MeshObject produces <HOME>
• HttpShell now recognizes “not set” value because “” is now a valid value for a MeshObjectIdentifier
• added JspoXXXTags
• removed now unnecessary org.infogrid.jee.store project
• carry incoming request in DefaultJeeNetMeshObjectsToView
• Allow both bean and identifiers in AbstractRelatedTag and subclasses
• always use multi-part encoding for Jspo forms so file uploads are always possible
• make JSP tags more consistent that use MeshObjects: can now specify MeshObject directly, MeshObjectIdentifier (direct and as String), or name of bean
• added MeshObjectBlessedByTag
• allow optional parameters on Jspo and Jspf tags
• Support $1.23 in addition to 1.23 USD for CurrencyValue
• support different capitalizations for MeshObjectBlessedByTag
• SingleMemberTraversalTag for convenience
• additional keywords for HttpShell
• allow HttpShell to override/specify direction of a redirect after shell execution
• additional callback afterAccess for HttpShell
• allow immediate activation of called JspoTag through property on call
• Added Json viewlet. New viewlet that will write a direct acyclic object graph in Json to level n.
• added custom tag that makes it easy to iterate over the intersection of two traversal sets
• allow MeshObjectBlessedByTag to write to a variable instead of to stdout
• rollback transaction when HttpShellHandler throws HttpShellException
• added MissingArgumentException that turns out to be generally useful
• Created custom tags to help matching URL arguments easily
• fixed incorrect processing of RuntimeException in HttpShell (particularly from HttpShellHandlers)
• DataType would sometimes invalidly truncate HTML output with maxLength specified. Hope this implementation is better.
• slight refactoring of JeeViewlet-internal API to make it easier to remove URL parameters from Viewlet’s default Post URL

MeshWorld/NetMeshWorld-related:

• Show number of MeshObjects in the MeshBase in AllMeshObjectsViewlet
• round borders for webkit-based browsers too
• NetMeshWorld has no business depending on AclBasedSecurity at this time
• introduced property AnetMeshBase.ALLOW_NON_LOCAL_MESHOBJECT_CREATION to allow non-local NetMeshObject creation — useful for test instrumentation, for example
• created ModuleInventoryViewlet
• added app resource file to MeshWorld and NetMeshWorld
• Added Sweeping button into NetMeshWorld.
• Added ability to filter MeshBase content in AllMeshObjectsViewlet
• Enable AllMeshObjectsViewlet to view empty sets.
• Added properties file for Log4jConfigurationViewlet
• add direct link to PropertySheetViewlet from AllMeshObjectsViewlet
• Added filtering by home proxy to AllNetMeshObjectsViewlet
• NetMeshWorld uses AllNetMeshObjectsViewlet, not AllMeshObjectsViewlet now
• added “requires Javascript” via <noscript> to templates in example and test apps
• Sort EntityTypes alphabetically in AllMeshObjectsViewlet

A new viewlet was recently added to the build called JsonViwelet that renders a graph of objects in Javascript Object Notation (Json). It’s packaged in a new module called org.infogrid.jee.viewlet.json to isolate its dependency on “Jackson” - an open source Json parser, from the rest of the ig-ui modules, which is checked in to the ig-vendors package.

This viewlet was provided to help you write an Ajax user interface. Of course, it does not solve the whole problem of creating a modern browser experience which includes visually rendering objects in a browser window, and synchronizing them with the server running Infogrid. However it does kick-start the process and is one less issue to deal with.

There is already support for Json in the tree, from a Viewlet in the ig-ui MeshWorld test apps. called ObjectSetViewlet.jsp. As its name suggests it is a java server page implementation. It’s packaged conventionally in the web tree in v/viewlet/objectset/ObjectSetViewlet/text/json so that it can be dispatched when you provide a mime type of “text/json” in the lid-format parameter of the request URL. As its name suggests it’s dispatched on a collection of objects returned by traversing from the object identified in the request URL - e.g. http://localhost:8084/org.infogrid.meshworld.net/nnnnn?lid-format=viewlet:org.infogrid.jee.viewlet.objectset.ObjectSetViewlet&lid-format=mime:text/json&lid-traversal=org.foo.Test/Entity_Collects_AnotherEntity-S

This serves as a useful example for how to develop your own Json viewlets, however you could soon end up with several special purpose jsp viewlets that could say render the specific object in the request as Json, render it with and without its meta information and so on … These different rendering requirements are based on how you intend to use the objects into which the Json response gets transformed when it’s evaluated in the browser.

So it seemed like a better idea to provide a universal viewlet that could fulfill all these requirements by being passed URL arguments to control its behavior. It works by performing a transitive closure on the object graph from the object specified in the URL - without looping. Whenever a cycle is detected it writes the object’s “id”  but not its contents, to stub the graph so that the client has a reference it can deal with later.

Unless constrained by the arguments, its default behavior is to dump everything about every object that is encountered in the traversal. So be warned - if you have a very interconnected graph you could end up dumping the whole contests of your meshbase.

The JsonViewlet takes the URL arguments:

• level=n|0 - 0 means no limit and will follow every relationship
• dateenc=func|string - determines how Date is rendered, either as a function or String
• ignoreblessing=entitytype_name_regex - names of blessings to be ignored
• meta=[no|false]|[true|yes]|only - whether or not to dump meta information
• trimsa=[yes|true]|[no|false] - trims the subject area name from the property names

Bold = default.

Level, determines the depth of the traversal. Since the object identifiers are returned for the next lower level you could provide code in your javascript client to later make another call to return the next levels in the graph.

Dateenc, controls how IG TimeStamp’s are rendered in json. Json does not provide a way to encode dates, so there is a choice to do this either as a string or as a javascript Date constructor method signature. They both work - you choose.

Ignoreblessing, has the effect of not dumping the values of the specified blessing. This is primarily useful for removing the Probe model properties which bless probed entities. This is control information that is probably not needed by a client - unless you are say writing a probe manager. You can pass it a regular expression or the fully qualified class name, and the argument can be repeated or comma-delimited to specify as many of these as you like.

Meta, gives you the option of either supressing the meta information or only dumping the meta information about an object. Javascript is a typeless language and probably does not care too much about the meta information. However there are some client frameworks that are wrestling with the issue of java-javascript mapping and doing things like simulating fixed class hierarchies - so maybe the meta information might be important.

Finally there is trmsa - which simply trims the subject area name off the front of property type names. These strings can get a bit long and often they are not needed if you’ve specified the property type name uniquely.

There is some error checking of these arguments, and if any error is encountered a Json error is returned as an “id” and “msg.” There should be no other errors returned.

Json returns values as either numbers or strings, therefore blobs are treated as references and returned as URL’s that reference the specific property, and request the PropertyViewlet do the rendering.

In order to use this viewlet you’ll need to make sure it’s added to the application’s viewlet factory, using something like this:

ret.add(JsonViewlet.choice(realToView, ViewletFactoryChoice.BAD_MATCH_QUALITY));

So please give it a try. It would be nice to build out more components needed to build a rich client experience.

InfoGrid trunk has been upgraded to NetBeans7, but for the time being, we remain with Tomcat6. Unfortunately the NetBeans guys have not exactly made it easy to remain compiling JSPs against the Tomcat6-supported specs instead of Glassfish.

In the relevant InfoGrid projects, we set the relevant properties in project.properties for Tomcat6. They are picked up when running the build from the command line.

If you find yourself with compilation errors when compiling from the IDE, the following trick might help:

• Find your system-wide NetBeans 7.0 properties default file, e.g. ~/.netbeans/7.0/build.properties
• In that file, find the properties that start with libs.jsp- such as libs.jsp-compiler.classpath.
• Comment them all out!

Given the sequence in which NetBeans reads in those property files (system-wide, private project-specific, all project-specific), they might override the correct settings otherwise.

The recent workshop on Graph Databases in Barcelona sparked an interesting debate among vendors of graph databases about how to accelerate graph database adoption.

I don’t think that debate has been resolved yet. Perhaps this post will help a bit.

Some opinions that I heard were:

• there are significant differences between the various graph database products on the market (e.g. graphs, property graphs, properties on edges or not, hypergraphs etc.). Unless they are more similar, customers will fear lock-in and not buy any of them. Here is a variation:
• relational databases only took off once there was agreement on the SQL standard among vendors. We need to cooperate to create a similar language, otherwise graph database adoption will not take off either.
• most potential users of graph databases have never heard of them. How could they use any if they don’t know they exist?
• even if possible users know of graph databases, they do not know what the use cases are because use cases and success stories have not broadly been documented.

What do you think the biggest obstacles are for graph database adoption?

It always looked like it was only a matter of time until the object database companies would try and become graph databases. Perhaps that is what they should have been all along. I’m speaking as somebody who tried several products almost 20 years ago and decided that they were just too much hassle to be worth it: graphs are a much better abstraction level than programming-level constructs for a database.

Today, Objectivity announced “Infinite Graph”, a:

strategic business unit is tasked with bringing [a] enterprise-ready and distributed graph database product to market

(I took the liberty of eliminating the “marketing” superlatives from the quote; the entire press release has a very generous sprinkling of them.)

Actually, they only announced a beta program, which I signed up for. InfiniGraph.com says:

X:\> BETA IS NOW OPEN

But then, on the screen behind, they say:

Over the next several days, we’ll be preparing our installer and documentation for distribution to the InfiniteGraph community. Stay tuned, and feel free to participate in the discussion on our beta blog!

Well, well, the difficulties of a launch. So I don’t know yet what they created. But it’s good to see another player legitimizing graph databases as a category. So, welcome Objectivity!

This release contains some major improvements, in particular to the way the object graph is mapped to and from the web. Download or browse documentation here.

General:

• improved stability and error reporting
• lots of bug fixes
• more tests
• removed symbolic links from SVN; was an endless source of frustration

Core:

• renamed TraversalDictionary to TraversalTranslator: it can be much more dynamic than a dictionary
• implemented KeywordTraversalTranslator with fixed translation keywords
• implemented XpathTraversalTranslator with a pseudo-subset of Xpath
• introduced AllNeighborsTraversalSpecification instead of null TraversalSpecification; introduced StayRightHereTraversalSpecification
• MeshObject’s userVisibleName now always returns value of a Property called “Name” if the MeshObject has one; this seems a sensible default for many applications
• MeshObjectIdentifierFactory now has a pointer back to the MeshBase to which it belongs. Downside is one cannot use the same instance of MeshObjectIdentifierFactory for multiple MeshBases any more.
• got rid of MeshStringRepresentationContext, which partially overlapped with the purpose of MeshStringRepresentationParameters; only historical reasons can explain why we had both
• removed title/target/additionalParameters argument from HasStringRepresentation.toStringRepresentationLinkStart; now handled as parameters in StringRepresentationParameters
• simplified StringRepresentation of PropertyValues and related
• additional pre-defined StringRepresentations e.g. HttpPost
• distinguish between formatting Properties (which may be null) and PropertyValues; no more muddling with funny MeshObject context parameters; formatting is now performed via DataType
• use correct ClassLoader to load ResourceHelper default properties file during Module initialization
• correct initialization of ResourceHelper in module.adv
• expanded MeshObjectSet and MeshObjectSetFactory API

Identity-related:

• refactored LID implementation to use an “instructions” based approach to the pipeline, instead of an exceptions-based approach. This is more flexible for users of the module.
• fixed typo about credential vs. credtype.
• renamed LidPersona to LidAccount; more natural to talk about it using that name
• added LidAccountStatus and SiteIdentifier to LidAccount for multi-tenancy
• factored account and session-related concepts from org.infogrid.lid.model.lid into new SubjectArea org.infogrid.lid.model.account.
• new custom tag library that deals with identity

Model-related:

• fixed code generation bug for descriptions of values in EnumeratedDataTypes
• code generator to generate static constants for all EnumeratedValues.
• added AccountCollection to account LID model
• expanded TestModel to cover both optional and mandatory PropertyTypes; renamed PropertyTypes correspondingly
• improved Test model for more comprehensive testing
• better user-visible strings for EntityTypes Bookmark, Account and WebResource

Viewlet/GUI-related:

• Viewlet framework and tag library extensions for including Viewlets in Viewlets; updated Viewlets accordingly; now allows in-context editing, change of viewlet types etc. for included JeeViewlets; no contiguous TraversalPath from top required
• support REST-ful URLs on hierarchical Viewlets e.g. GraphTreeViewlet with multiple Viewlet alternatives in contained Viewlet
• removed iframe in (Net)MeshWorld; hierarchical Viewlets is better approach
• various HTML fixes and improvements related to the rendering and editing of PropertyValues
• removed rootPath on all custom tags; not needed or used
• sanitized formatting of Identifiers; it’s still not totally sane but a lot more so
• eliminated PropertyValueTag.{css,js} and replaced with PropertyTag.{css,js}
• removed -moz-opacity CSS value per recent Firefox updates
• fix HTML doctype to make IE more happy
• BlobViewlet to get its PropertyType from URL argument not POST argument
• footer element for MeshObjectSetIterate tag
• do not print iterateHeader and iterateFooter when set has no content in MeshObjectSetIterateTags
• default POST behavior is now redirect-to-GET on same URL, so browser refresh is not as awful
• added missing setter methods on StructuredResponse
• created SetSizeTag to print the size of a MeshObjectSet in JSP
• slight changes how MeshObjects are shown on screen by default (dropped annotation in which non-standard MeshBase they are)
• overflow: auto; to support long CSS floats
• added orderBy property to setIterate JSP tags
• added ability to sort in the inverse direction
• created propertymeter JSP tag for bar graphs or temperature graphs based on Properties
• default sorting in JSP MeshObjectSet tags is by user-visible String
• don’t set domains on cookies when run from localhost; makes life of developers hard
• generate Javascript for PropertyValues
• eliminating unnecessary projects by moving their code into other projects:
  • org.infogrid.jee.rest -> org.infogrid.jee.viewlet
  • org.infogrid.jee.rest.net -> org.infogrid.jee.viewlet.net
• renaming projects:
• org.infogrid.jee.rest.net.local -> org.infogrid.jee.viewlet.net.local
• org.infogrid.jee.rest.net.local.store -> org.infogrid.jee.viewlet.net.local.store
• org.infogrid.jee.rest.store -> org.infogrid.jee.viewlet.store
• renamed meshObjectLoopVar to loopVar in custom tags
• added arrivedAt property to Viewlet
• enctype attribute on safeForm is all lowercase
• created SaneUrl, new supertype of SaneRequest that allows to reuse API for URLs and servlet requests; slight API naming changes httpHost vs. server; allows us to get rid of OverridingSaneRequest nonsense
• DEFAULT_LINK_START/END_ENTRY now consistently on StringRepresentation
• removed RestfulRequest, replaced with a MeshObjectsToViewFactory that directly translates SaneRequest into MeshObjectsToView
• an instance of MeshObjectsToViewFactory must now reside in Context
• removed NetViewletDispatcherServlet; not needed any more
• removed most redundant methods on Viewlet; better have one clear way how to do it only
• upgraded ViewletFactoryChoice: now HasStringRepresentation and contains MeshObjectsToView; this means unfortunately that ViewletFactory setup in applications needs to pass MeshObjectsToView to their choices()
• ViewedMeshObjects now keeps reference to MeshObjectsToView that it took its data from
• removed unnecessary request attributes like JeeViewlet.VIEWLET_STATE_TRANSITION_NAME: can be obtained via Viewlet
• made MeshObjectsToView an interface and subtyped to JeeMeshObjectsToView and NetMeshObjectsToView for cleaner model
• renamed getMeshObjects to getViewedMeshObjects for consistency
• ViewletState has moved from JeeViewlet to JeeViewedMeshObjects; added isDefaultState

The database industry is not used to databases that can generate events. The closest the relational database has to events are stored procedures, but they never “reach out” back to the application, so their usefulness is limited. But events are quite natural for graph databases. Broadly speaking, they occur in two places:

• Events on the graph database itself (example: “tell me when a transaction has been committed, regardless on which thread”)
• Events on individual objects stored in the graph database (example: “tell me when property X on object Y has changed to value Z”, or “tell me when Node A has a new Edge”)

Events on the GraphDB itself are more useful for administrative and management purposes. For example, an event handler listening to GraphDB events can examine the list of changes that a Transaction is performing at commit time, and collect statistics (for example).

From an application developer’s perspective, events on the data are more interesting:

An example may illustrate this. Imagine an application that helps manage an emergency room in a hospital. The application’s object graph contains things such as the doctors on staff, the patients currently in the emergency room and their status (like “arrived”, “has been triaged”, “waiting for doctor”, “waiting for lab results” etc.) Doctors carry pagers. One of the requirements for application is that the doctor be paged when the status of one of their assigned patients changes (e.g. from “waiting for lab results” to “waiting for doctor”).

With a passive database, i.e. one that cannot generate events, like a typical relational database, we usually have to write some kind of background task (e.g. a cron job) that periodically checks whether certain properties have changed, and then sends the message to the pager. That is very messy: e.g. how does your cron job know which properties *changed* from one run to the next? Or we have to add the message sending code to every single screen and web service interface in the app that could possibly change the relevant property, which is just as messy and hard to maintain.

With a GraphDB like InfoGrid, you simply subscribe to events, like this:

MeshObject patientStatus = ...; // the node in the graph representing a patient's status
patientStatus.addPropertyChangeListener( new PropertyChangeListener() {
    public void propertyChanged( PropertyChangeEvent e ) {
        sendPagerMessage( ... );
    });
}

The graph database will trigger the event handler whenever a property changed on that particular object. It’s real simple.

Pere Urbon published his e-mail interview with me about InfoGrid:

This is the fourth in his series on graph databases:

• Sones interview
• VertexDB interview
• HyperGraphDB interview
• InfoGrid interview.

It’s rather apparent that while these projects are all GraphDBs, they differ substantially in what they are trying to accomplish, and why, and therefore how they do it. This is a good resource for developers investigating GraphDBs and trying to understand their alternatives.

InfoGrid is now on YouTube at InfoGridDotOrg.

Occasionally we’ll post a software demo. The first one is there already: a screencast of the MeshWorld example application for the InfoGrid graph database. It shows:

• How to create and delete MeshObjects (ie. nodes)
• How to relate them to each other (ie. edges)

It follows the FirstStep example, except that MeshWorld is a web application, while FirstStep is a command-line application. There’ll be more to come.