Notes pattern : Inversion of logging

A typical requirement in applications is to keep a log of changes. The simplest form is to keep fields with the author and date/time of the latest change (which has the distinct disadvantage not knowing what has been changed by whom). The next level would be to keep the list of all authors and change times (still leaving the question " what unanswered) followed by very sophisticated tools like Audit Manager or similar home grown solutions.
All these solutions have in common, that they "merely" record changes made (typically in a query or post save event) after they happened (From a big picture perspective, a log on query-save is technically before, but after the trigger that will commit the change). While this works reasonable well for audit, adding another typical requirement calls for a different solution. More often than not the parts of a (workflow) document which can be altered by any given user change in the course of an [workflow] application. While Author fields protect a document at large, safeguarding individual fields becomes more tedious, especially when you can't group them in access controlled sections (e.g. in a web application).
In a related use case: documents could be updated by users, who have current information, but don't "own" the documents. Typically they send an email to the document owner (if that is visible) asking for the update of that information. Asking somebody to update data into some other system, that involves copy & paste from an eMail *is* a sign of a broken process. The solution to that seemingly contrary requirements: "update of certain fields" only and "update by anybody" can be solved using a pattern that I christianed " Inversion of Logging":
Instead of logging what has changed an application using this pattern creates a change request, stating what will be changed. This request is then processed by a central system and checked against system constraints. If an authorized user requests a change, the changes are applied without further user interaction. The change request is kept as record. If an unauthorized user requests a change a workflow is kicked of to determine if the request is suitable for approval to be then routed to the data owner (or data guardian) for approval. Once approved the changes are applied to the main document.

To make this internal flow transparent and pleasant to the users a few considerations need to be made:

• Default Access to Database is Author, but documents don't contain Author fields. So users can create new documents, but can't edit existing documents.
• You need to handle QueryOpen (mode=edit) and QueryModeChange events (from read to edit) in Notes clients as well as ?EditDocument in web applications. Ideally to a user it looks like going into edit mode, so the inner workings of the solution stay transparent.
• You need a mail-in database (Default access=Depositor) to collect the change requests. If the interval of the "if new mail arrives" agent is too long for you, you might consider an XPage and a @URLOpen agent as faster trigger option (more on that in another post)
• You need to have a rule book for what fields can be updated in which application by whom based on what document status. This sounds awfully like business process engineering and as a matter of fact: it is.
• You need to make a call how do you want to handle pending changes. For one you probably want to display a hint on the form. That costs you one @DBLookup (which should be bearable). You could also include the pending changes into the working copy of the document. The interesting question here: how to handle the flow if such a pending change is rejected. Or - if you base the working copy on the existing document, what if the changes requested match a pending change request (e.g. waiting for approval) and would get auto-approval. So you need to plan for a potential resequencing.
• Don't resort to email as the default notification mechanism. Rather add a record to a notification application (using a web service for that seems like an excellent idea) where a user could define how (s)he wants to be notified and what eventual notification exceptions (e.g. delegation of authority) exist.
• This pattern is suitable for master data management and workflow applications. You might not want to use it for transactional applications, while you might very well use it for updates to the pricelist driving the transactional application.
• Plan your archive well. You can keep the current archive in a profile in the change request database and switch based on calendar, size or number of entries
• The change request database should have a short deletion stub lifecycle (default is 30 days, cut it to 5) since you create and delete a lot of documents during the course of the application. You could consider to delete/recreate that database during a time-out period (or use 2 different database names based on the day of week, so you can recreate savely on alternate days.
• Make sure that in a multi-server environment the apply-changes agent is running only on one server. Make sure to plan for agent fail-over in a cluster environment.
• Provide an easy UI for data integrity checks. I don't think an automatic check would be needed in most of the cases. It might be smart to record the archive databases in the main document, so lookups will be faster. However you want to have an "Auditor" function that does a deep check. Eventually you would send a request to a mail-in database, so the UI isn't blocked while a lot of archives get queried.

As usual YMMV.