Today, as we write application code
we generally are not dealing with immutable state/data, we are applying application logic to state/data so that it transforms from one known state to another known state in a consistent, all or nothing fashion. Today, when performing this in code that does not involve Transaction Managers or Resource Managers, we use Try/Catch exception handling statements to mimick the "Commit" and "Rollback" concepts of transactions. The difficulties with such an approach is that as we scope certain types of work and/or scope more than a few simple actions into a unit of work, the level of complexity in nested try/catch code grows and can easy reach a point where it cannot truely provide the "Commit" and "Rollback" concepts of a Transaction.
For example,
take the Try/Catch below
try
{
// operation(stateX)
// operation(stateY)
// operation(stateZ)
}
catch
{
// cleanup after states X, Y, and/or Z, as something has gone wrong and we need to get to a previously known state
}
The challenges with using try/catch in a Commit/Rollback style are some of the following:
-
If stateX is shared between between more that one execution contexts, another execution context may read an inconsistent state and hence a "dirty" read, as the scoped unit of work may fail
- What if an operation on stateY deletes/alters a resource and how do we "Rollback" in the catch.
- How do we coordinate with resources that are transactional, i.e. if stateX represents a call to a transaction queuing/messaging or database system. How we coodinate between the different resources and unit of works!
- and there is lots of other problems
While all of the above can be eventually handled in code, with careful planning and structure, but what is the essence of what we are trying to achieve in the code and does it form a pattern! It is the "Commit" and "Rollback" concepts of a Transaction and if we factor the patterns, interfaces, contracts and interactions of Transacional Processing into the code we have more flexibility and agility. Please note I am not suggesting that every try/catch has a "Commit" or "Rollback" focus, but we sometimes use try/catch to group actions into a unit of work in which we determine the outcome of a unit of work by interrogation (using if statements) and do not negotiate/coordinate the outcome of the unit of work (as an example). Also, to provide the most flexibility and agility, we need for the actions that are grouped into a unit of work to be able to be aware of the (transactional) context. We also have the problem that try/catch is not context aware, i.e. it only knows a single execution context, and while dealing with a single execution context may suit us some times, we need to be able to handle both concurrency management and recovery control at the same time!
One approach to providing a cleaner implementation, is to factor out the interrogation and make it the negotiation/coordination of the outcome from the application logic and for an intermediary to work as a coordinator between the resources being used and the application logic. The coordinator controls the enlistment of resources within a scoped execution context (as defined by the application code) that we want to be transactional. We then need to define a number of common interfaces and interaction semantics, so that the use/enlistment of one or more resources in the scope/context of a unit of work is clearly defined. This can be achieved by having methods/semantics/behaviors common to all resources so that the different stages of moving/transforming from one known state to another known state as a unit of work or as a part of a unit of work, happens in an all or nothing fashion.
All of this fits quite nicely with the 2 phase commit (and even the one phase commit) protocol as to how the "Commit" or "Rollback" semantics of units of work can be achieved.
Oppps ....... Hmmmm ..... since I thought I was going to be able to do this is 3 to 4 postings, I seem to have underestimated and need one or two more. I think I will leave it here for now with this posting and follow up with a part 4,5,6
In general, when we think of Transactions we tend to think of database transactions. Databases are a great example of how a data server can implement transactions and therefore handle concurrency, provide isolation and provide consistency. The concept of what constitutes a Transaction expands beyond that of a Database Transaction and with this we have more and more areas where the abstraction and contracts that make up a Transaction can help us with the problem of how to handle concurrency management and recovery, as we apply application logic to state and consistently move from one known state to another.
So when we look at a Transaction, what are main parts involved? At a base level it is composed of three parties, i.e. an Application, a resource manager and a transaction manager. So how they are involved in a transaction and what is their role in a transaction?
The Application is pretty self explanatory, it’s your code. Its role in a transaction is to define the scope of a transaction, perform read/write operations on a resource manger and specify whether it views the state as being consistent. A resource manager is a component that manages a stable data storage system (i.e. it always shows a consistent view of data/state), and participates in the two phase commit and recovery protocols with the transaction manager. A transaction manager tracks which resource managers participate in the transaction, provides an interface to the application – start, commit, rollback and also, coordinates the transaction outcome between the resource managers. The transaction manager has the responsibility of properly handling error conditions, on behalf of an application, recover from them and ensure that the resource managers are always in a consistent state.
With today’s technologies the Resource Managers that typically get used are those of databases systems and queue managers. But is there other times when you need to be transforming/moving from one state to another? There certainly are!
The basic premise of Transactions is the coordination/negotation of an outcome that groups actions into a unit of work that transforms from one known state to another. As the transformation of state, the coordination/negotation of an outcome, or the grouping of action into a unit of work are something we do very often in application logic, is this not a Transaction.
So when do we get involved in the transformation of state, the coordination/negotation of an outcome, or the grouping of action(s) into a unit of work? What are we trying to achieve, how do we generally do it today and is there a better way to do it?
I've kicked off Part 3, to share my views on this and hopefully provide some insight!
When thinking about Transactions, I think it is fair to say that Transactions are one of the most powerful and frequently used paradigms in Enterprise Computing today. With Transactions, we have a collection of patterns that deal with how concurrency management and recovery control can be handled in possible shared/concurrent environments/contexts and it is this that ensures data/state correctness. It is the key concepts and abstractions of Transactions maintaining data/state consistency in possible concurrent state/data access environments (where failures may occur) that have allowed us as developers and architects to be largely released from having to deal with the tricky aspect of maintaining the correctness of state in concurrent environments in an enterprise application.
Two of the very interesting aspects of Transactions is how trusted they have become and how we perceive them. Concurrency Management and recovery control are some of the more tricky aspects of Enterprise Computing, but yet when leveraging the Transactional paradigm and executing units of work in a concurrent environment in the context of a transaction, we are not overly concerned about isolation or failure as much of the work is done for us by the Transaction Paradigm. This is a testament to the level of trust and adoption that Transactions have achieved as an abstraction and how it has evolved into a framework of runtime services that takes most of the responsibility for concurrency management and recovery control from us as developers and architects. The paradigm of Transactions can be seen as a set of patterns, Interfaces and Contracts that can be leveraged to simplify the transformation from one known state to another and without these Interfaces and Contracts; a lot of code, time and effort has to be spend in making sure that data/state transformations happen in a consistent and all or nothing fashion.
It is these key abstractions in the form of runtime services or frameworks that have provided great productive gains for us as developers and architects, as it allows us to focus on business logic and allows for the underlying runtime services to deal with concurrency management and recovery control. Therefore as we implement application logic, by applying units of work to state, we want the transformation from one state to another to be applied in a consistent (all or nothing) fashion, as well as wanting to be isolated from other parallel activities. As I mentioned in my
How should we think of Transactions ?
blog posting, I like to think of the Transaction paradigm as a collection of patterns, interfaces and contracts that can be leveraged to tackle the issue of data/state correctness as we group actions into unit of works to transform from one known state to another. Some of this collection of patterns, interfaces and contracts are short running transactions that in the main adhere to the ACID properties but there is a rich portfolio of transactional types that we do not focus on that much and/or maybe perceive as not meeting the ACID properties (see
How should we think of Transactions ?
and
Volatile Transactions & In-Memory Transactions - do they have ACID, ACId or ACI properties?
)
Today, the most common example of how the Transaction paradigm can provide concurrency management and recovery control is Database Management Systems (DBMS). Databases (aka DBMS) are themselves another key and often used architectural piece of Enterprise computing. Database Management Systems (DBMS) provide a paramount role in how an Enterprise may store, retrieve and share state/data. But we sometimes draw an association between Databases and Transactions that is so close that they are often perceived as being interchangeable, i.e. Transactions are DB Transactions and nothing else. The close associations between databases systems and transactional processing have been fundamental in bringing the concept of Transactional Processing to the fore, but in my honest opinion I think that the close association may have also brought a narrowing of perception of what are Transactions and Transactional Processing.
I have put together a series of blog postings to share my current views/thoughts! After talking to a few folks about Transactions, it is very interesting to hear the different views and I'm not sure if there is a one size fits all for what is a transaction, but this is the thing about a paradigm and a snappy acronym, it can have a life of its own
Maybe we interchange the paradigm, acronym’s and implementations into one fixed view and its this fixed view that maybe causing the issue!
In my last blog posting,
How should we think of Transactions ?
I mentioned Volatile transactions and how I currently think they can be perceived in different ways! Do they have ACID, ACId or ACI properties?
In case you do not know what Volatile Transactions (aka In-Memory Transactions) are, they are transactions that do not persist to a durable store such as a hard disk (ouch! we have multiple definitions of durable to content with
). They are transactions that are done completely in memory and do not incur the I/O cost of writing to a hard disk a Transaction log or the actual known data/state.
What is interesting about Volatile transactions is that they are sometimes perceived as not being Transactional, because they are viewed as not meeting the ACID properties, especially the "D" property. I suspect the main reason for this view is because there are no Transaction logs and/or no data stored to hard disk.
So when you look at a typical definition of the ACID properties (like what I have listed below), some of the reasons why this view maybe held become apparent:
(NB. this is an amalgamation/synopsis of many definitions I have seen)
A
- Atomicity - Transactions are atomic (all or nothing)
C
- Consistency - The transaction is a correct transformation of state. Consistency constraints that are defined on the underlying data servers are preserved by a transaction.
I
- Isolation - Each transaction should appear to execute independently of other transactions that may be executing concurrently in the same environment.
D
- Durability - Once a transaction commits, it’s updates survive even, if the system goes down
ok, so a Volatile Transaction meets the ACI of the ACID properties, but not the D as defined above. But is this ("Once a transaction commits, it’s updates survive even, if the system goes down") a true reflection of what Durability means? Many similar variants of this definition gets used on a frequent based and they bring specifics to the definiton that are not about the failure of meeting the ACI properties or making the "Commited" value available/public but focus on the failure of media or a system. In my opinion the failure of media or a system is not an aspect of the paradigm or indeed part of the ACID properties, but an aspect of how the paradigm can be implemented and/or a requirement of the longevity of the state being persisted/determined. It is perhaps the fact that the best known implementation of the data server feature of the transactional paradigm is a database that has led us to shape the ACID properties to the specifics of databases. But perhaps the Transactional paradigm has more to offer than the correctness of data/state transformations on a database. The transaction paradigm has a core focus on simpilifing how we can achieve the correctness of data/state transformations where we may have to deal with concurrency management and recovery control and this can be applied to any transformation of state.
Perhaps a different way to look at the durability of ACID is where the definition of ACID as an acronym came from and how some of the leading experts in the industry define them. As far as I know the ACID acronym was first coined by Haerder and Reuter (1983). The definition I like to use is that by Gray and Reuter (1993). It is the following:
A
- Atomicity - A transaction's changes to state are atomic: either all happens or none happen. These changes include database changes, messages, and all actions on transducers.
C
- Consistency - A transaction is a correct transformation of the state. The actions taken as a group do not violate any of the integrity constraints associated with the state. This requires that the transaction be a correct program.
I
- Isolation - Even though transactions execute concurrently, it appears to each transaction, T, that others executed either before T or after T, but not both
D
- Durability - Once a transaction completes successfully (commits), its changes to the state survive failures.
When you read this definition of Durability (and the ACID properties overall), it in my option explains the feature richness of transactions and how it can be applied to the persistence of the transformation of state and also how it applies to the other properties such as A, C and I. The state survives failures by the data server ensuring that when the transaction commits a new durable state is persisted/stable/available/public and if something goes wrong (such as not be able to ensure Consistency) that the data server returns to the previous known state. In essence, this is a contract that guarantees the persistence and stability of the state and that the outcome of a transaction is known/public and usable in subsequent transactions, as a known/correct state. A good definition I have seen is that once a transaction is committed, it cannot be abrogated.
With Durability being about the persistence, stability and making public of “commited”/known state as we transform from one known/correct state to another, it allows us to leverage the transaction paradigm in many scenarios. For example when using Volatile Transactions, we are leveraging data servers that store data on Volatile media, such as RAM and providing the transaction paradigm in-memory. With the common misunderstanding that Transactions are exclusively for databases, the question arises about the “D” property.
For me the Durability aspect of a transaction is present in a Volatile transaction. The question is whether its “D” or “d”. Since most transactions are based on the X/Open Transaction processing model, the requirement for log managers or lock managers in resource managers (data servers) are not central or core to implementing the transactional paradigm at a base level. In relation to the usage of Volatile media for Transactions? in my opinion the requirement of the media used by the data server is that it is able to present a stable/persistent/public state once "Commit" has occured.
So I believe that Volatile Transactions meet all of the ACID properties and that its ACID with a "D"
phew! long post.
I guess the crux of the issue is the 2 uses of "durable" when talking about Transactions. One is the "D" in the ACID properties and the other is the property of the data/state/media being used (and how it is stored). The "D" in the ACID properties is more about making a "commited" value available and the other is about an implementation requirement of how long (and/or resilient) a known/committed state must be.
So now the news is out, there is going to be a PDC 2005
There are currently no more details available
, but you can
signup
for updates as the details develop.
Last February we had a great Forum event for Architects involving a full day of architectural content with excellent industry and Microsoft speakers.
This event will be back in Dublin on January 27th. The topic this year is interoperability and integration. There will be great speakers and this is an excellent way to meet other architects from across Ireland. If you are an architect and interested in attending please register here.
http://msevents-eu.microsoft.com/CUI/EventDetail.aspx?EventID=118753459&Culture=en-IE
Bill
I'm just back in my office again after having given my course, "Patterns with .NET". I love giving this course, and it was just as much fun as usual (if not more) this time. One thing I particularly like is that I always get so many interesting ideas from the participants.
For instance, this time two participants, Ellen Lippe and Atle Haugan from Trondheim in Norway, asked me if they could reuse any of the DDD concepts I had been talking about when they use DataSets. I honestly hadn't thought about it in this way before, since I have mostly seen DDD as a way of structuring a Domain Model [Fowler PoEAA]. After a while they helped me to understand that in some cases combining DDD and DataSets might be a good compromise. For instance, a DDD Aggregate could be structured as a Table Module [Fowler PoEAA] wrapping a DataSet. The repositories could deal with the lifecycle of these aggregates too, and so on.
I think we agreed that it is a bit unnatural in some situations, but life is full of compromises, and, depending upon the situation, this might a very good one!
My session (in Swedish) about TDD is up on MSDN TV now. You find it
here
.
Well, what can I say? My throat was giving me grief, and it was my first time in a studio, and...
:-)
There is a recently started architecture-focused forum called SwedArch and already there are lots of lively discussions taking place. Unfortunately (or not, depending upon your language skills), it's only in Swedish.
:-)
Go
here
.
My current pet project is called NWorkspace and is (or, at least, might become) what I call adapters for different persistence solutions. Nope, it's *not* another OR Mapper for .NET, which might come as a shock!
:-)
Instead, what NWorkspace does is try to reuse the functionality of the current OR Mappers. Pretty wild, huh?
First of all, let's take a step back. As you might have noticed, if you are following my blog, I'm currently working on a couple of DDD projects. What I have noticed is that I find myself writing some silly code purely to be able to run automatic tests against a stub which simulates the database and against the real database (when it has been built).
Sure, I know, I can run the majority of tests against the Domain Model without dealing with persistence at all. I do that, and that's not the problem here. The problem is when I take the next step, so to speak, and want to test the interaction where repositories are involved.
What also happens is that I want to let users try out the application before the database has even been created. What they get instead is just a stub, so that they can get a feel for the application and provide feedback early on. This allows me (and the team) to make pretty radical changes to the Domain Model and UI without having to make any changes to a database schema as well. It's also extremely easy when it comes to distribution and installation for early demos.
You may be wondering what this stupid code is. Well, assume I use a custom stub for tests without database, and NHibernate for example as the OR Mapper for tests with the database. Typically, what I do then is create a specific interface for each repository, such as ICustomerRepository, for example, which I implement by the classes CustomerRepositoryStub and CustomerRepository. The code in those classes is pretty similar but different, so I need two of them. They get information from the consumer as a constructor parameter, such as a stub instance, in the case of the stub implementation, and an ISession, in the case of NHibernate. The CustomerRepository can then use the ISession for stating database questions and for associating instances with the Unit of Work.
When it comes to the CustomerRepositoryStub, there is no database against which to ask questions, so the stub-repository will iterate lists for filtering out the requested instances instead. Pretty different, as I said.
What also happens is that this affects the test code. It's not just how the instantiation of the repositories is done that differs, but how the transaction is dealt with, etc.
So, what I usually get is three test classes (one is a base class) per repository, one repository interface and two repositories (in two separate assemblies). Not an extreme amount, perhaps, but it doesn't feel too good.
So, I started thinking about how hard it would be to create some abstraction for which I could create implementations for different persistence providers. This would mean that I could code against that abstraction in both the tests and the repositories, and in doing so reduce the amount of dumb code. I could probably see the NH-interface as that abstraction and let the stub implement it, or I could implement a stub dialect for NH... However, I thought this might be a generic problem for several OR Mappers that I would be working with, so I started another route. That's where NWorkspace comes in.
So, NWorkspace is mainly a couple of simple interfaces (and some helpers), which are implemented by adapters so that NHibernate can be used under the surface. Or my stub implementation, or probably many other OR Mappers as well, if adapters are written for them. However, please note that I don't think this will be used for switching between different OR Mappers in the same project. It might help a bit, but definitely not the whole way, even if you switch between similar OR Mappers that are both POCO-style, for example. There are a lot of semantic differences that NWorkspace won't be able to hide. The main goals of NWorkspace are instead something like:
- Reducing the amount of dumb code when writing tests against both stubs and databases.
- Simplifying the API for consumers of OR Mappers (when a simple API is all that is needed).
- Providing a "standard" API (including querying) which will make it cheaper for developers moving between different OR Mappers in different projects.
When I write the tests against IWorkspace (which is the main interface of NWorkspace), I only need one test class per repository (or at least "all" the code could move to the base class). I also only need one repository per aggregate, and I can (if I want to) skip the repository interface. I can also move back the repository into the Domain Model assembly again.
My friend
Mats Helander
considered something similar to NWorkspace a couple of months ago, which he called NORM. Here, the idea was to define a couple of standard interfaces which .NET OR Mappers could implement to support that standard. NWorkspace takes a different approach and doesn't require OR Mapper vendors to implement anything. Instead, anyone who is interested could write an adapter for adapting any OR Mapper to NWorkspace. Sure, it will mean some performance overhead, but probably not too much for most cases.
If you're wondering about the name, it's inspired by
Christoffer Skjoldborg
's and my project called Valhalla. There, our class, which has the Unit of Work and the Identity Map, is called Workspace. NWorkspace (or its adapter implementations) won't typically *have* Unit of Work/Identity Map, but will use those in the mappers instead.
NWorkspace is little more than an idea right now, and is definitely not ready for real use. I have written a small proof of concept and I'm very eager to move on. If you are interested, watch this blog. I'll be talking about the basic API in a few days' time.
?
Last night a group of top developers from the Minneapolis area had dinner with
David Chappell
. Great company, great food and great conversation; what more could you want?
The conversation ranged wide, from whether parents should perpetuate the myth of the Tooth Fairy and Santa all the way down to how Whidbey’s System.Transactions compares to the transactional functionality in BizTalk Server.
At one point during the conversation David mentioned that there’s not only an impedance mismatch between objects and relational data, but also between users and objects and between services and objects. While the user-object mismatch is somewhat understood, we’re just now starting to learn the nature of the service-object mismatch.
(The OO-relational impedance mismatch is well-known and is discussed in many OO books, perhaps most succinctly in
David Taylor’s book
)
I think that his observation is accurate, and it is actually the topic of an article I submitted to
TheServerSide.net
a few days ago, so look for that to stir some controversy in the next few weeks :-)
I also heard that some people have read my blog entries on remoting (
this one in particular
I think) and somehow came to the conclusion that I was indicating that Indigo would be binary compatible with remoting. To be clear, as far as I know Indigo won’t be binary compatible with remoting. Heck, remoting isn’t binary compatible with remoting. Try accessing a .NET 1.1 server from a 1.0 client and see what happens!
What I said is that if you are careful about how you use remoting today you’ll have a relatively easy upgrade path to Indigo. Which is no different than using asmx or ES today. They’ll all have an upgrade path to Indigo, but that doesn’t imply that Indigo will natively interoperate with remoting or ES (DCOM).
(Presumably Indigo will interop with asmx because that’s just SOAP, but even then you’ll want to change your code to take advantage of the new Indigo features, so it is still an upgrade process)
What does “be careful how you use remoting” mean? This:
Use the Http channel
Host in IIS
Use the BinaryFormatter
Do not create or use custom sinks, formatters or channels
If you follow these rules you
should
have a relatively direct upgrade path to Indigo. Will you have to change your code? Of course. That’s what an upgrade path IS. It means you can, with relative ease, update your code to work with the new technology.
For more information go to
www.lhotka.net
.
Read a funny post recently,
"If Building Architects had to work like Software Architects".
You may have seen it before. It got me thinking again about the area of requirements. Having recently completed some house building work where I had commissioned a building Architect it started me thinking.
Many people comment after reading the above ditty, that software Archs have a long way to go to match the professionalism/process/skills of building architects. I have to agree that software architects do not yet have an industry standard qualification - consequently we lack the social/industry status this affords. The building architect I employed was able to use her 'seal' from the national regulatory body on her correspondence - and charge me a set fee based on their guidelines!
When working with my building architect we obviously started with a brief. This was a short document listing the high level requirements. If it were an IT project the closest thing I can compare it to would be a project goal statement e.g. Convert basement into two bedrooms with rear access and new windows. It was as high level as that. The next step was to draw plans, these were extremely detailed and the builder could essentially build off these plans. So building architects have refined their specification process to produce a set of highly refined diagrams. Crucially these diagrams can be read and understood by the clients (me) and the builder. What’s the best we've got in IT - UML?
My point (
finally
J
)
, the builder architect does not labour under the detail of non-functional requirements. e.g. Security, reliability, maintainability, reusability, usability etc. At no stage did we have a discussion about say - security. Obviously the standard was to put windows and doors in that have locks! Also I assumed that insulation and damp proofing would be done in accordance with best practice - obviously I talked about this with the architect but we never got to the level of talking about U values etc - I trusted her and the builder to do the right thing. Of course trusting builders and architects has been the ruin of many before so the other thing on my side was the building regulations that specify minimum insulation levels (sound and heat), plumbing standards etc. etc. So building has, over time, evolved a process whereby regulations and habit ensure buildings are built to a certain standard using a common set of materials and conventions. As IT Architects we are heading in this direction and in my opinion we need to quickly accelerate past the standard of workmanship and process in the building trade (which I believe to be inefficient with too many trades and skills fragmentation – more about this another time).
Finally - the big difference between house building and system building - in my house I could see everything! So I could see insulation in the ceiling joists damp courses in the floor etc. Our customers can’t see our security or reliability features nor can they understand our diagrams/schematics. Until these non-functional requirements become standard (or regulated) our requirements and design processes will be bogged down trying to specify them and our customers will be bamboozled trying to understand what we are selling them.
With over 40 Webcasts on: why it matters to the business, common strategies and methods, and guidance on specific implementation scenarios between the major platform players, this is going to be quite a comprehensive series.
You can pre-register for the Webcast series at
http://www.interopmonth.com/
and in a fews weeks you should receive a schedule on the sessions, speakers and topics.
Some highlights you can expect:
·
David Chapell (.NET)
and
David Chapell (J2EE)
to key note the webcast series and end the series.
·
Information on managing a multi-platform data center
·
Information on integrating Java with Windows applications and clients
·
Using web services to extent the life of legacy systems and provide efficient information exchange and functionality
If you have a look at the
Transaction Specification Index Page
on MSDN, you will see that:
- WS-Coordination was updated in November 2004.
- WS-AtomicTransaction was updated in November 2004.
- WS-BusinessActivity was updated in November 2004.
They are well worth a read.
Also, Luis Felipe Cabrera, Christopher Kurt and Don Box have updated
"An Introduction to the Web Services Architecture and Its Specifications"
|