Transactions - The concepts - Part 3

Today, as we write application code we generally are not dealing with immutable state/data, we are applying application logic to state/data so that it transforms from one known state to another known state in a consistent, all or nothing fashion. Today, when performing this in code that does not involve Transaction Managers or Resource Managers, we use Try/Catch exception handling statements to mimick the "Commit" and "Rollback" concepts of transactions. The difficulties with such an approach is that as we scope certain types of work and/or scope more than a few simple actions into a unit of work, the level of complexity in nested try/catch code grows and can easy reach a point where it cannot truely provide the "Commit" and "Rollback" concepts of a Transaction.

For example, take the Try/Catch below

try
{
 // operation(stateX)
 // operation(stateY)
 // operation(stateZ)
}
catch
{
  // cleanup after states X, Y, and/or Z, as something has gone wrong and we need to get to a previously known state
}

The challenges with using try/catch in a Commit/Rollback style are some of the following:

- If stateX is shared between between more that one execution contexts, another execution context may read an inconsistent state and hence a "dirty" read, as the scoped unit of work may fail
- What if an operation on stateY deletes/alters a resource and how do we "Rollback" in the catch.
- How do we coordinate with resources that are transactional, i.e. if stateX represents a call to a transaction queuing/messaging or database system. How we coodinate between the different resources and unit of works!
- and there is lots of other problems

While all of the above can be eventually handled in code, with careful planning and structure, but what is the essence of what we are trying to achieve in the code and does it form a pattern! It is the "Commit" and "Rollback" concepts of a Transaction and if we factor the patterns, interfaces, contracts and interactions of Transacional Processing into the code we have more flexibility and agility. Please note I am not suggesting that every try/catch has a "Commit" or "Rollback" focus, but we sometimes use try/catch to group actions into a unit of work in which we determine the outcome of a unit of work by interrogation (using if statements) and do not negotiate/coordinate the outcome of the unit of work (as an example). Also, to provide the most flexibility and agility, we need for the actions that are grouped into a unit of work to be able to be aware of the (transactional) context. We also have the problem that try/catch is not context aware, i.e. it only knows a single execution context, and while dealing with a single execution context may suit us some times, we need to be able to handle both concurrency management and recovery control at the same time!

One approach to providing a cleaner implementation, is to factor out the interrogation and make it the negotiation/coordination of the outcome from the application logic and for an intermediary to work as a coordinator between the resources being used and the application logic. The coordinator controls the enlistment of resources within a scoped execution context (as defined by the application code) that we want to be transactional. We then need to define a number of common interfaces and interaction semantics, so that the use/enlistment of one or more resources in the scope/context of a unit of work is clearly defined. This can be achieved by having methods/semantics/behaviors common to all resources so that the different stages of moving/transforming from one known state to another known state as a unit of work or as a part of a unit of work, happens in an all or nothing fashion.

All of this fits quite nicely with the 2 phase commit (and even the one phase commit) protocol as to how the "Commit" or "Rollback" semantics of units of work can be achieved.

Oppps ....... Hmmmm .....  since I thought I was going to be able to do this is 3 to 4 postings, I seem to have underestimated and need one or two more. I think I will leave it here for now with this posting and follow up with a part 4,5,6

 

Transactions - The concepts - Part 2

In general, when we think of Transactions we tend to think of database transactions. Databases are a great example of how a data server can implement transactions and therefore handle concurrency, provide isolation and provide consistency. The concept of what constitutes a Transaction expands beyond that of a Database Transaction and with this we have more and more areas where the abstraction and contracts that make up a Transaction can help us with the problem of how to handle concurrency management and recovery, as we apply application logic to state and consistently move from one known state to another.

So when we look at a Transaction, what are main parts involved? At a base level it is composed of three parties, i.e. an Application, a resource manager and a transaction manager. So how they are involved in a transaction and what is their role in a transaction?

The Application is pretty self explanatory, it’s your code. Its role in a transaction is to define the scope of a transaction, perform read/write operations on a resource manger and specify whether it views the state as being consistent. A resource manager is a component that manages a stable data storage system (i.e. it always shows a consistent view of data/state), and participates in the two phase commit and recovery protocols with the transaction manager. A transaction manager tracks which resource managers participate in the transaction, provides an interface to the application – start, commit, rollback and also, coordinates the transaction outcome between the resource managers. The transaction manager has the responsibility of properly handling error conditions, on behalf of an application, recover from them and ensure that the resource managers are always in a consistent state.

With today’s technologies the Resource Managers that typically get used are those of databases systems and queue managers. But is there other times when you need to be transforming/moving from one state to another? There certainly are!

The basic premise of Transactions is the coordination/negotation of an outcome that groups actions into a unit of work that transforms from one known state to another. As the transformation of state, the coordination/negotation of an outcome, or the grouping of action into a unit of work are something we do very often in application logic, is this not a Transaction.

So when do we get involved in the transformation of state, the coordination/negotation of an outcome, or the grouping of action(s) into a unit of work? What are we trying to achieve, how do we generally do it today and is there a better way to do it?

I've kicked off Part 3, to share my views on this and hopefully provide some insight!

Transactions - The concepts - Part 1

When thinking about Transactions, I think it is fair to say that Transactions are one of the most powerful and frequently used paradigms in Enterprise Computing today. With Transactions, we have a collection of patterns that deal with how concurrency management and recovery control can be handled in possible shared/concurrent environments/contexts and it is this that ensures data/state correctness. It is the key concepts and abstractions of Transactions maintaining data/state consistency in possible concurrent state/data access environments (where failures may occur) that have allowed us as developers and architects to be largely released from having to deal with the tricky aspect of maintaining the correctness of state in concurrent environments in an enterprise application.

Two of the very interesting aspects of Transactions is how trusted they have become and how we perceive them. Concurrency Management and recovery control are some of the more tricky aspects of Enterprise Computing, but yet when leveraging the Transactional paradigm and executing units of work in a concurrent environment in the context of a transaction, we are not overly concerned about isolation or failure as much of the work is done for us by the Transaction Paradigm. This is a testament to the level of trust and adoption that Transactions have achieved as an abstraction and how it has evolved into a framework of runtime services that takes most of the responsibility for concurrency management and recovery control from us as developers and architects. The paradigm of Transactions can be seen as a set of patterns, Interfaces and Contracts that can be leveraged to simplify the transformation from one known state to another and without these Interfaces and Contracts; a lot of code, time and effort has to be spend in making sure that data/state transformations happen in a consistent and all or nothing fashion.

It is these key abstractions in the form of runtime services or frameworks that have provided great productive gains for us as developers and architects, as it allows us to focus on business logic and allows for the underlying runtime services to deal with concurrency management and recovery control. Therefore as we implement application logic, by applying units of work to state, we want the transformation from one state to another to be applied in a consistent (all or nothing) fashion, as well as wanting to be isolated from other parallel activities. As I mentioned in my How should we think of Transactions ? blog posting, I like to think of the Transaction paradigm as a collection of patterns, interfaces and contracts that can be leveraged to tackle the issue of data/state correctness as we group actions into unit of works to transform from one known state to another. Some of this collection of patterns, interfaces and contracts are short running transactions that in the main adhere to the ACID properties but there is a rich portfolio of transactional types that we do not focus on that much and/or maybe perceive as not meeting the ACID properties (see How should we think of Transactions ? and Volatile Transactions & In-Memory Transactions - do they have ACID, ACId or ACI properties? )

Today, the most common example of how the Transaction paradigm can provide concurrency management and recovery control is Database Management Systems (DBMS). Databases (aka DBMS) are themselves another key and often used architectural piece of Enterprise computing. Database Management Systems (DBMS) provide a paramount role in how an Enterprise may store, retrieve and share state/data. But we sometimes draw an association between Databases and Transactions that is so close that they are often perceived as being interchangeable, i.e. Transactions are DB Transactions and nothing else. The close associations between databases systems and transactional processing have been fundamental in bringing the concept of Transactional Processing to the fore, but in my honest opinion I think that the close association may have also brought a narrowing of perception of what are Transactions and Transactional Processing.

I have put together a series of blog postings to share my current views/thoughts! After talking to a few folks about Transactions, it is very interesting to hear the different views and I'm not sure if there is a one size fits all for what is a transaction, but this is the thing about a paradigm and a snappy acronym, it can have a life of its own   Maybe we interchange the paradigm, acronym’s and implementations into one fixed view and its this fixed view that maybe causing the issue!

 

Volatile Transactions & In-Memory Transactions - do they have ACID, ACId or ACI properties?

In my last blog posting, How should we think of Transactions ? I mentioned Volatile transactions and how I currently think they can be perceived in different ways! Do they have ACID, ACId or ACI properties?

In case you do not know what Volatile Transactions (aka In-Memory Transactions) are, they are transactions that do not persist to a durable store such as a hard disk (ouch! we have multiple definitions of durable to content with ). They are transactions that are done completely in memory and do not incur the I/O cost of writing to a hard disk a Transaction log or the actual known data/state.

What is interesting about Volatile transactions is that they are sometimes perceived as not being Transactional, because they are viewed as not meeting the ACID properties, especially the "D" property. I suspect the main reason for this view is because there are no Transaction logs and/or no data stored to hard disk.

So when you look at a typical definition of the ACID properties (like what I have listed below), some of the reasons why this view maybe held become apparent:

(NB. this is an amalgamation/synopsis of many definitions I have seen)

A - Atomicity - Transactions are atomic (all or nothing)

C - Consistency - The transaction is a correct transformation of state. Consistency constraints that are defined on the underlying data servers are preserved by a transaction.

I - Isolation - Each transaction should appear to execute independently of other transactions that may be executing concurrently in the same environment.

D - Durability - Once a transaction commits, it’s updates survive even, if the system goes down

ok, so a Volatile Transaction meets the ACI of the ACID properties, but not the D as defined above. But is this ("Once a transaction commits, it’s updates survive even, if the system goes down") a true reflection of what Durability means? Many similar variants of this definition gets used on a frequent based and they bring specifics to the definiton that are not about the failure of meeting the ACI properties or making the "Commited" value available/public but focus on the failure of media or a system. In my opinion the failure of media or a system is not an aspect of the paradigm or indeed part of the ACID properties, but an aspect of how the paradigm can be implemented and/or a requirement of the longevity of the state being persisted/determined. It is perhaps the fact that the best known implementation of the data server feature of the transactional paradigm is a database that has led us to shape the ACID properties to the specifics of databases. But perhaps the Transactional paradigm has more to offer than the correctness of data/state transformations on a database. The transaction paradigm has a core focus on simpilifing how we can achieve the correctness of data/state transformations where we may have to deal with concurrency management and recovery control and this can be applied to any transformation of state.

Perhaps a different way to look at the durability of ACID is where the definition of ACID as an acronym came from and how some of the leading experts in the industry define them. As far as I know the ACID acronym was first coined by Haerder and Reuter (1983). The definition I like to use is that by Gray and Reuter (1993). It is the following:

A - Atomicity - A transaction's changes to state are atomic: either all happens or none happen. These changes include database changes, messages, and all actions on transducers.

C - Consistency - A transaction is a correct transformation of the state. The actions taken as a group do not violate any of the integrity constraints associated with the state. This requires that the transaction be a correct program.

I - Isolation - Even though transactions execute concurrently, it appears to each transaction, T, that others executed either before T or after T, but not both

D - Durability - Once a transaction completes successfully (commits), its changes to the state survive failures.

When you read this definition of Durability (and the ACID properties overall), it in my option explains the feature richness of transactions and how it can be applied to the persistence of the transformation of state and also how it applies to the other properties such as A, C and I. The state survives failures by the data server ensuring that when the transaction commits a new durable state is persisted/stable/available/public and if something goes wrong (such as not be able to ensure Consistency) that the data server returns to the previous known state. In essence, this is a contract that guarantees the persistence and stability of the state and that the outcome of a transaction is known/public and usable in subsequent transactions, as a known/correct state. A good definition I have seen is that once a transaction is committed, it cannot be abrogated.

With Durability being about the persistence, stability and making public of “commited”/known state as we transform from one known/correct state to another, it allows us to leverage the transaction paradigm in many scenarios. For example when using Volatile Transactions, we are leveraging data servers that store data on Volatile media, such as RAM and providing the transaction paradigm in-memory. With the common misunderstanding that Transactions are exclusively for databases, the question arises about the “D” property.

For me the Durability aspect of a transaction is present in a Volatile transaction. The question is whether its “D” or “d”. Since most transactions are based on the X/Open Transaction processing model, the requirement for log managers or lock managers in resource managers (data servers) are not central or core to implementing the transactional paradigm at a base level. In relation to the usage of Volatile media for Transactions? in my opinion the requirement of the media used by the data server is that it is able to present a stable/persistent/public state once "Commit" has occured.  So I believe that Volatile Transactions meet all of the ACID properties and that its ACID with a "D"

phew! long post.

I guess the crux of the issue is the 2 uses of "durable" when talking about Transactions. One is the "D" in the ACID properties and the other is the property of the data/state/media being used (and how it is stored). The "D" in the ACID properties is more about making a "commited" value available and the other is about an implementation requirement of how long (and/or resilient) a known/committed state must be.

 

 

A Fun Stack Trace

The stack trace below (snapshot taken at a breakpoint in [WebMethod] "HelloWorld") shows that I am having quite a bit of programming fun these days. Server-side ASP.NET hooked up to a MSMQ listener.  

simpleservicerequestinweb.dll!SimpleServiceRequestInWeb.Hello.HelloWorld() Line 53 C#
system.web.services.dll!System.Web.Services.Protocols.LogicalMethodInfo.Invoke(System.Object target, System.Object[] values) + 0x92 bytes 
system.web.services.dll!System.Web.Services.Protocols.WebServiceHandler.Invoke() + 0x9e bytes 
system.web.services.dll!System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest() + 0x142 bytes 
system.web.services.dll!System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(System.Web.HttpContext context) + 0x6 bytes 
system.web.dll!CallHandlerExecutionStep.System.Web.HttpApplication+IExecutionStep.Execute() + 0xb4 bytes 
system.web.dll!System.Web.HttpApplication.ExecuteStep(System.Web.HttpApplication.IExecutionStep step, bool completedSynchronously) + 0x58 bytes 
system.web.dll!System.Web.HttpApplication.ResumeSteps(System.Exception error) + 0xfa bytes 
system.web.dll!System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext context, System.AsyncCallback cb, System.Object extraData) + 0xe3 bytes 
system.web.dll!System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest wr) + 0x1e7 bytes 
system.web.dll!System.Web.HttpRuntime.ProcessRequest(System.Web.HttpWorkerRequest wr) + 0xb0 bytes 
newtelligence.enterprisetools.dll!newtelligence.EnterpriseTools.Msmq.MessageQueueAsmxDispatcher.MessageReceived(System.Object sender = {newtelligence.EnterpriseTools.Msmq.MessageQueueListener}, newtelligence.EnterpriseTools.Msmq.MessageReceivedEventArgs ea = {newtelligence.EnterpriseTools.Msmq.MessageReceivedEventArgs}) Line 33 C#
newtelligence.enterprisetools.dll!newtelligence.EnterpriseTools.Msmq.MessageQueueListener.ReceiveLoop() Line 305 + 0x2b bytes C#

>
(c)2002-2004 Clemens Vasters. You want to build service oriented applications on the Microsoft .NET Frameworkand are not quite sure how?
newtelligenceWorkshops get you going with concrete, pragmatic guidance on today's shippingtechnology stack, keeping tomorrow in perspective. Available on-site, world-wide.

Codus

Codus is a comprehensive code generation tool for object-relational mapping. It takes an existing database and automatically generates all of the code for updating, deleting, inserting and selecting records. In addition, it creates web services for distributed programming, strongly-typed collections, and a full set of unit tests.

The tool is available as a Beta version for the moment here

 

Neat Little SQL Server 2000 Enterprise Manager Tip

I've been using SQL Server Enterprise Manager since 1998, but just today realized that you could, from the Tables listing, hit Ctrl+C on a table name and then go to a text editor and hit Ctrl+V and get the appropriate CREATE TABLE SQL syntax for the “copied” table (which includes constraints).  Prior to this, when I've needed a table's T-SQL definition I've always right-clicked on the table name, gone to All Tasks and then selected Generate SQL Scripts.  This latter approach affords more options (such as scripting multiple database entities, scripting dependent objects, adding DROP statements, etc.), but the simple “copy-paste” technique is a fast way to just grab the create script for a single table.  (This handy shortcut works for stored procedures, too!)

On an aside, I noticed today that my blog, since its inception, has had over 1,000 comments from readers.  (And I'm very certain those are high signal to noise ratio posts, since I am a bit pedantic about deleting/preventing blog comment spam.)  A hearty thank you to those who have helped improve the content on my blog by adding your own commentary, anecdotes, experiences, and suggestions.  Keep 'em coming!

Potential Cross-site Scripting Attack when entering in a Color

Theres a potential XSS attack to watch for when allowing entry of Color values.

What's the Largest ASP.NET Application You've Worked On?

Whenever working on relatively large projects, I often wonder to myself what size projects other developers around the ASP.NET community are working on.  One of the ongoing consulting projects I am currently working on is a large intranet application to help a company automate a lot of processes that were all paper based.  It's the largest project I've ever worked on myself, with around 75 tables in SQL, several hundred sprocs, about 85 classes abstracting common functionality, over 5,000 LOC for the DAL classes, and over 300 hours of billed work (spread over 1.5 years).

I don't mean for this to be a pissing contest, as I know my project (which has one developer - me) trembles to the sizes of projects worked on by large teams, but I am curious as to the sizes of projects those few kind soles who read this blog work on.  So... you've seen my size, show me your's!

A Gaggle of Information on Self Publishing

Brian Bischof , author of the successful self-published book Crystal Reports for .NET , has provided a great resource discussing the ins and outs of self-publishing, which Brian calls Self-Publishing Tips .  These tips cover the breadth of the self-publishing venture, from marketing, to editing, to page layout, to cover design, to printing.  Brian points out estimated costs for each step, suggested companies, and provides a behind the scenes glimpse of how publisher, distributors, and retailers work in promoting and selling books to the public.  A must read if you've ever considered self-publishing or just want to know more.

Blogs Feed API

Matthew Cosier has working on an API to stream and interpret ATOM 3 and RSS 2 feeds.

You can easily build an aggregator from this becuase you can read multiple feeds in one time before analysing them.

Should be released soon but I also recommend Marcus's tool B.ag , used on Developers.ie for the blogs.

Interesting to see that web aggregators are back in force, instead of using a proprietary application.

 

Non-visual Controls are not just Components

Thoughts on non-visual controls in ASP.NET Whidbey such as SqlDataSource, and the new NonVisualControl metadata attribute.

IIS6 and Windows 2003 (Part 6)

Another aspect of the new IIS architecture that I find very interesting is the new process model called Worker Process Isolation Mode (WPIM).
The new WPIM allows for the isolation of application code in a worker process from other worker processes and also from all of the other architectural pieces within IIS6, such as WWW Service, IIS Admin Service and HTTP.SYS.

With WPIM, this allows for Web Applications/Sites to be managed separately from other applications, and things such as taking an application offline or recycling can be done independently of other applications. It also allows for new possibilities for the administration, monitoring, management and configuration of an application. This allows for the following features in IIS6:

- Multiple Application Pools - This allows you to group certain applications to one (or more) processes that will service the requests for these applications. This allows for different configurations of application pools for different applications (e.g. recycling configuration etc..) and also allows for applications to be isolated from each other (an application pool can be recycled, stopped, started without affecting others).

- Worker Process Recycling - A Worker process can be recycled at a scheduled time, by meeting a certain criteria such as no of requests, by the W3 ADM determining that the worker process is unhealthy or by manually recycling the process. The recycling is done by the W3ADM service and since the requests can be queued by HTTP.SYS when a recycling event is occurring the effect on HTTP being processed is minimized.

- Rapid Fail Protection - this is where an application pool is stopped, due to multiple failures within worker processes being detected within a certain period of time

- Application Pool Health Monitoring - This monitors the complete application pool and all of the worker processes that are within a pool, for unhealthy applications. When a unhealthy application is found (by not responding to pings etc..) the application is terminated and another Application Pool is started to service HTTP requests

- Web Gardens - This is where an Application pool has more than one worker process. In a Web Garden, it is HTTP.SYS that distributes the HTTP requests across the worker processes in an application pool

- Application Pool Identity - Each application pool has its own configurable process identity. By default the identity used is the "Network Service Account" which is has limited rights and privileges.

When developing applications targeted at IIS6 it is important to know that these changes have taken place, as it is may have introduced a change that affects your application and this is especially the case if you are migrating applications from IIS5.x to IIS6.0. In the scenario that something new in the IIS 6 architecture is possible causing an issue, you have the option to revert back to the "IIS 5.0 Isolation Mode", which reverts back to an architecture that is more similar to IIS 5. It is important to note, that you should only really use "IIS5.0 Isolation Mode" if you really need it, as many of the features of IIS6 will be disable. The IIS 6.0 Technical Reference covers this in more detail.
 

 

Viewing WSE Trace Files

One of the neat features of Microsoft's WSE Toolkit is that with the click of a button you can have all incoming and outgoing SOAP messages from a client or service recorded in trace files.  These files, named by default InputTrace.webinfo and OutputTrace.webinfo , can be helpful for debugging or for gaining a deeper understanding as to the actual XML being scurried back and forth between a client and a Web service.  One thing that's always irked me, though, is that these trace files are simply appended to with each run.  This is fine and good, I guess, but it makes it hard to pick through these logs.

Every time I introduce WSE to my Web Services .NET students, I show them the tracing features, and have to sludge through opening the files in UltraEdit32, or Internet Explorer, and paging through the XML and finding the incoming and outgoing messages for the example we had just completed.  Well, I decided enough was enough, so I spent this morning whipping up a simply WinForms application that allows you to load in trace files and view individual messages from them, one at a time, as the following screenshot illustrates (in the screnshot I am viewing just Message #9):

I also made it so that you can view the input and output XML for a given message number, as shown in the following screenshot:

If you'd like, you can download the application and complete source code (C#).  I have to give a big caveat here, though: I am not an experienced WinForms developer by any stretch of the imagination, so you may find glaring UI errors, terrible design, and offensive source code.  Be forewarned.  I learned a lot of new things today and played around with things like isolated storage and resizing WinForm controls... things I've done zero times before, so the code may be littered with mistakes.  Also, I do zero exception handling, so a missing file, or invalid permissions, or malformatted XML, and kablamo , the app's gonna bomb out on you.

Ok, enough of a caveat.  Enjoy the program!

Cryptic WSE 2.0 Error (WSE032 "configuration error" and System.Net.Dns.GetHostName())

While I was teaching my .NET Web Services class tonight I stumbled upon an esoteric error with WSE 2.0 that I had not encountered before - an error that ruined two demos of mine.  I had tried the demos literally 15 minutes before class and theyh worked great.  When I had a full classroom, though, it crapped out.  The error message I got was the standard WSE032, saying:

An unhandled exception of type 'System.Configuration.ConfigurationException' occured in microsoft.web.services2.dll

Additional information: WSE032: There was an error loading the microsoft.web.services2 configuration section

This is a pretty common error message when something has gone awry in WSE.  For example, if you are using UsernameToken authentication, but mistype any values in the Web.config file you'll get this error.  What was a bit perplexing was that, (a) the demos had worked earlier, and (b) the stack trace of the error message referred to a problem with System.Net.Dns.GetHostName() .  As I said, my demos bombed out, but afterwards I did a quick search on Google and found this blog entry: WSE 2.0 (sp1 & earlier) - The mistory [sic] of error code WSE032 .

This blog entry had a fix for my problem.  It appears that WSE uses the GetHostName() method, and this caused some problems, perhaps because my computer's wireless connection died, or my DNS entries timed out, or something.  Anywho, the solution - which worked for me - was to add a line to the HOSTS file in C:\WINDOWS\system32\drivers\etc .  This did the trick.  Probably the most esoteric bug I've faced in my experiences with Web services.