I often do solutionwide versioning. As in generatinig the AssemblyInfo.cs files when building a the deploy target. Before I come that far I just code, build and test in Visual Studio, but go out to the command prompt when the fix is tested and done. Cause I gotta do git commit anyway.

The first thing I do is set a new version number in my MSBuild file. It uses MSBuild Community Tasks for a lot of things, generating AssemblyInfo files being one.

<PropertyGroup Label=”versioning”>
<versioningMajor>1</versioningMajor>
<versioningMinor>0</versioningMinor>
<versioningBuild>0</versioningBuild>
<versioningRevision>0</versioningRevision>
</PropertyGroup>

<AssemblyInfo CodeLanguage=”CS”
OutputFile=”ReportUpdater\Properties\AssemblyInfo.cs”
AssemblyTitle=”ReportUpdater”
AssemblyDescription=”ReportUpdater”
AssemblyCompany=”Systementor AB”
AssemblyProduct=”ReportUpdater”
AssemblyCopyright=”Copyright Systementor AB 2010″
ComVisible=”false”
CLSCompliant=”false”
AssemblyVersion=”$(versioningMajor).$(versioningMinor).$(versioningBuild).$(versioningRevision)”
AssemblyFileVersion=”$(versioningMajor).$(versioningMinor).$(versioningBuild).$(versioningRevision)”
/>

Anyway, one of the problems with git is we dont easily get a numeric version number to use in our fileversion. So what I wanted to have was something like this:

Some sort of connection to the last Git commit hash.

So therefore I had to code my very first MSBuild Task.

Download MSTask for Git Latest Commit here

binary is here http://github.com/downloads/aspcodenet/Git-Last-commit–version–as-MSBuild-Task/GitVersion.zip

How to use it:

<UsingTask AssemblyFile=”\code\tools\MSBuildTasks\GitVersion.dll”
TaskName=”GitVersion”/>

<PropertyGroup Label=”versioning”>
<versioningMajor>1</versioningMajor>
<versioningMinor>0</versioningMinor>
<versioningBuild>0</versioningBuild>
<versioningRevision>0</versioningRevision>
<LastCommit>not avail</LastCommit>
</PropertyGroup>

<GitVersion LocalPath=”$(MSBuildProjectDirectory)”>
<Output TaskParameter=”LastCommit” PropertyName=”LastCommit” />
</GitVersion>

<AssemblyInfo CodeLanguage=”CS”
OutputFile=”ReportUpdater\Properties\AssemblyInfo.cs”
AssemblyTitle=”ReportUpdater Last commit:$(LastCommit)”
AssemblyDescription=”ReportUpdater”
AssemblyCompany=”Systementor AB”
AssemblyProduct=”ReportUpdater”
AssemblyCopyright=”Copyright Systementor AB 2010″
ComVisible=”false”
CLSCompliant=”false”
AssemblyVersion=”$(versioningMajor).$(versioningMinor).$(versioningBuild).$(versioningRevision)”
AssemblyFileVersion=”$(versioningMajor).$(versioningMinor).$(versioningBuild).$(versioningRevision)”
/>

and I look in the CQRS example sourcecode, expecially for the ReadModelupdater http://github.com/aspcodenet/CQRS-demo-20100906/blob/master/Demo/ReadModelUpdater/Program.cs plus http://github.com/aspcodenet/CQRS-demo-20100906/blob/master/Demo/ReadModelUpdater/LogChaser.cs and what bothers me is what you do is polling!!! Polling against a SQL Server database? Why not have CommandHandler publish the events to a queue as well as storing it into the eventstore.

Fair question. And it deserves an answer as its one of the design choices I have made. Lets look at what it means:

That is much nicer. However the BIG problem we have here is now our CommandHandler – meant to execute behaviour on our domain (business logic) – now have another thing it is supposed to do. It has to send a message to the queue as well. From a design view I really like to keep things clear. One process one task. And the biggest problem is this: what happens if there is an error when sending it to the queue? Our business logic functions performed as according to plan but the outgoing msmq is malfunctioning for some whatever reason? That is totally unacceptable.

In practice it also means a longer total transaction length (maybe 2pc) since it now involves the outgoing queue as well.

I actually lied in my header about not using a queue. In effect our SQL Server database (eventstore) IS a perfect queue. By having each listener keep track of where it left off we have indeed a solid  publisher/subscriber  solution.

Now, in cases where we really need to publish to a “real” queue, like to an external system or whatever, we can do that as well. But we dont do it from the commandhandler. Look at this

In this example I have even modified the Readmodelupdater to read from the queue, but that was just to show you you can do that as well. My point is to keep commandhandler do nothing but business logic. Mimimize the risk of failure.

Last: the polling has a sleep(5000 milliseconds).   Where does that number come from? You simply have to ask the stakeholders. What is acceptable SLA. As we are doing CQRS we are allowing eventual consistancy. Now the question is how long before it has to be consistant.

Coincidence, yes, but still, it kind of gives me confidence this is the way we are heading. Databases as we know it today (3NF) are not here to stay forever.

Do I need to say I  pretty much agree with Orin on this matter.  Consistency boundaries based on behaviour is what makes up my models. Basically we need to stop thinking data and nouns. Why would we say the second model In Orins example is denormalized? Some might say it’s because we have “ProductName” in two places. Yes, if a “product” is the same thing as a “ordered product”. And it’s not 100% sure it is…as in “hanged the product name from “Thingamajig” to “Foomiester”, mighty confusing for me when I look at that order and have no idea what that thing was”

Where he kind of misses is taking the next step:  the real key is to being happy with the model as it is right now. Its good for commands and executing behaviour. Lets now just call it write model.

Reporting/reading data is another business. Generating a specific report model based on the changes made to the write model. So lets call the new model “read model”. Have it updated asynchronously from changes from write model.

Thing is for write model, denormalization is still a good thing to avoid (but not the end of the world either), but
when it comes to pure read models, denormalization is really ok.

All in all, the beauty of CQRS.

http://aspcode.net/no-crud-allow-for-saving-user-intent-in-your-gui

http://aspcode.net/the-problem-with-how-we-save-data-today

Now lets get into the how. Fiorst start by downloading the CQRS example from githib.

This pretty much implements the architechure layed out in this diagram

I will do some complimentary lowlevel posts later, but this one need to come first.

Choices I’ve made

- I have chosen Castle/Windsor for IOC

- I have chosen RhinoEBS as bus

- I have chosen SQL Server as eventstore. Although I do have another driver implementing the IEventStore interface which I use internally. I’ll might get into that in a later post

- I have chosen Protobuf-net for serialization

What does the application do?

Its a fake GymMember application. A Windows Forms GUI (GymGUI) allowing you to view/search the members. Doubleclick on a member to Upgrade/Downgrade the level.

GoldMember is implemented by business rule: I think it was like if trained 12 times in 30 days they get upgraded to gold member. If they then dont train at least 10 times in last 60 days they gets downgraded. The idea is to have (nightly) batch use the CheckGoldMemberStatus command to have it calculated each night. But as I said, much havn’t been done, point is to show the infrastructure.

Projects in solution

Bus – just a helper for command dispatching when chasing the eventstore (used by ReadModel updater)

Eventing – infrastructure for eventsourcing – and eventhandling in general

EventStoreSQL – implements IEventStore interface and is a driver for SQL Server

GoldGymMemberCommandHandler – command handler (look at picture). Receives messages from client(s). Executes domain behaviour

GoldGymMemberDomain – business logic

GymGoldMemberCommands – commands. Client sends commands to CommandHandler

GymGoldMemberEvents – events. Business logic (domain) publishes events. Please note that I use the eventstore database as queue – all eventhandling is done asynchronously and in other processes

GymGoldMemberRepositories – implementation of GoldGymMemberDomain.Ports.IGymMemberRepository

GymGUI – client app (WIndows Forms)

ReadModelUpdater – asynch process to update the read model

To get it running you need to:  run SQL Server database.txt to create the database tables, change app.config wherever there is a sql connectionstring to point to that database. Make sure MSMQ is up and running.

The key part is starting GoldGymMemberCommandHandler first so the queue gets created.

Typically you should have GoldGymMemberCommandHandler, ReadModelUpdater running ALL the time.

-

Let me ask you. When you are faced a new business problem, where do you start? Do you start by trying to model the database. Maybe with UML you try to create a third normal form model from which to base your business logic from. Maybe even using NHibernate or Entity Framework 4 to get the base classes and loading/saving/updating for free. Hehe, now all we need to do is implement the business logic

Now after a few month the boss comes telling you there has been security problems  in your application and you’ll need to start logging ALL changes done to it. Ok, since you are using SQL Server, why not whip up some clever triggers for each and every table. Whenever a column is updated our trigger will insert a row to a logtable, along with date, table, column, who dot it etc. Trust me, I actually have heard a developer being proud of managing to implement such a solution.

Lets start from the beginning: Why do we want to create our database in 3NF? Well, we have been told so. It’s reasonable fast for inserts/updates as there are no duplicates. So, to be honest, for our domain model is is not such a bad choice. There is of course a risk of having many many tables being involved in a single transaction, therefore needing multiple locks – and longer running transactions as well.

For pure reading, 3NF is NOT good. Wouldn’t the optimal table structure be one table per DTO, as in for each form our GUIs need to present, data has been precalculated and all we do is “select * from  formcustomers_dto1 where id=…”.  Instead, what we often do is call our business layer to load up the object graph needed, use something like AutoMapper or handcode the DTO creation where we do aggregates or simple just pick out the few fields actually needed. Or even worse, we send the whole entities to our client, edit data, send it back to our business logic  layer – then we use an ORM to calculate the delta (changes) and save it.

Ok, we can agree the type of databases we develop today (using UML/3NF approach) are reasonable for writing, bad for reading. Now go back to your own system and think about how many percent of the database accesses are made for pure reading or writing…

Ok, so far I’m just outlining some problems we have when we develop like we do today. Next, even bigger problem, is what I started off with. The fact that we typically just store current state. In some cases we do introduce “logging” tables. But all to often its too late. We add it after the fact. Simple, we have no possibility to foresee future business requirements , so why don’t we make it a rule to ALWAYS store a full log of state changes?

Simply because of costs. It costs diskspace. It costs development time, we need our DBAs to create logtables for each our “real” tables, there is also a storage cost. And what happens when we add a new field to our business class. We then need to add it to our logtable, and TWO ORM mappings must be  redone etc.

The answer to all these toubles are in a concept called eventsourcing. Now before I move on I wanna make it clear. Its a lot of fun coding – at a much lower level than needed for the ORM/SQL approach, and me being a tech nerd of course loves that, but the fact is, we do it for business reasons. Imagine the first example again with “the boss comes telling you there has been security problems  in your application and you’ll need to start logging”. You say, we already are. Tell me what I should look for and I can easily see who the offender was, what he did. Oh it did happen 10 years ago. No problem. Just tell me anyway, I’ll have it for you  without even needing to restore a backup.

And performance reasons. Imaging having a table where you never read/update data. All you do is insert. Thats pretty much all that ever ever happens inside a transaction the CQRS style.  Earlier (3NF style) a transaction might have involved ten or twenty different tables. Insert/update whatever with locking. Now we do insert statechanges. Thats it. And we have NO other processes/users needing to aquire readlocks on the table.

What many developers do when faced to create a GUI for such a task is create a generic data entry form, kind of like this:

Cool, its pretty generic, I can use it for new/update with a simple flag, and thanks to me being so clever using EF4/NHibernate/insert any ORM here, everything is set. I feed it/bind some data and then when I call SaveChanges it will be in my database. 30 minutes, Done!

So, whats the problem with that approach? Well for starters, all we have talked about is data. We havn’t even talk about behaviour, meaning what should be able to do with data. In this case we want to just modify the “Level” and “Is gold member” flag.

“Ok no problem. I’ll add some ReadOnly = true;” and were done again…

Ok. It seems like we have solved our problem. But, now lets say, a few months later the boss says, from next week we need to send an email to billing apartment when a member is being upgraded from Beginner to Intermediate.

And you say: “what, upgrade? Oh, you mean they change the “Level” field from Beginner to Intermediate. Now I see. ” And off you go hitting Visual Studio again, coding it into the very same form.  In your Save button handler, if (oldlevel == “beginner” && newlevel == “intermediate”) SendMessageToBilling…

I bet you already see what’s coming. Is there a risk of email being sent which shouldn’t?  Since all we have is CRUD (modify data) we face the risk of having other level changes (such as pure error corrections) being treated as “upgrades”.  Someone might have done an error when registering. Does that happen? Well, in most systems every day more or less. Bottom line: In our GUI there is no distinction between corrections and upgrades.

Ok, the example might be very madeup, but the trouble with CRUD system is having double meaning for the same “Save” button .  And these types of all-in-one-data-entry CRUD forms surely invites to these types of errors. And I am not talking about programming errors here, I am talking about real business problems.

If we instead go back to “what should be able to do” and behaviour, we start by asking “What are we supposed to do”?  And we might get answers like: We want to be able to upgrade a user from Beginner to Intermediate level, or Expert level, and when that happen an email should be sent to billing.

And that type of dialog should be the base for our GUI:

This makes the GUI much clearer, as in letting the user KNOW what the effects will be.  Now in our code we have clearly distinct executions paths for these veru different use cases.

Now where do the bashing on ORMS (“so clever using EF4/NHibernate/insert any ORM here”) come in?

That’s the next step. Instead of storing “current state” in our database, what about storing all statechanges instead. Causel GUI is one thing, when using an ORM we still leave it to the UOW of our ORM to keep track of changes – and it will simply calculate the delta (changes) and store what it need to store, so the end result in our database is still the SAME in our better GUI as in the first one.

All in all, so far with out “clear user intention GUI ” we havn’t won much I guess… Therefore, for tracing, and most important for business reasons – wouldn’t it be great to save be able to save ALL changes done. This way we get a log like

2009-12-01 LevelCorrected from a to b

2009-12-09 LevelUpgraded from b to c

That’s the nest article – as we get into saving events.

That scenario by itself might not be very common (it simply shouldn’t, but I am at early stages in my CQRS exploration, hence my decision to change a pretty technical aspect ).

Put that aside, the beauty of CQRS is it allows you to do application upgrades etc, typically just putting a new version in production – WHILE USERS STILL ARE USING IT.

CQRS could very well be the end of, “oh, I need to stay at work til late when all users have logged out to upgrade the system”.  To be honest CQRS by itself is not the answer, a loosly coupled architechture is (and it could be achieved by other means than adopting CQRS), but in my point of view CQRS really drives you (the developer) to  loosly coupled architechturing.

Here is a flowchart (click on it to see full size):

By utilizing persistant queues, and of course adopting the thinking of eventual consistancy, you could very well take down the “Business Logic” process (commandhandler) while your users keeps using the client – upgrade it, and start it up again. And it will start processing commands where it left off.

I had a base class
[code language="C#"]
EventBase
Guid AggregateId
DateTime Created
int SequenceNo
[/code]

My applications “events” inherited from this base class like
[code language="C#"]
public class ProjectActivatedEvent : EventBase, ICustomBinarySerializable
{
public ProjectActivatedEvent(Guid aggid, bool active)
: base(aggid, DateTime.Now)
{
Active = active;
}
public bool Active { get; set; }

[/code]

For storage of serialized data I used a SQL Server table (eventstore)

and serialized using Greg Youngs fast serializer (ICustomBinarySerializable) since its really fast and effective

My serialization code looked like
[code language="C#"]
public void WriteDataTo(SerializationWriter writer)
{
base.WriteDataToBase(writer);
writer.Write(Active);
}

public void SetDataFrom(SerializationReader reader)
{
base.SetDataFromBase(reader);
Active = reader.ReadBoolean();
}

[/code]

Now I wanted to do a major rearchitecturing of my code.

For starters, get rid of EventBase and introduce a Interface IEvent instead which all events inherits from.
Second: Use protobuf-net instead.

Basically:

[code language="C#"]

[ProtoContract]
public class ProjectActivatedEvent : IEvent
{
public ProjectActivatedEvent()
{

}

public ProjectActivatedEvent(Guid aggid, bool active)
{
AggregateId = aggid;
Created = DateTime.Now;

Active = active;
}
[ProtoMember(10)] public bool Active { get; set; }

//IEvent
[ProtoMember(1)] public Guid AggregateId { get; set; }
[ProtoMember(2)] public int SequenceNo { get; set; }
[ProtoMember(3)] public DateTime Created { get; set; }
//

}

[/code]
And there I was. I had two branches of my sourcecode, one the old way, one with the new way. Tests shows everything is working ok.
How can I now convert the (1500000) existing records in table eventstore to the new format?

The solution was to introduce JSon into the equation. So (using my old branch) I coded a app which loops all (old style) events, possible since the old code branch knows how to deserialize them of course,  and converts them to JSon. Storing them into a intermediate table.

Second: from the new code branch I simple coded a new app which deserialized the JSon (into my new classes), and stored them the
Protobuf way.

The beauty of JSon (de)serializer is that it simply works on field names. I did not change the names of my fields so
it was therefore possible to use it as a pretty effective intermediate storage.

It did take some time running these loops but at least it worked.

I hope to present some more articles on this (and related) matters but want to show you the the trick to make a generic
(de) serializer out of Json without the need to specify the type:
[code language="C#"]
class Json
{
static JsonSerializerSettings jsonSerializerSettings =
new JsonSerializerSettings
{
DefaultValueHandling = DefaultValueHandling.Ignore,
NullValueHandling = NullValueHandling.Ignore,
MissingMemberHandling = MissingMemberHandling.Ignore,
TypeNameHandling = TypeNameHandling.Objects
};
public static string Serialize(object theObject)
{

return JsonConvert.SerializeObject(theObject, Newtonsoft.Json.Formatting.Indented, jsonSerializerSettings);
}
public static TType Deserialize(string sInData)
{
return (TType)JsonConvert.DeserializeObject(sInData, typeof(TType), jsonSerializerSettings);
}
}

[/code]
The secret is TypeNameHandling.Objects. This adds a “$type:<typename>” in front of the JSon when serializing and
therefore it is able to create such an object when deserializing.
I still do have a TType, can’t remember why, since I stole thew code from an old project I made years ago, but the
point is you can send in Object or a baseclass as type.

I deserialize my data like:
Eventing.IEvent ev2 = Json.Deserialize<Eventing.IEvent>(row["data"].ToString());

And serialization was like
Eventing.EventBase b = …
string sa = Json.Serialize(b);

But first I thought I’d start with explaining why I feel I need a servicebus (at all) as doing that might show some light of what it’s good for, it might help you decide if thats suitable for you as well.

First ask yourself if your application is distributed at all. Even if it isn’t (for example I have a lot of plain client/server oldschool apps which are nothing but client app and database server) – it might gain some my making it distributed. I wont get into all the reasons why such a decision could be beneficial, as thats nothing I can say something about, but in general if you centralize your applications core code your making it available for other  type of clients, like web. Also scaling (out, not up) gets easier.

It might also be that you want asynchronous commands, where you delegate actions such as “Send this email to this person” I dont have time to do it cause I am calculating this very valuable rate or whatever right now. This also gives you the effect of a clear architechture where you split different things up.

A servicebus abstracts messaging and transports

What do I mean by that? First of all, a message could be seen as a serialized command.  Say I sant to send that command “Send this email to this person”. Then a message must be delivered to that service. Typically you setup (in configfile) the different endpoints (servers) you want to talk with and do something like bus.Send(new EmailCommand(…)) .

A servicebus also abstracts HOW the messages are being delivered. Do we want to use MSMQ or ActiveMQ? Thats also configurable – and typically also possible to extend (plugin) more transports.

A servicebus helps you with client AND server development

Above we looked at how the client programming becomes really easy. A servicebus should also help you with programming the server part (in our example the email service). It should be configurable the same way as the client. It should in short handle all the transport details, and even more it should allow you to just write the business logic for each command (type of message), not much else. No plumbing code.

A servicebus decouples messaging

Fire and forget. Just like delegation of work to a collegue. We should be able to Send the message. And go back to the more important things we were doing.

A servicebus should help setup pub/sub

Apart from the fact that transports are pluggable, you would still benefit from a bus even if you have a plicy to just use a single transport, like say MSMQ – or WCF for that part. Instead of programming directly against it, when you use a servicebus you gain the benefit of some nice message distribution scenarios. The bus lets you setup and handle publisher/subscriber scenarios, where you might have multiple listeners (servers) for a certain message.

A servicebus should help setup sagas

A saga is a serie of commands. Like a workflow. I wont get into this right now, but this is also a thing you wont get out of the box from the MSMQ api.

A servicebus goes well into CQRS scheme

I am now developing all my apps the CQRS way. The benefits and reasons for that is out of scope for this article

I attended a Greg Young session at NDC in Oslo and really had a feeling, thats the way applications should be developed. Its been a long time since I had that feeling, I have been comfortably developing my apps basically the same way for the last 8 years or so this is not only refreshing but also a lot of fun, me being the tech nerd I am…

So for about a month or so I have been trying to get a POC CQRS app up and running. Using nServiceBus and MSMQ as transport.

The whole experience (far from finished) can be read and full source can be downloaded from the CQRS NServiceBus example site