Bits that bite

Another case for code contracts

2012-01-27T16:10:00.000-08:00

I stumbled upon one of the many articles on inversion of control (IoC). Interesting enough, one of the arguments the author used for utilizing interfaces was that it enabled you to safely change the actual implementation of the interface later on. I have to disagree on that point. In fact, in some way I think that depending upon a specific class rather than an interface is more 'honest'. Not saying that you shouldn't use interfaces: IoC is a very powerful and useful pattern. One issue I have with interfaces is that there aren't many interfaces that actually define a contract in terms of pre-conditions, post-conditions and invariants. Without a contract you can still only guess what the interface provides to you. If you rely upon a specific class, you at least recognize that you are dependent upon the implementation details of that class. When you rely upon an interface without a contract, you are basically saying "I rely upon an interface that has no clear specification, but I trust anyone that implements it not to violate my interpretation of the interface." Again, I can only hope that the code contracts currently provided by Microsoft Research will become a core part of .NET.

During the "meet the experts" session at the Windows Build conference I talked to some Microsoft employees about model checking. Their first reaction was, with a somewhat surprised look: "Do you develop driver software?" My answer was "Nope", but it was a clear indication that formal methods are still only generally accepted in very small pockets of our industry. Maybe over the next couple of years Microsoft will finally be able to make some of these formal methods readily available to all programmers.

Back to basics

2012-01-10T22:16:00.000-08:00

One of the projects that I work on involves customization and modification of an existing system. The original system was however not architected for change. Beyond the rather long list of architectural and implementation issues that surround the system, one of my colleagues noted that the system was really a world by itself. This brought me back to the thought that larger systems have organic aspects to them.

In the end, simple organisms are favored above complex organisms when disaster strikes. Cockroaches are for example more likely to survive nuclear holocaust than humans. Are simple applications more likely to do well in a changing environment? I have lately been leaning strongly toward limiting custom frameworks and not 'over architecting' applications. The closer you can stay to mainstream constructs that you can expect any programmer to know, the better. As part of the 'less is more' strategy, I have for example ensured that the interaction with Windows Identity Foundation is completely transpartent to the programmer. We have also started to refocus on basic concepts, like encapsulation, contracts and decoupling and making the developer aware of these concepts in an intuitive way through the IDE. I understand why the industry aspires to extreme test driven continuous integrated agile development :), but I also don't want to loose sight of the basics.

Code Contracts and more ...

2011-10-20T11:19:00.000-07:00

There is so much fun to be had and so little time. A list of some of the areas I have been working on (some professionally, some for my own fun):

Code Contracts: A Microsoft Research project that brings pre- and post-conditions and invariants to .NET. Although I understand the limitations, I have been very impressed with the capabilities it brings to the table. For example: after adding code contracts to existing code, the suggestion was to add a restriction on the size (at least 20) for an input parameter of type string. Rather odd requirement, which turned out to be the consequence of some substring calls inside the method in question. Obviously it was time to weaken the pre-condition. I can deeply appreciate a tool capable of pointing these issues out.
PEX: Yet another Microsoft Research project that brings automated whitebox testing with high code coverage to .NET. In combination with code contracts, it rocks. You will never have any excuse for an 'object is null' error again.
Kinect for Windows 7: Yet another other Microsoft Research project bringing the Kinect to Windows. What else can I say, NUI is finally available for the masses and the way we interact with machines has changed for the better.
Windows 8 METRO applications: After my visit to the BUILD conference, I have been trying to keep the momentum going and get more familiar with the new Windows 8 METRO applications.

Ofcourse I still have actual projects going on and fun questions to answer like for example 'What does code quality mean to us?' Luckily Fall has finally kicked in and I have two additional hours a week available that are normally spent on mowing the lawn. That will make a big difference :).

Arduino ...

2011-09-23T09:07:00.001-07:00

Been tinkering around with the Arduino boards. It is a blast! They are somewhat of a pain to interface with, but it's been good to learn in detail about bluetooth profiles and wiring :). You can do some wild stuff with these boards, so go and have fun: figure out how to open your garage door from your Android or change tv channels from your laptop.

My time at 'ask the expert'

2011-09-15T19:13:00.000-07:00

Even though you can't really expect answers that actually mean a lot, it is a great way of interacting with some of the presenters/MS developers. So my questions were:1) With me working for a retail company (blah-blah), can we expect motion (not touch, but for example kinect) to become a first class citizen? Mixed response: one guy said just use a webcam and have the user hold lights in their hands so your software has something to track through the webcam. My answer: really? An other actually found the idea quite fascinating and although it is not there right now, he said that use-case scenario's for kinect outside the xbox are definitely being considered. I liked the answer of the last guy better :). My plea: if you are pushing these slates as gaming devices, support 3D and kinect from win8! And then we can also use standard, affordable, hardware in retail environments :). And please provide standard gestures (culture sensitive? lol, thinking about different finger arrangements meaning different things in different cultures) with the kinect. Just like touch, a set of default movements would be helpful (restart is jumping up and down while having an angry face ? :))2) SPIN like model checking in VS. To my surprise they are having a talk about this tomorrow! They are using data paths (which is slightly different), but hey, it's at least a form of model checking :). I mentioned the ability to generate test cases out of your model checker, but that didn't seem to resonate.3) Analysis of large amounts of historical data in the cloud. The answer: unless you can figure out how to use blob data, you will pay a dear price for storage or band-width. That figures :).4) Suspend mode: while your app is running, save state, so you don't run out of your limited time to save things once you are suspended. The user can choose a certain number of apps to always run, but no guarantee that will be your app! Also: you can schedule tile updates while the app is running or gets notified of suspension. This way you can still provide the user some sort of message after a given amount of time. However: if there is a server task running and your app gets suspended, you will no longer be able to track progress of that server task :)

Windows 8

2011-09-15T07:23:00.000-07:00

I am spending this week at the Build conference where Microsoft just released their preview of Windows 8. After the initial message of "everything is okay, you can still use what you know", we had some more in-depth sessions yesterday. And here is, at a high-level, what they have done. They have taken COM and made it a "first class" citizen in .NET. Their new WinRT layer is actually COM. But COM is all about interfaces, and .NET is all about objects. So what they have created is metadata for all the winRT interfaces and mapped those into object oriented equivalents that are exposed in .NET. Thus you no longer need to deal with COM interop when talking to winRT (their framework takes care of it for you).

The other messages is that everything has to be async if it takes more than 50 ms to access.

Here is the rub. Last night I set out to build a new METRO style app to see what it was like. I set my goal low: read some XML in from a file and display it in a grid. Been there, done that, shouldn't be that hard, right? Here are some of the challenges:

The first challenge is getting the right to access a file. I was storing the file in the document library, so I figured there was probably some kind of privilege setting required in the new application manifest to allow access. And there was :). But after changing all that good stuff I still ended up with a blank screen. Huh. They mentioned that if you didn't have the right privileges set, you would get some kind of an error. I am yet to see that, the UI was just blank and no error other than in your debugger. Anyways, after more digging into one of their sample apps, I figured out you also need to specify in the application manifest the specific file extensions that your application will be associated with. If you don't, you will get an 'access denied' error.

Second challenge was actually reading the data. Obviously, since it could be more than 50 milliseconds :), that is an async call. So you go into their new 'Windows' namespaces and find the classes to read a file async. Not that bad. However, here is the rub: all of the winRT functionality is mapped into interfaces. Remember that this is actually COM that is interface oriented? So I like using the classes in the System.Xml.Linq namespace for reading a stream into an xml document. Oooppsss ... you are not getting a stream back from the async file read, you are getting one of the WinRT specific interfaces back. So after some digging I find this new 'Windows' based xml document class that does allow you to process the interface that winRT returns.

Even more fun, winRT doesn't allow you to just inject regular .NET classes, it specificly wants winRT classes in many scenario's.

If this was confusing: sorry, it is also somewhat confusing to me :). Right now it feels like you have to two frameworks awkwardly rolled into one: the .NET framework and the WinRT .NET framework. And they both for example have a way of dealing with Xml documents. So you are choice is to put all these conversions together between winRT and the 'desktop' .NET classes you are familiar with, or you live in a divided house where there is yet another way of working with an XML document. Developers are require to once more split their personality. You already had a WinForms, ASP.NET, WPF and SL personality, now you can start to raise your winRT personality.

Implicitly typed variables

2011-06-11T11:13:00.000-07:00

Recently I got into a discussion about implicitly typed variables in C#. It was my fault, I asked others for their opinion :). The general observation was that implicitly typed variables are mostly syntactical sugar and a sign of lazy coding. Except, of course, when you use anonymous types.

There is however an other, and I think very valid, reason to use implicitly typed variables that goes beyond anonymous types or lazy coding (yes ... lazy coding can be a valid argument :) ). In short: implicitly typed variables allow you to make the weakest assumption about the type of the variable. And one common problem in applications is making stronger than needed assumptions and copying those assumptions across your code.

To give a more concrete example:

var s = myObj.GetValue();
//….
string myText = s.ToString();

versus

int s = myObj.GetValue();
//….
string myText = s.ToString();

In the first we only require in our code that the value returned by GetValue() implements ToString(). In the second, we actual make the much stronger assumption that GetValue() will return an integer. If that return value is ever modified to be a real, we will need to modify the second implementation, even though the fact that we changed from integer to real did not impact what the code does.

To give a second example:

string[] values = anObj.GetValues();
foreach(string s in values) {}

versus

foreach(var s in anObj.GetValues())
{
}

The second implementation only requires the return value of GetValues() to implement IEnumerable. The first requires GetValues() to actually return an array of strings.

To recap, I think implicit variables have a valid use. I have always felt that if a language has a feature, there must be strong reasons for barring it from your code. I don't see a strong reason to bar implicitly typed variables from code. I actually see some advantage in using implicitly typed variables when you look at assumptions and contracts in your code.

Windows 8 commotion

2011-06-06T11:08:00.001-07:00

Microsoft announced Windows 8 and mentioned HTML5+JS+CSS often ... but no word of XAML or .NET. So that has caused some commotion. In the end, I get why you would want HTML5 to be the presentation layer for your desktop apps as well. For one thing: how much time is MS spending on web versions and desktop versions of exactly the same thing? Even if you can get them about 95% the same, that's still a big gain.

But it is a huge paradigm shift. And I have this uncomfortable feeling around browsers, even if it is just their rendering engine, becoming part of the OS.

Whatever the future holds I guess :) . MS will not give developers more info until their Build event in September, so until then all we can do is utilize the best practices of keeping UI, UI logic and business logic tiered.

Shared Services

2011-06-02T11:30:00.000-07:00

Since I really enjoy technology, I can endure a lot of organizational pain. But lately I have been starting to clearly notice some of the negative side effects that a shared service organization can have. I cannot remember ever having to deal with such a large number of forms, requests and hand-offs (business users, business user representatives, BSA's, QA, architects, developers, support ...). All would be fine if there was not such an insane amount of time required for completing all these interactions: meetings to plan resources across these services, meetings with the service providers on outstanding issues, KT (knowledge transfer, ... don't even get me started on that one, brain transplants are the only quick solution to that one ...), waiting for service requests to be completed, coordinating amongst different service providers ... Insanity is only one more shared service away. What once was a task that took maybe 2% of your time, is now a shared service that takes 20% of your time. So the advantage would be that we are capturing all this knowledge ... Yep, but a lot of it is not captured or only captured in small fragments scattered across all the service providers.

I am not specialized in organizational structures, so I assumed it was just me. Luckily, there is at least one other person with similar observations:
https://www.qualitydigest.com/inside/quality-insider-article/shared-services-paradox-it-increases-costs.html#

To quote:

John Seddon of Vanguard Consulting discovered that variety of demand is a key leverage point for services. All the simplification, standardization, and centralization that support cost reduction and improvement projects lead to failure demand (i.e., demand caused by a failure to do something or do something right for a customer). It’s an inside-out, top-down way of thinking that makes service management oblivious to the impact and total costs.

When services are shared, demands are often found to be different, even though the function may be the same. Combining like functions (e.g., contact centers, HR, finance, and IT) that have different customer demands does not decrease demand. Failure demand, which typically runs between 25 percent and 75 percent, and the variety of demands are still present: They are “shared” but not reduced.

In industrialized design, back offices are ever-present. This functional design is rooted in the scale thinking that accompanies industrialized thinking. To free up customer-facing front offices, work is passed to back offices. The front office too often can’t actually help a customer and so the waste begins. This waste comes in the form of hand-offs, rework, duplication, and loss of continuity, which increases the time to get service. The act of sharing services increases the scale of this design and, in turn, increases the waste. More failure demand results from customers who don’t receive service quickly, accurately, or sometimes at all. Costs rise as a result.

This doesn’t mean that all sharing of services is bad. However, when we:
• Accept back offices or other industrialized designs
• Ignore the variety of demand in services
• Assume economies-of-scale thinking
• Merge functions in the name of simplification, standardization, and centralization

We run the risk of making a costly mistake.

An other person is suffering under the same illusion ... I am not alone! :). The little nugget I found in the above article was that the argument for a shared service is often reducing cost through scale. As the article points out however, cost is not in scale, but in flow.

Service organizations see the visible cost savings that can be achieved through shared services. What they don’t see are the hidden costs of poor design, and the poor flow that results. The problem is that costs are not in scale, but in flow.

This is the truth given to us by Taiichi Ohno, mastermind of the Toyota Production System. Economies of flow are superior to economies of scale. Whereas scale focuses on reducing transaction costs, an economy of flow focuses on end-to-end value as defined by the customer.

Back to the coding

2011-05-14T10:25:00.000-07:00

The last couple of weeks there has been a lot of work related to planning projects and roadmaps. In the middle of all that, writing some code keeps me sane :) . One of the outcomes of the roadmapping exercise was that WPF/Silverlight is in our future. I don't like the term 'rich UI', because it's not that a web based interface is not 'rich' (certainly with some of the new HTML 5 features that is no longer the case). To me it is more about the ability to have a stateful client that is written in a compiled language like C#. We have a large demand for stateful, highly functional, user interfaces. A common scenario is that a large model has to be re-evaluated on the fly as the user enters new values. With javascript and AJAX this is still a challenge and results in poor performance and maintenance issues.
Since there is also already a WPF/MVVM project in-flight, I have taken this as an opportunity to dig into MVVM frameworks. That is one way of learning a new technology :). The challenge is that Microsoft does not have direct IDE support for MVVM. PRISM is out there, but it tends to be a little heavy weight. After evaluating some of the other frameworks available, I ended up with Jounce (http://jounce.codeplex.com/). Jounce became my experiment of choice, since it is actually very close to some of the initial thoughts I had around MVVM. It utilizes MEF for managing the different parts of the model. With MEF it's a straight forward exercise to dynamically load additional parts and have them automatically snap-in to your application. MEF is a natural match for the loosely coupled architecture that MVVM promotes.
The first thing I did was rip out the existing event aggregator in Jounce and replace it with a Reactive Framework based event aggregator. Since Reactive Framework has hit the market, there is no longer an excuse to roll out your own pub/sub implementation. It also removes some of the ugly code needed for having the event aggregator manage the subscriptions. Within the Reactive Framework the publisher is not responsible for managing subscriptions.
It also turned out that MEF keeps a reference to each object it instantiates. With non-shared view models this results in a memory leak. A shared view model is a singleton: there is only one instance of the view model during the life of your application. A non-shared view model can have multiple instances during the life of the application. Each instance should be free for garbage collection once it is no longer utilized. Since MEF keeps a reference to each object it instantiates, non-shared view models will not be garbage collected. In general, the Jounce support for non-shared view models seems like an afterthought. The Jounce interfaces have many assumptions in them that only apply to 'shared objects'. Strengthening the support for non-shared view models resulted in the following changes:

Heavy modification of the interfaces in Jounce to be agnostic to the part creation policy. You can now simply use the MEF part creation policy for indicating the shared status of a view model. I choose for never having shared views, since views are driven by the view model and should only be bound to a single view model instance. To keep MEF from creating a reference to the instantiated non-shared view model, some additional plumbing is needed for creating a temporary child container. The use of a temporary child container is described in detail in the MEF documentation. The main thought is that the MEF reference to the instantiated object is released when the child container is disposed.

Several classes were referencing view and view model data that was imported through MEF. I consolidated that into a single class that exposes the information through an interface.

A side effect of having a non-shared view model is that MEF recomposition will not occur. MEF does not keep a reference to the view model, so it can't perform recomposition for the non-shared view model.

A common requirement is support for different cultures. The original Jounce implementation drives menu names and tooltips through an attribute. This is a rather convenient approach, but it puts up severe challenges for internationalization. So I provided menu/tooltip routes (very similar to the other routes that can be configured in Jounce) that allow you to dynamically get menu/tooltip text. With additional support for a 'Language' property on the base view model and resource file management, this provides a rich set of options for internationalization. As a bonus, I added a public event (through the event aggregator) that notifies listeners when the culture changes and support for a 'CultureDependentAttribute' on properties, so that the we can fire 'OnPropertyChanged' automatically for all culture dependent properties when the culture changes.

The other observation I made was that many of the quick start solutions provided with Jounce are not always clean examples of code that I would like to see in actual production environment. One lesson I have learned is to provide clean examples. Examples are there to guide people that are learning the framework. Chances are they will not recognize that you were just quickly providing a rough outline instead of an actual example of how to correctly utilize the framework. So I spent some time cleaning up the quick start solutions while I was testing my changes to Jounce.

Independent of the question if I will end up utilizing Jounce or not, analyzing and improving a framework is always a good exercise. You end up getting deep insight into underlying issues and solutions while at the same the time sharpening your awareness of the consequences of design choices.

During a discussion with an architect last week I brought the following point up: "Have you ever NOT ended up regretting [for any number of reasons] cutting significant corners on your architecture?" His answer matched my experience: "I have always ended up regretting it."

Awesome

2011-04-12T13:25:00.000-07:00

I have to admit that software development is pretty awesome. In what other job can you take quotes directly from the Sunday comics and seriously use them in work conversations? I love Dilbert. As part of "don't take things seriously all the time", I have also been reading "Why Software Sucks" by David Platt. My favorite part: Your software has zero value in and of itself. Nothing. Nada. Zero point zero zero. How bad things get made:

General incompetence: I’d hire the worst people, pay them poorly, give them bad equipment and unpleasant working conditions, and yell at them often.

Unclear purpose: I’d never explain the goals, never rationalize the non-existent business plan with the non existent product plan, and randomly change my mind about important things every few hours. Once a week, in the morning, I’d act just sane enough to gain people’s confidence, only to obliterate it in the afternoon.

Unplanned design: I’d wait as long as possible to think about what the customer’s experience should be, so that decisions that most impact the people I’m designing for have the fewest resources, the most constraints and the lowest possible probability of a quality outcome.

Poor engineering: I’d demand people build things that frequently fail in dramatic, surprising and dangerous ways.

Make customers miserable: Our only team motto will be “They must suffer more”. We’d watch the Monty Python Spanish Inquisition scene to master all of our chief weapons, ensuring maximum suffering for every customer we have.

How good things get made:

Gift mentality: Good things in the world come from people that have the gift mentality. They do care about who they are designing for. They are sincere about trying to build something that will satisfy a person’s needs. They are willing to expend time and energy refining their thinking and developing new skills so that when they are finished they can sincerely offer what they’ve made to the world as a good thing. They see their work as a deep expression of generosity and as an attempt to live up to their own ideal of quality and workmanship.

Thoughtful and passoniate: Good programmers, designers, architects or creators of any kind are simply thoughtful. They are so passionate about making good things, that they will study any discipline, read any book, listen to any person and learn any skill that might improve their abilities to make things worthy of the world. They tear down boundaries of discipline, domain or job title, clawing at any idea, regardless of its origins, that might help them make a better thing.

To directly apply it to one of the projects I have been working on lately:

Your (MMVM) framework has zero value in and of itself. Nothing. So don't bother programmers with silly code and plumbing to make it work. Just make it work, so the programmers can focus on the actual application.

A change will do you good

2011-02-23T19:08:00.000-08:00

Although I have been enjoying diving into reactive framework, parallel extensions, EF 4.0 and tooling (TFS), I got a request to see if I could do some work on a PDA for someone. Completely different platform and programming language, ... but actually quite fun. Wouldn't want to be the one though to have to maintain a large code base for a PDA. Way too many breaking differences between OS versions and different hardware that it runs on.

Touch

2010-10-20T19:28:00.001-07:00

Having a blast with the new touch SDK beta for Windows 7. And it seems that the intend is that the next generation Surface will also support this API. In the end: now that we have social media, we want more social computers and software. No longer the boring, keyboard and mouse, in the corner of your office, black box. Nope. Sleek. Integrated in every fiber of what we do. A nod here and a general wave there and I just finished my work kind of deal.

http://blogs.technet.com/b/microsoft_blog/archive/2010/10/08/more-like-us.aspx

More on software factories

2010-07-14T19:20:00.000-07:00

It seems to be a rather popular point of view that implementation is an almost mechanical process once you have the specification written. An assembly line in the style of Henry Ford: put the bolt on, next project. The observation I would like to make though is that putting the bolt on a car is a task that has very limited variation and is repeated exactly the same for each car. In software development it is best practice to put these tasks in a library that we re-use across projects. The challenge in software development is that projects are uniquely different. And that the requirements are usually very unclear. It's more similar to Henry Ford telling an assembly worker: "I would like a bolt and nut thingy somewhere along the exhaust pipe to keep the thing from falling off". The only way to get a clearer specification is to be as precise in our specifications as car manufacturers are: use 3D modelling and simulation tools to determine the exact strength and location of the nut/bolt to minimize failure. Why does it pay off for car manufacturers to invest in such a precise specification where the location, dimensions, materials and strength are determined in such detail? Because after doing that they will be able to mass produce the product easily and limit liability (with failing parts and other ugly accidents). And still, as we have recently seen, they will at times fail in building a product that meets all specifications. In software development we can only reach that level of detailed specification by formally specifying the application and performing model checking. I love that approach. But it's not very popular for mainstream business applications. It requires highly specialized individuals and large amounts of time. So we resort to being less precise and incrementally refining our attempts to build something that seems to at large meet requirements. Obviously, anyone that functions as 'a human model checker' is going to be wildly more successful during that process than someone that needs instructions like 'put bolt 1 on nut 2' .

Soon to be a commodity

2010-03-29T20:17:00.001-07:00

3D games for X-Box, Wii, etc.: going to be a huge market. Touch screen for PC's. When I see how easy it is for my kids to use a touch screen while they are still rather clumsy with a mouse, I know it's going to grow market share rapidly.

In the end, this will apply to business applications. 3D exploration of data with touch and sensory input. Seamless integration of different wireless devices and services that you can simply connect to by waving your hand. Enhanced virtual shopping or augmented reality in store.

I might be a little off, but I'd say it's a likely and nearby future.

Programming and art

2010-02-22T08:01:00.000-08:00

I have heard many people say that programming is an art. I have thoughts on that statement, but they are truly not that relevant. The following does inspire me to add more art to programming. Lets make programming languages more poetic and make technical specifications and functional specifications more enjoyable reading: http://www.cs.cmu.edu/~dst/DeCSS/Gallery/decss-haiku.txt And some background: http://www.loyalty.org/~schoen/haiku.html This has turned into an internet classic, showing the obvious demand for more art in programming :). Next time you need to write code or documentation, write a poem or make a painting. The world will be a better place.

PDC 2009

2009-12-07T18:32:00.000-08:00

It's always great to be able to take a step back from the daily grind of the projects and take a look at the bigger picture. Microsoft PDC 2009 provided me that opportunity this year. The nutshell that I got out of it:

User interfaces are starting to see a revolution. We are quickly moving beyond mouse and keyboard to touch and sensor (heat, movement, camera [for instance as a scanner], voice, location, etc.) enabled interfaces. The Microsoft Surface, for instance combined with medical applications or interactive business intelligence, is a great example. More intuitive interaction with complex data or processes is the final goal.
Cloud computing will grow quickly. It provides a compelling on-demand environment that can integrate with your local data center. A service oriented architecture is a natural fit for scalable load-balanced cloud computing.
REST-ful services are changing the service landscape due to the ease with which they can be consumed. The Microsoft Open Data Access services with query capabilities are being fully integrated with SharePoint 2010, making SharePoint increasingly a data integration platform. The goal is being able to create mash ups of any data on any client or device.
Parallel computing, either at cloud level or processor core level, is finally becoming a common scenario. Tools and programming environments are increasingly supporting parallel processing.
Silverlight is at the point that I am seriously considering it a an alternative for business web applications. Working with large amounts of data is increasingly supported out-of-the-box. Silverlight 4.0 will also provide the option of running the application out of the browser.
AppFabric is the new service layer for applications, providing functionality like caching, workflow and service bus. I can see the AppFabric layer growing in the future with additional services that applications can utilize.

EF against DB2 z/OS

2009-10-22T18:23:00.001-07:00

The joy of it all :) . Turns out that the v9.7 EF provider from IBM doesn't generate correct insert statements against z/OS. I can't even tell how often my habit of putting wrappers in place has saved me a lot of trouble. The EF caching provider from CodePlex that I put in place provided an easy place to fire an event just before a DBCommand is cached/executed. Some quick & dirty string manipulation and the insert statement is fixed. While doing that, also handled SQL statements that contain 'length', 'starts with' and 'ends with'. The IBM provider doesn't handle those, but they are somewhat handy to have. An other option was waiting for the IBM product support to turn around and fix all this stuff, but I thought it was better not to hold my breath for that one.

Without a doubt, the IBM DB2 provider has serious limitations at this point. So why use EF when it's that difficult against DB2? As always, there is talk about getting rid of DB2 z/OS as a DB platform. Don't see that happening any time soon, but if I have the opportunity to develop an application DB platform independent, I probably should. On top of that, EF makes development a lot faster and easier once you have it in place. Can't even explain how happy it makes me to have the compiler help me when the data model changes :) . The additional framework I built on top of EF also provides features like audit trails. That happened to be a requirement, and within a day I had a screen built on top of the audit trail that exposes entities and properties under 'friendly names' and allows you to navigate relationships in the audit trail. Being able to use that saves so much time and bugs compared to an audit trail that would have to be custom coded for each entity.

I have noticed though that the projects are starting to become less challenging to the point of not challenging at all. It's simple execution of established patterns. By itself good, but I think it's part of a series of signs that it's time to move on (on a big scale). Noted on the list of things to do quickly.

Living the mainframe

2009-09-22T12:08:00.001-07:00

The joy of connecting Entity Framework to DB2 on z/OS. The IBM plug-ins are marginal at best (can't filter on schema within a database and actually generates the schema name within the code :( ), the IBM website is insanely difficult to navigate and the logic behind the naming and packaging of their products is a well kept secret ... but it least it works in the end :) .

On a seperate note, looks like I'll be going to Microsoft PDC this year. I have found DevConnections and PDC a great way of staying current and updating our internal roadmap for the .NET platform. As a working developer, you don't have much time to play with new technology, so PDC provides me a great overview of what I can start using right now (and will actually hold up in production environment) and what I need to plan for going forward.

Entity Framework Provider Wrappers

2009-08-31T21:04:00.000-07:00

The following open source project is hopefully a preview of the extensibility that is to be added to the Entity Framework: http://code.msdn.microsoft.com/EFProviderWrappers . Had to make a couple of adjustments and fixes, but after that it runs fine and speeds up data access quite a bit. I have mostly looked at the caching, still need to spend some time on the tracing. Fixes made to the caching code:

Modifications are not cacheable. Results of an insert should not be cached :) .
Extended the cached data so that more methods in the wrapped reader could be implemented. For instance the field names and schema.
Don't cache when the data reader hasn't been read to the end.
Fixed the loading of assemblies (done to find the referenced EF files) so it works better with GAC-ed assemblies.
Fixed a rather nasty issue in DbCommandTreeScanner. This class implements the visitor pattern for a command tree and is needed to determine the affected entity sets. However, in certain large statements (yes, the question is if it is wisdom to use those, but that's a different issue), this resulted in a stack overflow. Auch. By pruning constants out of the visited expressions early on, I was capable of avoiding the stack overflow.

Besides that, I extended the caching policy so that the expiration can be specified by entity. When multiple entities are affected, I take the smallest expiration across the affected entities.

With a little work, the provider wrapper turns out to be rather useful.

Six sigma due diligence

2009-08-19T12:37:00.000-07:00

Questions to ask before applying six sigma:

Does your data have a normal distribution? See http://www.pyzdek.com/non-normal : "Observe that the best-fit curve analysis shows the process to be capable of meeting the requirements easily, while the usual (normal) method shows the process to be incapable. Of course, the opposite situation may also occur at times." If not, make sure you use a curve that fits the data. From the same article: "For instance, most business processes don’t produce normal distributions. There are many reasons why this is so. One important reason is that the objective of most management and engineering activity is to control natural processes tightly, eliminating sources of variation whenever possible. This control often results in added value to the customer. Other distortions occur when we try to measure our results. Some examples of “de-normalizing” activities include human behavior patterns, physical laws and inspection." Based upon this, applying six sigma (which is all about eliminating the sources of variation) can actually cause the distribution no longer to be normal.
Is Six Sigma relevant for my process? See http://www.davenicolette.net/articles/six_sigma.html . "The similarities between software solutions are at the level of the general patterns that may be observed in the "environment." The individual solutions themselves are quite unique. In software development we use a solution "a million times over, without ever doing it the same way twice." A quality control mechanism that seeks to minimize variation applies to a process in which the solutions are done the same way twice, ten times, or a million times. Therefore, such a mechanism is fundamentally at odds with the basic nature of software development. "
If six sigma is applicable, what are you measuring? Popular measures seem to be the number of defects, size and time. The process that finds the defects is arguably not a fully repeatable process.

Defects are not randomly found in software. Their location and when they are found is strongly related to the usage of the software.
I have spent a blue Monday doing QA and the one paradigm that we lived by was: finding defects is not the goal of QA. Very fundamental rule. However, when the number of defects becomes our main measure, aren't we starting to violate that rule? Our measures (multiple, that can be expressed quite differently) should express compliance to the goals or specifications we set.
There is no 'default set of tests' that we can apply to each project. So the measured defects between two different projects are the result of different tests. That's quite different from a widget that we apply the same set of tests to every time we make it.
It's also not always clearly defined when something is a defect. If it wasn't specified, should it be counted as a defect? In some cases it would just be an enhancement. However, if it was an oversight in the specification, would that make it a defect?. Or is a defect simply "not implemented as specified" ? A first requirement is a consistent way of labeling defects across projects. This is different from a well defined widget that has been fully designed and tested before going into production and has a set of tests applied to it with well defined expected outcomes. Software is more like the widget prototype than a mass produced widget.
The size of the software is not a true measure of complexity. The complexity of a task is not only related to the task but also to the skills of the people executing the task. Take for example a team of software developers that has been successful developing large corporate data driven applications. Then have them develop a small module for an embedded operating system. Due to the completely different nature of the project, you'll probably see a lot more defects per line of code, even though the overall size is smaller. And are you sure that measures like size can be compared across projects that use different technology stacks? Finally, using size as a measure of complexity feels to me almost as a last resort: we can't find anything better to measure than the lines of code. A final acknowledge that we fail to even grasp what complexity means for software development.

Befor applying six sigma, make sure you brush up your degree in applied mathematics :) . And apply six sigma to your six sigma project. Define your measurements. Apply your measurements. Analyze the fit of your measurements. Improve your measurements. Control six sigma and don't let it get out of hand. Repeat as often as needed to make sure your measurements still fit your process.

Validation within the Entity Framework

2009-08-05T08:54:00.000-07:00

During last years DevConnections in Vegas I could definitely feel a big divide about the Entity Framework. Some people were diving right into it, others were rather critical. Certainly people that had been betting on LINQ to SQL. I think whenever Microsoft makes a large structural change in it's preferred data access layer, there are bound to be some waves. Personally, I never really got into datasets. Useful in some scenario's, but my preference truly is an object layer like the Entity Framework. So I have taken a deep dive into the Entity Framework, actually developed a large system in it, and came out alive and breathing (rather than floating belly up). Definitely some gotcha's, certainly in the first release of EF, that you have to pick up on, but overall workable.

One of my preferences is to always put an additional layer between a framework like EF and the actual application. Kind of a mini-framework, where you can easily deal with oddities, bugs and often used extensions, without copying it left and right through your code. Part of that framework that we developed on top of EF is a validation engine.

Part of some work I have done in the mid 90's on automatic generation of test cases was the idea of making validation an integral part of the application. The actual communication would continue to be checked against the contract. As a side note, that same work also included a note on applying the capability maturity model to testing. We threw the idea out as not being a practical thing for the study we were doing. Enter T-Map about two years later ( http://www.tmap.net/Home/ ). Oh well, never said I am doing this work because of my keen business instinct. The idea was applicable to a communications bus. You can validate the actual communication against a model of what it is supposed to be and pinpoint any differences. It can also be helpful in plugging in test mock-ups. A similar example closer to the .NET home is the code contract library added to .NET 4.0 (see http://msdn.microsoft.com/en-us/library/system.diagnostics.contracts(VS.100).aspx ).

The validation framework does something similar for the interface between the application and the database. By providing a predefined set of validation rules we reduce the number of mistakes made in actually implementing the validation. You can also code custom validation rules for more complex scenario's.

I quickly discovered a rub though that gave me the typical 'duh' Erlebnis. An invariant defined in your contract is an invariant in any state. If you state that within a class property A is always bigger than the value of property B, that is going to be always enforced. Sounds like what you want, but what when you are assigning values to an object of that class? Programmers will quickly become annoyed once they figure out that they always have to make sure this holds. For instance: property A has value 1 and property B has value 0. You want to assign property A value 3 and property B value 2. You won't be able to do that without violating the invariant unless you assign the value to property A first (an easy way around this is a simultaneous assignment operator, but C# doesn't have that). If you have many classes with this issue splattered all over them, you will see your programmers transform from quiet geeks to raving monsters. So the validation framework is capable of selectively applying rules based upon the state of the entity.

A core assumption of the validation framework is that everything is a function. So rather than evaluating a value of a property, we evaluate a function on the value of a property, that also can consider the object that the property belongs to and the evaluation context (which contains a reference to the EF object context) . It's flexible enough that I only had to implement one custom validation rule for the whole project. All other validations were covered by configuring instances of the standard rules that the framework provides.

Very early on I specify the rules (mostly in XML, seldom a need for custom roles that are coded). As the code is developed the validation acts as a constant reminder of what the stored data should adhere to. The nice thing about the Entity Framework is that the object manager allows you to access old and new values rather easily, making it easier to develop fast and complex validations. During development the rules are refined. Sometimes I feel like loosening up the rule, but most of the time I end up changing to code to comply, since it is the right thing to do :) . In production, the rules are still validated and constantly check the data going into the database, catching odd scenario's that were not covered originally.

On top of that it is rather easy to add audit trails (automatically track who changed what fields) and support standard fields. For instance, if it's standard to have fields to track who created/updated a record and when it was created/updated, you can easily implement a standard interface and automatically fill these fields as the entity framework saves changes.

I definitely enjoyed working with the Entity Framework and it has lifted development standards way above the "we call it a business layer, but really it just provides easy access to our stored procedures". We have been capable of performing complex changes in a very short amount of time. The major pains are probably:

Integration with UI components through databinding is sometimes awkward or inefficient. I truly hope that the integration will become good enough to be able to generate complex LINQ to Entities queries automatically with efficient use of paging.
Explicit loading. Wrapped that in some additional methods and using some extension methods to make that easier and not have the code littered with explicit load statements.
Attach/Detach: the rule is that an attach automatically cascades through the associations while a detach doesn't. The object oriented approach of the Entity Framework came to good use when implementing some what-if scenario's. But some of the results of that evaluation had to be saved while temporary artifacts should never end up in the database. That caused some rather ugly scenes with additional DeepDetach methods to clean everything up. Having two different contexts (one that contains the changes you want to save and one that you don't want to save) is an option, but since objects can't be in two contexts that also causes some additional code. Related to that is also the rule that you can't save changes in a context until all reads have been completed. That makes 'pipe line' processing where you save intermediate results more complicated.

Feature teams

2009-07-30T10:14:00.000-07:00

A healthy amount of distrust should be exercised toward 'golden bullets' that are launched to address the software crisis. To quote Edsger Dijkstra ( http://www.cs.utexas.edu/users/EWD/transcriptions/EWD13xx/EWD1305.html ):

The required techniques of effective reasoning are pretty formal, but as long as programming is done by people that don't master them, the software crisis will remain with us and will be considered an incurable disease. And you know what incurable diseases do: they invite the quacks and charlatans in, who in this case take the form of Software Engineering gurus.

Yes, he was quite a personable fellow :). On the other hand, I think it is okay to recognize that for a large portion of the world, the software crisis is a happy place, not a crisis at all. You get things done quickly with lower labor cost, you make $$$ even when it doesn't work. You sell a new version with new features that promises all things better, that also don't work as well, but at least it's by definition better than the previous version. It's like accepting the black plague as a welcome source for generating funds to support all kinds of medical activity.

So with interest I have been reading about feature teams (
http://www.infoq.com/resource/articles/scaling-lean-agile-feature-teams/en/resources/Larman%20Vodde%20Feature%20Teams%20-%20InfoQ.pdf ). And large parts of it just make sense. In the light of some recent developments, I love the fact that it recognizes social structures. A work place is not just about work. It's people spending a lot of time together performing tasks. It's a social structure in essence. It's high school revisited. Don't know what your high school was like, but there were probably small groups of students that hang out together. And you definitely had 'us' against 'them'. So what happens when you organize architects, developers, business analyst and testers all in different groups? You set the stage for scenes right out of high school. The teachers are the architects, the popular group are the business analyst, the outcasts are the developers and I guess the testers end up being the cool crowd. Setting up feature teams that are cross disciplinary just makes sense. Certainly since I strongly believe the following to be true about architecture:

Architecture is done on a project by project bases. It's not something you just write and talk about, you practice it on each project you do.
Architecture is not technology independent. Building a wooden bridge is architecturally different than building a concrete bridge. True, some high-level similarities exist, but architecture goes a lot deeper than just some high-level similarities.
In a field that changes so quickly, architects have to practice what they preach. As an architect you have to know that your guidance is effective. Without practicing it yourself, you do not fully realize the impact your guidance has on the actual execution. And observations made during implementation should be translated back to architectural enhancements.

The piece on feature teams is good enough for me to buy the book "Scaling Lean & Agile Development: Successful Large, Multisite & Offshore Products with Large-Scale Scrum" (Larman & Vodde, Addision-Wesley). I hope the rest of the book will be just as pleasant. Other goodies that definitely seem to apply:

Parkinson's law: As long as the manager's prestige and power are tied to the size of his budget, he will be motivated to expand his organization. This means that the architect group will compete for more architect resources, development group will compete for more developers, etc ... Feature groups are small groups that contain all these functions and that are directly tied to features of the final product. It naturally balances the mix without competition between each group for additional people.

Making things is about making people. So how much high quality training did you get this year? How much focus is there on your personal and professional development?

Project managers select people from a specialist resource pool and release them back when finished. This gives rise to projectgroups or feature projects, usually with matrix management. In such organizations one hears people called 'resources' as though they were machine parts and human or team dynamics had little importance on productivity or motivation. [...] In practice, resource pool and feature project management has disadvantages:

lower productivity due to non-jelled project groups
lower motivation and job satisfaction
less learning
lower productivity due to multitasking
lower productivity and throughput due to increased handoff and delay waste
lower productivity and increased handoff and delay due to physical dispersion
lower productivity and higher cost due to more managers

[...] There us a very close relationship between the structure of a system and the structure of the organization which designed it. I even observe the same relationship between a business and it's IT department. Software development is a rather abstract business. And truly, people don't deal well with horribly abstract concepts. So as soon as we can touch a common similarity (or what is at least observed as a similarity) with a more concrete activity that is well-known to the business, there is a huge sigh of relieve. And after that sigh, it's just not a good idea to point out where the similarities end.

View the entire life cycle of the software. Don't chop the life cycle up in development as one part and maintenance as an other. You will notice that you start shifting work around between the two activities to fit your needs. If you run out of development resources, you shift it to maintenance (lower quality, incomplete features, etc.). If you are doing too much maintenance, you delay the work to the next development cycle. True question is what the value of the application is to the business during it's life cycle. What's the cost (including for instance missed savings due to none implemented features) and what's the gain.

Test Driven Development

2009-07-22T07:08:00.001-07:00

I always need to take a deep breath when this subject comes up. I don't want to pretend that I know the answer to the best method of developing software. I would deem it highly unlikely that there actually is one single best method. If you have found the best method, my opinion on the subject obviously is no longer relevant. If I would have found the best method, I wouldn't write only a couple of sentences in a blog entry about it.

I did however work for a while as a test consultant and have thus some understanding of testing and QA. I also did some work on deriving test cases from formal specifications. And opinions start to take shape after a while. Opinions are like weeds in that respect. Come to think of it, opinions are also appreciated like weeds. Rather than writing a lengthy entry on my opinion on the whole thing, I'll provide a link to an article that captures the essence of my opinion in rather compact format: http://www.eiffel.com/general/column/2004/september.html . In case the link gets broken:

Test or spec? Test and spec? Test from spec!

Which came first? As an intermediate result towards a more general research goal, it has recently been demonstrated that the egg precedes the omelet. A full proof falls beyond the scope of this modest EiffelWorld column, but here is the idea: you can construct the omelet from the egg, but not the egg from the omelet. (The chicken is covered by a separate lemma.) The reader will already have jumped mentally to an important special case; for omelet read test, and for egg read specification, particularly in the form of Eiffel contracts.

We are being told from some quarters that you can't specify anyway, and that development should be "test-driven". Eiffel programmers know from daily practice and from observation of the libraries that the first proposition is false: precise specifications in the form of contracts, even if they cover only part of the functionality, radically change the software development process. In the new scheme, tests play a critical role (see another little article of a few years back, "Quality first" at:http://www.inf.ethz.ch/~meyer/publications/computer/quality_first.pdfwhich I believe already suggested a few of the good ideas of agile methods), but tests are not a replacement for specification: they follow from these specifications. It is indeed remarkable to see how the presence of contracts can drive the testing process. If a slogan is needed and two can do, I will venture "Contract-driven testing" and "Test-obsessed development". That works very well -- you should test all the time, with the intent of finding bugs -- but it's not a reason to drop specification and design. Specification and design are what propels both the testing process and the test cases themselves.

Going from specifications to test is one-way: you lose the abstractions. A specification describes the general properties of a computation, for all possible inputs; a test addresses one particular result for one particular input. From the general you can deduce the particular, but not the other way around: even a billion tests don't reveal the insight -- the abstraction -- of a specification. Omelets beget no eggs.

For an Eiffel programmer, testing consists of turning on contract monitoring, exercising components, and waiting for a contract to fail. Since any test oracle can be expressed by a postcondition, but no collection of test oracles can replace a specification, we lose nothing, but we gain a great advantage over naïve (contract-less) test-driven development: retaining the fundamental abstractions that underlie our programs and their individual modules.

In some later installment of this column focused more on research, I'll discuss how it is becoming possible to automate the component testing process completely, test case generation included. Suffice it for the moment to note that while we should be grateful to our extreme friends for helping to rehabilitate the much maligned process of software testing, we -- Eiffel programmers -- know something they don't seem to have realized yet: you can test from specs, but you can't spec from tests.-- Bertrand Meyer

Use cases, and tests derived from them, are a great tool for exploration. A specification, or for instance a mathematical proof for that matter, does not just appear out of thin air. It can start many ways, for instance as an observation of a particular instance or scenario that inspires an idea. You can call that a use case. And some use cases can get you to rethink what you thought of as a good specification. Just as a single counter example can cause a mathematical proof to bite the dust (as I side note, I did enjoy reading "Proofs and Refutations: The Logic of Mathematical Discovery" by Imre Lakatos ( http://www.amazon.co.uk/Proofs-Refutations-Logic-Mathematical-Discovery/dp/0521290384 ) ). Test-driven development can be a useful tool for learning about the specifications. But don't take it as an excuse not to bother with even trying to understand the fundamental abstractions that underlie the software.

As an example, I would like to reference http://bitsthatbite.blogspot.com/2009/07/five-orders-of-ignorance.html (wondering if referencing my own blog entries within my blog entries will up my score in search engines). Obviously, for the test case with the lambda containing the multiplication, the original code works well. The code was more than likely developed with the multiplication operator in mind as the test case. However, the contract of the Fold method captures an abstraction beyond just multiplication: you can pass lambda's that apply other binary operators like addition, subtraction or division. It is easily shown that the original implementation does not handle every binary operator correct.

Also don't forget the refactoring step in test-driven development. I have seen bad code. The images are still floating in my memory banks. Introduce as many variables as you can when you are not sure. Don't remove code you no longer use or shouldn't have written in the first place. Keep tagging on to the same method until it's several pages long. When you happen to be bored, make sure you spread out your abstractions across as many classes as you can create within 30 minutes. Refactoring doesn't help at that point. Jumping of a bridge might, but that's rather final. The little trash can on your desktop might be your best friend. An unexplainable hard disk crash on all development machines is all that can save you in explaining to management why you need to rework the application. In short: the fact that you plan on refactoring later should not open up the doors to bad practice.

Here's my dilemma: the above story just doesn't go over very well as a casual conversation at the coffee machine. I tried. Doesn't work. So if you ask me about test driven development, I'll say: "It's marvellous. Sugar?"

Five orders of ignorance

2009-07-21T10:34:00.000-07:00

The five orders of ignorance, published in a short article in the 2000 issue of the Communications of the ACM, states that "software is not a product, but rather a medium for the storage of knowledge". It then goes on to recognize five levels of ignorance:

0th Order Ignorance (0OI)—Lack of Ignorance.You know, you have the answer.
1st Order Ignorance (1OI)—Lack of Knowledge.You know what you don't know, you have the question.
2nd Order Ignorance (2OI)—Lack of Awareness. You don't know what you don't know.
3rd Order Ignorance (3OI)—Lack of Process.You have no means, or process, for resolving your lack of knowledge.
4th Order Ignorance (4OI)—Meta Ignorance. Not aware of these 5 levels.

Obviously, level 4 is bliss, but reality might come and kick you in the teeth. Level 3 represents agony. Level 2 is bliss and level 1 is just hard work, but at least you can apply the processes from level 3. So we have seen many efforts at level 3, processes and methodologies for resolving your lack of knowledge. One of the methodologies for acquiring knowledge that I have not seen highlighted very much is formal analysis. It's about as rigorous as it gets, highly structured and rather unforgiving.

As an example, I was reading the following article http://www.switchonthecode.com/tutorials/csharp-tutorial-the-lambda-operator . In essence:

public delegate int FoldIntDelegate(int a, int b);

public int Fold(FoldIntDelegate fid, params int[] list){ int result = 1; foreach (int i in list) result = fid(result, i); return result;}

int val = Fold((a, b) => a * b, 1, 3, 5, 7, 9);

A nice, solid, article. I wondered however why the Fold method was not recursive. That's kind of what I would expect when using functions/lambda's, since you have no foreach loops in pure functional programming languages. Obviously, C# is not a pure functional programming language, but you can toy around with it. So writing a recursive version requires you to think about proof by induction. Ah, by going through the formal process of induction you rather quickly figure out that the Fold function above assumes that 1 is the unity for the operation you are applying. Which is true for multiplication, but for

int val = Fold((a, b) => a + b, 1, 3, 5, 7, 9);

that's certainly not true. Fold does not force you to only use multiplication as the operation being applied, since we are passing the operation through the lambda expression. In fact, Fold((a, b) => a + b, 1, 3, 5, 7, 9); will always be incorrect with the above implementation. So you can't just assume that 1 is the unity. Nor can you assume that when not enough arguments are passed for a binary operator, 1 is a correct answer.

Why care? Fold is a public method. Somebody will sooner or later think you can use Fold on operations other than multiplication and before you know it, people wonder why the software developers can't even add a list of integers correctly.

So when you remove the two mentioned assumptions from Fold, the code becomes a little more complicated. Since Fold definitely only applies in it's current form to binary operations (because of the signature required for the lambda expression), you would have to require that at least two parameters are passed to Fold.

Finishing this up, it could look like:

public int Fold(Func aFid, params int[] list)
{
if (list.Length > 2)
{
return aFid(list.First(), Fold(aFid, list.Skip(1).ToArray()));
}
else
{
if (list.Length < 2) { throw new Exception("Binary operation requires at least 2 arguments.");
} else { return aFid(list[0], list[1]); }
}
}

Ofcourse this could be even rewritten to the point where it's actually required to have at least two parameters of type int and then call a private version of Fold that takes just a list of parameters (with the value of the two parameters of type int concatenated to the front of the list). That would be cleaner since the public contract of Fold then requires you to at least pass two parameters.

An other assumption that Fold makes that might not be clear at first glance is that of an associative operator. The assumption is that (2*3)* 5 == 2*(3*5). For lambda x=> x-y, Fold would give us 6 - ( 3 - 2) = 5. Question is, would you expect fold to return (6-3)-2 = 1 instead?

Obviously, I am taking a small, and excellent, sample of the use of lambda's somewhat over the top, but hopefully the point is clear: (semi- , since I didn't actually write any formal derivations) formal analysis has it's uses.

Lambda's are so cool, I gotta wear shades

2009-07-20T20:28:00.000-07:00

I have to admit, I got rather giddy when I heard that C# was going to support lambda expressions. Part of the .NET community didn't share my feelings and felt that C# was being overloaded with syntactic sugar. Some of them didn't even spell lambda correct. I'll forgive that one though :). I'll only make three points, since there is no point in making this blog entry several pages long.

Look at lambda calculus (yes, my favorite pass time before I thought that being a junior Oracle developer was way cooler than being a doctoral student): http://en.wikipedia.org/wiki/Lambda_calculus and tell me again that it's only syntactic sugar. Alonzo Church, a true math genius, developed it as part of an investigation into the foundation of mathematics. It is also capable of expressing any algorithm. Syntactic sugar? C# is more like syntactic sugar than that lambda's are like syntactic sugar.
Lambda's have an expression tree representation. That means you can build them dynamically. And they are also great as key selectors. For instance x=>x.MyProp selects the value of MyProp. However, you can also easily extract the fact that x has a property called MyProp. And at compile time, this is checked. That's nice, because it allows me to avoid the old "referring to properties by their name through a string" problem. If the property is renamed, you have to make sure you also change all the places where you referred to the property as a string. For instance, in the Entity Framework the include statement requires you to refer to a property path as a string. That's rather annoying because it becomes one of the few spots that the compiler doesn't help you track down problems when you change your model. By using the lambda expressions to generate a property path, the compiler will help you. One issue left is when you go through a collection in the include statement (for instance Prop1.Collection.Prop2), but composition helps you out there (although it might not be the most elegant solution). I'll blog more about my adventures with the Entity Framework, but lambda's have helped me create a 100% compile time checked business logic layer using the Entity Framework. That makes it easy to change things quickly (like a couple of hours for a major refactoring) without breaking things (0 issues caused by the refactoring out of the gate).
Recently got to explain the following code I wrote to some people:

protected static Func<T,V,W> Partial<V,W>(U eventArg, Func<T,U,V,W> aFunc) { return (target, context) => aFunc(target, eventArg, context); }

What it does is take a function that has three arguments of types T,U and V and a result type of W. Then it applies the argument of type U and it results in function with only two arguments (of types T and V) and a result of type W. Why would you want to perform a partial function application like this? Lets say aFunc is a general validation function that takes some configuration information (for instance what property to validate and what the maximum allowed value is for that property). The partial application passes the configuration information in and hides that from the outside world. The outside world now only sees a function with two arguments. Works kind of like a private field. The fun part is that the Partial method can actually also be written as a lambda expression. Note that the parameter types of target and contex are being inferred. Try that with an anonymous delegate. See http://blogs.msdn.com/ericlippert/archive/2009/06/25/mmm-curry.aspx for a rather lengthy blog entry on this.

A helpful bog can be found here http://blogs.msdn.com/ericlippert/archive/tags/Lambda+Expressions/default.aspx .

Writing PDF from stream to IE6 from ASP.NET

2009-07-20T15:13:00.000-07:00

I *very much dislike* (stronger words come to mind here) things that you just happen to have to know about browser quirks and plug-in quirks. The amount of time you can spend trying to figure it out is horrible and, besides basic trouble shooting skills, there is not much you can do to reason your way out of it. Enter: IE6 and PDF's from a web server that uses gzip compression on aspx pages. Horrible. Getting the Open/Cancel/Save dialog to show is challenge number 1. It requires a very specific set of HTTP headers that's near impossible to guess (because there are quite a few possible combinations of HTTP headers out there).

Challenge number 2 is the gzip compression. IE6 doesn't understand that it should unzip before passing the PDF on to the plug-in. I think you can also resolve this issue by putting a gzip indicator in the PDF file, but since the PDF file was being generated by an other tool, that wasn't an option.

So, for issue 1 I finally pulled the following code together, after Googling it extensively:

public static void ExportReportToPDF(HttpContext aContext, string aFileName, bool aAttachment) {
byte[] result = ... stuff to generate the PDF.;
if (result != null) {
aContext.Response.ClearHeaders();
aContext.Response.ClearContent();
aContext.Response.Clear();
aContext.Response.ContentType = "application/pdf";
aContext.Response.AppendHeader("content-length", result.Length.ToString());
aContext.Response.Expires = -1;
aContext.Response.AddHeader("Content-Transfer-Encoding", "binary");
if (!String.IsNullOrEmpty(aFileName)) {
aContext.Response.AppendHeader("Content-Disposition", String.Format("{1}; filename=\"{0}\"", aFileName, aAttachment ? "attachment" : "inline"));
};
aContext.Response.AddHeader("Pragma", "no-cache");
aContext.Response.AddHeader("Cache-Control", "private");
aContext.Response.BinaryWrite(result);
aContext.Response.Flush();
aContext.Response.Close();
aContext.Response.End();
}

To address issue 2, you can use an ashx handler and reference it through a hyperlink. Assuming you don't have gzip compression turned on for the ashx.

Easy in the end, but getting the headers straight was a nightmare, only to be helped by an hour of Googling the issue.

Enter this century

2009-07-20T15:03:00.000-07:00

Okay, I have avoided blogging for a long time. I don't have a drive for publishing and I usually talk to the people I like to talk to. No need for writing stuff down.

However, I have noticed that many blogs out there are very helpful in resolving the type of 'stuff you just happen to have to know' issues in my field. Yep, the nerd field that is. I have a healthy dislike of anything technology related, I don't even have a quartz watch (kind of because I used to play guitar a lot, the watch band bothered me and I didn't want to deal with storing the thing, but whatever the motivation is, I don't). I can't keep track of where my cell phone is and never seem to remember to charge the thing. So actually I am too geeky to be a nerd. To get past this point, if these blogs help one person out there avoiding complete meltdown, it's time well spent on my part :).

So where's the content you might wonder? Well, the true test for all of this is whether I actually create the time to create the content.