City planning and slum control

Since we are talking about Enterprise architecture as analogous to city planning, I thought I can bring in my developing world perspective about how slums develop in a city despite having a nice city planning guide, in order to bring out importance of having policies for dealing with slums as part of city planning guide.

In developing world, we are quite used to slums springing up in a city despite having a city planning guide. A slum is a microcosm of a city, but without any planning or vision. It is like a city within city, with its own ad-hoc rules and regulations. It does provide some useful services to rest of the city. No doubt it is sign of broken governance but it is not just that. Slums survive and thrive because they are needed for proper functioning of city and they do provide some value to the city. However slums are not desirable. City planners must contain and eventually get rid of them.

Similar situations arise in enterprise IT world, despite having a nicely laid out enterprise architecture, and strong governance. This article discusses some such cases of deviation, but lays the blame squarely on ineffectiveness of governance. While that may be true in some cases, sometimes circumstances force one to bypass guidelines and governance. So what we also need is a framework to rectify situation, post-facto. In city planning analogy terms, we need a proper slum control and removal policy.

To illustrate, let me give this example.

In a large enterprise, they have a proper enterprise architecture defined, as intended by Annie Shum's paper mentioned in this article. There are guidelines and governance mechanism. Based on this a multi year IT plan is drawn. One of the elements of the plan is to build subject area specific data related services. The build of these services are governed by principles laid out in enterprise architecture viz. Build should be iterative; TCO should be low, resultant services must be maintainable/flexible/performant etc. It is also tied to business benefits and this capability is sponsored by a business facing project which is planning to use it in a year’s time. Now this programme is midway and another business sponsor comes up with a proposal to build a slightly different set of services for same subject area (where attributes are very specific to the problem at hand, their granularity is not generic enough and their currency is very specific to problem). This new business sponsor has a solid business case; he is promising to add millions of dollars to bottom line, if this new set of services are made available within a short span of time. There is a very small window of opportunity from business perspective and it does not matter if underlying capability does not follow any of the enterprise architecture guidelines/principles or governance processes, as long as that window is utilized. The matter reaches highest level of decision making apparatus within enterprise (e.g. CEO) and you know who would win the argument. So this set of services are built which does not fit the plan laid out by enterprise architecture, it may not use technologies suggested by enterprise architecture (procuring them in time for this solution, is an issue), consequently it may not use solution architecture laid out by enterprise architecture guidelines.

In short this is a slum that is getting developed in this nice city (viz. enterprise IT) governed by city-planning guide (viz. Enterprise Architecture). And as an Enterprise Architect if you protest too much, I am sure you would be shown the door. You cannot argue with millions of dollars added to bottom line. So what should an Enterprise Architect do?

1. Do nothing. Which would mean more such slums spring up and then it is just question of time before governance and consequently enterprise architecture breaks down totally. So it does not appear a viable option.

2. Demolish the slums. This is possible if what was built, was one off capability and not required on an ongoing basis. So as soon as the utility of this capability diminishes below a certain threshold, get rid of it, completely. If required, by being very stern.

3. Rehabilitate the slums. This is the only option if what was built is not a one off capability but is required for enterprise on an ongoing basis. The reason it bypassed enterprise architecture guidelines and governance was to catch a business benefit specific window of opportunity. One cannot avoid such incidents. What we now need is to refactor this capability as per the enterprise architecture guidelines and governance processes. We need to bring in this capability in mainstream of the enterprise IT. In short we must rehabilitate the slums (if required by some amount of redevelopment).

There may be hybrid options based on specific situations, but an enterprise architecture plan as city planning guide, must provide for this eventuality as well. I have seen this type of issue cropping up more than once. So it is difficult to dismiss it as statistical outlier, not worthy of a strategic response.

Is single version of truth achievable?

Did you have a programme in your IT organisation to build 'the single view' of data for some subject area, say Customer or a Product? These types of porgrammes have different names Book of Records, System of Records, Single version of Truth and so on. Have you experienced the agony one goes thru to try and create a single view, acceptable to different view points that exists in an organisation? There always will be some view point, which would want some data at different level of granularity or different level of currency or both, from rest of the view points.

No I am not talking of CDI/MDM. The problem I am talking about is about defining what 'the single view' of a particular subject area should look like? Even after you decide how your single view of subject area should look like, there are further challenges of collecting, reconciling, cleansing and hosting data. Thats what CDI/MDM predominantly addresses. But who helps you in deciding what the right single view of particular subject area? Frankly, nobody.

So isn't a single view of a subject area, a misnomer? Let me make a logical argument. even when we build an IT system, we start with a nice third normal form data model. But the non-functional requirements, such as performance, scalability makes us abandon the third normal form data model and introduce level of redundancy, what we poularly call denormalisation. And we live with it.

Why not take a similar approach to data, at enterprise level. Lets accept the fact that, the different view points are not always reconcillable, and the single view of subject area is impossible to achieve. Why not build a fit for purpose single view of subject area. And there can be as many single views as there are different purposes. Some of the purposes may collaborate with each other and can reuse each others single views. So in reality there can be less number of single views of subject areas than the number of purposes. Ofcourse, you need to create mechanisms to keep these different single views in synchronisation. Well why create one level of indirection, why not go to multiple view underlying this single view directly? May be using a service facade? Well I have answered these questions in one of my earlier posts . so you would need these fit for purpose subject area single views. You can use MDM/CDI technolgies to build them.

And if you dont believe me then I have a bridge to sell you ;-)

Challenges in using services as integration mechanism

Another topic of interest for me is services as integration mechanism. ESB has been touted as next best thing for integration. To me ESB is a facade, which provides a clean interface to host of services, by abstracting myriad of source systems implementing those services. In doing so, it will face challenges. The main challenges I see in practice are -

1. Efficiency
   
2. Data quality
   
3. Adaptation
   

Lets examine them one by one.

Efficiency of services dealing with bulk data is a serious problem. I am talking of inquiry services here. The complexity of handling bulk updates using services are just unimaginable. ESB essentially is a federated architecture and hence techniques like caching to improve efficiency are hard to implement. This challenge needs to be addressed properly for ESB to succeed.

Data quality within the source system, which is abstracted by service facade is not guaranteed to be of highest order. A service facade needs to deal with these data quality issues gracefully. ESB, somehow needs to identify the data quality issues and feed neccessary actions back to source systems for correction of those issues. Source systems need not alwys be capable of correcting the data quality issues, which needs to be handled by ESB.

Service facade tries to provide an uniform interface to myriad of source systems. In doing so, it takes a least common denominator approach. It tries to define service shema, which is possible to be implemented in all source systems. This need not be best service definition. If this has to be avoided then there needs to be adaptation mechanism, which allows service implementation adapted to standard form, by extending functionality of source system within ESB.

These challenges need to be addressed within ESB for successful implementation.