Google Chrome OS - I told you so

The Google Chrome OS announcement is a 'I told you so' moment for me. Nothing pleases an analytical mind than when the prediction borne out of analysis turns out to be true. There are enough doubters, doubting motives, timings of this announcement. But I always thought this would happen. Google is not at war with Microsoft, rather it is enabling the uptake of 'the Cloud'. As for enterprises, I see every reason to latch on to this trend. This presents an opportunity for enterprises to free themselves from clutches of desktop OS, its incessant upgrade cycles, security vulnerabilities and limitations it imposes.

I remember a conversation I had with one of my friends regarding the same subject just after Chrome browser came out. When Chrome browser came out, every one was treating Chrome as just another browser and questioning need for one more browser. Then I had stuck my neck out and contended that it was not just a browser but a prelude of things to come. It was indeed a light-weight front-end to a light-weight desktop OS. I was not sure how fast this would happen. It looks like it is happening faster than I thought. And my second contention is that it will enable enterprises move to the Cloud, faster.

In my opinion the consumer market will be conquered by the smart phone and netbook. The enterprises on the other hand will only need netbook class devices than smart phones, but not with the bloated OS of yore. Chrome OS with a netbook class device will be a perfect choice for enterprises. Current legacy can be hauled off to cloud and exposed as a plug-in on Chrome browser. And there is no reason why back-end services, yet to be built and which will reside in the Cloud, need to be provided only by Google. In fact, this Chrome OS - with open standards based front-end, will bring in much more transparency and Independence from OS, for service providers. Chrome OS, in essence is taking desktop OS out of equation within enterprise IT.

Coming to think of Chrome OS, it has potential to change the enterprise IT, the way switching technology changed telecommunications. Now in the days of packet switched network, circuit switching looks so much of a retarded idea, even though it was the only possibility during initial days of telecommunication. But when packet switching became practical it changed telecommunication beyond recognition; changing the architecture and business models. Similarly a computing device capable of doing everything locally and storing all it needed - locally, looks so 'Stone' age in the age of ubiquitous network connectivity and elastic compute/storage capacity. In that sense Chrome OS with a browser front-end is set to change the architecture and business models of enterprise IT forever. Desktop OS had its run, now its time for it to move on and vacate the place for Chrome OS. Mind you demise of dominance of desktop OS does not necessarily translate into demise of Microsoft, the same way the demise of dominance of mainframes did not kill IBM.

Indeed we live in interesting times. We have an year to prepare and fine tune our strategies and architectures. Don't miss on this opportunity.

Retrospective

Here is a retrospective on some of the blog posts.
Incidentally most of them are also top content on this blog per reader's choice.
So if you are a recent visitor, there might be a gem you may have missed ;-)

1. Agile, iterative or waterfall
   
2. City planning and slum control
   
3. Centralised service delivery team
   
4. Requirements management is crucial
   
5. Challenges in using services as integration mechanism
   
6. Requirements elicitation
   
7. SOA in enterprises or Hype 2.0
   
8. BPM or workflow
   
9. SOA and demonstrating ROI
   
10. SOA Fable: Do you wear a watch
    

SOA - Dead or alive

I had doubted SOA hype years ago in this post about SOA hype. Recently a small storm was raised when Anne Thomas Manes of Burton group blogged about untimely demise of SOA. But curiously, I am not so pessimstic anymore. My feeling is that SOA as an arhcitecture style is now better understood for what it is, than a magic technology silver bullet intended to solve all of enterprise IT problems. This short post from Ali is quite insightful in that context. What it is suggesting is that SOA has moved on to next step on its evolutionary path. Which is a good sign.

I always believed the true value proposition for SOA was not sharing (a.k.a. reuse) of services, and definitely not cost reduction. As sharing is important for provider who has mass consumers adding further value to digitised services provided. In enterprise scenario this may not be of much importance, unless you are in business of provision of digitised services. But sharing was a necessary step before SOA moved on to next stage of evolution where it makes more sense for enterprises, which are not in business of providing digitised services to masses for further value addition.

This is because sharing of services requires service contract standards(WSDL) and service discovery standards(UDDI) albeit in weak form. It also needed mechanism, to determine what is to be shared and at what granularity, to be established. These standards and technologies are in place and mature now. With these in place, implementing shared service is quite straight forward. It also set stage for next step of this evolutionary journey.

The next step of evolution, viz. flexible services where flexibility is at hands of providers, is progressing as I am blogging. The charge is led incidentally by progress made in BPM world. The more we know about processes, process composition and orchestration especially in a human task centric processes, we will make more progress on flexibility aspet. It would throw up its own set of standards and technologies, e.g. BPMN, Services Fabric based on industry models etc. So having process standards and technologies create need for limited sharing of services, which now can be provisioned using standards and tools in place already. At this stage benefits of SOA start becoming visible for enterprises. It would show up as agile and flexible processes. Processes which can be measured, monitored and flexed easily. When this becomes mainstream then stage will be set for ultimate goal of SOA, viz. flexibility at hand of consumer (a.k.a virtualisation).

I believe for virtualisation to become mainstream we would need ability to procure, provision & manage services in a location independant manner(a.k.a. in the cloud). We would need some form of semantic compatibility guarantees on the fly (kind of expected in semantic web). At this stage of evolution we may also see an ITIL like methodology evolving for business service management. I also believe it is not necessary for all services to be virtualised. What services need to be virtualised in what context would remain a very specific decision for each organisation. (Or you end up finding your business getting disintermediated using the services provided by you).

So don't let all the doom and gloom news get on to your nerves. It may be just current economic climate reflecting in people's feelings. It may not be possible anymore just to declare an IT project as 'SOA' and get funding from business. But from an enterprise architecture viewpoint SOA looks well set to be de-facto architectural style of IT in future.

Cloud or SaaS?

In response to my earlier post Apoorv  has pointed that there are questions on viability of SaaS model. My take is that SaaS is a commercial model whereas Cloud computing is an architectural approach. One can deploy cloud computing in an enterprise need not be as SaaS. In the same vein one can deploy SaaS with traditional tools, over proprietary infrastructure without cloud computing.

I feel Chrome has hastened the cloud up-take in enterprise. SOA has been found useful for integration of apps and sharing of services. SOA also promotes a vision of composite apps, where ultimate control of composing apps is put in end user's hand. SOA has not realised that vision yet. In my opinion cloud computing is a required enabler for this composite apps vision propagated by SOA. Without Cloud computing combined with SOA,  realising the composite application vision is very difficult - if not impossible. 

I also believe cloud may have positive impact on SaaS as a model. SaaS as commercial model may have viability issues. Again, I dont have enough data points, but my gut feel is a pure commodity applications can be successfully deployed in SaaS model. Trick is to make many users to accept it as commodity without any customisation. Does such pieces of commodity applications exists within enterprise application space? I believe so. But carving them out and putting them in SaaS mode is a challenge more in terms of organisational  inertia than a technology challenge. Without sufficient scale SaaS model is indeed doomed. The question is who buckles first, orgnisational intertia to change or the surviving capacity of SaaS providers.

So does cloud computing has future? Definitely. Does SaaS model have a future? May be, if SaaS providers can build the required scale by somehow overcoming orgnisational inertia. 

BPM or Workflow ?

This post referred in Todd's post has resurrected an old debate. What exactly is the value provided by BPM over traditional workflow solutions?

Those who have worked with traditional workflow solutions know, how there is no concept of end to end process in workflow solutions. The workflow solutions were mainly ground-up automation tools to complete various process steps executed in back-end systems. Even if you put in the metric collection framework (which I dont believe is a trivial task) on top of it and gained an end to end process visibility, you could not do much with it. Changing the processes in those ground up tools was not as easy as in the modern BPM tools.

Again all BPM tools are not the same and may have issues in implementation. But thats no reason not to move to correct technology option. Ofcourse there may be reasons not to move to BPM technology. (e.g. you have limited budget and BPM is not your highest priority pain point). But the argument that, workflow solution with metric collection framework on top of it is equivalent or better than BPM tool, in my opinion is not valid.

BPM tools are closed loop solutions. You can define processes, operationalise them, analyse them while in operation and take corrective actions. Moreover BPM tools are open standards based. The process steps to underlying services integration, is thru SOA and that too is becoming part of standards. It would be better to hitch your ride on open standards based closed loop solution which integrates well with underlying services, than go for complex solutions involving multiple proprietary tools.

SOA Fable: Thermometerman

This story is courtesy James Gardner. The story points out that how a shared service provider sticks to service level agreements (SLA) rather than service consumer's satisfaction with service. The SLA often define objective criteria to measure some attribute of service. And attributes covered by SLA do not essentially imply the satisfaction of consumer. Providing personalised consumer satisfaction, to a large consumer base is considered uneconomic. So service provider sticks to some objective criteria acceptable to large group of consumers and tries to make it a golden mean between service consumer's satisfaction and economy for service provider.

The reason for doing so is economical. Service providers segment their service consumers and for each segment the provider decides which service attributes that segment is sensitive to. Service provider then goes on to optimise those attributes alone for that segment.

In doing this segmentation however, service providers may lump a diverse group of consumers with diverse aspirations, motivations, intentions and temperament, together. The service attributes deemed important for entire group may be sum total of service attributes important to these smaller groups. But these smaller groups may like some other attributes to be addressed which may be missed by the segmentation. So instead of achieving of golden mean between economy and satisfaction it may lead to widespread consumer dissatisfaction.

The alternative to this is to allow consumer to choose which service attributes are important to them. In essence allow service consumer to define their SLA, even on the fly. The availability of such mass customisation ability is the moral of this story.

In an SOA, the service provider can allow consumer to choose the attribute of service which consumer wants optimised, and even allow this optimisation to take place in every interaction if it makes sense. Some combination of attributes are difficult/uneconomical to achieve.

e.g. Service availability and service currency can both not be made 100% at the same time When you provide 100% service availability, you may have to sacrifice currency a little and vice-versa.

Allowing such mass customisation ability might mean more work for service provider. Provider may have to provide different implementations to satisfy different combination of service attributes. Provider may charge differently for each of the combination as well to take care of economics. Whether this is done offline via service contracts or online via an architectural layer doing service attribute brokering, should be left to provider.

SOA Fable: Do you know the intent?

Before advent of SOA, enterprises opened up their central systems to a limited set of users, thru initiatives called 'Extranets'. The idea being stakeholders can have a limited visibility into workings of enterprises which would help them in their businesses, which in turn help the enterprise.

So this giant automobile company opened up its inventory to its distributors, and distributors could place orders for spare part based on inventory they saw. Soon the company realised that some distributors were misusing the facility to hoard critical spare parts when inventory was going low, and making extra buck by selling those parts to other distributors for a premium.

Company stopped this facility. To company's dismay some clever distributor found another way around. He started placing orders for ridiculously large number of parts, system started returning error response with a suggestion for maximum number he could order. The maximum number being level in inventory. This was worse than earlier situation. Not only distributor was getting the information he wanted, he was loading the central system unnecessarily - by continuously running this service for different parts and forcing error response.
Then because of some unrelated change, company stopped giving correct level of inventory in error response. So this distributor started making a lot of orders starting with a high number and gradually coming down to correct level, while doing a binary search. This was worse than earlier situation. It has started creating a lot of orders into system, which launched associated work-flows and then cancelling them - which launched even more work flows.

Till the last situation arose, the problem was handled as IT capacity problem and mis-deeds of distributors were not caught. The last hack tried by distributor nearly broke the back of central systems. Which launched massive investigation to uncover these mal-practices.

What has this got to do with SOA?
Well, one vision for SOA is that SOA can make enabling services available to consumers. Once consumers have enabling services available, consumers can compose the applications they need. Even if we keep aside the doubts about how does one define these enabling services, the issue of consumer intent is a big issue. It need not be always a malafide intent. Some consumer running an innocuous service, multiple time to get the data set he wants in order to do analysis he wants, can break the back of transactional systems. This has nothing to do with services policy. The consumer is willing to abide by policies set for the service, but his intent is not the one envisaged by service provider. And in this era of 2.0 this is bound to happen. SOA needs to take cognizance of this issue and handle it. No, CAPTCHAs are not a solution when services are meant for consumption by machines rather than humans.

Moral of this story is, in a true SOA, the architecture must

1. have a mechanism to diagnose violation of normal intent by consumers
2. provide alternate service implementations for genuine exceptional intent
3. provide a seamless switchover from normal intent to exceptional intent
4. provide incentives to promote normal intent and disincentives to reduce exceptional intent - when it makes sense to do so.

SOA Fable: Do you wear a watch?

Most of us wear watches, and except for those who wear it as a piece of jewellary, we expect a service from this piece of equipment. And what is the unique service provided by this equipment? It lets us know what time it is! Wait, hang on. Aren't there enough service providers, providing this very service, around us? Well every car has a clock in it. every house has many clocks, every PC has a clock. Even every mobile phone, PDA, iPhone has a clock. Why do people wear watches, anyway?

What are your expectation as a consumer from a possible service telling you waht time it is?

Availability: Service must be available when consumer needs it.
If a consumer relies only on mobile phone and battery runs out. The service is not available when consumer may need it.

Reliability: Service response must be reliable and should not require double checking.
A consumer is not sure of the time shown by a clock at public place, he needs to double check it with another reliable source anyway.

Accessibility: Consumer can access the service whenever he needs it.
A clock in laptop in backpack is not easily accessible in a crowded commuter train.

Trust: Consumer trusts that service is backed up by accountability on part of a service provider. When it is a question of life and death, one can trust service provided by one's own piece of equipment to be accurate, reliable and available than a general purpose service. As a consumer one tend to trust no one but oneself as most accountable service provider.

This is a very important insight which helps one prepare a proper versioning policy.
Without that trust versioning schemes can be mis-used for creating specialized services.

In enterprises, since SOA is mandated, project owners will use services. But they will make sure that they get their own private version of a service. This is quite easy by getting a veto power over life cycle of a service and mis-using governance for this sake. Assume a service version 1 is in use. Second consumer wants to create another version, because it has additional needs. This version 2 is derived from version 1. Now first consumer wants an upgrade to his existing service. But he is not willing to accept service version 2 as base for its next version. He will find any execuse to make sure he gets a service version 1.1 rather than 3. This pattern will keep repeating. And soon there will be a lot of versions, changing only in second qualifier. So you will get, what I call parversion (parallel version) anti-pattern.


SOA governance must make sure it guards against this anti-pattern and creates appropriate policies and controls to minimize and eliminate its occurences. Morover governance must recognise that this is a symptom and root cause lies somewhere else.

Moral of the story: Consumer is willing to bend rules to get an acceptable, reliable, accessible and trustworthy service. Watch out for such rules violation and fix root cause.

Not whether but when to REST

There is a debate starting to rage about whether REST or SOAP/XML-RPC is better choice for services. Following is my take on REST v/s SOAP/XML-RPC debate, in traditional enterprise computing scenarios.

From whatever I have read till now, my opinion is that REST is quite close to a distributed object oriented abstraction than a service oriented abstraction. Following table tries to bring out similarities between REST and OO abstraction.

REST	OO
In REST there are resources	in OO you have Classes and Objects
In REST there are HTTP methods(PUT, DELETE, POST, GET ) for lifecycle management	in OO you have facilities provided by OO language (Constructor, Destructor, method invocation, accessor methods)
In REST resources keep application state information	in OO object represents the state
In REST type safety is provided by XML schema	in OO type safety is provided by pre-shared class definitions (e.g. using .h files or java .class files)
In REST dynamic invocation is possible because of repository	in OO dynamic invocation is possible because of reflection.

Of-course REST provides a more open distributed object oriented mechanism than say CORBA or EJB. It does it by usage of XML schema for marshalling/unmarshalling and open protocol like http (as against dcom, iiop or rmi).

But it is bound to face the some of the problems that distributed object oriented mechanisms faced. e.g. granularity of objects, scalability related issues, differing consumer experience based on lazy or early instantiation of resource graphs (object graphs in OO).

REST is an interesting way of implementing distributed object oriented mechanism, and there are times this abstraction is better suited than pure service oriented abstraction. So in my opinion debate should not be either REST or SOAP/XML-RPC, but when to use REST and when to use SOAP/XML-RPC. The limiting factor for time being is availability of tooling and skills. Over period of time these will develop and then, within enterprises both can co-exist.

SOA - Necessary and sufficient

SOA is heralded as the 'must have' for business agility. I agree to a point. SOA is necessary but not sufficient to achieve the highest degree of business agility. Let me explain, why I think so.

In service oriented world, information systems try to be congruent with business world, providing information services in support of business services. The business organisations provide business services in order to carry out the business activities. These business services are steps within business activities and they use information services provided by underlying IT infrastructure.

However underlying IT infrastructure is not amenable to this business service oriented paradigm, fully. At implementation level, IT infrastructure has to deal with non-functional properties, such as responsiveness, scale, availability, latency, cost, skills availability etc. That imposes certain restrictions on implementations. E.g. For scale reason we normally separate behaviour and data. Behaviour (as represented in business logic and business rules) scales differently than data (and data stores - databases, file systems). That’s why in a typical distributed information system, there are more database servers than servers dedicated for executing business logic.

In service oriented world, information service provided by information systems need to mask such implementation issues. The idea that SOA will provide business agility will hold true, iff information services enable business services, use disparate information systems seamlessly. In SOA world, business services should lend themselves to rapid re-organisation and redeployment, in terms of business activity volumes, business responsiveness, speed of new product/service introduction etc.

The current thinking seems to be that a set of open standards, enabling integration between disparate information systems is all that is needed. With such integration mechanism, one can create a facade of a business service, using underlying disparate information systems. Hence the emphasis is on XML schemas, in-situ transformations, service choreography and to extent mediation [between required business service and provided information service(s)].

To me this is part of solution. It is the necessary condition but not sufficient.

As I had posted in past, one really does not know what should be granularity of information services. If you provide too granular information services, you would be better at reorganising but will be hard pressed to meet non-functional parameters. If you provide good enough services for current usage, satisfying non-functional parameters, you will have tough time reorganising. So for all practical purposes, for any business service change, there are possible information service related changes, rather than just reorganisation of information services.

That would mean, the agility of business service reorgnisation comes down to the change management in information systems. If you make pragmatic decisions in favour of speed of change, it leads to duplication and redundancy. If you try to keep your information systems pure and without redundancy, you sacrifice the speed of change.

So the key appears to be

• getting your information services granularity just right for all possible reorganisation that would be needed by business. You cannot really know all possible business changes, but you can know up to a certain time horizon. So that you are just re-organising your information services rather than redeveloping.
  
• if this is not possible or considered risky, you can take a re-factoring oriented approach. And incrementally build the service definitions.
  
• and whenever you change information systems (because despite your best efforts business came up with a change that is not possible with current information service definition), use MDA or Software factories (or any other conceptual to implementation mapping technology) to effect the change from conceptual business services onto its IT implementation. This would bring down the time to make changes. And also would enable you to make pragmatic decisions, because even if there are duplications and redundancies at implementation level, the conceptual field is clean and pure.
  

That would be complete SOA for me.

Service Aversion to Service Orientation

Well I have a slightly different take on Service Averse Architecture. It is based on my experience with Banking Financial Services and Insurance (BFSI) industry and may not be generalised to other industry segments. The information technology (IT) was introduced in BFSI to improve operational efficiencies. If you look at the value chain, within BFSI, viz. manufacture-market-sell-service-retire a product or a service, IT was primarily required to take care of ‘service’ part. As long as IT expenditure was less than the operational efficiencies it provided, enterprises were happy, notwithstanding delays and budget overruns. Since IT was not commoditized then, whoever could cross the barrier to entry, benefited from IT (despite cost and time overruns of IT).

Interestingly enterprises within BFSI were always ‘Service oriented’ in their business. They did provide specific services to their stakeholders. The problem was always with the information systems they used to support these services they provided. There was a big mis-alignment between services that business provided and info systems they used to provide these services. These info systems were always monolithic and closed. It was these info systems, which distorted underlying service culture of business. And these ill-fitting information systems were result of what Todd would call ‘project culture’.

Interesting point is how business which itself operated services for its stakeholders, was taken over by this project culture and created ‘service averse architecture’ in information systems. It was mainly due to, aura and geeky culture associated with IT. The business leaders did not understand IT, but understood its importance. So they gave a free reign to IT leaders. Initial IT leaders did not have much understanding of underlying businesses, so they were in the mode, “Tell us exactly what you want done, and we will do it!” Unfortunately what business wanted done was always a small piece of big puzzle. Hence multiple monolithic closed information systems, handling parts of services that business was delivering to its stakeholders, were developed.

Now that IT is commoditized and barrier to entry is lowered considerably (well buying a mainframe used to be a momentous decision for CEO and now any IT related decisions are hardly made by CEOs), cost and time have become important. And, IT has penetrated the other aspects of value chain, notably market, sell and even manufacture (which uses business intelligence tools). So IT has become more important to business at the same time business has become less tolerant of IT’s pitfalls.

Also, over the years IT folks started understanding business in more details and they started asking “Why do you want it done this way?” rather than just following orders. It is, what my friend Ravi would call a shift from output oriented to outcome oriented mindset. So when business and IT finally started coming closer to each other, they started appreciating need for alignment between two. SOA in my opinion is vehicle for that. SOA helps IT recast itself, in business terms.

Most of the organisations out there have ‘Service Averse Architecture’ within their information systems. And the organisations that are doing transitions to SOA are the ones where the IT leaders have made that paradigm shift from output to outcome-oriented mindset. These are the leaders who understand importance of business and IT alignment and how SOA can help achieve that.

Unfortunately leaders buying into SOA vision is just part of the story. It would mean enterprise is willing to make transition to SOA, but whether it will be done successfully or not, depends on changing entire organisational culture from undue competition to more of trust and co-operation.

Centralised service delivery team

Typically in business IT world, budgeting and portfolio definition phase determines what business benefits should be targetted for in a given period and what is the maximum cost that can be paid to achieve those benefits. Business projects are then executed to deliver business benefits while minimizing cost and time to get those benefits. So notwithstanding the various benefits of building reusable services for enterprise, projects tend to do point solutions. Mainly because of these budget and time considerations.

In response to this situation, one of the best practices that has evolved in SOA implementations is to have a central team implementing services for business projects. This has been a good idea. It can address many of the governance challenges. Mostly because this central service team (lets call it services delivery team - SDT) is managed and governed by principles set by enterprise architects for SOA, rather than budget and cost considerations alone.
However there are a couple of challenges in putting this to practice.

• Having a right engagement between project team and this SDT is a big challenge. This engagement model breaks or makes the future of SOA, in the enterprise. It is vitally important to get this model right.
  
• Getting the composition of this SDT right is another challenge, that must be addressed.
  

Initially the SDT was set up as an island, which owned the services and its interfaces. The SDT utilised existing capabilities or whenever necessary got the capabilities built. The engagement between project team and the SDT, was mainly of an outsourcer - outsourcee type. The project team defined its requirement and threw those over to SDT. Which then worked out the interfaces (keeping in reusability in mind). Then further outsourced building of capability required (if ncessary), to teams maintaining the systems which might provide the capabilities. The SDT had only application architects and high level designers apart from project managers. So the role of this SDT had become that of a systems integration designers and SOA standard tools had become SI tools.

This mode of working had turned into a waterfall life-cycle model, partly because the way engagement worked. This started to add a lot of overheads in terms of time and effort to project estimates. At the same time, benefits were not immediately realised for projects. As a result project team started resisting this engagement model, which in turn was viewed as rejection of SOA by project teams (which it was not). When project managers are measured by budget and schedule compliance alone, it was natural for them to resist this engagement model which took away the control of deciding how much budget and schedule risk they can afford to take. So SDT too are facing problems.

I think a new co-sourced engagement model needs trying out. In this model, services delivery team works as part of project team, governed by same budget and schedule consideration. But they are also governed by SOA principles too. So when these two governance principles contradicted, comporomises were required to be made. These compromises were inevitably are made in favour of project. Rarely in case of strategic services, the compromises are made in favour of services. These compromises, in their wake, will leave what I call 'non-'services. Some of these non-services need to be refactored into services by a part of service delivery team, which gets its funding straight from CIO. This team has limited funding, hence could only turn so many non-services into services, resulting into a back-log. Over a period of time, size of back-log would give a fair understading of the costs involved in making tactical compromises against strategic imperatives. This model needs to run for some time, for its efficacy to be known. For this model to work, it would need strong SOA governance framework and a strong testing facility in place.

Any sharing of experience in this area is most welcome.

City planning and slum control

Since we are talking about Enterprise architecture as analogous to city planning, I thought I can bring in my developing world perspective about how slums develop in a city despite having a nice city planning guide, in order to bring out importance of having policies for dealing with slums as part of city planning guide.

In developing world, we are quite used to slums springing up in a city despite having a city planning guide. A slum is a microcosm of a city, but without any planning or vision. It is like a city within city, with its own ad-hoc rules and regulations. It does provide some useful services to rest of the city. No doubt it is sign of broken governance but it is not just that. Slums survive and thrive because they are needed for proper functioning of city and they do provide some value to the city. However slums are not desirable. City planners must contain and eventually get rid of them.

Similar situations arise in enterprise IT world, despite having a nicely laid out enterprise architecture, and strong governance. This article discusses some such cases of deviation, but lays the blame squarely on ineffectiveness of governance. While that may be true in some cases, sometimes circumstances force one to bypass guidelines and governance. So what we also need is a framework to rectify situation, post-facto. In city planning analogy terms, we need a proper slum control and removal policy.

To illustrate, let me give this example.

In a large enterprise, they have a proper enterprise architecture defined, as intended by Annie Shum's paper mentioned in this article. There are guidelines and governance mechanism. Based on this a multi year IT plan is drawn. One of the elements of the plan is to build subject area specific data related services. The build of these services are governed by principles laid out in enterprise architecture viz. Build should be iterative; TCO should be low, resultant services must be maintainable/flexible/performant etc. It is also tied to business benefits and this capability is sponsored by a business facing project which is planning to use it in a year’s time. Now this programme is midway and another business sponsor comes up with a proposal to build a slightly different set of services for same subject area (where attributes are very specific to the problem at hand, their granularity is not generic enough and their currency is very specific to problem). This new business sponsor has a solid business case; he is promising to add millions of dollars to bottom line, if this new set of services are made available within a short span of time. There is a very small window of opportunity from business perspective and it does not matter if underlying capability does not follow any of the enterprise architecture guidelines/principles or governance processes, as long as that window is utilized. The matter reaches highest level of decision making apparatus within enterprise (e.g. CEO) and you know who would win the argument. So this set of services are built which does not fit the plan laid out by enterprise architecture, it may not use technologies suggested by enterprise architecture (procuring them in time for this solution, is an issue), consequently it may not use solution architecture laid out by enterprise architecture guidelines.

In short this is a slum that is getting developed in this nice city (viz. enterprise IT) governed by city-planning guide (viz. Enterprise Architecture). And as an Enterprise Architect if you protest too much, I am sure you would be shown the door. You cannot argue with millions of dollars added to bottom line. So what should an Enterprise Architect do?

1. Do nothing. Which would mean more such slums spring up and then it is just question of time before governance and consequently enterprise architecture breaks down totally. So it does not appear a viable option.

2. Demolish the slums. This is possible if what was built, was one off capability and not required on an ongoing basis. So as soon as the utility of this capability diminishes below a certain threshold, get rid of it, completely. If required, by being very stern.

3. Rehabilitate the slums. This is the only option if what was built is not a one off capability but is required for enterprise on an ongoing basis. The reason it bypassed enterprise architecture guidelines and governance was to catch a business benefit specific window of opportunity. One cannot avoid such incidents. What we now need is to refactor this capability as per the enterprise architecture guidelines and governance processes. We need to bring in this capability in mainstream of the enterprise IT. In short we must rehabilitate the slums (if required by some amount of redevelopment).

There may be hybrid options based on specific situations, but an enterprise architecture plan as city planning guide, must provide for this eventuality as well. I have seen this type of issue cropping up more than once. So it is difficult to dismiss it as statistical outlier, not worthy of a strategic response.

CBD, SOA, whats next?

Most engineering disciplines have managed to split their problems into one that of components assembly. Individual components are well defined and engineered to satisfy a very concise set of requirements. The system level problem then reduces to selecting proper components and assembling them to satisfy overall system's requirement.

Doing things this way has its benefits and it is well established by other branches of engineering. Software engineering is trying to do the same, but without much success. Atleast in business IT world. We have tried component based development (CBD) and now trying SOA. But we are not any closer to the pannacea of assembly of components.

What is that constraint, which is preventing software engineering to move to next level? Implicit in this question is an assumption that software engineering wants to move to a component assembly world. Well, I am not sure that is true either. And I am not alone in this thinking, see this post.

So the problem is not entirely technical. It is also of business motivation, and viable business models. In other branches of engineering, craft became engineering, when it promised to bring prosperity to all stakeholders. In software engineering, what is that business model, which will let multitudes of component makers flourish and big guns will just concentrate on component assembly? Is it standardisation effort, that is lacking? Is ultimate end IT user really interested in such component assembly world? Because in business IT world, the enterprise IT is main differentiator for an enterprise, which prevents it from being commoditized as a service or product provider.

These are the hard questions that industry must grapple with. Otherwise, every couple of years we will have, a wave, be it CBD or SOA and as an industry we'll not move much.

Each such wave (CBD, SOA) takes us further than today. But to make that transition to next level we need an industry wide push and not vendor specific attempts. And this must include all stakeholders - viz. business IT consumers, service providers and product vendors. Perhaps OMG can take lead?

SOA: Reuse, of Interface or Capability?

SOA reusability debate is bit confused. For more clarity, we must understand the difference between service interface and capability behind it. A service exposes some interface which is backed by a capability. From a service consumer's perspective the reusability of interface is enough. Whereas from a builder's perspective the reusability of capability is more important. It is the later which is difficult (or impossible in some cases). It is possible that same interface can be mapped to multiple capabilities. For example an interface can be mapped to multiple capabilities based on Quality of Service (QoS) needs specified in interface. (Well, this is not possible right now in most implementations). The QoS needs like performance, scalability, availability, currency, data quality etc. would determine the binding of interface to capability. Sometimes even functional needs of services may force an interface to be bound to multiple capabilities developed over period of time. The problem is how does one control the proliferation of overlapping capabilities.

The reason why overlapping capabilities get developed are not straight forward. Sometimes it is weak governance, sometimes it is for QoS purposes, sometimes it is trade-off made to achieve results for a business facing project. This post by Todd Biske has elaborated about the last aspect. In practice one can address governance issues but capability duplication cannot be totally eliminated. The QoS needs will force multiple capabilities for same interface in some cases. There may be business drivers which will force a tactical trade-off to achieve larger business benefits. Well, one way to avoid trade-offs is to plan for the future, which I have suggested in this post. Still such overlapping capabilities will get built.

So what should one do? Well, one way out is to refactor and streamline the capabilities that are built. Here SOA would help, as consumers are not affected when capabilities change, as long as interface is same. So one should be refactoring and cleaning capabilities, because up-front reusable capabilties are hard to achieve in a working IT organisations. Business leaders must realize this reality and make funding available for these kinds of activities.

Enterprise IT Oracle

It is very surprising that so many competent and smart people come together in some Enetrprise IT organisation, then create a mess that is unmanageable and beyond their collective capabilities. It is not always a case of inaptitude or incompetence of individuals, but it is the nature of the beast. If we were to understand the problems of enterprise IT, we should try to analyze the root causes.

To me enterprise IT is driven by four major drivers,
1. Own business needs
2. Competative scenario
3. Regulatory compliance
4. Technology changes

These four drivers have different rate of change. Cost of not attending to those changes, is different too. Hence priority to attend to these changes is different. Enterprise IT is like an assembly line in action. And to make matter worse, it is an assembly line which can't be stopped for maintenance. So the four drivers combined with the fact that you never get any breathing space to fix enterprise IT, results in many tactical decisions which finally leads to mess of unmanageable proportions.

Ability of enterprise IT to respond to the drivers is constrained by capability of its own IT organisation, capability of its suppliers and capability of other stakeholders. Enterprise IT does not deal only with planning and design of IT but also about building, governing and sustaining too. This requires collective effort from IT organisation, suppliers and stakeholders, hence any mechanism to avoid the mess must have participation of all.

To avoid the mess EA must plan for future based on these four drivers and remember to make that plan as flexible as the rate of change within most important driver of the drivers listed above. When plan changes, it can accomodate important of the latest changes within other drivers too. The capability of IT orgnisation, supplier and stakeholders puts constraints on any plan that is created. This capability building also must be addressed within the plan itself. This planning is based on tracking of drivers and constraints. A sub-organisation within enterprise architecture community must own this. This organisation can then aptly be called Enterprise IT Oracle (An Oracle is a person or agency considered to be a source of wise counsel or prophetic opinion; an infallible authority. Not to be confused with Larry Ellison's Oracle corporation).

SOA, funding and hidden gems

There are some good posts on services and their characteristics. The main thought of these posts, is to bring out characteristics of an enterprise service. There is a comparison between city planning and SOA. A thought encourages different granularity of services to co-exists with an anlogy to niche retailers v/s Walmart.

However the way projects are funded in enterprise IT will hinder this approach. Enterprise IT, as far as funding goes, is governed more like a communist polit-buro than a capitalist entreuprunial system. There are no venture-capitalists ready to fund (seemingly) wild-vackey ideas. They would rather go with proven ways of doing things. So the idea of having niche services co-existing with run-of-the-mill enterprise services is a non-starter. That does not mean it will not happen.

Traditionally departments moon-light on their budgets and create fundings for these niche capabilities (albeit not in services form), but then those capabilities remain outside perview of enterprise architecture and remain in back-alleys, hidden in enterprise hiddenware. Which also prevents proper utilisation of these capabilities. Same thing can happen in an SOA. Departments will create niche services, and somehow fund them. But these services will be below the radar for rest of the enterprise.

An enterprise architect, has to live with this fact of life and provide means to unearth such hidden gems and bring them back to EA fold for governance. As mentioned in the posts mentioned above, a collection of such niche services may be a viable alternative to a coarse grained enterprise service, only if we know such niche services exist.

Solving the funding issue, to borrow a term from Ms. Madeline Albright,is beyond payscales of most enterprise architects and best left to business leaders to figure out!

SOA and demonstrating ROI

Typically large enterprises in BFSI space have a conservative cost benefit accounting practices, focused on accountability. Within these enterprises IT departments are often flogged for not achieving ROI. This also makes it a major area of concern for an Enterprise architect.

In 2007 trend analysis by IT experts, one of the major trends mentioned is about difficulty in demonstrating ROI. Traditionally demonstrating ROI was a problem in IT shops. With stovepipe architecture, at least you could attach your costs to relevant information systems and could figure out benefits accrued because of that information system. Whether it made sense or not, to have particular information system was obvious. Of course problems related to intangible benefits accounting was always there.

The situation became difficult with sharing of infrastructure be it client server systems and lately with EAI. However with SOA gaining traction the issue of demonstrating ROI is becoming even more difficult to handle. There are several issues, ranging from, when a project considered delivered, how costs are measured, apportioned and paid for, to what constitutes benefit and how to measure it. Most of the time IT folks are on defensive, in these debates.

There are problem with measuring both, the fixed costs incurred while building these systems as well as running costs incurred while running these systems. On fixed cost front, the problem arises mainly because of the reuse of infrastructure and non-functional services in a seamless manner. Please note this is different than any sharing of infrastructure you may have seen earlier. Earlier even with sharing, there were clear cut boundaries between users. Now with SOA, the boundaries are blurred, as far as infrastructure and non-functional services goes. One does not know how many future projects are going to use these services (with or without refactoring). Hence they have no clue how to apportion the fixed costs to users. This sometimes turns projects away from using the common infrastructure, as the up front costs are deemed too high. If yours is the first project you rue the fact that you have to build the entire infrastructure and own up all the cost. The enterprise needs to have a clear policy for these types of scenarios so that projects know how cost is going to be charged.

On running costs front various cost elements, e.g. ESB, network connections, underlying applications etc. need to be metered at the granularity which makes sense for the accounting purposes and which helps tagging metered data with a user. Here user is meant from a cost benefit accounting perspective and not actual business user. This metering is not easy. Most of the times underlying IT elements are not amenable to be metered at the level of granularity which helps you connect the metered data with users. Sometimes, metering at such granularity adds an un-acceptable overhead so as to breach non-functional requirements. I have not seen any satisfactory solution to this problem. People have used various sampling and data projection techniques. These are unfair in some scenarios and costs get skewed in favour or against some information systems. The applications part of this run-time metering is relatively easy but it still has problems of adding overheads and breaching non-functional requirements. So people use sampling and projection techniques, here too. But luckily there is not much seamless reuse of application services hence these sampling does not skew costs drastically.

As for benefits the debate is even more intense. The debate begins with the definition of when the project is deemed complete and starts accumulating benefits. e.g. In a business process automation project using SOA, with iterative delivery, one may automate some part of business process which results into some benefit, but end to end business process may actually suffer during this transition, because it may have to carry on with manual work-around. So how does one measure benefits in this case? With intangibles, there is a perennial problem of how does one measure intangible benefits? Sometimes even measuring tangible benefits is difficult, because the data for comparison is unavailable. As with costs, to measure benefits one needs metering at various levels for measuring elements of benefits (e.g. time, resource usage etc.). All the metering issues faced by cost measurement are also faced by benefits measurement as well.

So the key problem is working out semantics of costs and benefits with the business folks, putting a programme in place for their measurement in conjunction with SOA. If SOA is combined with a measurement programme, then it may be possible to demonstrate the ROI with these agreed definitions. This measurement programme is peculiar in the sense that for deployment it can be clubbed with SOA but it has its own separate governance needs, apart from SOA. So it needs to be handled appropriately. This is more than BAM and covers even IT aspects too. May be we need a new acronym, how does BITAM (Business and IT activity monitoring) sound?

SOA in enterprises or Hype 2.0

If dot com in enterprises was hype 1.0 then surely SOA in enterprises is coming very close to becoming hype 2.0 . The way SOA has been touted as next best thing to happen to mankind since sliced bread brings it closer to that dubious distinction. The vendors are promising all kinds of things from flexibility, adaptability, re-use to lower costs if you use their merchandise to do SOA. SOA is good as long as decision makers can seperate hype from reality. I for one will be very saddened if SOA goes the some way as dot com hype. Following discussion is to seperate hype from reality so that decision makers have correct expectation, to enable them to move along the path of sustainable SOA.

1. Myth of reusable services

In my experience as architect I have never seen as-is reuse of a business service implementation. Some amount of refactoring is needed for it to be reused. The refactored business service actually harbours multiple services under a common facade. For a service to be as-is reusable it needs to be so fine grained that it will have problems related to non-functional attributes of services. Just to give an example, if I had a business service providing customer details along with his holding details given a customer identity, then I have couple of options in its implementation.

I) I can build it as a composite service composed of more granualar services for customer detail and holding detail.
II) I can build a monolithic service for providing both customer and holding details

Now remember the lesson we learnt in managing the data. Always do the join at the source of data, because at the source you know more about actual data and can do many more optimisations compared to away from source. (Remember the old adage don't do join in memory let RDBMS handle it?). So from a non-functional perspective (scalability and performance), option II) is very attractive and some times mandatory.

No doubt, option I) gives me more re-usable service. But it still does not give me absolutely reusable service impementation. For example if I need the customer details with holding details for three different kinds of execution scenario, viz.
a) an on-line application for customer service,
b) a batch application to create mass mailer and
c) a business intelligence application to understand customer behaviour (with holding as one of the parameters).

Even though I have more granular services, all of them are not usable in all these different execution context. I cannot simply call the granular services in a loop to get the bulk data needed for scenario b) and c) above. So the re-usability is restricted by execution context.Of-course you can throw hardware at this problem, to solve it. But then your costs escalate and any savings you made by reusing software will be more than offset by hardware costs. So just because you organise your software in terms of services (which essentially specifies the contract between user and supplier and nothing more), you are not going to get re-usability. It will enable re-usability within an execution context but not universal re-use. So if we treat Services as explicit contract specification between users and suppliers then we should attempt to reuse these contracts. This however does not automatically translate to implementation reuse.

2. Myth of composite applications

This myth is related to the myth above. In most other engineering disciplines, the real world components are standardized and higher level solutions are typically component assembly problems. Not so in software. Even if we have services, their assembly does not necssarily behave within accepted parameters, even though a single service might behave OK. So composing implementations, to arrive at a solution is not so straight forward. Many vendors will have you believe that if you use their software, most of your software development will reduce to assembly of services. This is not true for following reasons. What is the correct granularity and definition of services is known to user orgnisation than vendor. These service defintions are dictated by user organisation business practices and policies. Each organisation is different, so a vendor can never supply you those service definitions. If a vendor does not know how the services look like and what their properties should be, how on earth is he going to guarantee that composition of such services will behave in desired manner? And as outlined in point above, the implementation reuse is a big problem. So even on that front vendors can not help you. So the composite application will remain a myth for some time now. The vendor sales and marketing machinery will show you mickey mouse applications built using composite apps scenario. But demand to see atleast two productionized composite apps, where majority of constituent services of apps are shared between those two. My guarantee is, you wont find any.

So is SOA a BIG hype about nothing. Not exactly. It does provide following benefits.

1. Manageability of software with business alignment

The single most important contribution of SOA is that it connects software with business. In an SOA approach, one can make sure that all software is aligned with business needs, because all software is traceable to their business needs. The whole edifice of building, maintaining and measuring utility of software will revolve around business services in an SOA approach. So it becomes easier to maintain focus on business benefits (or lack thereof) of software. With the traceability it provides, software becomes a manageable entity from being an unwieldy and randomly interconnected monolith. And there is some reuse possible in terms of non-functional services (such as security, authentication, personlisation etc.).

2. Ability to seperate concepts from implementation

The next important contribution of SOA approach is the focus it brings on seperating interface from implementation. The logical extension of this approach is to seperate conceptual elements from platform elements. So if you are using SOA approach towards software development, you have necessary inputs to create a large scale conceptul model of your business. You just need to filter out platform specific stuff from the interfaces you defined. You can further distill these interface specifications to seperate data and behaviour aspects. These are really reusable bits within your business. It is very easy to figure out how exactly these reusable bits can be implemented on different implementation platforms. This will give you necessary jump start for your journey towards a conceptual IT world.

So in my opinion SOA is good and it is the way to go. But not for the reasons stated by vendors. It is not going to make software drastically cheaper nor going to make software development drastically faster. Its just a small step in a long journey towards making enterprise software an entity managed by business folks rather than IT folks.

Services as contract

Recently we had a visit from Bertrand Meyer (Eiffel fame). He mentioned how Eiffel langugae helps make contract explicit between user and supplier. He also elaborated in what all different ways the contract definition can be used. (One of the interesting uses he showed was that of 'push button' testing). When I asked him that how does the language discover the contradictions in contract specification, he said "It does not". Which was kind of disappointing, a wrong contract specification, howevere faithfully implemented by the implementor, is no good.

This incidentally is the point I made in one of my earlier posts. Not only, one must make the contract explicit, as most SOA proponents are proposing, but also we must have ability to discover contradictions in the contract specification. We must have ability to see, if multiple contracts can co-exist and satisfy some common properties. It is not easy. However this is an important aspect of contract specification and must be addressed. The service definition should define how the compatibility would be established between various services (which essentially are contracts) and governance must ensure that this continues to be the case. Such defintion time checks, will help prevent costly budget and time overruns not to mention prevention of service proliferation and governance nightmare.