Challenges in using services as integration mechanism

Another topic of interest for me is services as integration mechanism. ESB has been touted as next best thing for integration. To me ESB is a facade, which provides a clean interface to host of services, by abstracting myriad of source systems implementing those services. In doing so, it will face challenges. The main challenges I see in practice are -

1. Efficiency
   
2. Data quality
   
3. Adaptation
   

Lets examine them one by one.

Efficiency of services dealing with bulk data is a serious problem. I am talking of inquiry services here. The complexity of handling bulk updates using services are just unimaginable. ESB essentially is a federated architecture and hence techniques like caching to improve efficiency are hard to implement. This challenge needs to be addressed properly for ESB to succeed.

Data quality within the source system, which is abstracted by service facade is not guaranteed to be of highest order. A service facade needs to deal with these data quality issues gracefully. ESB, somehow needs to identify the data quality issues and feed neccessary actions back to source systems for correction of those issues. Source systems need not alwys be capable of correcting the data quality issues, which needs to be handled by ESB.

Service facade tries to provide an uniform interface to myriad of source systems. In doing so, it takes a least common denominator approach. It tries to define service shema, which is possible to be implemented in all source systems. This need not be best service definition. If this has to be avoided then there needs to be adaptation mechanism, which allows service implementation adapted to standard form, by extending functionality of source system within ESB.

These challenges need to be addressed within ESB for successful implementation.

What is a service?

I see a lot of material about SOA but not much thought has been given to what exactly is a service. SOA puts a lot of emphasis on the infrastrutucre which makes development and usage of services possible. But how does one define (useful) services? What must be its granualrity? Is it really possible to have stateless services in a typical enterprise scenario(barring text book credit check service)? What I mean by real stateless service is that a service may have a clean input output specification, but can it be performed out of context? If not, who sets up the right context? Possibly another service? Where, then, this dependency specified in service specification, if services are self contained?

Let me give an example.
In insurance world a cusotmer is given a proposal based on his/her requirements. And when proposal is accepted and premium paid, it becomes a in-force policy. Now if I have a service which accepts propsal and payment details as input and returns in-force policy identifier as output, do I have a valid service?
This service cannot be performed unless a proposal was made, which might have required execution of some other service. So how do I add this dependency in my service specification?

Now somebody might say, that this is mere choreography of services. I would argue otherwise. Choreography of services is possible, if they can be choreographed together. Where do I specifiy this choreographical compatibility?

A more general question I would like to raise is that if services are explicit contract between provider and consumer then just specifiying input, outpt and exceptions sufficient to describe the cobtract fully? Shouldn't we add context in which this contract is valid? How do we specify this context?

The reason I am so woried about it is because, the great promise of SOA is promise of service as means of integration in a large enterprise. In a large enterprise there are sufficient challenges to get the data definitions to be compatible and right. But even after getting data definitions right, services may not fulfill the promises becuase they also need right context. Consumer cannot be sure of using a service, if he/she does not know what context the service executes in. This will result in cloning and re-implementation of services rather than re-use. Soon there will be proliferation of services and they will become as unmanageable as point to point interfaces.

Incidentally I feel this is the reason, in first place, why people create point to point interfaces.