SOA From the Bottom Up - The Best Approach to Service Oriented Architecture

The general attitude of business organizations towards Service Oriented Architecture, or SOA, has changed significantly over the course of the term's existence. When SOA first made its appearance as a buzzword in the early 2000s, enthusiasm for the new model quickly reached a fever pitch. Companies with big infrastructure problems were so sure that SOA was the fix they'd been waiting for that they were willing to pour millions of dollars into massive top-down SOA initiatives with long, hazy ROI timelines.

By 2009, things had changed. Service Oriented Architecture was no longer the belle of the ball, to say the least. The vast majority of the sweeping top-down SOA initiatives that had been launched with such high hopes had failed miserably, leaving companies millions of dollars in the hole and years behind on architectural improvements. Some studies estimate that as few as 20% of the SOA initiatives launched at the peak of the model's popularity were ever been fully realized. The negative backlash towards SOA was so immediate and strong that one industry analyst went so far as to post a mock-obituary for SOA on their blog in January of 2009.

In the face of so much failure, the backlash is perhaps understandable. However, it couldn't be more off-base. Far from being dead, SOA is more relevant than ever.

The same infrastructure problems that existed in the early 2000s still plague companies today, and with today's economic climate demanding more agility from companies that want to stay at the forefront of their industries, finding a way to implement SOA is crucial. Meanwhile, those companies that did manage to successfully complete their SOA initiatives - Bechtel being the most frequently cited example - saw exactly the incredible ROI that was promised at the outset of the process.

From this, we can conclude one thing: the top-down, drop-everything approach to SOA is to blame for the perceived failure of the model, not SOA itself.

In this article, we'll take a look at some of the reasons why these early top-down SOA efforts failed, and how tools like Mule as an ESB are making the holy grail of SOA a reality for many organizations, using a new bottom-up model of SOA adoption.

To CIOs faced with the task of managing increasingly complex infrastructures, the benefits offered by Service Oriented Architecture sounded like a dream come true - costs would be slashed, developer and business productivity would increase, and the company would be prepared for an agile future.

The big change introduced by the SOA model was architectures designed around services, rather than applications. The concept of services - small, independent pieces of software that executed a single task for whatever program called upon them - was nothing new, having been in use in enterprise infrastructure since the 1980s. What  SOA brought to the table was a vastly increased scope of use for these small units of functionality.

At the time, enterprise infrastructures were becoming increasingly bloated and unwieldy. New business services or automation needs were usually handled by developing new in-house software. These new programs often duplicated functionalities that already existed in other internal programs.

For example, if multiple programs required credit check information, each program would duplicate all the code required to perform the credit check (or in the worst cases, use a different implementation altogether). Each new program represented an additional codebase that the company's IT team would be responsible for supporting, as well as additional overhead for the network. In other cases, the complexity of building a new application in-house would result in expensive outside contract work that might not integrate smoothly with other existing programs.

SOA aimed to solve these problems by shifting internal application development practice towards the creation of reusable components called services.

First, the company would make a comprehensive map of the actual functionalities they needed from their infrastructure - what were the tasks that all of these custom programs had been created to automate in the first place?  How did they relate? What kinds of data formats and protocols had to interoperate?

Next, the company would determine how each of these functionalities could be expressed not as a single application, but as a collection of services. For example, an ordering system would not be a thought of as a single functional piece, but as a logical combination of credit card handling services, inventory maintenance services, customer data services, and more. From this assessment, the organization would be able to identify those services that were common to every application, and build them in such a way that the same service would be reusable in every application.

Once these services had been created, the various applications that the company had been using before could be recreated with minimal duplicated code by using these common parts. As an additional piece of the plan, any functionalities specific to a new application would also be created as new services, and be made available for reuse by any later applications that might require them.

SOA would create an ecosystem of actively updated components of business logic, that could be quickly linked with minimal amounts of new code to create ad-hoc programs to handle any business need, no matter how specific.

While this bold plan for implementing SOA looked great on paper, when companies attempted to put it into action, they quickly ran into difficulties. Most of the problems were caused by similar sets of naive assumptions about organizational behavior that were included in every top-down adoption plan.

To understand what went wrong, let's take a look at some of the flawed ideas that cause so many top-down SOA initiatives to fail.

Top-Down SOA Wisdom: The SOA "Adoption Team" selects and purchases a proprietary SOA Governance product. Development teams will then learn and use this product, both to re-design all existing systems and to design future projects.

Why It Doesn't Work: Massive expense combined with a lack of developer input and vendor lock-in is a recipe for disaster.

In the top-down SOA model, companies often wanted to pass the complex task of SOA adoption to a single team. This team would then be responsible for driving all aspects of adoption. In the days when SOA was the hottest buzzword around, these departments were under heavy pressure to put an SOA in place as soon as possible, and SOA vendors were more than happy to prey on these fears. As a result, the first step in the SOA process for many companies was the purchase of a multi-million dollar SOA Governance Framework.

There are three problems with this approach. First, it virtually guarantees vendor lock-in. While vendor lock-in is sometimes tolerated by companies in application server products, where the loop of interoperability is fairly closed, it has absolutely no place in an integration architecture. It's hard enough to make accurate predictions about how your needs may shift in the future without having made a multi-million dollar commitment to a single company's roadmap. SOA is about what YOUR organization needs - not what a vendor tells you that you need. Don't forget that your needs aren't just a list of systems that need to work together - your solution needs to make things easier for your developers and users, too.

This brings us to the second problem with the top-down model - developer adoption. Your development team isn't sitting around waiting for the chance to implement SOA for you - in fact, in addition to their regular workload, they're probably also kept busy putting out the day to day fires that already plague your network. The effort required to switch to a new model is not trivial on its own. When coupled with a mandate from on high to use a new tool simply because that's what the company has purchased, the task can become insurmountable. Just a few bugs or design flaws in the SOA tool can be enough to make a busy development team less than enthusiastic about the whole project.

Finally, let's talk about the money. SOA is a big change. Making a huge initial investment in a single product is a sure-fire way to kick your entire organization into panic mode, when what you need is a clear, orderly plan that you can implement incrementally, with plenty of input from all arms of your organization. This allows you to ensure that each part works perfectly - integration, services, best practices, adoption - without interrupting your day to day operations or overloading your teams.

Top-Down SOA Wisdom: SOA means an organization-wide paradigm shift, and everyone's efforts rely on everyone else's. Thus, the whole shift must happen simultaneously.

Why It Doesn't Work: The majority of organizations do not have the resources to drop everything and focus on SOA. SOA that falls from the sky is a pipe dream - well-planned incremental adoption is not.

When dealing with a task as complicated as implementing SOA, the amount of changes that need to be made can be daunting. It's tempting to think of the situation as a "catch 22" - we can't start using SOA without writing the services, and we can't write the services without understanding our SOA model. There's only one way out of this catch 22 - the drop everything, rip and replace SOA model, where everything, from development processes to hardware is changed simultaneously.

The problem? This approach is statistically proven to fail. For most organizations, choosing this model is a surefire way to kill your SOA plans.

Fortunately, SOA is not as much of a Catch 22 as it seems. From a top-down perspective, SOA can seem like an irreducibly complex initiative. But from the bottom-up, SOA is a manageable, sensible proposition. We've seen this time and time again in the Mule user community. Good developers understand the value of service-oriented development.

Open-source ESB technologies such as Mule allow teams to follow best practices for SOA without a heavyweight governance model, building out RESTful interfaces that can be reused right away, and will integrate seamlessly with any SOA governance model as the company moves forward. Sometimes, a new Web Service isn't even what you need - if you have a well-designed solution in place already, simply use Mule components to quickly connect it to the rest of your architecture, and move on to an area where the initial outlay associated with building a new service yields a bigger margin of value to your organization. Begin evangelizing your teams today, get them hooked, and then gradually introduce smart, lightweight SOA Governance at a pace that matches your actual available developer resources.

Top-Down SOA Wisdom: The SOA Service Repository saves developers time by giving them reusable components. That's why teams must keep all the information in the repository up to date.

Why It Doesn't Work: The point of SOA is to make development easier, not load developers down with menial tasks. Your SOA solution should automate the cataloging of services.

Like many assumptions made by top-down SOA advocates, this approach is based on the idea that developers are a resource, not teams of skilled professionals. Think of the switch to SOA as a sale you're making to your company. The value proposition is faster development, ease of management, and less time doing tedious integration work. That's why it is a case of serious cognitive dissonance to make your developers responsible for keeping your repository up to date - you're basically saying they will become more productive and do less tedious work by doing additional tedious work.

A good SOA Governance model always makes things easier and reduces complexity. Using Mule as an ESB, our users have built some amazing, bottom-up SOA enabling tools - things like Java classes with metadata that automatically populate the repository with service information, or REST integration of the repository, placing all services directly in front of developers.

When combined with an approach that does not require millions of dollars of lock-in dollars as its first step, this means you can add additional complexity-reducing tooling as SOA technology continues to mature or real-world pain points surface, future proofing your architecture and continually improving your ROI.

Top-Down SOA Wisdom: The key to successful SOA is an organizational culture shift towards "virtuous" architecture decisions.

Why It Doesn't Work: The key to successful SOA is a plan made up of clear, achievable goal sets with well-defined benefits.

Yes, "virtuous" was really a word that was used by top-down SOA advocates to describe the adoption process. The idea was that SOA was so feel-good that everyone would adopt it not only as a time-saving technology but as an ideology.

This is a nice way to think, but it's also a good way to sink your SOA effort by leaving your team in the dark. SOA is not about ideology. It's about doing things in the simplest, most efficient way. Teams are motivated by clear, achievable development goals that have proven, clearly defined benefits. Ditch the top-down SOA soft sales pitch, and show your teams how simple changes in the way they think about development will result in greater productivity down the line.

After over 10 years of failed SOA efforts, it’s clear that the traditional top-down philosophy for SOA is outmoded and outdated – a new approach is needed for today’s organizations to see real value.

The focus should be on making things easier for everyone, not about virtuous architecture; on improving existing organizational structures and processes rather than wholesale re-engineering; on implementing pragmatic tools at the work-group level rather than crippling them with bloated multi-million dollar governance tools.

Mule as an ESB is an integration framework that just works. Lightweight tools like Mule as an ESB completely change the cost equation, as well as the adoption pattern of SOA, allowing development teams to implement SOA-enabling projects in a bottom-up fashion. In fact, Mule is so simple that we've seen some developers successfully implement service-oriented development and follow SOA principles without knowing (or caring) that they are, in fact, doing SOA.

No SOA salesman needs to come knocking on the door pitching SOA, and you don’t need to justify big expenditures for proprietary software licenses and training classes (not to mention the new hardware or upgraded development systems that might be required to run the typical vendor stack). You recognize a problem (or an opportunity), discover how others are currently building similar solutions, and get started learning which available tools fit your project the best.

Don't be scared of Service Oriented Architecture, and don't wait.

Download Mule as an ESB today, and start building your SOA solution the right way - from the bottom up!