The Challenge — a fast business cycle and a highly distributed network

The consumer tax services business faces challenges like no other industry. Every year, the business needs to be equipped, staffed, and fully operational for 4 months out of the year and virtually dark for the remaining 8 months. For H&R Block, with over 13,000 offices and over 100,000 seasonal employees, every tax season represents a virtual “mini-Y2K” of IT activity.

During tax season, there is a very limited window to capture the mindshare of the customer. Even with 13,000 locations, it is impossible for H&R Block to ensure that there is capacity in the right place and at the right time.

“If our tax customers come through that door, and they can’t do their taxes right then and there, they’ll walk across the street to any of our competitors,” said H&R Block senior architect, Christopher Ginn.

In order to differentiate from the competition, H&R Block embarked on a strategy called Virtual Tax Services. Instead of trying to optimize capacity around local pockets of demand, they wanted to get work to locations where there is capacity. The vision is to create a virtual network of tax professionals, who can serve customers regardless of the tax pro’s location.

With such an extensive network comprised of so many nodes, not every location has full always-on connectivity. Many smaller offices might have consumer-grade DSL, or even dial-up connectivity. As a result, an architecture with a centrally hosted application at its core was not realistic. Instead, H&R Block needed to find a solution that pushed intelligence to the edge in a distributed fashion, with asynchronous communication back to the core that could withstand interruptions in connectivity.

“One thing that is very important for Block,” said H&R Block senior architect Dan Cahoon, “is that there doesn’t have to be phone lines, there doesn’t have to be communication – as long as there’s power, we still can do taxes.”

Finally, with so many locations, any solution would need to be cost effective, repeatable and manageable across thousands of deployments. For these reasons, open source started to look like a very obvious choice.

The Solution — a Virtual Tax Network

After rolling out a pilot to 60 locations, H&R Block is now in the process of deploying to all 13,000 offices, spread geographically across the U.S. In each location, there is an instance of Mule, integrating with the office file server and doing some work load-balancing.

Customer tax documents are scanned at the location and the work is routed to the Mule service bus, where it is commuted to the H&R Block’s Tax Vault system (built on Alfresco open source CMS), where the VPRO application distributes the work. The bus also integrates open source workflow and collaboration engines (jBPM and Jive forums), and over time,
a rules engine will be integrated as well as enhanced security.

In the core, Mule is demonstrating a high level of scalability and performance. Interfacing directly with the core applications, Mule is serving many of the functions of a traditional application server.

“We process a huge number of transactions through the one instance of Mule that we have,” said Ginn. “Basically every AJAX request we get from the VPRO application comes through Mule. Every command that’s processed, everything that users see and do, comes through Mule.”

H&R Block has already seen benefits from the asynchronous and non-persistent architecture. In the early part of production, the network experienced some unplanned downtime. When engineers brought the system back up, all of the queued messages resumed where they left off, with no lost information.

Mule’s lightweight footprint and simplified programming model made it easy for H&R Block to get up and running quickly and achieve a fast ROI. The pilot project started on November 15 with five developers and rolled out to 60 offices with production data, all by January 25.

Said Ginn, “With no question, Mule has been hands down the best decision we have made.”