The Challenge – Expensive and Unreliable Fundraising Infrastructure
As a nonprofit organization dedicated to funding blood cancer research, LLS has spent many decades finding creative ways to raise money for its cause. With programs like Team in Training, Light the Night Walk, the Leukemia Cup Regatta, LLS has been a pioneer of event-oriented fundraising, leveraging the passion of individual fundraisers. Its flagship program, Team in Training (TNT), has grown to become the world’s largest endurance sports training program, raising more than $850 million since its founding in 1988.
The TNT program allows amateur endurance athletes to help raise funds for LLS. In exchange, the program provides personalized fitness training and sponsored trips to endurance events around the globe, including marathons, triathlons, and century (100-mile) bike rides.
TNT’s earlier efforts at online fundraising were made in partnership with a third-party outsourcer, who provided an off-the-shelf hosted fundraising application. For a fee of 7% of each transaction, the outsourcer provided a personalized fundraising website for each program participant and collected funds from donors, remitting them to LLS on a monthly basis. This arrangement allowed LLS to access the online channel relatively quickly, allowing the organization to focus on things other than IT infrastructure. Over time, however, the solution started to become a victim of its own success.
“Over the last three or four years, we have seen a real spike in the participation of Team in Training,” said Osman Mazhar, senior architect at LLS. “At the same time, with the increasing maturity of online fundraising, there has been a massive shift of fundraising from offline channels to the online channel.”
As increasing numbers of fundraisers joined the TNT program, the outsourced application began to have difficulty scaling in concert. The site would experience frequent outages and performance issues, and as a result, confused users would often see their transactions fail or mistakenly make duplicate donations.
The benefits to LLS’s IT organization started to become less clear. The weak integration between the outsourced application and LLS’s internal systems meant duplicative functionality and higher maintenance costs. Because there were multiple owners of the infrastructure, LLS’s IT group was also burdened with managing customer support calls, often for issues that stemmed from the outsourced solution. Finally, the 7% transaction fee, in addition to the lost interest carry due to the delayed transfer of funds, meant that LLS’s fundraising costs were prohibitively high.
Over time, it became clear that LLS needed to make some changes in order to reduce fundraising costs and improve the user experience. The crux of this plan would be to regain control of the infrastructure end-to-end, developing a set of rich user applications and integrating them with the existing back-end systems. Because donors would be performing important financial transactions, the system would need to be highly reliable – with no messages lost, robust exception strategies, and an active/passive failover architecture to achieve full redundancy.
The Solution – Scalable and Reliable ESB Infrastructure
Using Adobe Flex to create a rich user interface, LLS built a series of front-end applications, allowing individual participants to register themselves and manage their own fundraising, and providing TNT chapters with tools for administration. The team brought together Mule ESB Enterprise and ActiveMQ at the core, creating a backbone to integrate the front-end applications with a core set of reusable services called Mission360, as well as a legacy VB-based campaign management system called Society Central. Mule’s out-of-the-box connectors allow the applications to access the file system and SMTP services for sending email.
To ensure reliability, LLS leveraged Mule’s built-in component and connector exception strategies. In the case of a component failure, the component exception strategy is configured to retry the message until it succeeds. The messages are also persisted in ActiveMQ via Mule’s JMS endpoint to ensure that no messages are lost, even if the ESB shuts down for any reason. If ActiveMQ fails, the connector exception strategy is configured to persist messages in the file system until ActiveMQ comes back online. In addition, all messages are logged for auditing purposes.
This new architecture and organization has resulted in a set of robust systems and processes, designed to scale with increasing donations for years to come. The system is designed and tested to handle peak loads of over 24,000 messages per hour. In the first 4 months of production, the system processed more than 500,000 messages with zero message loss.
“Since we went live with the solution, we have seen 100% uptime, and almost unnoticeable CPU utilization from Mule,” said Mazhar. “This reliability has really freed the IT team to spend more time on innovation rather than fielding support calls as we did in the past.”
The business-level benefits resulting from these changes are even more remarkable. By eliminating transaction charges from the hosted application provider, LLS has reduced fundraising fees from 7% with the outsourced provider to a mere 2% currently for credit card processing fees.
“The savings we have realized by building this infrastructure has really affirmed the success of this venture,” added Mazhar. “Every dollar we save on fundraising costs means more money going for research and patient services.”
Furthermore, user feedback on the interface and system performance has been overwhelmingly positive. The improved functionality and performance have helped to drive over $10 million in donations in the first 4 months alone.
Over time, LLS will leverage TNT’s scalable platform to bring its other fundraising programs on board, further improving cost efficiencies by eliminating redundancies. In addition, LLS plans to implement management and monitoring tools like Mule HQ™ to improve visibility and control, as well as begin to evaluate Mule Galaxy™ for artifact management and governance. Finally, with more IT resources now available for innovation, the team plans to develop and integrate value-added community features such as blogs, discussion boards and forums, to continue enhancing the user experience and driving stickiness.
Choosing Mule and Working with MuleSoft
As the LLS team evaluated various ESB technologies for its new infrastructure, it was looking for a flexible choice that could adapt to the existing infrastructure. Mule quickly stood out from the pack, given its broad support for multiple protocols, as well as its strong integration with the Spring framework, which is used throughout LLS.
Mule’s open source model fit well with LLS’s internal IT strategy for using open source technology where possible. After evaluating several open source options, Mule established its leadership position, as evidenced by its strong adoption and robust user and developer community. With further validation from industry analysts, Mule rose to the top as the clear choice.
The team was challenged by the LLS Board of Directors to meet an aggressive 3-month timeline to complete the project. To make sure that their Mule implementation was “first time right,” the LLS team decided to bring in MuleSoft Consulting to help with their architecture and implementation. The consulting engagement was designed to be pragmatic, structured as a tactical 2-week project.
During this time, MuleSoft Consulting acted as an extension of LLS’s team, performing a full design review and providing architectural recommendations. The project included documenting functional and technical requirements, reviewing specific Mule configurations, as well as assisting with the technical implementation itself. In addition, the team emphasized knowledge transfer on Mule configuration best practices, such as exception handling and failover, performance tuning and testing.
Given LLS’s performance and reliability requirements, the team chose to develop on Mule Enterprise, the enterprise-class version of Mule. With enhanced features for enterprise production deployment, along with an additional layer of professional QA, Mule Enterprise gives the LLS team peace of mind. The technical support services that come with a Mule Enterprise subscription add yet another layer of protection, making sure that any issue can be resolved quickly, minimizing downtime and impact to customers.
“LLS is committed to leveraging open source technologies where possible, but having a support subscription offering for those products is equally important to us,” concluded Mazhar. “These are mission-critical systems handling significant amounts of money, so we want to have somebody to lean on in case anything goes wrong. For LLS and our beneficiaries, it can literally be a matter of life and death.”