Moving Beyond "Messages Per Second": Evaluating CEP Products the Right Way
By: Daniel Chait
The CEP market is a relatively new and fast-changing one, with lots of vendors touting similar products. When looking at the field of products and determining which to choose, I have seen many of my customers become confused and bewildered about how to even evaluate the various offerings. Meanwhile, the primary factor most often touted by vendors, journalists, and industry analysts is messages per second (MPS). However, MPS is not the right aspect to focus on. There are three basic problems with using MPS to drive purchasing decisions: first, most of the numbers touted in the press and in marketing literature are largely meaningless since they do not account for realistic or consistent use cases; second, at this point almost all of the major CEP products perform at adequate rates for all but the most demanding applications; and third, there are other, more important issues at play.
To be sure, performance should be a primary consideration. However I would argue that performance consists of many dimensions, and to focus on vendor-provided throughput numbers only, without considering the whole performance landscape, would be a mistake. Certainly, the ability to process events with high throughput and low latency is a laudable goal. However, there are additional performance considerations that firms should consider. One simple consideration is maximum latency; that is, even if the total average latency of a system is quite low, if it periodically experiences pauses or lags, that can cause a serious competitive disadvantage, as competitors and latency arbitrageurs are certain to take advantage of this. Therefore a system which integrates well with the hardware and operating system underneath it to provide upper limits on latency periods, or which otherwise effectively manages peak loads, will yield great returns.
The performance of the non-core parts of the system (adapters, plugins, calling out to custom libraries) can also have a dramatic impact on the overal performance of a CEP application. After all, any non-trivial application will likely include third-party libraries, calls out to custom pricing routines, database queries, custom adapters, and other external connection points. How well do those perform? Are they in-process or out-of-process? Do they require marshalling data into or out of a JVM? Can database connections be pooled? The answers to these sorts of questions will have significant impact on the real-world performance of your applications.
A third, often overlooked, additional performance consideration is performance monitoring and reporting. The ability to accurately see what a system is doing in the field, as opposed to how it performs on the vendor's product sheet or in your testing lab, is a vitally important characteristic. As many of my customers have experienced, performance problems can be among the most difficult and vexing to detect, diagnose, and resolve, since they often depend on certain very specific conditions to surface. Sometimes, a problem may only appear after a system runs in production for several hours, making it that much more problematic to reproduce, track down and correct. A CEP platform that enables a much finer-grained level of detail onto what the application is doing and what is going on under the covers provides a real and unique benefit to firms when they need it most; when the systems are actually up and running in production, supporting real-time business activities.
Aside from performance there are three additional considerations - infrastructure fit, application development model, and total cost of ownership, which firms must consider. Infrastructure fit refers to how well the platform fits into their overall environment. Many of us in the industry are familiar with systems that, while they may provide robust performance, are difficult to connect with, requiring specialized tools and environments. The ability for a system to fit naturally and cleanly into a firm's larger IT ecosystem is a critical consideration. If an event processing platform cannot be easily integrated into the surrounding systems its utility is greatly curtailed. The important considerations here are cross-platform, cross-language, network-independent, extensible, and simple API's. If a system provides those, it can be quickly leveraged within the surrounding environment. Additionally, firms should look at the set of adapters available. Can the CEP engine connect to your messaging bus? Does it come with the right market data adapters? How easy is it to build new, custom adapters to your in-house systems? These questions also play a role in determining the right infrastructure fit.
Application development model refers to the ease of actually building and deploying applications in the system. This is a crucial, but often-overlooked, concern. After all, you will buy the platform but once; you will continually develop new applications on it for many years to come. In this industry, there are a bewildering variety of programming models, from rules-based engines to functional programming paradigms, which can be challenging for developers to become proficient in. Some products provide a clean and simple programming model, allowing application developers to represent events easily, and write the business logic for them in a familiar programming language using standard tools. This stands in stark contrast to many other systems, which, variously, depend on a mix of arcane programming languages, inflexible memory and programming models, and confusing or complicated debug and deployment stories. The end result of an ideal event-processing application is that the business logic is a clean, declarative expression of the relevant business rules, yielding a system infinitely easier to develop, maintain, and extend. Additionally, a robust partner or developer community will be an important factor in your ability to find expertise at building your applications.
Lastly, firms need to look at total cost of ownership (TCO). TCO implications are evident in the way a platform itself must be deployed and managed. Look here for a robust method of configuration and management. Is it a well-known and widely understood paradigm? Will your existing system administrators understand and be familiar with it? Will your IT management infrastructure be compatible with it? And of course, in such a critical piece of technology, the reputation and ability of the vendor providing the product to support a global, mission-critical system cannot be overlooked.
Over the past few years I've had the distinct perspective of watching many firms go through the process of selecting a CEP product. As the product space matures, and possibly narrows, over the next few years, the choices may become easier. For now, I encourage everyone to consider the range of factors above when judging these products so as to align your CEP platform with your business goals.
Daniel Chait is a Managing Director at Lab49, a consulting firm focused on building new custom applications for customers in the financial markets. He can be reached via email at Daniel@Lab49.com. To learn more about Lab49 visit www.lab49.com.
.