I am writing this blog right after a very successful Meetup in Chicago, thanks to the CapitalOne team hosting us in their very nice office in downtown Chicago and even more so traveling from the West Coast to talk at this Meetup.
This indeed was a very proud moment in our history. A financial services company talking about a very tough business problem, one which has real $$ and customer satisfaction impact. One of the top 5 Credit Card issuers and providers in the US. A company whose requirements were so stringent in terms of latency (2 ms) and uptime (5 9’s), an unprecedented SLA for a distributed Big Data application.
One visit to the Capital One office in San Francisco and you don’t feel that you are in a â€˜traditional’ financial services company. No mahogany offices. No intimidating board rooms. No ties and suits. Just a bunch of young men and women sitting around modular modern furniture, discussing, white boarding on writable walls. Brainstorming, the next cool disruptive idea that will change the customer experience with banks and financial services. Another group standing around a tall table with coffee mugs, in a well stocked kitchen full of healthy, organic snacks and munchies. This is where cool products are built. Vibe of a typical San Francisco start-up, not boring financial services, banking kind of company. I didn’t explore beyond this area, but I bet there would be some kind of Xbox/ Wii, Lego bricks and foosball table lurking right around the next corner.
It was a team of people in this office who looked at Apache Apex and thought it might be what they need to overhaul an existing architecture that â€˜kinda sorta worked’, umm, but was not built on modern data principles, was not agile and nimble, took years to update, and ways data could be breached ultimately would outgrow the platform. They looked at Apex and compared it with a set of other tools and technologies (see the figure below on the number of technologies considered) which provide or claim to provide similar functionality, i.e. take in data that’s coming in fast and furious, process it in very very short amount of time (we are talking not minutes, not even seconds, but MILLISECONDS) and make a decision. They found that most of the tools and technologies did not even make the initial cut. Finally the three they down-selected, Apache Apex, Apache Flink and Apache Storm on which they built a working application, threw data at it and started to test the results. Apache Spark did not make the cut to the final bakeoff.
The key criteria for their evaluation were:
- A very high performance system. Peeling the onion, what this meant
- A system which can provide a very very low latency for decision making (40 milliseconds)
- Ability to handle large volumes of data and ever changing rules (1,000 events per 20 milliseconds burst)
- 99.5% uptime. Which is about 1.5 days in the YEAR
- Ability to integrate with existing business and financial systems
After they put this through the test and a bake-off, the actual results were even better. The 40 milliseconds became 2 milliseconds. The 99.5% uptime became 99.9995% and 1,000 events burst became 2,000 events burst at a net rate of 70,000 events/second.
- Have a roadmap that is relevant to them, and more importantly they have the ability to contribute and influence
- Be Enterprise Class/ Enterprise grade (unlike most open source projects, which are ahem exactly that- just open source, not created for the enterprise)
- Have a healthy, vibrant and diverse community and committers, i.e. not controlled by one vendor
This was the topic of the Meetup In Chicago and they walked the audience in great detail about their evaluation process and ultimately their findings and why they chose Apex. Some people claim that folks showed up to the meet-up for the Gino’s pizza (a shout out to Gino’s for the great pizza), but I think they really came to see and understand the evaluation process by the Capital One team and what makes Apex special and a league above the other players.
The topic of the Meetup said it all: Next Gen Decision Making in < 2 ms. Yes you read it right – LESS THAN TWO MILLISECONDS
<Look at the slides on Slideshare>
Some great comments and feedback from the Meetup
“The meetup was informative. I am a graduate student and I have tried spark and flink till now. Will definitely try apex now. Thanks to the organizers!”
“Speaker should work on projecting his voice, or hold the mic closer. He was very knowledgeable, but just a little hard to hear. With that said, the content and demo was great! Overall, I enjoyed this presentation very much and I look forward to great things from Apex.”
“Hi Charu, It was great session on Apache Apex held at Chicago. Thanks for organizing it.”
For your convenience
Join our Apex Meetup (please join it as per your geographical location)
Look at details and pictures on this Meetup with Capital One