Being now back and totally immersed in the world of data analytics and the subtleties of streaming, real-time, at rest, batch, micro-batch, etc., it continues to amuse me to see the gap between vendor marketing and reality. Not to mention the delta between customers assumptions and the reality of their current solutions.
To help bridge this gap, I wanted to give you a simple, yet all too often quoted misconceived real-time use case in which both solutions below claim they have real-time analytics licked. But upon closer examination, one solution is like phoning a library to see if they have your favorite book to read, the phone is answered straight away, and the librarian has your book while in the other solution, the phone just rings unanswered…
Real-time analytics: Solution One
“Yup, we totally are doing it, in fact, we’re doing sub-second real-time. We collect all the information we can handle more than (pick your fav hero number), we normalize it and drop it into our data lake. We have granular details and offer 700 criterion we can apply to the data so our customers can run their specific rules against this source and get real-time insight back.”
Real-time analytics: Solution Two
“Yup we totally are doing it, in fact, we’re delivering sub-second real-time. We collect all the information we can handle more than (pick your fav hero number), we normalize it, allow our customers to run their specific rules against this data stream to get real-time insights back. At this point, customers can either take actions on the data in motion, in memory and in the moment or dashboard the results for human consumption. We can then drop the data stream into a data lake.”
One is poking stale data in real-time, and one is actioning a fresh dataset in real-time.
One is the tradition of Extract, Transform, Load (ETL) and one is inserting insight action when it most matters – Extract, Transform, Insight, Action, Load (ETIAL).
It’s a repeated mistake that many customers fall into, this confusion over real-time analytics. Neither solution is good or bad, but they are different and garner different results. Be clear about your business need. Are you fine to get real-time insights to data that’s old or do you want to take actions on data as it happens? Your call.