One Single Truth: Combining Your BI and Real-Time Analytics

While BI and real-time analytics traditionally fall under the general umbrella of “analytics,” they are still developed and delivered differently. The explosion of Big Data, along with the general growth of data around us, generates great new opportunities for businesses to utilize and leverage data. In this blog post, we elaborate on why the separation between these two systems should no longer exist in modern IT and how combining them together adds huge benefits to businesses.

The Difference Between BI and Real-Time Analytics

BI, or Business Intelligence, is usually used in offline analysis for decision making. Generally, BI data does not have to be acquired in real time, or even near real time. The results involve visualization tools and offline data mining modeling that supports decision-making. BI analysis involves large sets of data, especially when dealing with Big Data.

The purpose of real-time analytics is to inform application owners about user activity during an online user’s session. This information acts as insights that can be leveraged by recommendation engines to promote real-time actions (e.g. opening a chat window or sending mobile push notifications). Real-time analytics is usually performed on the user level, not on large data sets, but at a very high velocity when it comes to Big Data.

BI and real-time analytics pertain to separate data sources and are usually not built on the same infrastructures. In the traditional IT world, developing these two systems requires distinct skill sets and they are generally used by different teams within a company. Building a BI system typically involves a data warehouse with aggregated data and a user profile, from which models are built. In contrast, real-time analytics usually requires an in memory user profile so that data can be aggregated in real time (e.g. specifying tasks that are performed and the number of times they’re performed). This means that action can be taken when a user’s attributes are determined, such as making an offer. Actions are performed during a user’s session, so there is no lag until the user comes back.

The problem is that there is no real connection between BI and real-time analytics.

Why Combine Them?

Real-time visibility is extremely valuable to businesses, however its value can be further enhanced by combining offline information that is extracted from data that already resides in structured databases and data warehouses.

There are great benefits to having BI and real-time analytics under the same umbrella, connecting to each other with bidirectional feedback. Together, they provide the value of heavy batch analytics in addition to real-time events that are part of a real-time action engine’s decision making process. For example, bidirectional feedback enables churn prediction scoring with a very sophisticated algorithm offline. Combining this with real-time events adds a great deal of value. A platform that provides both of these together provides the confidence of knowing there is a single point of truth, the user profile in both places will be the same and calculations that are done in the real-time system serve the offline analytics system (and vice versa, as in the case of scoring, churn prediction, lifetime value, etc…).

Data collection in BI systems is usually done using engines, like Hadoop, for batch processing, which make the data irrelevant time-wise; it cannot be used for anything in real time because it would be too old, from several hours to even a day late. Accessing real-time data, requires an agent and a server side engine in order to make decisions regarding the current state of affairs. So decisions are made based on whatever data is received from customers’ current sessions without all of the history in their user profiles, which would enable a much more informed decision. This is why the two systems were created, with the real-time system working with real-time data and the BI system working with historical data. However, new generation platforms, such as CoolaData’s, collect data in real time, ensuring that data is always accurate in real time, using the same mechanism for both purposes with the same source of data.

An Example

For example, let’s say we have an eCommerce website where a customer visits once to make a purchase, but doesn’t return again. By the end of the first session, we want to bring the customer back to make another purchase. We can do this by telling the customer in real time that their next purchase comes with free delivery. This may cost us money, but it saves our customer money and provides an incentive to return. This tactic should be used in a clever way, focusing on customers with good LTV (lifetime value) and a low chance of returning to our site. If a customer has a high chance of returning, the extra effort on our part is a waste of resources. This scenario requires both offline calculations and real-time decision making. By combining both approaches, we have the best of both worlds. Categorizing this customer under a high LTV and propensity user profile is done by analyzing users who have behaved similarly in the past, which is carried out by building a model scenario offline. The next step is then to match the behavior to the right profile in real time and decide when to offer a free delivery coupon.

By providing this offering under one infrastructure, allowing BI and real-time methods to feed off each other, our effectiveness increases, resource utilization decreases and the quality of our insights is enhanced. It’s a win-win situation, saving time and resources.

Get a demo to learn more on CoolaData’s unique analytics platform.

Share this post

Leave a Reply

Your email address will not be published. Required fields are marked *