## Introduction

In the world of big data, enterprises have a common problem. They have large volumes of data flowing into their systems for which they need to observe historical trends in real-time. Consider the case of a digital advertising publisher that is receiving hundreds of thousands of click events every second. Looking at the history of individual clicks and impressions doesn’t tell the publisher much about what is going on. A technique the publisher might employ is to track the total number of clicks and impressions for every second, minute, hour, and day. Such a technique might help find global trends in their systems, but may not provide enough granularity to take action on localized trends. The technique will need to be powerful enough to spot local trends. For example, the total clicks and impressions for an advertiser, a geographical area, or a combination of the two can provide some actionable insight. This process of receiving individual events, aggregating them over time, and drilling down into the data using some parameters like “advertiser” and “location” is called Dimensions Computation.

Dimensions Computation is a powerful mechanism that allows you to spot trends in your streaming data in real-time. In this post we’ll cover the key concepts behind Dimensions Computation and outline the process of performing Dimensions Computation. We will also show you how to use Data Torrent’s out-of-the-box enterprise operators to easily add Dimensions Computation to your application.

## The Process

Let us continue with our example of an advertising publisher. Let us now see the steps that the publisher might take to ensure that large volumes of raw advertisement data is converted into meaningful historical views of their advertisement events.

### The Data

Typically advertising publishers receive packets of information for each advertising event. The events that a publisher receives might look like this:

```
public class AdEvent
{
//The name of the company that is advertising
public String advertiser;
//The geographical location of the person initiating the event
public String location;
//How much the advertiser was charged for the event
public double cost;
//How much revenue was generated for the advertiser
public double revenue;
//The number of impressions the advertiser received from this event
public long impressions;
//The number of clicks the advertiser received from this event
public long clicks;
//The timestamp of the event in milliseconds
public long time;
}
```

The class **AdEvent** contains two types of data:

**Aggregates**: The data that is combined using aggregators.**Keys**: The data that is used to select aggregations at a finer granularity.

#### Aggregates

The aggregates in our **AdEvent** object are the pieces of data, which we must combine using aggregators in order to obtain a meaningful historical view. In this case, we can think of combining cost, revenue, impressions, and clicks. So these are our aggregates. We won’t obtain anything useful by aggregating the location and advertiser strings in our **AdEvent**, so those are not considered aggregates. It’s important to note that aggregates are considered separate entities. This means that the cost field of and **AdEvent** cannot be combined with its revenue field; cost values can only be aggregated with other cost values and revenue values can only be aggregated with other revenue values.

In summary the aggregates in our **AdEvent** are:

**cost****revenue****impressions****clicks**

#### Keys

The keys in our **AdEvent** object are used for selecting aggregations at a finer granularity. For example, it would make sense to look at the number of clicks for a particular advertiser, the number of clicks for a certain location, and the number of clicks for a certain location and advertiser combination. So location and advertiser are keys. Time is also another key since it is useful to look at the number of clicks received in a particular time range (For example, 12:00 pm through 1:00 pm or 12:00 pm through 12:01 pm.

In summary the keys in our **AdEvent** are:

**advertiser****location****time**

### Computing The Aggregations

When the publisher receives a new **AdEvent** the event is added to running aggregations in real time. The keys and aggregates in the **AdEvent** are used to compute aggregations. How the aggregations are computed and the number of aggregations computed are determined by three tunable parameters:

**Aggregators****Dimensions Combinations****Time Buckets**

#### Aggregators

Dimensions Computation supports more than just one type of aggregation, and multiple aggregators can be used to combine incoming data at once. Some of the aggregators available out-of-the-box are:

**Sum****Count****Min****Max**

As an example, suppose the publisher is not using the keys in their **AdEvents** and this publisher wants to perform a sum and a max aggregation.

**1.** An AdEvent arrives. The AdEvent is aggregated to the Sum and Count aggregations.

**2.** Another AdEvent arrives. The AdEvent is aggregated to the Sum and Count aggregations.

As can be seen from the example above, each **AdEvent** contributes to two aggregations.

#### Dimension Combinations

Each **AdEvent** does not necessarily contribute to only one aggregation for each aggregator. In our advertisement example there are 4 **dimension combinations** over which aggregations can be computed.

**advertiser:**This dimension combination is comprised of just the advertiser value. This means that all the aggregates for**AdEvents**with a particular value for advertiser (for example, Gamestop) are aggregated.**location:**This dimension combination is comprised of just the location value. This means that all the aggregates for**AdEvents**with a particular value for location (for example, California) are aggregated.**advertiser, location:**This dimension combination is comprised the advertiser and location values. This means that all the aggregates for**AdEvents**with the same advertiser and location combination (for example, Gamestop, California) are aggregated.**the empty combination:**This combination is a*global aggregation*because it doesn’t use any of the keys in the**AdEvent**. This means that all the**AddEvents**are aggregated.

Therefore if a publisher is using the four dimension combinations enumerated above along with the sum and max aggregators, the number of aggregations being maintained would be:

NUM_AGGS = 2 x *(number of unique advertisers)* + 2 * *(number of unique locations)* + 2 * *(number of unique advertiser and location combinations)* + 2

And each individual **AdEvent** will contribute to *(number of aggregators)* x *(number of dimension combinations)* aggregations.

Here is an example of how NUM_AGGS aggregations are computed:

**1.** An **AdEvent** arrives. The **AdEvent** is applied to aggregations for each aggregator and each dimension combination.

**2.** Another **AdEvent** arrives. The **AdEvent** is applied to aggregations for each aggregator and each dimension combination.

**3.** Another **AdEvent** arrives. The **AdEvent** is applied to aggregations for each aggregator and each dimension combination.

As can be seen from the example above each **AdEvent** contributes to 2 x 4 = 8 aggregations and there are 2 x 2 + 2 x 2 + 2 x 3 + 2 = 16 aggregations in total.

#### Time Buckets

In addition to computing multiple aggregations for each dimension combination, aggregations can also be performed over time buckets. Time buckets are windows of time (for example, 1:00 pm through 1:01 pm) that are specified by a simple notation: 1m is one minute, 1h is one hour, 1d is one day. When aggregations are performed over time buckets, separate aggregations are maintained for each time bucket. Aggregations for a time bucket are comprised only of events with a time stamp that falls into that time bucket.

An example of how these time bucketed aggregations are computed is as follows:

Let’s say our advertisement publisher is interested in computing the Sum and Max of **AdEvents** for the dimension combinations comprised of **advertiser** and **location** over 1 minute and 1 hour time buckets.

**1.** An **AdEvent** arrives. The **AdEvent** is applied to the aggregations for the appropriate aggregator, dimension combination and time bucket.

**3.** Another **AdEvent** arrives. The **AdEvent** is applied to the aggregations for the appropriate aggregator, dimension combination and time bucket.

#### Conclusion

In summary, the three tunable parameters discussed above (**Aggregators**, **Dimension Combinations**, and **Time Buckets**) determine how aggregations are computed. In the examples provided in the **Aggregators**, **Dimension Combinations**, and **Time Buckets** sections respectively, we have incrementally increased the complexity in which the aggregations are performed. The examples provided in the **Aggregators**, and **Dimension Combinations** sections were for illustration purposes only; the example provided in the **Time Buckets** section provides an accurate view of how aggregations are computed within Data Torrent’s enterprise operators.

Download DataTorrent Sandbox here

Download DataTorrent Enterprise Edition here

## Leave A Comment