Big Problem – Cyber Threats

How secure is your network? And your data? Was your network breached recently? Has confidential information been compromised?

Even when almost everybody agrees to the part that Network Security is a vital organ of the IT infrastructure system. Yet, there are newer worries that underline just how critical it is to take network security seriously.  With the changing face of the internet — from desktops to handheld devices and from data centers to cloud computing and IoT — network security too, has changed colors.  All this has resulted in sudden surge in respondent networks & internet activities. And hence cyber threats has also been seen rising. More and more devices, networks and people are vulnerable to cyber threats like phishing, attacks and malwares.

Reports  show that nearly all organizations faced some sort of a security breach in the last few years. If this isn’t worrisome enough, the 2015 Trustwave Global Security Report  highlights a grimmer picture — nearly  81%  victims were not aware that they were breached.

According to the 2015 Trustwave Global Security Report,  98% of applications tested  were vulnerable  to attacks, whereas 95% of mobile applications have at least one known vulnerability. Nearly 43% breach investigations were reported by the retail sector, with a 42% in the e-commerce  sector, because both these sectors were characterized by high volumes of payment activities.

There is more. The 2015 Cyber Threat Defense Report  shows that nearly 71% of respondent networks were breached, recording a 9% rise from last year.

With ease of use by mobile devices, more and more non-technical folks are getting involved in online purchase/payment options with less knowledge and awareness against fraudulent activities and common security practices. 2015 Cyber Threat Defense Report has also mentioned that there has been significant rise in mobile device threats, nearly around ~60%, which is quite huge. And vulnerable mobile applications makes it even bigger and immensely difficult to tackle.

So what are you doing to fight security breaches?

The most common network security solutions are  Firewalls, Intrusion Detection System (IDS),  Intrusion Prevention System (IPS),  Unified Threat Management (UTM), and Security Information and Event Management (SIEM). However, these solutions are perimeter-centric defense mechanisms  and  they play an essential role in defining preliminary defense strategies for networks.

Limitations & challenges

As the statistical analysis suggests, conventional mechanisms to deal with Network Security are not sufficiently enough. Perimeter-centric defense mechanisms are primarily designed to protect the network, and not the data that flows. With Cloud Computing & IoT, the definition of a perimeter  is becoming hard to concretize. In the new age context, the perimeter-centric strategies are  inadequate for sophisticated attacks, with no mechanisms for  data-in-motion security.

With the kind of Data & High-speed networks and ever increasing bandwidth, conventional mechanisms are finding it difficult to scale-up. Effective changes in network usage patterns have added another level of challenges. Retail, e-commerce like online industries are in booming phase and vast majority of people are using their mobile devices to purchase online. All these have been changing dimensions of Networks and simultaneously of Network Security of course. Cloud Computing & IoT adds up to network traffic at a very large scale.

Even though there is need for analyzing everything that goes out and comes in your network, user experience can’t be compromised. Variety of things happening on internet puts enough challenges in front of manual or defined set of rules. Need of the hour is Behavioral Analysis, Predictive Analysis – machines talking and learning, staying ahead in time,  learning from mistakes and building intelligence to defend network & data from possible external attacks.

There are also unique set of challenges per network or type of network. Security approaches must also face challenges from the network itself, specifically, the type of network. SQL Injection Attacks, for instance, affect data at the execution level. However, No-SQL databases are naturally resilient to such attacks. The same might be targeted at the point of authentication. With the proliferation of the internet, maintaining the security of the network, and the data within this network is becoming increasingly complex. This is especially so in the world of cloud computing and IoT, where conventional security measures are fast proving to be ineffective. Newer security measures are a must, but not at the cost of user experience. A degeneration in the quality of experience could seriously affect business value.

Future in Security – Think big data, Think big insights

When you think big data, you think Hadoop. With Hadoop 2.0, processing is going beyond MapReduce. Programmers now have the flexibility to write distributed applications in the  Hadoop 2.0  ecosystem. The time has come to productize Hadoop, and employ it to fight security concerns.

Zero Trust Model

After the U.S.  Government issued directives via Foreign Policy Cybersecurity Executive Order  13636 Improving Critical Infrastructure Cybersecurity, organizations  are on the verge of migrating  to the Zero Trust Model. A glaring example of this move is Google’s initiative called  BeyondCorp. The Zero Trust Model is an aggressive network security approach for  accessing  network resources using secure channels such as  SSL for communication, authentication  and authorization,  and generating logs for network activities. Compliance with this model effectively translates to large sized data churns.

Big Data & Network Security

Along with conventional methods for handling  network security, the next generation requires  Threat Intelligence using Behavioral Analysis  of the network  and data patterns. Behavioral Analysis of malicious activities, content or files shared or accessed in a network can be used to build intelligence  for handling vulnerabilities and security loopholes.

Capturing Network flows related information using netflow  provides an insight  of the usage patterns and trends that an  enterprise network is following. The grasp of statistical information provides  knowledge of the type of network security solutions required., Applying this knowledge can be one approach to dealing with  security concerns.

To gather network flow information,  packets are captured, higher level protocols  are decoded  to get a clear idea of what is happening in the network. Deep Packet Inspection (DPI) engines in Big Data  will soon  be the next step in  understanding the tiniest  detail of  a network.

Big Data Analytics & Security

According to Gartner,  by the year 2016,  25% of global giants will have adopted big data analytics  for at least one security or fraud detection  scenario. Big Data Analytics & Predictive Analytics  are  already playing an important role in ensuring security of network and the data flowing thereof. However, much still remains to be done in the way of exploring predictive analytics for foreseeing vulnerabilities, and thus staying ahead of attacks. An example is OpenDNS’s Umbrella Security that uses predictive analytics for detecting probable attacks.

Security issues, analyzed in real time

DataTorrent RTS recognizes the myriad challenges of the network security domain. It is designed to handle large volumes of data, while addressing in the best way, the challenges that network security handling poses.

Critical events lose its value over the period of time. It is equally important to attend each such event in real time. DataTorrent RTS comes with an array of operators that can be used for scenarios such as  Analytics, Behavioral & Predictive Analysis, Malware Analysis,  and Security alerting in real time. DataTorrent RTS is available as an Open Source platform, as well as a licensed edition along with  a proprietary set of operators that can be used to fulfill different security usecases. What’s more, developers can write their own code to handle situations and scenarios typical to their business needs.

DataTorrent has unified batch & real time streaming platform. The same platform can be used for real time analysis and batch processing for any further detailed analysis. Along with competitive advantages over other real-time streaming platforms, there are 2 uniquely identifiable, distinguishing, major factors DataTorrent has in Big Data Network Security Domain:

  1. Operators or logical building blocks can easily be added or modified. So Security Analysts can quickly automate some of their work.

  2. DataTorrent has released core engine under ASF as Project Apex. Independent development of fraud detection or Intellectual Property related business logic can remain with end user.

NetFlow Analyzer using DataTorrent RTS

Real time analytics of network security incident detection can be achieved using SIEM logs or Netflow information. SIEM logs emit information about events, Security Analytics platform can consume them for  anomaly detection. The entire solution can effectively detect brute-force attacks, for example,  similar failed attempts within certain time-window. The same can be used for much complex set of events after performing Behavioral Analysis.

In this example,  NetFlow Analyzer is  built using the DataTorrent RTS platform.


NetFlow Collector/Acceptor  is the input operator in this scenario. Also Network Traffic Analysis and Real-Time Analytics considered here are specific for NetFlow data.

Real Time Security Alerting using DataTorrent RTS

DataTorrent RTS can effectively detect, and record alerts of malicious activity/content or anomaly detection. There are various use-cases and each one has its own set of significances. Here we are trying to list down very few/basic of them.

ArcSight CEF  Events Listener for SIEM logs


Messages that are generated in different formats.  The  Common Event Format (CEF)  is one of the most commonly used format. After parsing, CEF events represent information about network flow. Such  log messages can be used for generating alerts in the event of certain conditions having being met. CEF Events Listener is an  input operator. CEF Parser would convert CEF to POJO and Rules would be applied on POJOs.

Malware Threat Analysis using DataTorrent RTS


Files that travel in a network can be infected, or might carry with them malicious data. This can be primarily identified using malware analyzers such as VirusTotal. The Malware analyzer accepts data using the dtIngest application, and proceeds to compute the hash value (MD5, SHA1, or SHA256) during the file being ingested and later look up for File Reputation in the database from these Malware Analyzers.

Network Threat Analysis


This is slightly complex use-case, wherein major part comes with DPI (Deep Packet Inspection) and today vast majority of protocols being used. Existing Open Source or Proprietary DPI Engines can be used to Protocol decoding purpose.

DataTorrent RTS can be leveraged to build as many modules as the security and vulnerability scenarios. To learn more about DataTorrent RTS, visit