Jayendra Pathak
NSS Labs
Jayendra Pathak, Chief Architect and Head of Offensive research, brings a wealth of expertise in malware, phishing, and exploit analysis. Jayendra and his team maintain a comprehensive database of threats prevalent in the wild and have built an automated live testing infrastructure that runs with minimal supervision. Along with NSS Labs Researcher Mohamed Saher, he also built an exploit-hunting tool — BaitNET — which gathers real-time intelligence on these exploits. Prior to NSS Labs, he was a research assistant at University of Houston where he was pursuing his MS degree. A native of Nepal, Jayendra worked as a computer engineer for the government of Nepal for 4 years prior to coming to United States. A true researcher, Jayendra's hobby is to scan the Internet for threats and try to determine how those threats affect users. He has a BE in computer engineering from Nepal Engineering College and an MS in Management Information System from University of Houston. Jayendra-Pathak-Huntion-Threat-Hunting-In-Real-Time-Using-a-Streaming-First-Approach_AusCERT2018

Huntion - Threat Hunting in real time using a streaming first approach

Why are we doing this? Enterprises have security controls that generate logs in different formats. Tools that exist in the current market either don’t have the ability to consume and process those logs formats or are quite expensive (Splunk). The inherent problem of these tools is that, data is not streamed in real time to be processed and aggregated. This paper reports on a new system that attempts to address these shortcomings by utilizing low cost open source tools/modules and enable the enterprise to do event processing at a massive scale. The proposed approach does threat hunting for tailored hunting missions on event data ingested from a variety of formats across wide security controls. What does the system do? The functional solution provides the ability for users to define, manage and control every use case as a hunting mission. A hunting mission consists of Plan, Build and Run phases. Every aspect of a hunting mission is defined in a special DSL defined and executed from a hosted threat hunting control system. During the Plan and Build phases of a hunting mission, the filtering, annotation, extraction and correlation rules are developed using an iterative process, based on initial event sample sets. During the Run phase, the system receives and processes continuously streamed input events. Outputs are also configurable by the user. Streaming consumers can receive processed detected events continuously or the user can also do on-demand result generation to visualize outputs or extract outputs for further analysis. How does the system do it? The architecture of the system fulfills the following requirements:
  • It supports both on-demand and streaming consumers of results.
  • It scales to handle massive event throughput and incremental internal data generation
  • It supports diverse input mechanisms, including the Syslog protocol, control system APIs, manual feeds and a system-provided messaging interface.
  • It is cost effectiveThe as-implemented architecture of the system employs open source technologies to achieve the above requirements. Specifically, the following components play an important role:
  • A message processing system that executes distributed processing graphs to process filtering, annotation, extraction and correlation rules, parsed from configurations defined in the internal DSL by users.
  • A columnar database engine that is optimized for massive write rates that is used to produce intermediate results.
  • An infrastructure-as-code platform that facilitates distributed work scheduling and management that serves as automation engine for the central mission control application.In this presentation, we will elaborate on the details of the system design and provide feedback on the results achieved on actual hunting missions.