Detecting Distributed Scans Using High-Performance Query-Driven Visualization

K. Stockinger, E. W. Bethel, S. Campbell, E. Dart, K. Wu


Modern forensic analytics applications, like network traffic analysis, perform high-performance hypothesis testing, knowledge discovery and data mining on very large datasets. One essential strategy to reduce the time required for these operations is to select only the most relevant data records for a given computation. In this paper, we present a set of parallel algorithms that demonstrate how an efficient selection mechanism -- bitmap indexing -- significantly speeds up a common analysis task, namely, computing conditional histogram on very large datasets. We present a thorough study of the performance characteristics of the parallel conditional histogram algorithms. As a case study, we compute conditional histograms for detecting distributed scans hidden in a dataset consisting of approximately 2.5 billion network connection records. We show that these conditional histograms can be computed on interactive time scale (i.e., in seconds). We also show how to progressively modify the selection criteria to narrow the analysis and find the sources of the distributed scans.

full text of LBNL-60053 (PDF)

Appeared in Supercomputing (SC06), Tampa, Florida, USA, November 2006, IEEE Computer Society Press.
Accelerating Network Traffic Analysis Using Query-Driven Visualization
Network Traffic Analysis With Query Driven Visualization -- SC 2005 HPC Analytics Results [movie]
Interactive Analysis of Large Network Data Collections Using Query-Driven Visualization

More research work by John Wu
Bitmap Index
Connected Component Labeling
Eigenvalue Computation
Inforamtion available elsewhere on the web
Google Scholar
Contact us

John Wu