Accelerating Network Traffic Analytics Using Query-Driven Visualization

E. Wes Bethel, Scott Campbell, Eli Dart, Kurt Stockinger, Kesheng Wu


Realizing operational analytics solutions where large and complex data must be analyzed in a time-critical fashion entails integrating many different types of technology. This paper focuses on an interdisciplinary combination of scientific data management and visualization/analysis technologies targeted at reducing the time required for data filtering, querying, hypothesis testing and knowledge discovery in the domain of network connection data analysis. We show that use of compressed bitmap indexing can quickly answer queries in an interactive visual data analysis application, and compare its performance with two alternatives for serial and parallel filtering/querying on 2.5 billion records' worth of network connection data collected over a period of 42 weeks. Our approach to visual network connection data exploration centers on two primary factors: interactive ad-hoc and multiresolution query formulation and execution over n dimensions and visual display of the n-dimensional histogram results. This combination is applied in a case study to detect a distributed network scan and to then identify the set of remote hosts participating in the attack. Our approach is sufficiently general to be applied to a diverse set of data understanding problems as well as used in conjunction with a diverse set of analysis and visualization tools.

full text of LBNL-59891 (PDF)

Published in Symposium on Visual Analytics Science and Technology (VAST), Baltimore, Maryland, USA, October 31 - November 2, 2006, IEEE Computer Society Press.
Network Traffic Analysis With Query Driven Visualization -- SC 2005 HPC Analytics Results [movie]
Interactive Analysis of Large Network Data Collections Using Query-Driven Visualization

More research work by John Wu
Bitmap Index
Connected Component Labeling
Eigenvalue Computation
Inforamtion available elsewhere on the web
Google Scholar
Contact us

John Wu