IDEALEM
Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures

Summary Data is dynamically reduced with a novel pattern searching method based on statistical similarity. Compressed/reduced data has accurate representation for the original data, i.e., decompressed/reconstructed data has the same statistical data distribution as the original data. The method supports event/feature detection directly on the compressed data.
Fig. 1	Fig. 2
Fig. 3
Fig. 1 shows micro PMU (Phase Measurement Unit) data from power grid electricity measurement from one of on-site switches at LBNL (in collaboration with Energy Technology Area). In Fig 2, the data compression ratio (original size in bytes / compressed size) in this use case is 95.23 with only one history buffer, and can be achieved using only 64K bytes of memory. Compared to gzip, IDEALEM compressed data size is under 2% of gzip-compressed data size in bytes. Fig. 3 shows another micro PMU data from power grid electricity measurement from one of on-site switches at LBNL (in collaboration with Energy Technology Area), which results the compression ratio of 242.3 with 255 history buffers. IDEALEM is an implementation of the data reduction and pattern searching algorithm for streaming data based on Locally Exchangeable Measures, US Patent no. 10,366,078, "DATA REDUCTION METHODS, SYSTEMS, AND DEVICES", 2019. Software is available for commercial and non-commercial use.

By checking this box, you acknowledge that the software and material on this web site is a non-commercial, academic, non-profit use only. "IDEALEM: Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM)" Copyright (c) 2016-2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. BY DOWNLOADING, INSTALLING, OR USING THE SOFTWARE, YOU ARE INDICATING YOUR ACCEPTANCE OF THE TERMS AND CONDITIONS HEREIN. Full text of LICENSE Full text of COPYRIGHT

Before going to download page... The lab needs to collect the software distribution info. Please fill in the information below and press Go! button. Name Institution Email

Related links:

• "DATA REDUCTION METHODS, SYSTEMS, AND DEVICES", US Patent no. 10,366,078, 2019
• Similarity-based Compression with Multidimensional Pattern Matching,
  ACM Workshop on Systems and Network Telemetry and Analytics (SNTA), 2019
• Multidimensional Compression and Pattern Matching, Poster
  Data Compression Conference (DCC), 2019
• Dynamic Online Performance Optimization in Streaming Data Compression
  IEEE International Conference on Big Data (Big Data), 2018
• Statistical Data Reduction for Streaming Data
  2017 New York Scientific Data Summit (NYSDS), Data-Driven Discovery in Science and Industry, 2017
• Improving Statistical Similarity Based Data Reduction for Non-Stationary Data
  International Conference on Scientific and Statistical Database Management (SSBDM) 2017
• Expanding Statistical Similarity Based Data Reduction to Capture Diverse Patterns, Poster
  Data Compression Conference (DCC) 2017
• Novel Data Reduction Based on Statistical Similarity
  International Conference on Scientific and Statistical Database Management (SSDBM) 2016
• Relational Dynamic Bayesian Networks with Locally Exchangeable Measures
  LBNL 6341E, 2013
• Efficient Data Reduction Method with Locally Exchangeable Measures IB-2013-133
• IDEALEM presentation sides (Nov. 14, 2016)
• IDEALEM iPhone demo at Supercomputing 2016

Questions or comments:

Credits:

We gratefully acknowledge the collaboration and helpful suggestions from colleagues and friends for this project. In particular, we thank Emma Stewart, Sean Peisert, Chuck McParland, Reinhard Gentz, Mahdi Jamei, Ciaran Roberts, Sebastian Ainslie, and Horst Simon.

Disclaimer
LICENSE
This web page is maintained by the Scientific Data Management Group
Lawrence Berkeley National Laboratory

Last updated: Wed Jul 27 13:52:21 PDT 2022