Scientific Data Management Group, Lawrence Berkeley National Laboratory
Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures
- Data is dynamically reduced with a novel pattern searching method based on statistical similarity.
- Compressed/reduced data has accurate representation for the original data, i.e., decompressed/reconstructed data has the same statistical data distribution as the original data.
- The method supports event/feature detection directly on the compressed data.
IDEALEM is an implementation of the data reduction and pattern searching algorithm for streaming data based on Locally Exchangeable Measures, US Patent no. 10,366,078,
"DATA REDUCTION METHODS, SYSTEMS, AND DEVICES", 2019.
- Fig. 1 shows micro PMU (Phase Measurement Unit) data from power grid electricity measurement from one of on-site switches at LBNL (in collaboration with Energy Technology Area).
- In Fig 2, the data compression ratio (original size in bytes / compressed size) in this use case is 95.23 with only one history buffer, and can be achieved using only 64K bytes of memory.
- Compared to gzip, IDEALEM compressed data size is under 2% of gzip-compressed data size in bytes.
- Fig. 3 shows another micro PMU data from power grid electricity measurement from one of on-site switches at LBNL (in collaboration with Energy Technology Area), which results the compression ratio of 242.3 with 255 history buffers.
Software is available for commercial and non-commercial use.
By checking this box, you acknowledge that the software and material on this web site is a non-commercial, academic, non-profit use only. |
"IDEALEM: Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM)" Copyright (c) 2016-2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.
BY DOWNLOADING, INSTALLING, OR USING THE SOFTWARE, YOU ARE INDICATING YOUR ACCEPTANCE OF THE TERMS AND CONDITIONS HEREIN.
Full text of LICENSE
Full text of COPYRIGHT
Before going to download page...
The lab needs to collect the software distribution info.
Please fill in the information below and press Go! button.
- "DATA REDUCTION METHODS, SYSTEMS, AND DEVICES", US Patent no. 10,366,078, 2019
- Similarity-based Compression with Multidimensional Pattern Matching,
ACM Workshop on Systems and Network Telemetry and Analytics (SNTA), 2019
- Multidimensional Compression and Pattern Matching, Poster
Data Compression Conference (DCC), 2019
- Dynamic Online Performance Optimization in Streaming Data Compression
IEEE International Conference on Big Data (Big Data), 2018
- Statistical Data Reduction for Streaming Data
2017 New York Scientific Data Summit (NYSDS), Data-Driven Discovery in Science and Industry, 2017
- Improving Statistical Similarity Based Data Reduction for Non-Stationary Data
International Conference on Scientific and Statistical Database Management (SSBDM) 2017
- Expanding Statistical Similarity Based Data Reduction to Capture Diverse Patterns, Poster
Data Compression Conference (DCC) 2017
- Novel Data Reduction Based on Statistical Similarity
International Conference on Scientific and Statistical Database Management (SSDBM) 2016
- Relational Dynamic Bayesian Networks with Locally Exchangeable Measures
LBNL 6341E, 2013
- Efficient Data Reduction Method with Locally Exchangeable Measures IB-2013-133
- IDEALEM presentation sides (Nov. 14, 2016)
- IDEALEM iPhone demo at Supercomputing 2016
Questions or comments:
Please send any comments or questions for this site to: sdmsupportlbl.gov
This program was constructed by
K. John Wu, and
based on an earlier algorithm and papers.
We gratefully acknowledge the collaboration and helpful suggestions from
colleagues and friends for this project. In particular, we thank
Sebastian Ainslie, and