Determination of an acceptable level of spectral data compression by Discrete Wavelet Transforms

Today, due to the ever increasing amount of data generated by analytical instruments, good compression methods are desired to keep computation time acceptable. The lower the volume and noise content of data, the easier it becomes to investigate and interpret the modeling results. Discrete Wavelet Transform (DWT) is an effective data compression and noise suppression tool. Compression can be performed at different levels, in each, the size of signal part of the data reduces to half the size. This work includes an approach for determining an acceptable level of compression of data where the aim is to achieve minimal loss of information and no significant change in the structure of data, which could mean, e.g. no loss in predictive ability or the effective rank of the data-set. The method is based on estimation of the Singular Values (SVs) from a data matrix and the Singular Values at each level of compression followed by the application of Median Absolute Deviation (MAD) of the correlation between original SVs and compression SVs as a simple statistical test for the determination of the optimum level of compression. We illustrate the method using FT-Raman data from aqueous solutions of three sugars (glucose, trehalose and sucrose) and NMR data from mixtures of three alcohols. A sudden change in prediction error sum of square plots from Partial Least Squares (PLS) modeling confirms the results from MAD statistics.