The entropy of a constrained signal: A maximum entropy approach with applications

The information content of signals subject to constraints such as the total power at a set of frequencies is well understood. This paper illustrates a general approach to the computation of the entropy of a constrained signal. Although general solutions will be derived, exact solutions may be difficult and so three approximation techniques are discussed. One is based on the cumulant expansion. The lowest-order results from this expansion have a simple form. The use of an asymptotic expression to compute the signal entropy is explored when the constraints are such that the number of states available to the signal is small. The role of direct numerical calculations is discussed. Results are discussed in light of the problem of computing the entropy of a signal constrained by its moments.The lowest-order terms in the cumulant expansion of the entropy are used to find a generalized correlation filter to detect signals subject to different constraints. The critical role that entropy has in determining the sensitivity and specificity of such a detector is discussed. Another application is the computation of the entropy of a signal subject to polyspectral constraints. It is demonstrated that the entropy change is governed by the higher-order coherence. Applications of these findings to the detection of signals subject to polyspectral constraints is discussed as are the limitations of the second-order cumulant expansion. More general expressions for the entropy of such systems are derived.