Analysis of Maximum-Likelihood Sequence Estimation Performance for Quadrature Amplitude Modulation

865 the channel. We investigate maximum-likelihood sequence estimation (MLSE) to optimally detect the impaired signal, which is also corrupted by additive Gaussian noise. In particular, we analyze the Viterbi algorithm for realizing MLSE and then proceed to find an upper bound for the BER. In this regard, the analysis to be presented simplifies, unifies, and expands earlier treatments of the same subject. The primary goal is to obtain a basis against which the performance of any suboptimum receiver may be compared. In so doing, we show QAM to be capable of providing performance close to the Shannon limit, an observation of considerable importance for communication satellite and terrestrial digital radio applications. Additionally, study of the mechanisms responsible for MLSE error generation provides insight into appropriate waveshaping to improve BER performance. One such waveshaping technique, wherein the spectrum is intentionally asymmetric with respect to the transmitting filter passband (e.g., QAMSingle Sideband), is shown to generally improve the BER performance. In Section II, motivation behind a study of QAM is illustrated, the QAM model is presented, and the MLSE algorithm is derived. Section III is devoted to the derivation of a BER outer bound, and in Section IV, this bound is applied to examples which demonstrate the inherent power of MLSE. We show that a BER of 1CT can be maintained at a transmission rate of 5 bits/s/Hz with an energy per bit penalty no greater than 1 dB compared against ideal nonoverlapping rectangular signaling; a four-pole Butterworth transmit filter is assumed in this calculation.