Adaptive Equalization of Channel Nonlinearities in QAM Data Transmission Systems

The prevalence of nonlinearities and their distorting effect on highspeed data transmission over voiceband telephone channels has long been recognized.1 The effect of nonlinear distortion on linearly modulated data signals is to introduce nonlinear intersymbol interference and 2589 reduce the margin against noise. For data rates above 4800 bps, nonlinear distortion is the dominant impairment on many voiceband telephone channels. Experimental studies have measured nonlinear distortion and related the observed error rates for specific modulation formats to this and other measured impairments.2-3 Estimation of performance for data transmission in the presence of nonlinearities can be carried out4 but gives little insight into the problem of receiver optimization, except for certain simple nonlinear channel models.5 Recognizing that nonlinearities in transmission channels generally coexist with linear elements such as filters, one is led to consider a general nonlinear receiver structure, based on a Volterra or Wiener kernel characterization6 of a general nonlinear system such as that proposed in Refs. 5, 7, and 8, the latter in connection with adaptive echo cancellation. In the present work, we extend this approach by generalizing the structure of a passband decision feedback equalizer, previously studied in connection with linear channel distortion,9 to process nonlinear as well as linear functionals of the incoming signal and prior decisions.* The new receiver structure is based on a model of a passband channel with quadratic and cubic nonlinearities, as well as linear filters.