Neural Network Applications to Digital Coherent Receivers with Soft-decision Forward Error Correction in Optical Communication

In digital communication systems, Forward Error Correction (FEC) techniques are employed to improve the bit error rate during data transmission over unreliable communication channels. FEC introduces redundancy, allowing the receiver to utilize this additional information during the error correction stage. The decoding algorithms leverage a probabilistic approach based on soft demodulation of the received symbols into bit log-likelihood ratios (LLRs), which provide a measure of the likelihood of different bits given the received signal. However, calculating LLRs becomes computationally expensive when dealing with large constellations and, in non-ideal Gaussian channels, it is difficult to accurately calculate them. In this work, we present various approaches for LLR computation in non-Gaussian channels and compare traditional algorithm-based equalization schemes with machine-learning techniques, employing soft demodulators to generate inputs for a low-density parity-check decoder. Specifically, a soft demodulator based on a neural network (NN) is introduced. Simulations demonstrate the effectiveness of the network in estimating LLRs from symbols with non-Gaussian statistics. The training phase is conducted in a channel-agnostic manner, enabling the network to effectively mitigate the effects of various types of impairments. One of the main challenges is to extract LLRs from channels with correlated phase and amplitude noise with unknown statistical properties, leveraging the correlation arising from consecutive symbols. Moreover, the investigation addresses the development of iterative structures, in which the NN-based LLR algorithm and the FEC decoder iterate between each other to improve channel estimation. Significant focus is placed on fiber nonlinearity, which is one of the most severe impairments for an optical communication system. The study focuses on compensating for intra-channel nonlinear effects by comparing traditional equalization methods with a novel NN-based approach, in order to mitigate nonlinear effects without needing complete knowledge of channel parameters.