Deep Learning–Assisted Chaotic Optical Communication With Adaptive Nonlinearity Compensation

ABSTRACT High‐speed optical networks have contributed to the increased demand for secure, reliable, and high‐capacity communication systems in which chaotic optical carriers provide high resistance to interception because of their broadband properties and natural randomness. Nonetheless, with these security benefits, chaotic signals are extremely noise‐sensitive, dispersion‐sensitive, and fiber nonlinearity‐sensitive. These seriously compromising signal quality and synchronization error, and current mitigation strategies that cannot cope with the complexity of chaotic waveforms, caused increased bit error rates (BER) and shorter secure transmission ranges. To overcome these issues, this paper introduces Deep Learning‐Enhanced Chaotic Optical Transmission System with Adaptive Nonlinearity Compensation (DLCOTS‐ANC), which combines a high‐dimensional chaotic carrier produced by Electro‐Optic Feedback Loop and modulated by Mach–Zehnder Modulators. A deep neural network at the receiver learns the nonlinearity distortion patterns and dynamically corrects the impairments in the system introduced by fiber. The system stability is demonstrated by the Lyapunov exponent analysis and DDE‐BIFTO. The results of simulation with OptiSystem and MATLAB show that the synchronization stability has been enhanced considerably. Moreover, a reduction of BER has been significantly achieved, and signal reconstruction has been improved compared with traditional methods of compensation. The adaptive learning model has been successful to control chromatic dispersion, self‐phase modulation, and cross‐phase modulation. DLCOTS‐ANC consistently outperforms existing methods, achieving the lowest BER (3.2 × 10 −4 –1.4 × 10 −3 ), minimal synchronization error (0.004–0.013), highest SNR (18.9–22.0 dB), and superior nonlinearity‐compensation efficiency (82%–92%). It further delivers the most stable chaotic dynamics with Lyapunov exponents 0.21–0.36, the best eye‐height (0.79–0.94), and significantly reduced chromatic‐dispersion residuals (0.34–0.82). Reconstruction accuracy is also enhanced, achieving the lowest MSE (0.006–0.0024), confirming DLCOTS‐ANC's dominant overall performance across all eight parameters.