Optical OTSM signal detection using RNN-based QRM-MLD for optical Rayleigh channels

Orthogonal time–space modulation (OTSM) is emerging as a promising two-dimensional signaling technique for massive multiple-input multiple-output (M-MIMO) communication; however, its performance is highly dependent on reliable signal detection under imperfect channel state information (CSI). Existing detectors, such as zero forcing (ZF), minimum mean-square error (MMSE), and maximum likelihood detection (MLD), as well as recent convolutional neural networks (CNNs), bidirectional long short-term memory (BiLSTM), and generative adversarial network (GAN) models, either suffer from significant bit error rate (BER) degradation or require high computational complexity in large antenna arrays. To address these limitations, in this study, we propose a recurrent neural network (RNN)-assisted QR decomposition-based reduced search maximum likelihood detector (QRM-MLD) for OTSM in 5G and beyond 6G systems. The algorithm combines lattice-based searching with adaptive sequence learning to improve robustness against Rayleigh fading and channel estimation errors. The numerical results for 64 × 64 and 256 × 256 M-MIMO configurations demonstrate a consistent 5–12 dB signal-to-noise ratio (SNR) gain at a BER of 10⁻3 over conventional linear and iterative detectors, even under 10% and 20% CSI mismatch. A comparative analysis with recent deep learning detectors reported in the literature shows an additional 2–7 dB performance advantage while maintaining a moderate polynomial complexity. These findings indicate that the proposed RNN + QRM-MLD method offers an effective balance between reliability and computational feasibility, making it suitable for high-capacity future wireless receivers, optical wireless communication links, and dense Internet-of-Things (IoT) environments.