Maximum likelihood multi-user MIMO detection with blind modulation classification

Abstract Multi-User MIMO (MU-MIMO) detection plays a pivotal role in modern wireless receivers, yet practical downlink deployments are severely bottlenecked when co-scheduled users employ unknown and highly heterogeneous modulation formats. This paper introduces a joint architecture that seamlessly integrates blind modulation classification with an adaptive non-linear MIMO detector. First, to overcome the latency of exhaustive classification, we propose a DMRS-anchored selective inference mechanism that mathematically guarantees high-fidelity priors while achieving an 85\% reduction in computational overhead. Subsequently, we formulate an adaptive lattice transformation that actively absorbs the geometric asymmetry of the diverse multi-user signals. By mapping these non-uniform constellations into a standardized integer search space, this mechanism enables an improved sphere decoding (SD) framework. We theoretically prove that this architecture reduces the node-expansion complexity to strictly O (1) per layer, completely circumventing the layer-specific sorting bottlenecks of conventional SD methods. Finally, 3GPP-compliant link-level simulations confirm that the proposed soft-output detector tightly bounds the ideal exact-ML performance in terms of both un-coded bit error rate (BER) and normalized throughput, underscoring its exceptional efficiency and reliability for practical MU-MIMO systems.