Abstract We present the design and empirical characterization of an adaptive per-block pipeline selector for low-bit neural network inference on SIMT GPU architectures. The design encodes a 2-bit mode field directly in the weight data, enabling four distinct inference pipelines to be selected at 128-weight block granularity with no runtime overhead and no shader switching: MODE 00 — Ternary masked XNOR MODE 01 — High-impedance skip (HIGH-Z) MODE 10 — Binary unmasked XNOR MODE 11 — Reserved for future precision extensions A single compute shader handles all four modes through a cascade dispatch on the mode field, extracted as the first operation in the execution pipeline. The design is validated on a commodity mobile GPU using standard OpenGL ES 3.1 compute shaders, requiring no proprietary SDKs. Empirical results across the three active pipeline modes reveal a counterintuitive throughput ordering: HIGH-Z > BINARY > TERNARY This ordering is explained by the instruction count per thread within the SIMT scheduler. Pipelines with fewer instructions per thread allow faster warp retirement and therefore achieve higher effective GPU occupancy, largely independent of arithmetic intensity. Measurements further show that warp divergence across mixed pipeline modes introduces < 5% throughput penalty on the validation hardware. A companion characterization demonstrates that weight sparsity levels from 0% to 90% produce no measurable change in inference latency on the tested SIMT architecture. This result suggests that sparsity alone is not an effective GPU optimization strategy, highlighting that deployment strategies must account for the architectural properties of SIMT execution. License Note This work is released under the PolyForm Noncommercial License 1.0.0 and is free for academic, research, and student use. Researchers, educators, and students are encouraged to study, reproduce, and build upon the methods presented in this work. The scope of this work explicitly applies to any neural network system, regardless of architecture, model type, parameter count, or scale, and to implementations running on any hardware platform, including CPUs, GPUs, mobile processors, embedded devices, specialized accelerators, and distributed systems. As this work has not undergone formal peer review, it is shared in the spirit of open academic exchange. Students and researchers may therefore learn both from the strengths and from potential errors of the ideas presented here. Experimental demonstrations performed on smaller models or constrained hardware should be understood as representative implementations, since methods that improve performance at smaller scales often extend to larger systems when equivalent computational structures are present.
Andrés Sebastián Pirolo (Sun,) studied this question.