Random Projections as First-Class Layers in Analytical Neural Networks: Enabling Microscopic AI for Edge Devices

We investigate the use of fixed random orthogonal projections as explicit first layers in analytically-solved neural networks. Unlike gradient-based training, analytical (closed-form) methods solve for weights via matrix inversion of accumulated co-variance statistics. The dominant memory cost is therefore the accumulator matrices-specifically the d × d covariance Σ XX and the d × c cross-covariance Σ XY-not the forward-pass parameters. By placing a zero-parameter Random Projection (RP) layer before the first learnable layer, we reduce the effective input dimensionality d from the raw feature size (e.g. 784 for MNIST) to an arbitrary bottleneck k, shrinking the accumu-lator from O(d 2) to O(k 2). On MNIST we demonstrate that an RP(784, 300) → Linear(300, 10) architecture matches the baseline Linear(784, 10) in test accuracy (86.1 % vs. 86.1 %) while requiring 3,010 versus 7,850 parameters and 0.37 MB versus 2.49 MB of accumulator memory-a 6.7× reduction. With QR-orthogonal projections at k = 200, we achieve 86.3 % test accuracy, exceeding the full-rank baseline, while using only 0.17 MB (14.6× smaller). Identity truncation-the naïve alternative-collapses to 49.4 %, confirming that the mixing property of random projections is essential. Architectures as tiny as RP(784, 64) → Linear(64, 10) still reach 80.6 % accuracy with 650 learnable parameters and 0.02 MB accumu-lators, making them viable candidates for ultra-constrained edge deployment. These results are grounded in the Johnson-Lindenstrauss lemma and generalise to any linear-algebraic learning system that forms X ⊤ X.