NeuronFabric: A Software Reference Architecture for On-Chip Transformer Training with Local Adam

To our knowledge, publicly documented accelerator architectures generally separate trainingcompute from optimizer state updates or rely on external memory/host orchestration. Inferenceonly chips (Groq LPU, Apple ANE, IBM NorthPole) do not support on-chip gradient computation;training accelerators (Cerebras, Tenstorrent) perform the weight update off-chip. Publiclydocumented training systems generally separate gradient computation from optimizer-statestorage or orchestration.This paper describes an attempt to fix that. We built a complete C# software prototype of atransformer that runs forward pass, backpropagation, and Adam weight update entirely in oneprocess with no external framework. The purpose is to establish that the math works and thenumbers are right, before committing to silicon.The model we trained is a 334K parameter autoregressive transformer (d=88, H=4 heads,f=264, L=4 layers, vocab= 256) trained on the Shakespeare corpus. The BF16W variant reacheseval loss 1.5426 within 80K samples (GPU FP32 oracle: 1.5224), with coherent character-leveltext generation confirmed.We also introduce BF16W: storing weights in BF16 while keeping Adam moments in FP32.For the target FPGA budget, BF16W becomes practically necessary to leave SRAM headroomfor activations. A 334K FP32 model with Adam moments requires 4.0 MB — exactly theZCU102 BRAM limit, leaving zero headroom for activation buffers. With BF16W it requires3.34 MB, leaving 660KB free.We describe the vocabulary-budget constraint we discovered during earlier experiments,quantify the BF16W SRAM saving, and lay out the FPGA training target as the next step. NoFPGA measurements are included in this paper; FPGA implementation and measurement areleft for future work.Code: github.com/Binoculars-X/neuro-fabric (release v1.1.0, commit e9ab47a).Publication purpose: This paper serves as a public architectural disclosure and softwarereference implementation for future FPGA/ASIC implementations of the NeuronFabric system,establishing prior art for the local Adam update architecture. NeuronFabric is a researchprototype and not a production accelerator