HDC-Brain: A 300M Hyperdimensional Language Model with Bipolar Codebook

This paper presents HDC-Brain v14.1, a 299M parameter language model built on hyperdimensional computing (HDC) principles as an alternative to the standard transformer architecture. The model replaces learned embeddings with a bipolar (±1) codebook trained via straight-through estimator (STE), and replaces the quadratic-parameter QKV projections with multi-head binding attention using only 12,288 parameters per layer — a 5461× reduction over an equivalent-width transformer. Additional components include thought loops (iterative multi-pass reasoning through shared blocks) and a parallel-scan HDC memory with learned mass/decay (constant-memory alternative to KV-cache). The model was pretrained on 3B tokens of FineWeb-Edu for 88 hours on a single RTX 3090, reaching a validation loss of 5.434 bits per BPE token (equivalently 1.25 bits per raw byte under our 32K tokenizer — 0.44 bits/byte behind SmolLM-360M and 0.13 bits/byte ahead of GPT-2-medium on the same FineWeb-Edu slice). After instruction finetuning on 591K filtered prompt-response pairs (75M tokens) drawn from OpenHermes 2.5, TULU-3, Alpaca-GPT4, Alpaca (×3), Dolly-15K, and WizardLM Evol-Instruct, the model reaches 3.521 bits per token on held-out instruction-format text and produces coherent responses in the instruction-following format — correctly answering simple factual prompts (e.g. "Paris" as the capital of France) while exhibiting typical small-model failure modes on arbitrary factoids, arithmetic, and code generation. The bipolar codebook enables 16 MB inference storage for a 32K vocabulary (versus 512 MB float32). Realising a corresponding compute advantage requires custom XNOR/POPCNT kernels, which are not implemented here; end-to-end latency measurements and binary-kernel inference are left as future work. Code: https://github.com/OlegPhenomenon/hdc-brainWeights: https://huggingface.co/olegphenomenon/hdc-brain-v14.1-base (pretrain), https://huggingface.co/olegphenomenon/hdc-brain-v14.1-finetune-v3 (instruction-tuned).