What question did this study set out to answer?

The aim is to examine the measurable internal state structures of various open language models.

May 31, 20260 citationsOpen Access

Internal State Geometry is Measurable in Small Open Language Models: A Cross-Family Survey

Key Points

The aim is to examine the measurable internal state structures of various open language models.
Survey of twelve open language models from five architecture families
Assessment of activation observations using a fixed-budget probing protocol
Analysis of module and channel counts across different architectures
Internal state structures observed with coverage between 90.6% and 97.05%
Module count varies by an order of magnitude, ranging from 23 to 84 modules per billion parameters
Channel count scales sub-linearly with parameter count at C ~ N_p^0.63

Abstract

I report a survey of dense activation observation on twelve open language models spanning five architecture families (BLOOM, BLOOMZ, Phi-2, Pythia, Qwen2. 5, InternLM-2) and parameter counts from 124M to 7. 6B. Every model in the qualified set produces an observable internal state structure under a fixed-budget probing protocol: coverage in the band 90. 6% to 97. 05%, total channel count from 523k to 1. 54M, hooked-module count from 147 to 253. Three cross-family patterns are reported: module-count-to-parameter ratio varies by an order of magnitude between architectures (23 to 84 modules per billion parameters) ; channel count scales sub-linearly with parameter count C ~ Nₚ⁰. 63 on this sample; the linear-to-norm module ratio is roughly 5: 1 across families. Small open LMs are not opaque substrates; they are dense, observable, and comparable.

Demander à l'IA

Bookmark

View Full Paper