Evaluating the Necessity of Pre-Training and Layer Depth Effects on Downstream Performance in Lightweight Multimodal Models

This study investigates the relative impact of multimodal pre‑training and Transformer depth on the accuracy–cost balance of lightweight vision‑language models trained exclusively with public data. A unified framework toggles pre‑training and varies encoder/decoder layers (encoder 0–8; decoder 12/24) across three benchmarks—HOI recognition on V‑COCO and clothing categorization + localization on DeepFashion2. Results show that costly pre‑training alters recognition/classification by ≤ 1 percentage‑point and yields only marginal localization gains despite requiring more than 200 GPU‑hours. Removing the encoder keeps HOI accuracy unchanged but lowers localization by 13 percent; a single encoder layer restores performance at trivial cost. Doubling decoder depth brings no benefit while adding 70–95 training hours. Consequently, an enc2‑dec12 configuration without pre‑training provides the best accuracy–cost trade‑off unless fine‑grained localization is paramount.