This dissertation investigates the implementation of Transformer-based architectures for time-series analysis on resource-constrained embedded Field-Programmable Gate Arrays (FPGAs) within the Tiny Machine Learning (TinyML) domain. The research follows a structured methodology, progressing from implementation feasibility to deployment-oriented optimization, and finally to fairness-constrained cross-architecture comparison, thereby connecting software-level model optimization with hardware-level realization. To establish implementation feasibility, the research first develops a compact Encoder-only Transformer architecture for efficient inference on embedded FPGAs. All core components are realized in integer-only format with low-bit quantization. The model is deployed using an end-to-end pipeline that integrates Quantization-aware Training with automated VHSIC Hardware Description Language (VHDL) code generation, Register-Transfer Level (RTL) simulation, and hardware synthesis. Results on representative time-series forecasting datasets confirm real-time inference on the AMD Spartan-7 XC7S15 FPGA within milliwatt-level energy budgets, while experiments on the Lattice iCE40UP5K FPGA expose hard feasibility boundaries imposed by limited logic and on-chip memory. Building on this feasibility foundation, this work advances deployment-oriented optimization by introducing hardware-aware configuration exploration. Deployment is formulated as a bi-objective optimization problem that balances predictive performance and energy consumption under strict resource limits. An Optuna-guided search loop is included in the deployment process. Each sampled configuration is trained, quantized, and then evaluated with RTL- and post-synthesis feedback. A staged, constraint-aware mechanism also brings in application-driven requirements for accuracy, latency, power, and energy. Experiments on several time-series tasks show that hardware-aware search can reliably identify deployable configurations with millisecond-level latency and sub-millijoule energy per inference on the Spartan-7 XC7S15 FPGA. To further improve deployability under tight constraints, Mixed-precision Quantization (MPQ) is investigated. We present an FPGA-friendly, requantization-free, cascaded, mixed-precision formulation, along with a hierarchical precision configuration spanning module-wise, layer-wise, and intra-layer granularity. To enable low-cost feasibility filtering before training and synthesis, we propose a resource-aware MPQ analysis based on a pre-profiled module-wise knowledge database. In parallel, the MPQ formulation is integrated into the Optuna-guided workflow to automatically explore mixed-precision configurations. This demonstrates complementarity between two approaches: (1) heuristic mixed-precision rules that expose quantization sensitivity and (2) an automated search that identifies compact configurations with balanced accuracy and energy consumption under hardware constraints. The deployment value of Transformer-based architectures is evaluated in comparison to alternative time-series models. Fairness-constrained, same-board evaluations conducted in two case studies indicate that the deployment value of Transformers on embedded FPGAs is context-dependent. In sewer overflow forecasting, quantized Transformers achieve competitive or superior accuracy when longer temporal contexts are available, but typically incur higher latency and energy consumption than Long Short-Term Memory models under identical constraints. For wrist-worn gait recognition, where strict wearable energy budgets and short latency requirements apply, compact one-dimensional convolutional models offer more favorable accuracy-energy trade-offs. These findings indicate that deployment value is determined by the interaction among model structure, task characteristics, and system-level constraints. In summary, this study enables systematic reasoning about feasibility, optimization, and deployment value for time-series Transformers on resource-constrained embedded FPGAs. It provides practical deployment knowledge, methodological tools, and hardware-grounded evidence for implementing Deep Learning in realistic TinyML systems.
Tianheng Ling (Wed,) studied this question.