A Study on the Application of Using Hypernetwork and Low Rank Adaptation for Text-to-Image Generation Based on Diffusion Models

Recent advances in the field of image generation have attracted attention due to the growing number of diverse data sources and test samples. A primary driver of this evolution is the application of neural networks, particularly for generating high-quality images from textual prompts. Despite the potential of diffusion models in this sector, they typically face computational challenges associated with vast datasets. This paper describes the research on two existing solutions: Hypernetworks and Low-Rank Adaptation (LoRA), both aiming to streamline and optimize the image generation process. While hypernetworks dynamically adjust model parameters based on the input text, increasing flexibility and performance, LoRA efficiently adapts the primary model style without requiring it to be trained from scratch. Using the Stable Diffusion 1.5 model as a benchmark, this research evaluates the influence of hypernetwork and LoRA modifications. The results indicate that both approaches provide efficient and highly accurate image generation, confirming their efficacy in contemporary image generation tasks.