Abstract Chinese poetry has been a cultural carrier of storytelling since ancient Chinese culture. Human poets convey their narratives through poems to connect with their audiences regarding scenes, characters, and related relationships and emotions. Although creative GANs for generating poems, lyrics, and metaphors are gaining popularity in recent years, existing studies rarely consider the feelings and relationships among characters in various scenes. Therefore, we propose an end-to-end approach, namely Video-Transformed Persona Poem Generation (VTPPG). VTPPG emulates the poet’s views and captures four qualities: the character’s actions, the character’s relationships, the character’s emotions, and the scenery’s emotions in the machine-generated Chinese quatrain. Accordingly, we conduct a qualitative analysis to compare VTPPG with state-of-the-art baselines, such as Jiu Ge, in terms of the four qualities: fluency, coherence, and meaning. Our results demonstrate that the video-based VTPPG outperforms the baselines by 8.25%. Furthermore, we conducted an in-depth analysis of drama scenes, such as those from Romance of the Three Kingdoms, under the four key qualities, and invited human poets in our evaluation. As a result, VTPPG demonstrates effective generations of expressive texts from videos, potentially facilitating creative democratisation in diversified multimedia contexts.
Wang et al. (Wed,) studied this question.