Lemmatization of long unit words of historical Japanese

Abstract Lemmatization plays a crucial role in digital humanities research, as it is essential for identifying the canonical forms of words, especially for morphologically rich languages such as Japanese. This article examines methods for estimating lemmas of long unit words (LUWs) in historical Japanese documents, which cannot be estimated by a dictionary-based method. After validating the estimation accuracy using an annotated corpus from the Heian to Muromachi periods (C.E. 794–1573), the results show that lemmas can be estimated with an accuracy of over 90 per cent for different surface forms. This suggests that the method can support and streamline manual annotation tasks. Additionally, it demonstrates that the estimation can be directly applied to digital humanities research. Specifically, by performing hierarchical clustering on unannotated Edo period (C.E. 1603–1868) documents, it became possible to focus more on stylistic features in the clustering process using the estimated LUW lemmas. As a result, a valid clustering outcome was achieved, reflecting reasonable classifications based on factors such as the author and the writing period.