Named entity recognition (NER) is a fundamental task in natural language processing. Recently, non-traditional nouns (known as complex NER) have increasingly emerged, including long noun phrases and ambiguous names, for example, Birds of Prey (and the Fantabulous Emancipation of One Harley Quinn), Among Us, and Chicago, which may refer to a city or a novel. Such rapidly growing entity names pose significant challenges for NER. Arabic NER research is usually limited to flat and nested entities, overlooking complex entities due to limited resources, the language’s rich morphology, and context ambiguity. Such tasks require high-quality annotated data, yet most existing approaches rely heavily on supervised learning, which depends on large amounts of labeled data. However, acquiring large annotated datasets is costly and labor-intensive. We construct our corpus by leveraging the superior performance of large language models (LLMs), which have driven recent advances in dataset generation. We propose an Arabic complex NER (AraCoNER) dataset with semantically ambiguous and complex named entities, using both gold and silver labels. We investigate several agent-based annotation frameworks in addition to the plain LLM to determine the most efficient annotator for our task. Then, we introduce LLMAAA+, an LLM-agent-based framework that integrates an LLM-powered agent as an annotator into an active learning loop to efficiently select what should be labeled. Instead of solely synthesizing the training data from LLMs, we enhance both the annotation and training phases to generate pseudo-labels using k-NN sampling for in-context examples. Such an approach ensures both efficiency and quality, with cost-effective and minimal human involvement. Our results show that combining an LLM (GPT-4) with a structured agent framework (Google ADK) yields the highest annotation accuracy, even with a limited number of annotated examples, supporting the proposed LLM-agent-based active learning framework.
Alruwaili et al. (Tue,) studied this question.