What question did this study set out to answer?

This research aims to evaluate a hybrid sparse-dense retrieval system for answering cultural multiple-choice questions.

May 18, 2026Open Access

CultRAG at SemEval-2026 Task 7: Hybrid Sparse-Dense Retrieval with Entity-Centric Knowledge Bases for Cultural MCQ Answering

Key Points

This research aims to evaluate a hybrid sparse-dense retrieval system for answering cultural multiple-choice questions.
Developed a six-phase pipeline utilizing hybrid BM25+FAISS retrieval and trust-weighted reranking.
Incorporated intent detection and country-aware filtering to tailor responses.
Evaluated system performance against a baseline LLM-only model.
The LLM-only model achieved 78.6% accuracy, slightly outperforming the full RAG system at 78.5% (p = 0.962).
Only 40.7% of questions were found answerable from the knowledge base, indicating high noise in retrieval.
Anti-leak prompt filtering in Phase 4 effectively reduced answer-anchoring artifacts.

Abstract

We present a trust-weighted Retrieval-Augmented Generation (RAG; Lewis et al., 2020) system for SemEval-2026 Task 7 (BLEnD) Track 2 (Ousidhoum et al., 2026), targeting English cultural multiple-choice QA across 30 countries. Built atop Llama-3.1-8B-Instruct (Meta AI, 2024), the six-phase pipeline integrates hybrid BM25+FAISS retrieval, country-aware filtering, intent detection, tiered routing, anti-leak prompt engineering, and trust-weighted reranking. The core finding is that RAG hurts rather than helps: the LLM-only baseline achieves 78.6% accuracy, outperforming the full system at 78.5% (McNemar's test, p = 0.962). Oracle analysis reveals that only 40.7% of questions are answerable from the knowledge base, explaining why retrieval introduces more noise than signal. The sole recovery comes from anti-leak prompt filtering (Phase 4), which mitigates answer-anchoring artifacts. Code: https://github.com/CultRAG/BLEnD-CultRAG.

Read Full Paperexternally

AI से पूछें

Bookmark

View Full Paper