A Benchmark for Accessing Cultural Bias in LLMs and VLMs Through Regional Sports Knowledge (Indian Institute of Technology, Patna)

This project aim to address whether AI models (LLMs and VLMs) are capable to answer local games related question that are deeply rooted in specific cultures and communities but are not widely recognized globally.

This project aim to address whether AI models (LLMs and VLMs) truly understand local games that are deeply rooted in specific cultures and communities but are not widely recognized globally. To explore it in depth, we produce a specialized QA dataset called CultSportQA. This dataset focuses on 30 culturally significant national sports—15 from India and 15 from France—that are often overlooked on the global stage. The dataset is available in three languages: English, Hindi, and French, and contains over 40,000 multiple-choice questions (MCQs). The proposed dataset includes different types of questions, such as history, rules, scenarios, and images, that require more than just basic facts understanding. The questions are culturally specific, making them harder to answer without a deep understanding of each sport. Since it covers sports from India and France in English, Hindi, and French, it also adds multilingual and cross-cultural challenges, making it a good test for AI systems in understanding regional sports. We plan to evaluate popular LLMs and VLMs using few-shot learning and chain-of-thought prompting on our proposed CultSportQA benchmark.