Publications
Preprints
OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!
Jingdi Lei*, Varun Gumma*, Rishabh Bhardwaj*, Seok Min Lim, Chuan Li, Amir Zadeh, Soujanya Poria
Preprint (2025)
The Role of Synthetic Data in Multilingual, Multi-Cultural AI Systems: Lessons from Indic Languages
Pranjal A. Chitale, Varun Gumma, Sanchit Ahuja, Prashant Kodali, Manan Uppadhyay, Deepthi Sudharsan, Sunayana Sitaram
Preprint (2025)
HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settings
Varun Gumma, Anandhita Raghunath, Mohit Jain†, Sunayana Sitaramâ€
Preprint (2024)
Published Papers
Towards Inducing Long-Context Abilities in Multilingual Neural Machine Translation Models
Varun Gumma*, Pranjal Chitale*, Kalika Bali
Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios
Millicent Ochieng, Varun Gumma, Sunayana Sitaram, Jindong Wang, Vishrav Chaudhary, Keshet Ronen, Kalika Bali, Jacki O'Neill
Contamination Report for Multilingual Benchmarks
Sanchit Ahuja*, Varun Gumma*, Sunyana Sitaram
PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data
Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, Sunayana Sitaram
🥇 Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology
Rishav Hada, Safiya Husain, Varun Gumma, Harshita Diddee, Aditya Yadavalli, Agrima Seth, Nidhi Kulkarni, Ujwal Gadiraju, Aditya Vashistha, Vivek Seshadri, Kalika Bali
METAL: Towards Multilingual Meta-Evaluation
Rishav Hada*, Varun Gumma*, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, SunayanaSitaram
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram
MAFIA: Multi-Adapter Fused Inclusive Language Models
Prachi Jain*, Ashutosh Sathe*, Varun Gumma, Kabir Ahuja, Sunayana Sitaram
MunTTS: A Text-to-Speech System for Mundari
Varun Gumma, Rishav Hada, Aditya Yadavalli, Pamir Gogoi, Ishani Mondal, Vivek Seshadri, Kalika Bali
IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
Jay Gala*, Pranjal A Chitale*, A K Raghavan, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar M, Janki Atul Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M Khapra, Raj Dabre, Anoop Kunchukuttan
An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models
Varun Gumma, Raj Dabre, Pratyush Kumar
PAMMELA: Policy Administration Methodology using Machine Learning
Varun Gumma, Barsha Mitra, Soumyadeep Dey, Pratik Shashikantbhai Patel*, Sourabh Suman*, Saptarshi Das, Jaideep Vaidya
