Smart MCQ solver: transformer ensemble with LoRA fine-tuning
Parameter-efficient fine-tuning (LoRA) of DeBERTa-v3 and RoBERTa for 5-option multiple-choice question ranking, achieving 0.74242 MAP@3 on 500 unseen questions.
- Primary Metric
- 0.74242 MAP@3
- Model Architectures
- DeBERTa-v3-base · RoBERTa-base (70:30 Ensemble)
- Tuning Method
- LoRA (r=8, alpha=16, dropout=0.1)
- Evaluation Corpus
- 2,000 Questions (500 Unseen Test Set)
Problem Formulation
Multiple-choice question answering requires ranking five plausible options for a given prompt. The competition metric was Mean Average Precision at 3 (MAP@3), which heavily rewards placing the correct option in the top rank while giving partial credit for second or third-place placements.
What I Built
Rather than full fine-tuning of all transformer weights, I applied Low-Rank Adaptation (LoRA) to adapt pretrained language models efficiently:
- microsoft/deberta-v3-base: LoRA rank r=8, alpha=16, dropout 0.1, targeted at the query_proj and value_proj attention projections.
- roberta-base: LoRA rank r=8, alpha=16, dropout 0.1, targeted at query and value matrices.
- Ensemble blending: Combined predicted probability distributions using a 70:30 weighted blend (DeBERTa-v3 : RoBERTa).
Results and verification
The 70:30 LoRA ensemble generalized effectively on unseen test data, scoring 0.74242 MAP@3 on 500 test questions on Kaggle. Weights & Biases experiment tracking logs and safetensor adapter weights are preserved as verifiable artefacts.
Stack
PyTorch · Hugging Face Transformers · PEFT (LoRA) · Weights & Biases · Python