Consistency Regularization for Extractive Question Answering: A Reproduction and Empirical Study with ELECTRA

Abstract

This study reproduces and adapts the idea of consistency-based semi-supervised learning, in the spirit of Unsupervised Data Augmentation (UDA), to a question answering (QA) setup using ELECTRA-small on the SQuAD v1.1 dataset. The original UDA method showed that enforcing prediction consistency between original and augmented inputs can improve robustness in text classification. A lightweight consistency loss is implemented at the level of start and end logits for extractive QA and evaluated against a strong fine-tuned ELECTRA baseline. Building on the Transformer architecture and drawing from related semi-supervised approaches such as Virtual Adversarial Training, Mean Teacher, and MixMatch, the study examines whether consistency regularization can improve extractive QA performance. To remain within realistic computational limits, a hyperparameter sweep over the consistency weight λ is conducted on a subset of the data, followed by training a single full model using the best λ. Overall, the consistency-regularized model slightly underperforms the full baseline in EM and F1 but remains competitive despite using fewer training epochs, while exhibiting broadly similar behavior across long-answer, short-question, and negation-focused slices. The discussion considers why the UDA-style signal appears less pronounced in the clean, in-domain SQuAD setting and outlines how a more faithful reproduction, such as stronger augmentations or out-of-domain evaluation, could produce clearer gains, particularly when combined with more recent pre-trained language models.