Semantic and Linguistic Based Short Answer Scoring System

Abstract

In natural language processing (NLP), automatic short answer scoring is an essential educational application. It can relieve the burden of manual assessment while enhancing the reliability and consistency of evaluations. These systems have shown good accuracy with the advancement of text embedding libraries and neural network models. However, the ultimate goal is to embedding given text (student responses) into vectors with coherence and semantics, and providing feedback to students. This paper presents a novel approach to address these challenges using semantic and linguistic-based embedding techniques. Specifically, we utilize XLNet, a transformer model, to convert essays into vectors. These vectors are trained on Long Short-Term Memory (LSTM) networks to capture the connectivity between sentences and their underlying semantics. To evaluate our approach, we employ our dataset, which comprises approximately 2500 responses from 650 students. This dataset is domain-specific and tailored to our specific requirements. Our model demonstrates outstanding performance on the training and testing datasets, achieving an impressive average QWK (Quadratic Weighted Kappa) score of 0.76. Additionally, our approach showcases superior results in comparison to other existing models. We further assessed the robustness of our models by testing them with adversarial responses, and the outcomes were found to be satisfactory.