Embodied Understanding of Large Language Models using Calibration Enhancement

Abstract

In our research pursuit, we explore the inherent capacity of Large Language Models (LLMs) to develop an innate understanding of the physical realm—an essential prerequisite for empowering embodied agents to adeptly navigate real-world challenges. This paper introduces an extensive dataset encompassing diverse physical scenarios, establishing AuPPLE (Augmented Physical Priors via Learned Enhancement) as a robust benchmark. It serves as a comprehensive evaluative framework for assessing and amplifying the physical intuition of LLMs, including scenarios involving free fall and projectile motion. Within this benchmark, questions are framed in various formats, spanning MultiQA, binary classification, and continuous number prediction, thereby facilitating a comprehensive evaluation of LLMs' proficiency in comprehending physical dynamics. Moreover, we conduct a fine-tuning process on LLMs like Flan-T5-Large and DeBERTa, employing succinct physics-based prompts to instill a nuanced understanding of environmental physics. Our empirical findings underscore a notable improvement in the performance of LLMs fine-tuned on these physics-centric scenarios, particularly when confronted with questions rooted in the intricacies of the physical domain. This substantiates the effectiveness of our approach, indicating that strategic fine-tuning through physics-based prompts, in conjunction with external methodologies, significantly reinforces LLMs' intuitive grasp of the physical environment and enhances their efficacy in addressing tasks with a distinct physical dimension.