Towards Reliable LLM-driven Mobile Robot Task Planning and Execution
Dion 311
Dr. Jiawei Yuan
jyuan@umassd.edu
Advisor: Dr. Jiawei Yuan, Department of Computer and Information Science
Committee Members:
- Dr. Yuchou Chang, Department of Computer and Information Science
- Dr. Gokhan Kul, Department of Computer and Information Science
- Dr. Liudong Xing, Department of Electrical & Computer Engineering
Abstract
Given the recent advances in large language models (LLMs) and their remarkable capabilities of natural language understanding and text generation, LLMs have increasingly been integrated into robotic systems, enabling robots to understand high-level human instructions, reason about task objectives, and generate code for robot execution. However, enabling LLMs to reliably understand high-level human instructions and produce executable robot operations remains challenging. For example, LLMs can misinterpret human intentions, forget task objectives, hallucinate unavailable robot functions, generate syntactically invalid code, produce logically inconsistent action sequences, or make unsupported assumptions about environment observation.
This proposed research first enhances the reliability of LLMs by enabling them to generate valid and feasible task plans and robot operational code, identify errors, and recover from failures. First, this research developed GSCE, a structured prompt framework to enhance LLM reasoning and generate reliable robot operation code. Building on GSCE, this research further enhances reliability by developing a closed-loop framework that evaluates the robot behavior and provides feedback for correcting generated code in simulation before its deployment on a physical robot. To reduce the configuration effort and execution time associated with specialized robotic simulators, this research further developed an LLM-driven static text-based simulation framework without dynamically executing the code in a physical environment or simulator during corrective code refinement. Moreover, to achieve reliable task planning and execution on mobile robots that host on-board LLMs, this research designed Ro-SLM, a framework that leverages prior knowledge to teach on-board language models and enable them to perform reliable task planning and execution with performance approaching substantially larger models.
Despite this progress, challenges remain as the diversity and complexity of robotic tasks continue to increase. In particular, performance may degrade on previously unseen tasks, long-horizon operations, and tasks that require complex reasoning and decision-making. Therefore, to further enhance the reliability of LLM-driven mobile robots, this research proposes to address: 1) problem solving, in which on-board language models will select, reuse, and orchestrate skills to complete tasks, enabling the robot to solve problems by recombining its learned knowledge rather than
memorizing complete task solutions. This direction aims to enhance the reliability of LLM-driven mobile robots under complex, diverse, and previously unseen tasks; 2) decision making, the on-board language model will review past experiences, anticipate possible future consequences and failures for the skill that the robot will conduct, and correct the errors before executing the robot operations, which enables the robot to make reliable decisions for the given task before executing the robot operations. This direction aims to further improve the reliability of LLM-driven mobile robots for tasks that require long-horizon operations and complex decision-making.
For further information, please contact Dr. Jiawei Yuan at jyuan@umassd.edu