By

Tanya Malhotra

July 26, 2024

In the domain of sequential decision-making, especially in robotics, agents often deal with continuous action spaces and high-dimensional observations. These difficulties result from making decisions across a broad range of potential actions like complex, continuous action spaces and evaluating enormous volumes of data. Advanced procedures are needed to process and act upon the information in these scenarios in an efficient and effective manner.

In recent research, a team of researchers from the University of Maryland, College Park, and Microsoft Research has presented a new viewpoint that formulates the problem of sequence compression in terms of creating temporal action abstractions. Large language models’ (LLMs) training pipelines are the source of inspiration for this method in the field of natural language processing (NLP). Tokenizing input is a crucial part of LLM training, and it’s commonly accomplished using byte pair encoding (BPE). This research suggests adapting BPE, which is commonly utilized in NLP, to the task of learning variable timespan abilities in continuous control domains.

Primitive Sequence Encoding (PRISE) is a new approach which has been introduced by the research to put this theory into practice. PRISE produces efficient action abstractions by fusing BPE and continuous action quantization. In order to facilitate processing and analysis, continuous activities are quantized by converting them into discrete codes. These discrete code sequences are then compressed using the BPE sequence compression technique to reveal significant and recurrent action primitives.

Empirical studies use robotic manipulation tasks to show the effectiveness of PRISE. The study has demonstrated that the high-level skills identified improve behavior cloning’s (BC) performance on downstream tasks through the use of PRISE on a series of multitask robotic manipulation demonstrations. Compact and meaningful action primitives produced by PRISE are useful for Behaviour Cloning, an approach where agents learn from expert examples.

The team has summarized their primary contributions as follows.

1. Primitive Sequence Encoding (PRISE), a unique method for learning multitask temporal action abstractions using NLP approaches, is the main contribution of this work.
2. To simplify the action representation, PRISE converts the continuous action space of the agent into discrete codes. These distinct action codes are arranged in a sequence based on pretraining trajectories. These action sequences are used by PRISE to extract skills with varied timesteps.
3. PRISE considerably improves learning efficiency over strong baselines such as ACT by learning policies over the learned skills and decoding them into simple action sequences during downstream tasks.
4. Research involves in-depth research to comprehend how different parameters affect PRISE’s performance, demonstrating the vital function BPE plays in the project’s success.

In conclusion, temporal action abstractions present a potent means of improving sequential decision-making when seen as a sequence compression problem. Through the effective integration of NLP approaches, particularly BPE, into the continuous control domain, PRISE is able to learn and encode high-level skills. These abilities show the promise of interdisciplinary approaches in increasing robotics and artificial intelligence, in addition to enhancing the effectiveness of techniques such as behavior cloning.

Tanya Malhotra

Tanya Malhotra is a final year undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.
She is a Data Science enthusiast with good analytical and critical thinking, along with an ardent interest in acquiring new skills, leading groups, and managing work in an organized manner.