Researchers Use Large Language Models to Help Robots Navigate

Because their method uses purely language-based representations, they can use a large language model to efficiently generate a huge amount of synthetic training data.

The technique can also bridge the gap that can prevent an agent trained with a simulated environment from performing well in the real world. This gap often occurs because computer-generated images can appear quite different from real-world scenes due to elements like lighting or color. But language that describes a synthetic vs. a real image would be much harder to tell apart, Pan says.

While this approach couldn’t outperform vision-based techniques, when they tested it they found that it offered several advantages.

For an AI agent, this is easier said than done. Current approaches often use multiple, hand-crafted machine-learning models to tackle different parts of the task, which requires a great deal of human effort and expertise to build. These methods, which use visual representations to directly make navigation decisions, demand massive amounts of visual data for training that are often hard to come by.

‘By purely using language as the perceptual representation, ours is a more straightforward approach.’
—Bowen Pan

The model repeats these processes to generate a trajectory that guides the robot to its goal, one step at a time.

Rather than encoding visual features from images of a robot’s surroundings as visual representations, which is computationally intensive, their method creates text captions that describe the robot’s point of view. A large language model uses the captions to predict the actions a robot should take to fulfill a user’s language-based instructions.

“Maybe this means that language can capture some higher-level information than cannot be captured with pure vision features,” Pan says.

In addition, their method could be applied more easily to varied tasks and environments because it uses only one type of input. As long as data can be encoded as language, they can use the same model without making any modifications.

For instance, a caption might say, “To your 30° left is a door with a potted plant beside it, to your back is a small office with a desk and a computer,” etc. The model chooses whether the robot should move toward the door or the office.

Pan’s co-authors include his advisor, Aude Oliva, director of strategic industry engagement at the MIT Schwarzman College of Computing, MIT director of the MIT-IBM Watson AI Lab, and a senior research scientist in the Computer Science and Artificial Intelligence Laboratory (CSAIL); Philip Isola, an associate professor of EECS and a member of CSAIL; senior author Yoon Kim, an assistant professor of EECS and a member of CSAIL; and others at the MIT-IBM Watson AI Lab and Dartmouth College. The research will be presented at the Conference of the North American Chapter of the Association for Computational Linguistics.

Solving a vision problem with language

But such models take text-based inputs and can’t process visual data from a robot’s camera. So, the team needed to find a way to use language instead.

Although this approach doesn’t outperform techniques that use visual features, it performs well in situations that lack enough visual data for training. The researchers found that combining their language-based inputs with visual signals leads to better navigation performance.

Published June 12, 2024, in MIT News.

منبع: https://www.qualitydigest.com/inside/innovation-article/researchers-use-large-language-models-help-robots-navigate-070324.html

This research is funded, in part, by the MIT-IBM Watson AI Lab.

Since large language models are the most powerful machine-learning models available, the researchers sought to incorporate them into the complex task known as vision-and-language navigation.

To streamline the process, the researchers designed templates so observation information is presented to the model in a standard form—as a series of choices the robot can make based on its surroundings.

“If the agent fails to reach its goal, we can more easily determine where it failed and why it failed. Maybe the history information is not clear enough, or the observation ignores some important details,” Pan says.

However, the researchers were surprised to see that combining language-based representations with vision-based methods improves an agent’s ability to navigate.

This is one area the researchers want to continue exploring. They also want to develop a navigation-oriented captioner that could boost the method’s performance. In addition, they want to probe the ability of large language models to exhibit spatial awareness and see how this could aid language-based navigation.

“By purely using language as the perceptual representation, ours is a more straightforward approach. Since all the inputs can be encoded as language, we can generate a human-understandable trajectory,” says Bowen Pan, an electrical engineering and computer science (EECS) graduate student and lead author of a paper on this approach.

To overcome these challenges, researchers from MIT and the MIT-IBM Watson AI Lab devised a navigation method that converts visual representations into pieces of language that are then fed into one large language model that achieves all parts of the multistep navigation task.

The large language model outputs a caption of the scene the robot should see after completing that step. This is used to update the trajectory history so the robot can keep track of where it has been.

But one disadvantage is that their method naturally loses some information that would be captured by vision-based models, such as depth information.

First, because text requires fewer computational resources to synthesize than complex image data, their method can be used to rapidly generate synthetic training data. In one test, they generated 10,000 synthetic trajectories based on 10 real-world, visual trajectories.

The technique uses a simple captioning model to obtain text descriptions of a robot’s visual observations. These captions are combined with language-based instructions and fed into a large language model, which decides what navigation step the robot should take next.

“One of the biggest challenges was figuring out how to encode this kind of information into language in a proper way to make the agent understand what the task is and how they should respond,” Pan says.

Advantages of language

Someday, you may want your home robot to carry a load of dirty clothes downstairs and deposit them in the washing machine in the far-left corner of the basement. The robot will need to combine your instructions with its visual observations to determine what it should do to complete this task.

Also, the representations their model uses are easier for a human to understand because they are written in natural language.