Rendezvous

Effective collaboration requires two-way communication. To enable the widespread adoption of collaborative robots, they need to be able to communicate effectively with non-expert human users. Natural language is an expressive and effective medium for communicating with humans, since it does not require additional training for the user. Natural language is already used in numerous robotics applications, including some ongoing projects within Prof. Lawson Wong’s group. However, most existing work focuses on one-way communication: from human to robot. In contrast, work on the reverse direction of robot-to-human communication is sparse. This research project involves an application that requires two-way communication via natural language.

This project aims to build a system that allows an agent/robot to instruct a human user about how to navigate from their current location to an intended destination. This involves at least the following tasks: (a) Localize the human within a known environment map. This requires understanding a description given by the user in natural language, and if the location is ambiguous, requires follow-up questions from the robot to disambiguate the user’s location. (b) Plan a path for the human to follow. It is unclear whether the shortest path is necessarily optimal in this case. Since the path will need to be conveyed to the user, it may be more desirable to pick a path that is easily described and not confusing, even if it is slightly longer. (c) Describe the path to the human using natural language. This is another part of the project that requires natural language generation. Similar to the previous point, this requires reasoning about whether the instructions are easy to follow. (d) Track the human as they are following the path. If the navigation task is complicated or involves many steps, it may be necessary to continuously localize the user (first part of this project) as they are navigating, to ensure that they are following the instructions correctly. Long paths may also require incremental instructions (e.g., giving the second direction after the first is completed).