Understanding and using the terminology was crucial for the team when communicating with the various robotic teams during the usability studies.  For example, the following lists and defines some domain-specific terminology that was utilized during the research and development process.

Activity: An action that the robot takes.  Example:  ‘turn left wheel.’

Sequence: Group of activities.  Example:  ‘drive forward’ is a sequence made up of low level actions such as ‘turn left wheel.’

Goal: Higher level activity of the robot, made up of low level actions.  Example: ‘inspect upper region of worksite’ is a goal made up of many sequences of activities.

Mixed Initiative: Type of robotic operation in which the robot is controlled either by the operator or by onboard artificial intelligence software.

Waypoint: Position on the rover’s path.  Waypoint is used to describe where the robot is traversing to.

Not only did the literature review help establish a domain-specific vocabulary, it also brought the team up to date on current robotic platforms.  These current research projects included robotic search and rescue teams, past and current NASA missions, and robotic systems deployed by the United States Army and Navy.  The team also became familiar with current robotic software systems.  The team discussed planning software used by the Artificial Intelligence Lab and JPL such as ASPEN and CASPER.  Different control techniques were also reviewed, such as Internet-based operation for future Mars missions.  Outside of the NASA domain, the team looked at the Army and Navy’s Detection Assessment and Response System (MDRS) that controls autonomous guard robots.

The third focus of the literature review was for the team to become familiar with current robotic research.  The research discussions centered around publications on human-robotic coordination, mission planning for multiple robots, utilizing current planning and scheduling software for autonomous spacecraft, and collaboration amongst scientists. 

An entire separate paper could be written summarizing the research findings from our literature review. Here, however, is a brief summary of the points most salient to our specific challenge:

Need for Better Support for In-Situ Re-Tasking
Our literature review supported our client’s request for us to address the challenge of in-situ re-tasking in remote robotic planning. Sequencing a remote robot is an extremely time-consuming and therefore costly activity for current mission operation teams. Without a flexible commanding method, or a more intelligent on-board autonomous system, valuable unforeseen science opportunities are lost. Many research projects are based on this concern for allowing mission operation teams to spend more time gathering useful science data and less time dealing with low-level activity planning.

A typical approach to this problem found in the literature involved increased and improved on-board-autonomy levels. For example, various current robotic research projects are looking at allowing operators to define and rank high-level science goals, and then allowing the robot, or an off-board AI system, to formulate the detailed activity-level plan to achieve these goals. Then, when the robot is carrying out the plan and communication with the robot is not possible, the robot would make real-time decisions to adjust its plan if and when opportunistic science elements were encountered. One example of this type of on-board system with dynamic sequence generation is NASA’s CASPER (Continuous Activity Scheduling, Planning, Execution, and Replanning).

While this increased-autonomy approach is certainly promising, it seems that it is many years away from being fully flight-ready. Based on the current state of robotic research, a ‘human in the loop’ will be required for dynamic re-tasking for at least the next few Rover missions.

Human-Robot Interaction Taxonomy

Before tackling the problem of human-in-the-loop dynamic re-tasking, we needed to understand the important issues involved in ‘tasking’ in the first place. You can’t re-task if you haven’t already ‘tasked’! Here are some important take-away suggestions and guidelines found in our literature review:

• The interface should provide the necessary information for the human to determine that an intervention – such as the human taking over a robot currently in auonomous mode - is needed.
• The interface should provide the necessary information for the human to gain a solid understanding of situation awareness: where the robot is and what it’s doing.
• The interface should fuse information whenever possible in order to decrease the cognitive load of the human operator. For example, if there is one area of the interface that displays the environment map and another area of the interface that displays the robot’s position within the environment, the user will be forced to fuse those pieces of information together in their heads.
• All interactions with the interface should be efficient and effective. Time is a critical resource when planning and re-planning remote robots.
• The interface design should be extensible enough to allow for additional robotic sensors and tools.
• The interface should provide the user with information about where the robot has already been.
• A map, or some sort of representation of the robot’s environment, should be given a prominent size and position in the interface.
• A key consideration is that even if the user is familiar with the GUI and it’s widgets, they still need to understand exactly how these input mechanisms will affect the actions of the robot.
• The operator should be able to rapidly identify the robot’s current position, state, and health.

All of these guidelines helped us focus our first-hand research and formulate our interaction and interface design ideas.

• “About Face: Rover Engineers Change the Rules for Driving”, NASA Web Site, July 16, 2004. Author unknown.
• "Activities of the NASA Exploration Team Human-Robotics Working Group", AIAA Space 2003 Conference. Author Unknown.
• Backes et al., “Internet-based Operations for the Mars Polar Lander Mission”, Proceedings of IEEE, April 2000.
• Baker, M., et al., “Improved Interfaces for Human-Robot Interaction in Urban Search and Rescue”, Proceedings of the IEEE Conference on Systems, Man and Cybernetics, October 2004.
• Balaram, J., et al., “Enhanced Mars Rover Navigation Techniques “, Proceedings of the IEEE International Conference on Robotics and Automation, pages 926-931, 2000.
• Chien, S., et al., "ASPEN - Automating Space Mission Operations using Automated Planning and Scheduling," SpaceOps 2000, Toulouse, France, June 2000.
• Chien, S., et al., “Using Iterative Repair to Increase the Responsiveness of Planning and Scheduling for Autonomous Spacecraft”, Jet Propulsion Laboratory. Publication unknown.
• Clement, B.J. and Barrett, A., "Continual Coordination through Shared Activities," Proceedings of AAMAS 2003, ACM Press 2003.
• Cohen, P., Firoiu, L., “Abstracting from Robot Sensor Data using Hidden Markov Models”, University of Massachusetts at Amherst, February 1, 1999. Publication unknown.
• Croft, E.A., Kuli?, D., “Estimating Intent for Human-Robot Interaction”, Proceedings of the International Conference on Advanced Robotics, Coimbra, Portugal, June 29 - July 3, 2003.
• Drury, J.L., and Yanco, H.A., “Classifying Human-Robot Interaction: An Updated Taxonomy”, Proceedings of the IEEE Conference on Systems, Man and Cybernetics, October 2004.
• Drury, J.L., Yanco, H.A., and Scholtz, J., “Awareness in Human-Robot Interactions”, Proceedings of the IEEE Conference on Systems, Man and Cybernetics, Washington, DC, October 2003.
• Drury, J.L., Yanco, H.A., and Scholtz, J., “Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition”,  Journal of Human-Computer Interaction, Volume 19, Numbers 1 and 2, pp. 117 - 149.
• Frenger, “Robot Control Techniques”, ACM SIGPLAN Forth Report, 1997.
• Inderieden, R.S., et al., "Overview of the Mobile Detection Assessment and Response System," DND/CSA Robotics and KBS Workshop, St. Hubert, Quebec, October 1995.
• Kaelbling, L., Shatkay, H., “Learning Geometrically-Constrained Hidden Markov Models for Robot Navigation: Bridging the Topological-Geometrical Gap”, Journal of Artificial Intelligence Research, March 2002.
• Laird, R.T., et al., “MDARS Multiple Robot Host Architecture”, Naval Command Control and Ocean Surveillance Center. Publication unknown.
• McCurdy, M., Tollinger, I., Tollinger, P., and Vera, A.H., "Collaborative Knowledge Management Supporting Mars Mission Scientists," Proceedings of CSCW 2004, ACM Press 2004.
• Nesnas et al., “Toward Developing Reusable Software Components for Robotic Applications”, JPL. Publication unknown.
• Steinfeld, A., “Interface Lessons for Fully and Semi-Autonomous Mobile Robots”, Proceedings of the 2004 IEEE international Conference on Robotics & Automation, April 2004.