Testing

Paper Prototyping

Our initial designs were tested using paper prototypes. These low-fidelity prototypes, consisting mainly of paper cutouts, sticky notes, and pipe-cleaners, allowed us to make many changes to the design early on. We performed needs validation with two geologists from the robotic reconnaissance field test by walking through the paper prototypes and asking questions to assess whether the features we had incorporated were addressing current science needs. We also performed “think-alouds” with a geologist and three engineers from the robotic reconnaissance field test, as well as four members of the NASA Ames HCI group. During think-aloud testing we created a testing script and defined scenarios and tasks for the users to complete. Users were asked to explore specified areas on the map with the instruments available to them, voicing their thought processes as they performed these tasks. These tests allowed us to assess the usability of the system with unfamiliar users, and also to narrow in and iterate on the ideal interaction for each feature.

Digital Prototyping

Digital prototyping allowed us to evaluate more complex functionality and user interaction that was not possible to do with the low-fidelity prototypes. Because development occurred in parallel to user testing, in many cases we were able to test what would eventually become our final prototype with various stages of functionality. In some cases, we also created mid-fidelity Adobe Flash prototypes of individual features--such as viewing notes in the task list, and creating a microscopic imager sequence--in order to quickly test their functionality before fully implementing them. We performed think-aloud testing with three geologists from San Jose State University, giving them the goal of locating scientific points of interest and traverse hazards, much like the geologists' goal at the robotic reconnaissance field test. We also performed think-alouds with four NASA Ames summer interns, giving them step-by-step instructions in order to test specific aspects of usability. These tests helped us refine the tool's usability, isolate unexpected bugs in the program, as well as refine the ideal interaction for each feature.

Operational Readiness Tests (ORTs)

We performed three ORTs in addition to the final landing day scenario. The ORTs are used primarily to assess whether or not our planning tool is successfully capturing the science team’s planning intent and conveying it to the flight team. We offered potential science interests to the science team, and restricting constraint information to the flight team. This forces the flight team to alter the plan in some way and allows our team to evaluate whether or not the science intent is being accurately understood and executed.

During the ORTs, we simulated a complete robotic reconnaissance field test. The science team was instructed to explore the map in order to assess interesting geological areas within the region, and also to evaluate the traversability of the terrain for a future crewed mission. The flight team was given operating constraints that would force them to alter the science team’s plan in some way before relaying it to the rover team. No communication between the science and flight teams was allowed outside of the planning tool. The ORTs allowed us to assess whether our tool was successfully capturing the science team’s planning intent within the planning tool itself. If the flight team changed the plan but was still able to capture the same data products requested by the science team, this was deemed successful. We used screen-capture and voice recording to document the participants' interactions with the tool, and followed-up each ORT with a qualitative assessment and debrief with all participants.