Process: Operational Readiness Tests

We planned our first ORT as a series of separate tests of each of our newly modularized interfaces in isolation, reporting a simple, generic problem. This problem was that of a frayed wire on a webcam, which was chosen for the clarity of the report's content (a real user would not be presented with unrecognizable items or jargon) and the ubiquity of the discrepancy (frayed wires were responsible for the Apollo 13 explosion). The camera and bar code reader were not yet ready and were tested in the 2nd ORT.

First operational readiness test

For this first test, we decided to force the user to navigate entirely through the keypad, by disabling the stylus. This would prevent the user from constantly homing between stylus for navigation to fields and keypad for entering into fields and ensure that expert keypad use of the device was possible. The user adequately completed the task without stylus.

Although we had created a simple tutorial to help simulate the effect of training on the end users, and lower the incidence of those novice-style errors that would be eliminated or significantly reduced through NASA's mandatory training process, we found it had limited effectiveness. Platform-dependent issues, such as the use of modal function keys, were interfering with the user's understanding of our interface.

This problem of separating the effect of the form factor and the interface presented a difficult quandary. On one hand, the form factor is a part of the design. On the other, it is for us, beyond certain vague bounds of size and shape, very flexible, yet at the same time impossible for us to iterate. At this late stage of our process, our dominant focus had shifted between our dual foci, from that of the handheld form factor (#) to that of the interface itself, and we needed a way to differentiate the two and test the latter. We decided that in the next ORT, a small and unrelated task should be performed in advance, allowing the user to gain familiarity with the form factor, but not the task script.

Two interesting observations also emerged from the test. First, the user remarked that the distinction between capital and lowercase letters is not important Ð meaning that the necessary input options on the keypad drop by about a third, reducing the incidence of confusion over modal shift buttons. Second, while we created a fake WAD for realism, this merely added confusion with the task script. Coupled with observations at JSC, it appears that there is a psychological divide between WADs and PRs that must be navigated if both interfaces are introduced to the technician in one handheld.

second operational readiness test

Our second Operational Readiness Test was dedicated towards determining whether our multiple-interface system could be made to not only demonstrate the completion of a problem report through a handheld device including our newly-available camera and bar code functionality, but also whether or not the results of that report would be sufficiently useful to allow the next person in the lifecycle to comprehend it. For this reason, we arranged a staggered sequence of several separate tests in various locations at various times, in order to test the movement of the report from person to person along its entire lifecycle.

The first two users were two technicians working in the wind tunnels at NASA Ames. The senior of the two was assigned the role of Quality, and the more junior technician played the part of the ordinary Technician. The junior technician reported a problem on the Technician's interface that was captured by the database and then forwarded onwards to the senior technician, who elaborated it using the Quality interface. We intended for a non-colocated engineer to use the Engineer interface later in the week.

The test found that the system worked. The Technician's report moved through the database and was comprehensible to the Quality user such that the tasks of both could be completed. Oddly, the users relied on the stylus at all times, even to press on the keypad.

In order to complete the user testing on the Symbol interface, a class of seven Aircraft Maintenance Training School technicians and their two teachers, visiting the Vertical Motion Simulator at Ames, were led to the PROPHET lab that evening, as a focus group. Local wireless was strong enough to test the Technician interface on the Symbol device. The purpose of their visit was specifically to test the interaction of the bar code reader and the camera. Other more minor observations about the interface as a whole, already considered tested by the more valid users above, were also recorded, but given lesser prominence in our results, and can be found in the appendix.

The result of the testing was that the camera and bar code reader were operationally ready. The technicians were encouraged that the bar code reader could be used in the place of signatures, which were, like stamping at KSC, a delay in their maintenance process. They were also interested in the reading of bar codes on parts, but warned that, at least in their field of general aviation, there would be considerable resistance from manufacturers to bar coding their products.

A secondary result of the user testing was the identification of a usability problem relating to the placement of the screen relative to the camera lens of the Symbol device. The lens is along the top edge of the device. Normal digital cameras place the screen on the opposite side of the device from the lens, but the Symbol makes the relationship a ninety-degree angle, which means that the user cannot attain a viewing angle that is both comfortable and undistorted.




Even a screen resistant to contrast problems at awkward viewing angles would still be distorted, and so the removal of the camera lens to the back of the device, opposite the screen, is recommended. The bar code reader, on the other hand, is best left at the top.

After the Technician and Quality interface testing was exhaustively completed, testing on the Engineer interface was done as well, several days later. Mainly the purpose of this final minor portion of the ORT was to test that the latest iteration of Quickmode was both functional and comprehensible.