> Overview Results

Overview

Our goal for the summer was to run 2-5 user tests each week. During the week of the PSLC Summer School, we ran nearly 20 tests in five days. This unique opportunity provided access to users from outside of Pittsburgh, who we feel are more representative of the future population of the PSLC. The data we collected from that week was so rich that it took another week to analyze and implement the changes it inspired.

When we transitioned to a flash prototype to prepare for the Summer School, the team split its tasks to make the most effective use of its members' skills. Three programmers worked to implement changes provoked by the results of user tests run by the remaining two members in the previous week. Each week, we tested whatever features were available, including the navigation between features. During the month following the PSLC Summer School, we were able to implement additional features to make the Data Shop more powerful, such as the ability to compare multiple learning curves at once. This was based on a reasonable comfort level with users' understanding of the basic features, based on their success in previous tests.

What follows is a collection of our specific user tasks, grouped by usability goal.
Download final task script (.doc)

Inter-Report Navigation

This first task was used to test navigation in paper prototypes early in the summer:

Examine the skill ëpolynomial factoring,í starting with a learning curve. If there are any spikes in the curve, find the problem which contributed to that spike that the most students saw and then provide the following:

1. the text of that problem
2. the three most common incorrect student responses of that problem

Later, we paired this task with comparing multiple curves in the flash prototype (see next goal: Comparing Learning Curves)

• Please select one of the skills that appear to be troubling. What might be causing the studentsí difficulty? What other information can you identify and/or do you need to further examine the issue?

Comparing Learning Curves

1. Please identify and jot down 2 skills on which the students seem to be improving. Of these 2, which skill are they mastering more quickly?
2. Are there any skills where the students appear to be having trouble?

Sample Selection

In part of the study, one of the classes was given diagrams along with the problems and another was given only text. In School 71, you want to look at how Class 1ís performance compared with that of Class 2. How would you set the Data Shop to examine the performance of only these two classes?

1. Please create a sample for Class 1 labeled ědiagramsî
2. Please create a sample for Class 2 labeled ěcontrolî

Error Analysis

1. What question had the highest number of incorrect responses?
2. Tutors provide feedback for expected incorrect student responses. Your advisor wants to make the tutor more helpful by increasing the amount of feedback for incorrect answers. Please find the most common unexpected incorrect response and give your advisor all of the information you can find about it.

Data Export

You want to study how students are using hints as they use the tutor. There is no standard report for this research question. With each hint on a particular step, the hints become more explicit. Extract all of the students who used hints on the problems. You will need the hint level information to perform your analysis.