Prototype P1 Testing
Test Protocol
For P1 testing, we conducted a week-long diary study with three groups (total of 15 participants) that used TeamMail as their primary email client. Each day we sent participants a daily task with a link to a web survey. They completed the same survey each day, evaluating quantitative data for how many tasks they sent or received, what features they used, the ease of use of the application, and general daily feedback. At the end of the week, we conducted a one hour focus group with each group, soliciting their feedback on the social aspects of sending and receiving tasks in a team environment.
Test Goals
Our main test goals revolved around learning how users in a team setting would use TeamMail as a collaborative tool:
- Will users send each other tasks?
- Will they create tasks for themselves and if so, when and why?
- Do they find the interface easy to use?
Key Findings
We categorized the observations from the diary study and both focus groups to identify several issues that validated our design or highlighted areas for iterative improvement. From over 145 different categories, the top categories are listed below:
- Tasks in email
- Completing tasks
- Validation
- Days Left Indicator
- Project Label
- Identifying tasks
- Receiving tasks
- Reminders
- Sorting
- Notification
- Posting
- Sending tasks
- Ease of use
Tasks in email
The majority of users created at least one task every day. We observed that tasks were used primarily as personal reminders, specifically when converting emails to tasks for personal To-Do's.
“I just used it for a personal task. My wife sent me the phone number for the insurance company and I had to call. So I just converted it to a task. Sort of a reminder. It already had the information in it to complete the task.” (FG2, 15:00)
One user discussed how email is a communication medium and tasks are work objects. In our early CI’s we observed how users would distill out of their inboxes, messages, etc. the critical information surrounding a task and place it in a separate visible space.
“I really liked how you can send tasks and receive them in the context of email, but I think of them as 2 separate things. I really want to have the tasks in a separate place. Because they are different things, it’s a way of tracking progress versus a means of communication.” (FG2, 9:39)
Some users questioned the definition of a task, and what merited a task designation.
“Sometimes it's hard to know what level of task to put into TeamMail. Bugs to fix are too low-level, and sometimes it seems silly to add longer term non-work tasks/reminders into an email client.”(Diary study, U23, 7/7/04
Completing Tasks
From our contextual inquiries we knew that users relished the moment of task completion and enjoyed the tactile nature of “crossing things off their list.” During our five-day study, approximately half (48%) of tasks were designated as completed. We observed that completing tasks was perceived as enjoyable to most users.
“It’s nice to have something where you feel like you completed something, versus erasing it.” (FG2, 15:10)
Users varied in how they managed their completed tasks. Some deleted them, some wanted to see what they had done, some converted them back to emails, and some wanted to file them in folders.
“I did not complete tasks, I un-tasked them and moved them to folders (my previous action for completing a task was moving it to a folder)” (Diary study, U40, 7/5/04)
“I never keep completed tasks in my inbox, but when looking through my archive folders, they make for nice indicators that draw my attention” (Diary study, U40, 7/5/04)
Users also appreciated the archival qualities of email with a task.
“One thing I thought was really useful was having a record of things.” (FG2, 16:15)
Validation
One of the key areas of product usability is creating an engaging experience for the user. Simply put, making something fun increases desirability.
“At first I did not know if I would like the yellow stars... I like 'em! They add a little fun to the program. It is nice to see a long row of yellow stars.” (Diary study, U44, 7/6/04)
Days Left Indicator
The days left indicator was the primary visual flag indicating that an object was a task. Users did not rely on the task icon in the far left column. Often, users did not even know that there was a distinguishing icon for tasks versus emails.
"I did use the time left indicator to see what tasks I had left to do. I noticed that it makes me want to keep things at 0 tasks... making it a little exciting to complete everything." (U44 diary study, 7-6-2004)
“What was in the left column?” (FG1, 16:00)
In viewing the Days Left indicator for a task due today, users did not like the perception that they were “already late” on a task they just received or created for themselves.
“I don’t like having zero days left when I have something due today. At all.” (FG2, 32:20)
“...having that thing sitting there, like “zero days! Zero days!” and it seems like I just made it 5 minutes ago.” (FG2, 33:25)
Some users wanted even stronger notification for overdue tasks.
“the negative time left doesn't seem to warn me enough that something needs to be done to that specific task” (Diary study, U24, 7/12/04)
“…. It was nice that they turned red but I would get to two days overdue and it would not flash at me or hit me and say “You didn’t do your tasks”. (FG1, 3:25)
Project Label
From the user survey, 24% of created tasks were given a descriptive project label. However, for those users who added the label, they commented that the project label was hard to see and that for sent tasks the project label resulted in naming convention confusion.
“Doesn't pop out at me” (Diary study, U31, 7/8/04)
“Project sorting needs to have a unified naming convention. I'm getting projects from people that are the same as mine, but spelled differently, so they sort differently. sometimes, they just use a different name, and so the project sort hides information that I'd need.” (Diary study, U42, 7/8/04)
Identifying Tasks
It was difficult for some users to distinguish between tasks and regular email based solely on the task and email icons. Users mentioned that they primarily used the Days Left indicator to identify tasks.
“I NEVER notice the difference between the mail icon and clipboard. The only indication of a task to me is the days left... even THAT i don't always notice.” (Diary study, U20, 7/8/04)
“I was looking for the dots.” (FG1, 16:34)
Receiving Tasks
All of our user groups had flat social structures. Many used tasks within their teams without hesitation, but offered comments that they would feel different in a hierarchical team.
“In our group, since the hierarchy was kind of flat, it wasn’t really a problem to send and receive tasks but I’d imagine that it would be different in a hierarchical organization. ”(FG2, 1:00)
Reminders
We observed that users need structure when they are overwhelmed.
“I find myself wanting to set event reminders along with tasks, even though they don't have a due date, exactly. If I'm really busy, I'll put everything in and use it to decide what I need to do. If I'm less busy, I usually only put in items that don't need my attention for a while.” (Diary study, U23, 7/8/04)
Users also requested increased interactivity to get their attention for overdue items.
“I like Outlook popup reminders. Pops up when you say you want it to, and you can say remind me in 5 min or I’m not going to deal with this for another day. But when it reminds you it’s in your face.” (FG2, 35:13)
Sorting
Many users continued to sort their email by the "received date," the same way they did prior to using TeamMail. However, some users found that sorting by the number of days left was an effective tool for organizing what needs to be done. Users also mentioned that they liked how they could easily discriminate tasks from email by using the sorting mechanism, making it clear what items they need to work on.
“(I sorted on tasks) All the time” (FG2 – 9:02)
“I like the days left view so much that I keep mail sorted by days left, not by received date. This makes it a little harder to locate new mail when it arrives” (Diary Study, U42 07-08-2004)
The utility of the sort mechanism was clear for many of the people who used the application but others found the sorting to be confusing. The problem that arose when sorting by the “Days Left” is that new emails that arrive might fall below the fold, or not naturally be in the position the user would be expecting them to be. Users might not immediately notice when new email arrived, causing potential problems with time-sensitive messaging.
“I still feel a bit unsettled by the sorting, trying to figure out the best way to have a the top, overdue tasks, then active, then completed, then my unread or most recent emails. I eventually find a sort that works, then I do send and receive and I have to re-do the sort order because the new bold email is in the middle - still not used to seeing it there.” (Diary Study U41, 07-05-2004)
Notification
There was substantial interest in the ability to have greater communication amongst teammates in the context of sharing tasks. Specifically, the ability to notify the originator of a task that the task had been completed and to allow greater visibility into the progress of delegated tasks.
“When someone sends me a task there is no way to respond back to them with the "completed" field filled so that they know I have completed the task” (Diary Study, U21 07-06-2004)
Posting
Editing a task was cited as a potentially beneficial feature to support. By supporting some method of annotating or posting to an existing task, it would allow tasks to be more than static documents, and would allow discussions and negotiations to take place around a task. This was communicated in diary studies as well as both focus groups.
“I'd like to markup existing tasks - both to elaborate on static emails coming to me, and to support real time work, and to support permanent but dynamic tasks (e.g., always have homework for a class, but what is it?)” (Diary Study U40 07-05-2004)
“So there’s like one shared task. As people do different things it would be nice to see the collective update on that, all in that one area.” (FG2 10:30)
Sending Tasks
We were interested in seeing how and when individual team members sent tasks to each other. 64% of users sent one task a day, and 33% sent two tasks a day. (Note: The majority of the groups who participated in our study had flat hierarchical structures)
One group sent several tasks to each other over the course of the week and found this to be a little awkward at first but commented that it soon became quite natural. One user expressed that she actually liked receiving task as it made it easier for her to keep track of the item. For the other group who did not use tasks with any regularity, they expressed concerns that in order to use tasks within a team, the social rules would have to naturally develop whether if it was acceptable to send a task to someone or not.
“I thought that in our lack of hierarchy it would be weird to send around, but actually it got less and less weird. At the end I was thinking it was pretty cool... It was an easy way of sorting and… I actually appreciated it when someone sent tasks.” (FG2 4:50)
Ease of Use
One of the main goals of P1 testing was to evaluate the ease of use of TeamMail. We were interested in users' overall experience with TeamMail and how natural sending tasks to others felt. Participants were asked in their daily journals to indicate on a seven-point Likert scale how they felt about the both the ease of use and their comfort level in sending tasks to others.
Note: Our user population consisted of Carnegie Mellon University undergraduate and graduate students who are extremely computer literate and to which email is an commonly used application.
