We landed on designing a conversational agent to facilitate an autonomous hands-on/test drive experience to provide customers with a better way to experience a car on their own terms. The hands-on experience involves any kind of in-person interaction with the vehicle.
A key insight that our research informed us of was that for both the customer and business, the in-store experience provides significant value and will continue to do so in the future. We hypothesized that focusing on a solution that takes advantage of this experience would differentiate CarMax in the evolving competitive automotive landscape and give them a unique advantage over their online-only counterparts.
Core to understanding how to design the best autonomous and informational test drive experience was first determining the interaction dynamic between the customer and agent.
We wanted to know if people preferred interacting with an agent that just answered their questions (reactive) or one that would also actively engage them with contextually relevant information and questions on the drive (proactive). We also wanted to know how much data people were comfortable sharing with the agent.
Building a multi-modal voice-based prototype necessitated that we think outside of the box in order to create a realistic conversational experience. This involved thinking about dimensions such as: technological feasibility, data usage, agent personality design, conversational design, interface design and more! We ended up deciding that the Wizard of Oz methodology was the best for our testing purposes
We started by defining what data the agent would need to be proactive, how that data would be captured, and mapped the actions that could be taken.
We created models for how it should interact with the participant to both answer questions and also proactively engage them about contextually relevant information.
We created a display to be used during the drive to help people quickly discover how to interact with the agent and provide feedback on the agent's current state.
We used terminal-based text to speech commands in order to simulate our assistant’s voice over a zoom call.
We had real CarMax customers (virtual) and other car buyers interact with a conversational agent on a test drive experience. Participants were allowed to ask the agent questions about the car's features, history, and the CarMax process. They were also engaged by the agent during the drive with contextually relevant information about the car based on their reason for buying and features of interest. For our in-person studies, we used a microphone and two cameras to capture real time input data, monitor the drivers, and simulate the assistant. For our remote studies, we used OBS to create a simulated test drive.
We started exploring this space by planning the design of our conversational agent and thinking of how we might use the time before the test drive of “kicking the tires” to help customers better prepare themselves for their test drive. In doing so, we landed on the idea of a guided tour around the car — one wherein our conversational agent would walk shoppers around the car to show off the features of the car and demonstrate their functionality.
We hypothesized that this experience of “kicking the tires” with our conversational agent would enable shoppers to learn more about the car and get information that would help make their test drive even more enjoyable.
We simulated a conversational agent guided tour of a car with information about both its history and features contextualized to be relevant to the participant's car buying needs. In our guided tour participants would learn about a high interest feature and then have the ability to ask any follow-up questions about the feature.
We used the same participants from the test drive study to see how this added experience could have improved their previous one and whether they would even use a guided tour while walking around the car. We used the same WoZ of method as during our test drive research to simulate the conversational agent.
With a specific interest in providing customers with an autonomous informational experience that helps them build confidence in their purchase decision, our primary goals were to understand….
The decision to test with shoppers’ real test drive cars required quick thinking and planning for our team. To actually run the test, we had to have all 5 team members working to pull it off. We started each test once we found out what car the shopper would be test driving — from there, we had to quickly update our research tool with data relevant to the car that was being driven. We’d then set-up the rig in the car and get the test started. While running the test, we had 3 team members running the experience. One member was controlling the tool that acted as the conversational agent, another was controlling the visual interface, and lastly another teammate researching facts about the car as they were needed throughout the test.
We tested our agent on 6 customer test drives. We started each test with a pre-survey to learn more about their interest in the car they were test driving and get at some of the context for their visit to the store. Then, after a brief onboarding, shoppers started with the “kicking the tires” experience and then, finally, the test drive. Each experience was capped off with a debrief interview.
Overall, we found the experience added value to customer’s in-store experiences and validated that this is something customers would actually use on their real test drives. We gained a lot of constructive feedback on how we could refine our experience and other considerations for the future. To top it all off we even “sold” 5 of the cars we tested with!
