Product Designer
Usability Test.png

SAM Usability Study

Seattle Art Museum Website

 

Overview

Usability testing is a valuable practice for observing how users interact with a system. It involves systematically observing user interactions under a controlled set of conditions. 

For this project, we conducted a usability study of the Seattle Art Museum (SAM) website. Our goal was to assess the overall ease of use of seattleartmuseum.org. By testing key tasks with our target user group, we were able to identify major usability issues and provide prioritized recommendations for improving the overall user experience of the site.

 

The Challenge

Our objective for this study was to asses the overall usability of seattleartmuseum.org. Specifically, we wanted to determine how easily users completed key tasks by defining success metrics and identifying any obstacles that prevented task completion. We aimed to gather a combination of observational and subjective data in order to understand the overall ease of use and effectiveness of the site.

Since we had limited data on the site’s primary user group, we started with the assumption that our target users were art enthusiasts in the Seattle area. 

 

My Role

I worked alongside three classmates on this research project. I was in charge of participant recruitment and coordination for our testing sessions. As a team, we all shared in the planning, execution, and data analysis stages, producing a research report and presentation as our final deliverable.

Tools: Morae, Tableau, Exce

Timeline: January 2017 – March 2017

Type: Class Project
 

 

Exhibits.png
 

The Approach

 
 

Usability Inspection

We started by conducting a usability inspection of the website using 2 techniques: a heuristic evaluation and a cognitive walkthrough. These are both quick and low-cost diagnostic tools to help us identify and prioritize usability concerns before involving actual users.

For the heuristic evaluation, we used Nielsen’s 10 Usability Heuristics to guide our analysis:

 
SAM heuristics.png
 
 

Cognitive Walkthrough

We also conducted a cognitive walkthrough to understand key tasks, contexts of use, and both ideal and error pathways for completing these tasks. The 5 key task pathways are illustrated below:

 
 
 

Usability Testing Kit

In order to ensure the consistency of each testing session, we created a usability testing kit that would serve as a guide for both moderator and observers. We crafted 3 different scenarios, encompassing a total of 5 tasks. We identified the objective metrics we wanted to measure, as well as the method of collection for each metric. We also assembled a pre- and post-study questionnaire, as well as post-task questionnaires in order to gather subjective data from our participants. 

 

Quantitative Metrics

The following spreadsheet served as a reference for our session note-takers:

 

Qualitative Metrics

We used the System Usability Scale (SUS) to measure subjective ease of use, including perceived effectiveness and satisfaction with the system:

 
 
 

User Testing

We recruited participants by distributing a questionnaire to our networks, asking about demographics, interest in art, and experience purchasing tickets and making donations online.

During the testing session, participants were presented with each scenario and asked to complete all 5 task using a think aloud protocol. One team member served as the moderator, while the rest of the team observed remotely from a different room. 

 

Our testing room set-up with refreshments, a camera which connects to Morae, and a testing script for the moderator.

 

Morae software enabled the rest of the team to observe the testing session remotely

 
 

Data Analysis

We collected data on task completion, ease-of-use rating, error rates, clicks per task, mouse movements, SUS survey responses, and qualitative responses. We also quantified the ease of use of the general system by averaging SUS scores across participants.

Samples of the data we collected are shown below:

 
 
 
 

Design Recommendations

For our final deliverable, we translated our findings into actionable design requirements: