路嘉锋/Jiafeng Lu
Smarthistory Mobile Website User Experience Study
Smarthistory is a free resource for the study of art history. Based on clients' demand, this study aims to analyze Non-Euro, Prehistoric and Start Here pages on the mobile website through Analytics and Eye-tracking.
Team Originators
Talented UX designers and researchers formed my team: Roey Wang, Srishiti Dokras, Shuyang Lin and me.
In collocation with the team, we conducted website analytics using google analytics and Hotjar, after that, we conducted Eye-tracking on our recruited participants. Ten users participated in tasks designed to improve the overall experience of navigating the mobile website, including landing pages, bottom navigation, layout difficulties and Start Here page.
My Role in the Team
My role in this project included designing, recruiting, and filtering participants, moderating Eye-tracking studys based on client concerns, analyzing test data and consolidating it into graphical illustrations. After that, differentiating major and minor findings. Synthesizing follow-up recommendations into the usability report, and building the presentation while concluding report contents to the client.
Project Objectives
We sought answers to the following Questions about Pre-History, Non- European and Start Here pages on Mobile Website.
-
How do users use or perceive the landing pages?
-
What are users' behaviors with the bottom navigation on the content page?
-
What layout difficulties are users experiencing when searching for information?
-
How do users perceive the Start Here page?
Landing Page
Bottom Navigation
Start here Page
Process
Client kick-off meeting
Eye-Tracking Study
Analyze Website Stats
Data Analysis
Recruiting Participants
Solutions
Screening Participants
To meet the client's needs, I, as part of the Research Operation representative team made a list of screening questions to find current college students and instructors that matched the most with high familiarity with art history, and were eye convenient to conduct eye-tracking. 10 participants were selected from 50 to conduct the Eye-tracking user test.
At the end, we recruited 10 participants, 8 are students from SVA, NYU and Pratt. 1 teacher from Partt and 1 more art industry employee.
Methodology:Eye-tracking
Eye tracking is a sensor technology that can detect a person's presence and follow what they are looking at in real time through fixation and saccades. The technology decodes eye movements and translates them into insights that can be used in a wide range of studies.
With Tobii eye-tracking machine that detects users' saccades and fixations, our team originators are able to collect quantitative and qualitative data with the help of methods including Retrospective Think Aloud (RTA), Gaze plots, Gaze replays, Heatmaps and Quantitative questionnaire data.
Creating Research Plan & Design Tasks
We designed our tasks for our eye-tracking study (based on Google analytics and Hotjar stats) to gain insight into how users perceive different pages and navigate the website to identify the issues they may encounter.
During the eye-tracking study, participants are allowed to think aloud and we asked them questions during the Retrospective Think Aloud (RTA) while showing them their screen replays.
Analyzing Data
Quantitative Data
Our team used the outputs from Tobii (Gaze replays, heat maps, and gaze pots) to find out what people tended to look at and in what time sequence they were looking at items on each page.
Furthermore, we marked and recorded participants’ success per task, and marked the time periods it took them to complete each task.
Qualitative Data
We also created a problem sheet to sort out and rate the severity of problems. I analyzed data from 5 participants I interviewed and found specific themes we saw in the qualitative data by grouping the findings into specific themes with an affinity diagram. Here we categorized them with different colors of note stickers.
Results and Findings
Overall, participants of the eye-tracking study felt that the mobile website was easy to learn information but not easy to locate desired information.
1.1:Users found the Beginner’s Guide section lack of relevancy to what they are looking for.
50% users expressed their unsatisfactory to the content in the Beginner’s Guide.
The only reason for them to click into the article is that they found keywords that is related to the task prompt in the title.
Recommendation
Reorganize the content display in the Beginners’ Guide by condensing existing articles and adding the searching guide.
Beginner’s Guide are for :
-
Users who are new to art history
-
Overview articles
-
Make each article more relevant by focusing on one topic.
-
New to the website.
Navigation and search guide
-
Location of useful features: search function, bottom navigation
-
Page layout
-
Classification of content
1.2:The section lacks of visual cues for users to engage.
Articles in the section look unclickable. 20% users expressed their confusion about links of articles.
Recommendation
Add visual cues to increase visual engagement and to indicate links.
-
Change the color of the title to blue and underscore them to signify clickable links.
-
Add an arrow icon next to each article title to encourage users to read more.
-
Add icons to increase engagement and present information in a more straightforward way.
2.1:Users found the information structure of the bottom navigation to be overwhelming and confusing.
50% users found the current classifications confusing and thought that there was “too much information” in the bottom navigation.
Recommendation
Increase the font size and the accordion target to make it easier for users to read and interact with the bottom navigation.
For better usability, change the signifier to a ‘caret’ as many users perceived the minus sign as a bullet point. According to studies, carets work best for mobile menus.
2.2:Users found it challenging to read the content on the bottom navigation due to the small font size on mobile devices.
3 users thought that the font size in the bottom navigation was not large enough and that information “should be bolded”
Recommendation
Reduce the amount of information displayed in the bottom navigation by including only the higher-level content categories.
Further, many users faced difficulty with understanding content categorized by time. There’s a potential to conduct further card-sorting test to help learn more about the information hierarchy and reorganize it.
3.1:The information display on the content page is difficult for participants to browse.
They want to have following features on the content page.
-
Content Overview
-
Font/Style differences between different sections.
-
For example, the “Notes” is in the same font as the main article, which makes the article too lengthy.
-
No link to the “Notes” area. Users have to do it manually by scrolling down.
50% users expressed their unsatisfactory to the layout of the content page.
They want to get desired information more efficiently, especially for browsing.
Recommendation
Increase browsability by adding the table of content and summary chart of the art piece at the beginning of the article.
-
Add summary section at the beginning of to inform users who are looking for basic information.
-
Add Table of Content to tell users what information are in the article.
-
Add links to notes, so users do need to find it manually.
Increase browsability by adding the table of content and summary chart of the art piece at the beginning of the article.
-
Change the color and font of the notes section to make it discernable from the main article.
-
Add hanging style to the section to signify its citation nature.
3.2:There is scope to improve the discoverability of the search feature for mobile users.
Hotjar data indicates that only 6% of users that click on “Search” are on mobile.
Cumulative heat map data of different pages shows that users do not notice the search icon.
This could be because pages are content heavy and users do not scroll all the way back to the top.
Recommendation
-
Allow the main menu bar to show when users scroll back up, and disappear when users scroll downwards.
-
Further, increase the search icon size and weight to improve visibility.
This allows for more opportunity for users to view the search icon without taking up screen space as users are reading.
4.1:The topic section is not designed intuitively; the classifications are not clear.
Content layout leads too much scrollings for users.
50% of users did not locate information within topics section as topics are too general.
4.2:The video and essay section is confusing due to so much mixed content.
The huge amount of information is not categorized, which requires users to scroll a lot (time-consuming) to find their wanted topics.
Recommendation
-
Add a guide at the top to teach users how to use this page.
-
Add a filter and search function in the video+essays section.
-
Reorganize the topic sections so that it’s more intuitive for users (beginners) to outline art history.
-
Further card-sorting/tree testing method is also recommended)
A/B Test Plan
Conclusion
The Smarthistory earned a weak SUS score for its Non-Euro, Prehistoric, and Start Here mobile pages, indicating low performance and requiring further improvements in the following aspects:
-
Reorganize the content displayed in the Beginners’ Guide and add visual cues to increase user engagement.
-
Reduce the amount of information displayed in the bottom navigation bar and increase its font size.
-
Make the search feature more prominent.
-
Redesign the Start Here page more intuitively and add a filter and search function under the Video + Essays section.