A Usability Test of M2 Interactive

By Jason Gerbes, Joshua Son & Paul Lee

Research

Six individuals participated in this research. Each participant was determined to be a potential user of M2 Interactive. A typical user is a middle-to-late aged person who reads magazines, has an interest in technology and owns a smartphone. As explained by Alexis Rabadan from Luminary Promotions (personal communication, 19 March 2015), "The app was designed to be simple enough to not need tutorials, as the fast paced lifestyle of an M2 reader does not have time for that."

Page 8 of the January 2015 issue of M2 magazine (seen below) includes a brief user guide for M2 Interactive: "Download the "M2 Interactive" app, open app & point your phone at the pages with "M2 Interactive logo" to watch videos, win instant prizes, interact with products & more."

M2 Magazine, January 2015 Issue - Page 8

Augmented Reality Background Information

There are four types of applications that are most commonly utilising Augmented Reality technology: advertising and commercial, entertainment and educational, medial and mobile applications.

AR, as described by Furht (2011), is mostly prominently used by marketers to promote new products online. Most techniques use markers that the users present in front of their camera. Once the marker is scanned, an object is overlaid on the user’s view.

For example, in the January 2015 edition of the M2 magazine, markers are placed throughout the magazine for readers to scan with the M2 Interactive application. Once the marker is scanned, an assortment of items, like a TV, tractor, or bomb, is displayed on screen. Some of these items can be interacted with. This interaction provides gives the magazine another dimension and allows it to have infinitely more content.

AR is still a relatively new technology and possible future applications are infinite. Advanced research includes the bionic contact lens, which is claimed by Babak Parviz (2009), a University of Washington assistant professor as “Looking through a completed lens, you would see what the display is generating superimposed on the world outside.”

Bionic Contact Lens — A prototype bionic contact lens. (Design Buzz, 2011)

Usability Test Procedures

Each participant was provided with a copy of the Participant Information Sheet, which outlines the testing process in detail. The participant's were given one week to decide if they wish to be part of the study, and signed a Consent Form upon decision to participate.

The participants were provided with the running M2 Interactive application on a smartphone and a January 2015 copy of M2 Men’s magazine. They were not given any usage instructions. The participants were asked to use the application as they would naturally assume to be correct.

Notes about any observations were recorded during the participant’s use of the app. When the participant had decided to end the test, a series of ten standard questions were asked. The participants were then give the opportunity to reflect on the observations recorded during their test.

The six independent tests were then collated, and our findings were derived from the results.

Usability Assessment Criteria

The M2 Interactive app is intended to supplement the magazine and enhance the user’s overall experience when reading M2 magazine. The application must be effective, efficient and make the user feel entertained.

The usability of the application was determined by the following key factors:

Intuitive design
A nearly effortless understanding of the architecture and navigation of the application.
Ease of learning
How fast a new user can accomplish basic tasks.
Efficiency of use
How fast an experienced user can accomplish tasks.
Memorability
After using the application, if a user is able to use it effectively in the future.
Error frequency and severity
How often users make errors, how serious the errors are, and how users recover from the errors.
Subjective satisfaction
If the user likes using the application.

(Usability.gov, 2015)

Backwards Gun — An example of poor usability design (Realise Design, 2015)

Post-test Questions Asked

Following the use of the M2 Interactive application, each participant was asked ten standard questions. Their answers were recorded on a question sheet. We asked a mixture of open and closed questions to encourage an unguided response. See the Question Sheet for an outline of the questions.

The closed questions helped build our understanding of the participants, such as 'have you used an Augmented Reality application before?', whereas the open questions gained an insight into the participant’s thoughts about the application, such as 'what would you change about the app?' and 'Why would you change it?'

Types of Observations

As the participant used the application, they were closely monitored by a member of the group. Details were recorded about their expressions, comments and difficulties throughout their experience. These observations were discussed with the participant following the study, and their responses were recorded.

The amount of time that each participant spent with the app and magazine was also recorded. This information provides an insight into the amount of engagement they had with M2 Interactive.

Types of Data Captured

Data captured from the testing is a mixture of quantitative and qualitative data. Quantitative data deals with numbers and options (e.g. 0 - 10, yes/no), it is information that can be measured. Qualitative data deals with descriptions (e.g. 'what did you like about the application?'), the data can be observed but not measured (Roberts, 1998).

The quantitive data we gathered was used to gain a better understanding of the participants, and the qualitative data was used to gain an insight into the participant's thoughts about the application.

Testing Control Measures

As suggested by Barnum (2010), the ‘thinking out loud’ procedure is more successful when the moderator is “effective and unbiased.” Barnum says, in order to be an effective and unbiased moderator, one must monitor their body language, balance their praise and ask ‘good’ questions. All of these practices were followed, to reduce bias in the responses and behaviours of the users.

To keep test conditions fair, the study was tested in a controlled environment with constant conditions i.e. similar lighting, same phone (iPhone 5C) and the same magazine. Each test followed the same testing procedures. Intervention only occurred when there were bugs or the application crashed, or when the participant struggled. These steps were necessary to eliminate bias between tests and develop an insight into the thoughts of the user.

Why were these measures required?
Controlled testing allows for a ‘fairer’ test. By creating less bias, we are able to develop a better understanding of what the intended user is really thinking. Instead of influencing or prompting the subjects with the idea of what is right or what is wrong, the user is able to decide whether their actions met their expected outcomes. The overall aim is trying to get inside their head without clouding their judgement. Thus, an understanding of how intuitive the application really is is revealed during this experiment.