OER Dev

User testing

Why perform user tests? 

Designing with the user and not for the user is vital if you want your digital initiatives to be successful. User-centered design, also known as design thinking or human centered design, focuses on understanding the needs of the users, challenges and personal characteristics. Through various types of interactions with the user, for example conversation, interviews and observation, you get to know the people you are designing for. The information gathered from these interactions leads to building, testing, retesting, designing and redesigning tools and content until they effectively meet user needs. Co-creating and designing with the users helps you address the specific context, culture, behaviors and expectations. This will help you build better tools, and create better content for the users. Designing together means to cooperate with users throughout the whole project lifecycle. Together you co-create solutions, and you continuously gather and incorporate feedback from the users to improve your product. 

What is user testing?

There are three different types of knowledge: Knowledge based on experience, user based knowledge and knowledge based on research. Knowledge based on experience is mostly subjective, since it is based on the private experiences and perceptions of the individual. User based knowledge is closely connected to the experience the user has, for example of being a student. Knowledge developed through research and development work, evaluations and other types of practically oriented research, is called knowledge based research (Nordahl, 2009). User testing is knowledge based research. 

User testing is one of several methods to search for evidence on what works, meaning what is more likely to give the wanted results. Findings from user tests give us different kinds of evidence. We learn about usability and accessibility of an interface, how content best should be put together to enhance learning outcomes, and the didactic and pedagogical reasons a collection should be based on.   

User testing is an important way to involve the users. An essential goal is to gain the insight you need to make or improve a product. An example would be to improve the usability of a website or a content type.

A user test

  • simulates a reel situation
  • has specific tasks
  • is a test where you observe the user (on site or remote)
  • is being used to evaluate the usability of a system or a product

There are two different methods of user testing: ordinary user testing and usability tests. While usability tests mainly focus on the functionality, for example a website and how to navigate on the site, user testing focuses on the actual need for this site. It is important to separate these two methods. Both are equally important, but provide different knowledge about the users (see table below).

User test Usability test
FocusThe need and value of concept, service, functionality and content for the user The user and her understanding of the interface 
AnswersThe needs of the users for this solution
Validates how well a concept and a solution work for the user
How well the users can use the solution as it is
ScopeOpenLimited
Should be performed….in an early phase and continuously in the last stages when a concept/solution is validated
After a test we can…change course or confirm direction optimize the existing design or details in the content    
Examples of test and focus areasGDL Read & Play:
What kind of interactivities and games engage the children? 
Do the reflection tasks and tasks related to a child’s own life help her understand the message of the book better?
Test of the GDL translation module: 
Do the users understand how to move from one page to another?
Do they know how to preview the content of a book?
Do they understand how to submit a book? 

Most user tests and usability tests result in some findings that are unexpected and not necessarily what you wanted to test in the first place. These can be interesting and important findings. You should therefore always be open to all feedback when you perform qualitative tests like these.  

User tests can be performed in different ways and are of different lengths. Guerilla testing, remote testing and tests of content or functionality are some of the different types of user testing (Toftøy-Andersen and Wold, 2011). 

An “ordinary” user test with High School students takes about 30 to 45 minutes. If you user test on younger children, though, you should, based on our experience, shorten the test to between 10 and 15 minutes. Also keep in mind that it takes time and effort to prepare a test script. The test script contains a plan for the run-through of the test with date, times and names of the informants. You must also decide on the scope of this test in particular, test criteria, success criteria and hypotheses for findings. 

The test script should be developed at least four weeks before the test. It should give an overview of all the tasks for the next weeks, time and date for when they should be done, and assigned a person responsible for doing the tasks. Examples of tasks can be what books on which levels the children should find, navigate in, talk about and read. 

It is crucial that the script and the activities are realistic to the user. If you test reading in a Read & Play, choose books that are suitable for children of a certain age, and also a topic that might interest them. If you only test the navigation of a book and the navigation on the website, it is still useful to choose books that are relevant and suitable for the age of the informants. 

A moderator will be responsible for running the test. It can be more than one moderator, but if that is the case, one moderator at a time should speak. The moderator can also be the observer, but if possible, there should be an observer in addition to the moderator. The observer can then focus fully on writing down the findings as the test proceeds. The observer should pay attention to what the user or users do (if more than one informant is present), what he says and the users´ body language.  Another thing to think about, is to have more observers taking part online, for example in a Google Meeting. This way, more than one or two people in the organization will get the experience of participating in a user test and in analyzing the findings. User tests can be recorded. QuickTime, Silverback or Lookback are all programs that can be used to document tests. 

Below is a step by step guide to a user test introduction:

  • Greet the informant
  • Welcome the informant to the user test
  • Ask her if she has participated in a user test before
  • Let the informant know the purpose and scope of the user test, for example: Today we are testing the math resources on the GDL
  • Let the informant know where you work, but at the same time let her know that you need honest feedback to improve usability and content
  • Let the informant know that you are testing the design, not her. She can not make any mistakes
  • You might also be testing a prototype. If that is the case, let the informant know
  • Ask the informant if she can think out loud while she is solving the tasks
  • If the informant is quiet for a longer time, she might be asked one or more of the following questions:
    • What did you just do?
    • Are you looking for something?
    • What did you want to do here?
  • Ask the informant if she has any questions before the test begins

After a test is finished, the moderator and the observer should ideally use 10 to 15 minutes to discuss the most important findings. They should also give each other feedback on the test – what went well and what can be improved in the next test. Things that might be changed are tasks, how to follow up on findings, or to be more patient with the user and wait until the user answers a question, not help out too early. If it is impossible to find time to discuss between the tests, time should be set aside after each day of testing to discuss findings and performance. 

A study made by the Dane Jacob Nielsen states that a group of five informants is the optimal number of participants in a user test. By having five informants, you will find about 80% of all the things that are not user friendly in the interface. Always make sure that the informants differ in needs, challenges and talents. Testing with informants who are visually impaired, hearing impaired or have other difficulties, should also be a focus. 

After a test you must make sure that a test report is written. The report should have a conclusion, a prioritized list of usability issues, preferable with solutions to the problems, document what worked well so no changes are made there, and information about informants and the tasks. 

%d bloggers like this: