When doing usability testing, I strongly believe that understanding the participants is a very important point that we shouldn't ignore. Here, the tester was a 23 year old female student in computer science, who self-reported a medium level of computer expertise.
Test objectives & context
The principal objective of my 1-person usability test was to identify areas in GNOME interface where users may encounter difficulties. Other objectives of this test were to determine if:
- The proposed tasks can be successfully accomplished
- The proposed tasks were clear for the user
- The given explanations clear enough for the user
As the participant was not previously familiar with GNOME, I started with an explanation of what GNOME was, and informed the participant about this usability test's purpose. In a usability test, we want to evaluate the user interface and not the tester's individual skills.
Before starting the test, I set some rules, asking the participant to "speak out loud" when doing a task. The participant also agreed to be recorded during the test, and was assured that the data was collected anonymously.
For the usability test, I gave the participant a single sheet of paper containing a list of 23 usability test scenario tasks. The duration of the test was about 30 minutes, including the introduction.
The test results
In order to evaluate the usability test results, I used Jim Hall's heat map technique, presented below. In this heat map, you can see that the participant was able to complete most of the tasks without much difficulty (green boxes), but she encountered some problems in accomplishing others (red boxes). Tasks that the tester was unable to complete are shown in black boxes.
The tester was easily able to search for an image in the system and change the screen and the lock screen background. The same happened with the tasks related to the browser, as the tester was already familiar with Mozilla Firefox. The tester was also easily able to play a song and repeat a song in the Music application, the tasks were also quickly and correctly accomplished.
However, during the test, some usability issues were revealed. The tester was not able to complete some tasks like bookmarking a page in a PDF document, or change the name of a bookmark. When testing gedit, the tester was unable to display line numbers in the file or to change the letters of a word to upper-case.
Furthermore, one of most difficult encountered issues was related to Nautilus, where the participant took a long time to discover the option to "bookmark this location". After this option was selected, the tester was unable to locate where the bookmark was saved. Indeed, when her attention was in the top-right corner in order to select the "bookmark this location" option, she did not see that the folder was added on the left-side pane in Nautilus(see screenshot).
|Bookmarking a location|
Another problem that the participant struggled with was the installation of the gparted program. Her first guess was to search for the keyword "install" in the GNOME search bar, but the displayed programs names were not explicit enough to convince her that she was following the right procedure. At first, she started the Synaptic package manager application, but then reacted with "no, this is not the right program". Then, she launched the Packages application, searched for "gparted" program and clicked on "Install Package", but after the button name changed into "Remove package" she thought that the program was installed. When she was asked to verify if the program was installed, she could not find it in the system. It was only after a closer look in the Packages application that she realized that she missed the "Apply settings" button, in the right corner of the window (see screenshot).
For her first interaction with GNOME environment, the tester successfully accomplished most of the basic tasks. However, I believe some of the difficulties in fulfilling the other tasks were due to the user's non-familiarity with GNOME.
Reflecting on the usability test itself, I realized that using a virtual machine with limited performance could have an impact on the test quality. The next time I do usability testing, I plan to use a dedicated machine.
Another uncovered issue was due to my scenario task design: some tasks were not explained well enough, so the tester sometimes had to ask about what a task was prompting her to do. The next time I do usability testing, I will put more focus on the scenario task descriptions.
To improve the quality of future usability tests, I would plan to develop more scenario tasks. For each category of GNOME features, tasks could be classified by their level of difficulty and designed for various types of subjects (for example people familiar with GNOME, or people familiar with Linux but not with GNOME, etc).
This is an interesting first test, although usability testing with only one person makes it difficult to determine if a product like GNOME has good usability.
In conclusion, usability testing can help GNOME to resolve its issues, by using testers' feedback to improve its features. This kind of testing can also help us to understand users and their expectations in order to offer a better user experience.
A few minor edits by me, for style and grammar. -jh