Let's lay some groundwork. From my previous usability test, I found four themes of successful usability in GNOME:
Testers commented that the programs seemed to operate more or less like their counterparts in Windows or Mac. For example, Gedit isn't very different from Windows Notepad, or even Microsoft Word. Firefox looks like other web browsers. Nautilus is quite similar to Windows Explorer or Mac Finder. To some extent, these testers had been “trained” under Windows or Ma, so having functionality (and paths to that functionality) that was approximately equivalent to the Windows or Mac experience was an important part of their success.2. Consistency
User interface consistency between the programs worked strongly in favor of the testers, and was a recurring theme for good usability. Right-click worked in all programs to bring up a context-sensitive menu. Programs looked and acted the same, so testers didn't have to “re-learn” how to use the next program. While the toolbars differed, all programs shared a familiar menu system that featured File, Edit, View, and Help.3. Menus
Testers preferred to access the programs’ functionality from the menus rather than via “hotkeys” or icons on the toolbar. For example, the only toolbar icon that testers used in the Gedit scenarios was the Save button. For other scenarios, testers used the drop-down menus such as File, Edit, View, and Tools.4. Obviousness
When an action produced an obvious result, or clearly indicated success—such as saving a file in the editor, creating a folder in the file manager, opening a new tab in the web browser—testers were able to quickly move through the scenarios. Where an action did not produce obvious feedback, the testers tended to become confused. The contrast was evident when trying to create a bookmark or shortcut in the Nautilus file manager. In this case, Nautilus did not provide feedback, failing to indicate whether or not the bookmark had been created, so testers were unsure if they had successfully completed the activity.And while it wasn't part of the test, users experienced problems with the "Activities" hot corner. I can generalize this into a fifth theme:
5. Unexpected behavior
Those who experienced the "hot corner problem" typically did so right away in the first exercise when attempting to use the program menus. While testers were able to recover from the hot corner, it definitely caused disruption several times throughout the test. None of the users in the usability test had used GNOME before, so they did not have previous experience in using GNOME. The programs they were evaluating in the usability test were running "maximized" so when they needed to use the program menus (say, the "File" menu) they would naturally bring the mouse towards the upper left corner of the screen, to the "File" menu - and then "overshoot" the menu, activating the hot corner. Suddenly, the screen changed completely - all the program windows from the test were represented as smaller "tiles" on the screen. Testers exhibited clear confusion, irritation, and surprise at this unexpected behavior, even when they experienced the "hot corner problem" multiple times throughout the test.My question in this usability study is How well do users navigate or understand the new design patterns in GNOME? The differences in GNOME between version 3.4 (my previous usability test, using Fedora 17) and GNOME 3.10 (Fedora 20) & GNOME 3.12 (latest release) appear to be largely "under the hood." With the exception of Gedit, the user interface differences appear minimal. Perhaps the largest change is the loss of "title bars."
The 4 (now 5) themes will continue
Despite UI differences from GNOME to Mac or Windows, I believe testers will comment that the programs are somewhat familiar. Familiarity will likely be the strongest theme: Gedit is not too different from Notepad, Web is similar to Firefox or Internet Explorer, Nautilus is not unlike Windows Explorer or Mac Finder, etc. Notes may be a new experience, however. But the fact that all programs act similarly, and share the same design patterns, will make the programs easier to learn. Once you figure out one program, you can apply that new knowledge to the other programs. The other themes will similarly continue. Obviousness will get positive comments due to improved messages and feedback in GNOME 3.10 & 3.12. But I expect usability issues with the loss of obvious menus, and more "hot corner" problems.Users will be confused about menus
In my previous usability test, menus were an important part of usability. Menus were also a theme of good usability, because GNOME 3.4 still used menus for most actions. But in GNOME 3.10 & 3.12, actions have been moved into a "gear" menu. Some "top level" or "global" actions may only be accessed from the application menu. The loss of these obvious menus will likely cause problems during the test. I expect users will discover some functionality by clicking on the "gear" menu. Having found that menu, I don't know that users will experiment with the application menu. So any functionality that must be accessed through the application menu may be rendered unusable.Users will think the test didn't go well, even if they were able to complete all the tasks
This is a visual rhetoric statement, and probably the most important. In rhetorical discourse, if you are performing a rhetorical ritual and the audience feels uncomfortable, it's because you left out an important part of the ritual.
An easy example is a wedding ceremony. There's nothing that requires the use of "I take thee" vows. A wedding is a form of a legal contract, and you need to have some language in there that defines the wedding, but "I take thee" statements are not part of that contractual ceremony. But they are part of the traditional ceremony, the ritual. If you leave out the "I take thee" vows, it's pretty much guaranteed that family and friends in the audience will feel uncomfortable, as though something was missing. They may not be able to figure out what was missing, but they will know something got left out. And that's when some family and friends may begin to wonder "did they really get married?"
I think it's the same with this usability test. The loss of certain visual cues, the missing direction provided by visually rhetorical devices (such as "title bars" and other common desktop metaphors) will make the testers uncomfortable. At the end of the test, when I ask how things went, I predict a common theme where users initially say they aren't sure, then shift their answer to "it didn't go well." And I think that will happen even if the testers were able to complete all of the scenario tasks.
This isn't to say that user interfaces can't ever change, that they need to remain static. But it does suggest that interface designers need to pay close attention to these visual cues. Changes in these areas must be done with care. Sometimes you can get away with total change by shifting the context (such as the new interface on tablets, versus the traditional "menus and windows" interface on desktops and laptops - most people weren't surprised when this totally different computer had a totally different interface). But in general, if it's a visual cue, a rhetorical part of the desktop interaction, you can only change it in stages.Specific problems with applications and tasks
I predict that "training" in other desktop environments (Windows or Mac) will translate to behavior expectation. In addition, while I attempted to reflect common user actions in the scenario tasks, I feel some of these actions happen so rarely in other desktop environments that the testers will be unfamiliar with how they might do them under a different environment. (How often do you change the default font in a text editor, or a sticky-notes program? Maybe once, when you first start using it, but never after that.) This will be exhibited by specific problems with applications and tasks, such as: changing the default font in Gedit and Notes, creating a shortcut to a folder in Nautilus, searching for a file that's not under the current folder tree, bookmarking a website in Web, increasing the font size in Web, and deleting all notes in Notes.Problems caused by possible bugs
It's possible that we'll uncover a few bugs along the way, and users may experience usability issues because of programmatic errors. One problem I've found is that if you install a program (Robots) using Software, then try to remove the same program, you get an error. You seem to have to exit the Software program after installing a program, then re-launch Software to remove it. That seems like a program bug. As a workaround, I may ask testers to exit the Software program before they try out the Robots game, so we don't trigger the bug in Software.