Friday, December 28, 2012

Usability wrapup

I wanted to share some thoughts about this usability project, and how I would apply it outside of open source software. The project was very interesting and insightful about how to improve the usability of open source software, but usability methods don't just apply to software.

At work, I am the Director of Information Technology for a small liberal arts university. As part of my role, I often interact with students and faculty. For example, last year we held several "listening sessions" to solicit direct feedback from our customers about the level of service we currently provide, and where we can improve. These listening sessions aren't very different in concept to usability testing. In usability testing, you gather input from your target audience, and use that input to inform a process.

Where usability methods are used in software, that process is the software development lifecycle to bring new improvements to the software. Where the same methods are used in listening sessions, that process drives improvements in the organization. These are strongly parallel.

Going forward, I now recognize you don't need very many people to provide effective and useful feedback. In our listening sessions, we were concerned about attracting a "sufficient" number of folks to the sessions to garner useful feedback. But now I know from Jakob Nielsen, it turns out 5 testers is usually enough to get you there, if you're just interested in finding the core problems for a product. Per Nielsen: "If you want a single number, the answer is simple: test 5 users in a usability study. This lets you find almost as many usability problems as you'd find using many more test participants."

And in retrospect, that rule would have applied equally well to our listening sessions. We started the sessions with 20-40 people, depending on attendance, and asked participants to identify three to four ideas that would raise the level of technology on campus. But from each roomful of contributors, we ended up with about a dozen unique ideas. There was quite a lot of overlap. Five participants at three or four ideas each would have generated 15-20 ideas. So yes, Nielsen seems to be correct, even for this different process: "Test 5 users in a usability study. This lets you find almost as many usability problems as you'd find using many more test participants."

That's a key takeaway from this usability study that can be applied to non-usability areas: it can be statistically valid to use a smaller sample size. Of course, if you have the larger group available and the resources to test them, certainly do so. But if you can only scrape together a few participants, you can get by with as few as five interested testers.

Friday, December 14, 2012

Interesting reading

A few links to share about usability testing methods:

Usability Testing Essentials: Ready, Set...Test! by Carol Barnum. A few folks have recommended this book to me under the incomplete title "Ready, Set, Test" so I had a hard time tracking it down. From the description on Amazon:

Usability Testing Essentials presents a practical, step-by-step approach to learning the entire process of planning and conducting a usability test. It also explains how to analyze and apply the results and what to do when confronted with budgetary and time restrictions. This is the ideal book for anyone involved in usability or user-centered design-from students to seasoned professionals.

"Current Issues of Usability Characteristics and Usability Testing" by Ayobami, Hector, Hammed. It's a short read (five pages, including the references) but provides a good summary of the issues. From the abstract:

A well given design’s attention to Learnability, Contextuality and Operationability as Usability characteristics issues of software, and information systems generally is an evolving and challenging area in the field of Information systems, and Human Computer Interaction as an area of expertise. The procedure of testing and evaluating usability as quality characteristics in information systems has equally attracted researchers and professionals more in the recent times due to the developers’ attempt to meet the diverse users’ needs in the development of usable information systems. Hence, the paper aims to bring to the limelight; these current issues of usability and usability testing, the trend of the challenges inherent in the usability testing and evaluation process, then suggest the necessary attention that must be provided by subsequent researches in the bid to solve these lingering design research problems.

The article quickly identifies the key issues of usability: Learnability, Contextuality and Operationability. If your design can master those qualities, you should be okay. This is echoed somewhat through my findings in open source software and usability, where open source software that exhibits successful usability shares these characteristics:

  1. Familiarity
  2. Consistency
  3. Menus
  4. Obviousness

Thursday, December 13, 2012

Usability testing tools

If you are looking at improving the usability of your website, one of the first steps is to see how your users are trying to use the site. You can start with your website statistics to get an idea of the "top ten" pages they use. You might even have additional analytics that reveal how long a user remains on the site, and where they are from.

But I can speak from experience that seeing where users are clicking provides a very compelling case. The Useful Usability blog lists 24 website usability testing tools that you may find interesting. I've mentioned Ethn.io in another post. At work, we have used Crazy Egg. You may also be interested in the open source software tool ClickHeat.

These tools help to generate a "heat map" of your users' navigation habits. Where are they clicking? How much of the screen can they see?

As always, this is just one tool that you can use to analyze the current usability of your website. Don't forget to start with a user-focused design with personas and scenarios, and to do a full usability analysis.

Thursday, November 29, 2012

A quick link

I haven't read through the actual study yet, but I wanted to share this paper: "Current Issues of Usability Characteristics and Usability Testing" by Ayobami, Hector, Hammed. The abstract:

A well given design’s attention to Learnability, Contextuality and Operationability as Usability characteristics issues of software, and information systems generally is an evolving and challenging area in the field of Information systems, and Human Computer Interaction as an area of expertise. The procedure of testing and evaluating usability as quality characteristics in information systems has equally attracted researchers and professionals more in the recent times due to the developers’ attempt to meet the diverse users’ needs in the development of usable information systems. Hence, the paper aims to bring to the limelight; these current issues of usability and usability testing, the trend of the challenges inherent in the usability testing and evaluation process, then suggest the necessary attention that must be provided by subsequent researches in the bid to solve these lingering design research problems.

Why not usability?

I mentioned this earlier, but I'll share it again:

Interesting to note a comment I see repeated when I ask open source software folks about usability: "I'm not a usability expert." But you don't need to be a usability expert to apply usability to open source software. As Redish said to me in our interview, you can learn a lot just by sitting down with a user, and watching them use the software. You don't need to be a usability expert to grab a friend or family member, and ask them to try out a scenario for 15 minutes.

But not everyone is so easily convinced of the value of usability testing. A List Apart discusses this topic (July 31, 2012).  From the article:

Like any research method, though, usability testing has its drawbacks. Most importantly, it isn’t cheap. In a classic study, you’ve got the expenses of developing the testing plan, recruiting and compensating the participants, sometimes even travel. Even remote testing—which has its limitations—can be expensive, because analysis of the testing data is still no easy feat. In fact, analysis is often the biggest component of the project. Going through all those testing videos and cataloging the qualitative data just can’t be sped up without undermining the integrity of the study.
[...]
Fortunately, we have other usability research methods at our disposal. The standouts, expert review and heuristic evaluation, are both easy to add to a design and development process almost regardless of budget or resource concerns. Both involve minimal costs, as compared to full-scale usability testing, with expert review in particular being scalable to almost any project. For projects with fuller budgets, both techniques can maximize the cost-benefit ratios of subsequent usability testing.


The article highlights "discount usability engineering" which is basically a "heuristic review" where a usability expert evaluates an information product (such as a website) across several areas, including:


  • Appearance: The appeal and effectiveness of the site’s look and feel, from major layout features to small typographical details.
  • Content: The quality and strategic significance of the site’s content—including not only page copy but page metadata, PDFs and other files, and element-specific content (e.g., image descriptions).
  • Interface Usability: Ease of use of the site’s interactive components, from the simplest navigation structures to the most complex forms.
  • Accessibility and Technology: On a basic level, the site’s ability to adapt to diverse user needs, encompassing not only accessibility guidelines compliance but also browser compatibility and mobile- and tablet-friendliness.


While I (personally) view in-person usability tests as being the most useful, if you have experienced usability experts who can help you, doing a heuristic review can identify the "low hanging fruit" on your design - which will result in a better end product.


Usability testing by flash mobs

I liked this article about "usability testing by flash mob" from Usability Testing Howto Blog. From the article:

Ever heard of Improv Everywhere? This was the UX equivalent. Researchers just appeared out of the crowd to ask people to try out a couple of designs and then talk about their experiences. Most of the interactions with participants were about 20 minutes long. That’s it. But by the time the sun was over the yardarm (time for cocktails, that is), we had data on two designs from 40 participants. The day was amazingly energizing. 

What an amazing idea! Using "flash mobs" and other on-the-spot venues for usability testing is a great innovation. If your test subject is willing to spare a few minutes, and with a "mob" of individual testers and evaluators, you can glean valuable information in a short amount of time.

The article lists ten suggestions to making "flash mob testing" work well:

  1. Organize up front
  2. Streamline your data collection
  3. Test the data collection forms
  4. Minimize scripting
  5. Brief everyone on the dos and don'ts of usability evaluation
  6. Practice intercepting people
  7. Do an inventory check before starting
  8. Be flexible!
  9. Check in
  10. Reconvene on the same day for a wrap-up

This idea of "flash mob" usability testing can be extended to other domains, too. You can do this type of impromptu usability testing on website visitors. Chad Fennell shared a few links with me, including a software package "Ethnio" that allows you to intercept visitors so you can recruit them for live UX research.

Ethnio and other similar packages will likely be used more frequently in future by web designers as they seek active usability feedback from users. It doesn't take much for a usability test, only 5 or so testers can identify usability issues or positive trends. Imagine a design team setting aside a day to intercept web visitors, asking them to respond to a quick (5 minute) usability scenario. How many visitors does your site get in a day? How many visitors are on your website right now? Even if only 1/4 of your visitors give you their 5 minutes, it won't take long to get useful feedback.

How many testers do you need?

When I was planning my usability tests, I optimistically hoped for a dozen usability testers. But "from the discussions I've had with thought leaders in usability testing, I may not need as many as a dozen folks to get usability good results - so I'll be satisfied if I only attract half that (five or six)."

For my usability tests, ten people signed up, but three were "no-shows," leaving seven people who participated in the usability tests. And it was amazing how quickly those few testers were able to "zero in" to the usability problems and highlight positive trends for "what worked well."

From Jakob Nielsen, it turns out 5 testers is usually enough to get you there, if you're just interested in finding the usability problems for a product: "If you want a single number, the answer is simple: test 5 users in a usability study. This lets you find almost as many usability problems as you'd find using many more test participants."

If you want to dig deeper or search for other information, you may need more testers, however:

As with any human factors issue, however, there are exceptions:
  • Quantitative studies (aiming at statistics, not insights): Test at least 20 users to get statistically significant numbers; tight confidence intervals require even more users.
  • Card sorting: Test at least 15 users.
  • Eyetracking: Test 39 users if you want stable heatmaps.

I've commented before that it doesn't take much to pull together a few testers and ask them to run through test scenarios that represent how the product is used.  Interesting to note a comment I see repeated when I ask open source software folks about usability: "I'm not a usability expert." But you don't need to be a usability expert to apply usability to open source software. As Redish shared with me in our interview, you can learn a lot just by sitting down with a few users, and watching them use the software. By extension, you don't need to be a usability expert to grab a friend or family member, and ask them to try out a scenario for 15 minutes.

What about joining a community?

This study of open source software and usability neatly intersects two of my interests: information design (and usability) with open source software.

I've been involved in the open source community for a very long time. I first became aware of "Free software" around 1992 (or 1993?) while I was an undergraduate physics student. I had several classes that required use of the campus computer lab, and the lab manager installed many Free software (such as the GNU C Compiler, the GNU Emacs editor, and other programs) for us to use on the central Unix systems. I found this software to be of equally high quality to the commercial "payware" programs that I had used up to that time.

Later in 1993, I decided that I could use Free software to run my computer, so I installed the Linux operating system on my small '386 PC. I've used Linux at home (and at work) ever since, although I continued to run DOS on my computer for the rest of my undergrad years.

During my undergrad years, I extended my interest in open source software. As a physics student, I had learned some computer programming - and had learned more about it outside of class. In 1994, I started work on my own Free software version of the DOS operating system. I first called this "PD-DOS" for "Public Domain DOS," I quickly renamed it to "Free-DOS" and later "FreeDOS" as much of the operating system was built on Free software.

FreeDOS was my first "real" entry into the open source community. Since then, I've contributed to other Free software and open source software projects, including operating systems, editors, tools, music programs, and games.

More recently, I became interested in information design, through my graduate studies. Usability is strongly related to information design - and in fact, is a method to improve information products.

So in a recent conversation with my advisor, I started looking to communities surrounding usability and "UX". One popular destination is the UXPA - Usability / User Experience Professionals Association (formerly "UPA"). I'm sure there are other communities that have formed around LinkedIn and other social media venues. I plan to join UXPA and LinkedIn communities, even if only as a "lurker" (reading and listening, while rarely posting or contributing topics). What other communities can you suggest?

Asking the right questions

An article "Modifying Your Usability Testing Methods to Get Early-Stage Design Feedback" on UX Matters discusses the early stage design questions for your usability test. It's a great quick read about giving a boost to your usability tests.

What about sharing with OS cons?

In a discussion with my advisor today, I considered the idea of sharing my findings with a broader in-person open source forum. There are two open source conventions that I can think of, which might be interested in "open source usability" as a feature topic:

O'Reilly Open Source Convention ("OSCON")
As its name implies, an open source convention sponsored by O'Reilly (the publisher of many useful technical reference manuals and how-to books). I attended an OSCON about ten years ago, chairing a birds of a feather session about FreeDOS. Topics at OSCON span a broad range, and I think a presentation on open source usability would be well attended.

Penguicon
An open source software (and science fiction) convention held annually. I was a Nifty Guest some years back, speaking about FreeDOS, Linux, open source in the enterprise, and building open source communities. The date and location for Penguicon 2013 haven't been set yet, but based on last year's schedule I would expect it in late April or early May.

Monday, November 19, 2012

Draft outline

While I am busy analyzing the usability test findings, I don't want to lose focus on the eventual outcome of this study: how can I help the open source community improve the usability of open source software? At the start of this project, I suggested an article in a journal or magazine related to open source software. I'd like to continue with that goal, using Linux Journal as a target.

LJ is a front-runner among Linux and open source magazines. First published in 1994, LJ used to be a traditional subscription print journal, but recently (2011) switched to electronic only. LJ focuses mostly on Linux, but discusses open source projects in general. Articles published in LJ will see the widest distribution in the open source software community. Senior Editor: Doc Searls. Associate Editor: Shawn Powers.

So, what would be the outline of such an article? Here's one draft outline: (Note: I've updated this a bit since yesterday. Adjusted 1, 7, 8 with new content.)

  1. A bit of background: we've all experienced open source programs that were difficult to use or confusing. Or you may have heard from someone, "this is a great program, once you figure out the menus, or once you learn how to use it." That's a problem. i.e. bad usability in many open source programs makes people unlikely to use them, OR the main uptake for open source programs in the non open source community is programs that are easy to use/have good usability, so how do we improve usability to make open source more widely used. - Make this a strong beginning with the thesis of the study, uses an anecdote to draw in the readers.
  2. What is "usability," and what is good usability?
  3. Why do so many open source programs have bad usability? In short, because most open source programs are written by developers for other developers. The functionality is most important; the menus and usability are generally second thoughts - if they are considered at all.
  4. How can developers improve usability? Ideally, you should start with user profiles and usage scenarios - but a usability test will get you there. Talk about the methods of usability: heuristic review, prototype tests, formal usability tests, questionnaires, etc. But you don't need to do anything big and complicated to get useful/helpful information - a simple in-person usability test is all you need.
  5. How to do a usability test.
  6. My usability test.
  7. What I learned from my usability test: what works in usability, what needs work. - What makes these programs usable, and how I determined that.
  8. Summary and next steps: how open source programmers can do their own usability tests, you don't need many users to get useful feedback. The challenge is how to incorporate usability into your project so it becomes part of the "culture." Summary should include wrap up of paper (this is what makes for good usability, etc.) as well as tease the next steps.

Sunday, November 18, 2012

What worked well?

I almost titled this post "What works in open source usability," but I think I'm not quite there yet. So for now, let me discuss what worked well in the usability tests this week, and the comments from the testers.

Testers were easily able to accomplish the following tasks:

Gedit (text editor)

  • Write a simple note. The scenario asked them to type in a short note (about two paragraphs) so I could observe them recovering from typos and errors.
  • Basic editing. Replacing a single instance of a few words, and move a sentence from one paragraph to the other.
  • Replacing text. This asked testers to replace all occurrences of two names with new spellings of the same names.
  • Saving a file as a new copy of the file, with a new name.

Firefox (web browser)

  • Search the web for a particular news site, and navigate to that site.
  • Set that website as the browser's default startup page.
  • Increase the font size.
  • Create a new tab or window, and navigate to a particular URL.
  • Save the page for off-line viewing.
  • Download an image from the website.
  • Make a bookmark to a website for later use, and give the bookmark a new name.

Nautilus (file manager)

  • Create a new folder.
  • Move a folder to a new location.
  • Give a folder a new name.
  • Delete a file.

So, what's common that made these scenarios go well for the testers? I can provide commentary, but let me first share a few of the testers' responses to the questions "Across all three programs, what went well? Can you suggest why this went well for you?":

  • "Replace All" worked well to replace Jeff with Geoff, and Jenny with Ginny.
  • Having right-click menus everywhere helped a lot.
  • All programs sort of look the same.
  • Firefox was "good" and straightforward, seemed easy to use.
  • Right-click options were good.
  • Having the "+" button in Firefox to make a new tab made it easy, and because it was next to the other tab made it easy to find.
  • Highlighting was good, especially in the editor.
  • Firefox is like other browsers.

In general, I see Familiarity as a theme. Testers indicated that the programs seemed to operate more or less like their counterparts in Windows, and remember that most (6 out of 7) testers used Windows as their primary desktop. Gedit wasn't too different from Windows Notepad, or even Word. Firefox is like other browsers. Nautilus is very similar to Windows Explorer. To some extent, these testers had been trained under Windows, so having functionality - and paths to that functionality - that was approximately equivalent to the Windows experience was an important part of their success.

Also, Consistency was a recurring theme in the feedback. Right-click menus worked in all programs, and the programs looked and acted the same.

Menus were also helpful. While some said that icons (such as in the toolbar) were helpful to them during the test, most testers did not actually use the toolbar in Gedit - except to use the Save button. Instead, they used the program's drop-down menus: File, Edit, View, Tools, etc. In fact, testers experienced problems when the menus did not present possible actions clearly.

And to extend the analysis slightly further, allow me to contrast with what didn't work well - Obviousness. When an action produced an obvious result, or clearly indicated success - such as saving a file, creating a folder, opening a new tab, etc. - the testers were able to quickly move through the scenarios. When an action did not produce obvious feedback, the testers became confused. (These problems were especially evident when trying to create a bookmark or shortcut in the Nautilus file manager, but the program did not provide feedback, did not indicate whether or not the bookmark had been created.)

    GNOME: this needs fixing

    One other thing to comment on before talking about what works in usability - in doing the usability test, 6 of 7 testers experienced difficulty with the GNOME "hot corner." In GNOME, you can open the "Activities" menu (basically, the "programs" menu) by clicking on the "Activities" icon in the upper left corner of the screen, or simply by moving your mouse all the way to the upper left corner of the screen (the "hot corner"). You can also do this by pressing the "Windows" key on your keyboard.

    Those who experienced the "GNOME hot corner problem" typically did so right away in the first exercise when attempting to use the program menus. While testers were able to recover from the hot corner, it definitely caused disruption several times throughout the test.

    None of the users in the usability test had used GNOME, so they did not have previous experience in using GNOME. The programs they were evaluating in the usability test were running "maximized" so when they needed to use the program menus (say, the "File" menu) they would naturally bring the mouse towards the upper left corner of the screen ... and then "overshoot" the menu, activating the hot corner. Suddenly, the screen changed completely - all the program windows from the test were represented as smaller "tiles" on the screen.

    Most of these users (5 of the 6 testers) recognized that they needed to click on the "tile" for the program they were in, to bring up the program again. But in doing so, I heard these quotes from the testers:
    • "Dammit!"
    • "Oops!"
    • "What?"
    • "Hey, what happened."
    • "Whoa."

    I think the quotes represent confusion and surprise at the hot corner. Testers frequently experienced the "hot corner problem" multiple times throughout the entire usability test. I am not sure why 1 tester happened never to hit the hot corner during the entire test.

    I'll add that a GNOME extension, "No Topleft Hot Corner," disables this hot corner behavior. However, users would have to know to seek out and install this add-on, as it is not available by default.

    Nautilus: what needs work

    In the Nautilus file manager usability test, users were asked to complete a series of short scenarios. Here are the ones that caused problems:

    "4. As you work on your project, you expect to go back to the folder frequently. A shortcut or bookmark to the folder would be handy, rather than having to navigate to it each time. Create a bookmark for Space Photo Project."

    In the preceding scenarios, the testers were asked to create a folder, move it, and rename it. As explained in the exercises, the folder would be used in a project that collected photos from space, stored under the Pictures folder.

    However, when asked to create a bookmark or shortcut to the folder, almost everyone (6) had problems. Quotes from the testers in this exercise:

    • "I'm not sure if it created the bookmark."
    • "It's not speaking to me." (Not telling me anything.)
    • "That didn't work."
    • "I'm looking for something that says 'Bookmark'." (Later commented didn't notice the Bookmarks menu.)
    • "Nothing shows up."
    • "I want to grab the address of the folder."
    • "Nothing shows up."
    • "Where did that go? I'm confused."
    • (To the computer) "Why do you not want to bookmark it?"
    • "Oops, did I do that wrong?"
    • "What about Edit - Make Link?"
    • "I don't think that did anything."
    • "Something should pop up to tell me."
    • "Nothing popped up, it didn't work."

    Even though I used the same term in the scenario ("shortcut or bookmark") as in the menus ("Bookmarks"), testers were unsure what to do. The most common action was to go into the Pictures folder, highlight the project folder with a single mouse click, then select Bookmarks - Add Bookmark. When the menu action didn't do anything, testers also looked in File and Edit menus.

    Two testers were eventually able to create a bookmark to the project folder. Of these, one first used the Help menu and searched for the answer - but admitted he would not have done that if he didn't know about the feature already. The other, having exhausted a number of menu actions, tried creating the bookmark from within the folder she needed to mark. And that, of course, is how you create a bookmark in Nautilus.

    It seems clear that Nautilus should display a message when the user tries to create a bookmark. If the action is successful, it should note the success. If the action fails, Nautilus should explain why in simple language.

    After the usability test, several testers asked how to create a bookmark, since they were unable to do this. When I explained how to create a bookmark, and why their earlier attempts to create the mark seemed to fail, they thought they could do it successfully the next time. (For those curious: selecting Bookmarks - Add Bookmark from within the Pictures folder, no matter what is highlighted, will attempt to create a bookmark for Pictures ... but Pictures is effectively bookmarked already as a default, so Nautilus doesn't take any action.)

    "6. Do you remember the image that you downloaded earlier, in the web browser exercise? The filename was freedos.png, but you don't remember where you saved it. Find the freedos.png file and move it to the Pictures folder, if it is not there already."

    In this scenario, testers were asked to search for a file. If I noted that they had (earlier) successfully downloaded the web image into the Pictures folder, I simply asked them to find the file.

    Two of the testers opted not to use the file manager to search for the missing file, but clicked on each folder in sequence until they found the necessary file.

    The other five testers used the Search button in the upper-right of Nautilus, but searching for the file was confusing for them. Quotes heard during this section:

    • "I'm not finding it."
    • "What else to search on?"
    • "Reload my search? Nothing happened!"
    • "Grrrr!"
    • "There is no general search?"
    • "Not found, huh."
    • "No option for my home."

    Only two users were able to use Search successfully. Of these, one happened to click on the Search button from their home directory. The other tried changing things in the drop-down search action and eventually picked a combination that worked.

    Saturday, November 17, 2012

    Gedit: what needs work

    It's easier to talk about what needs to be improved (the "deltas") than to analyze what worked well (the "plusses") so let me take a moment to discuss this for Gedit.

    "5. You decide the font in the editor is difficult to read, and you would prefer to use a different font. Please change the font to DejaVu Sans Mono, 12 point."

    In Gedit, testers were really confused about how to set the font and the color. Part of this confusion stemmed from thinking of the editor like it was Word, which makes it easy to change the font. But even Windows Notepad lets you change the default font using a Format menu item.

    Testers reported they were looking for a menu item called "Font". Quotes from users during this part of the test:

    • "Looking for Font."
    • "Looking for the font change menu."
    • "Not showing a Font menu."
    • "Where do I change the font?"
    • "Looking for a Font menu, but it's not in the toolbar."
    • "Is it an 'input method'?"

    While looking to set the font, testers looked in all the menus, but mostly in the File, Edit, View, and Tools menus.

    A few testers (3) eventually discovered how to set the font, but all of these took several minutes to do so. Upon discovering the correct menu item (Edit - Preferences - Font & Colors) to change the font, two testers actually exited the menu because they didn't know how to change the font. They did not realize that they needed to uncheck the "Use the system fixed width font" item.

    "6. You decide the black-on-white text is too bright for your eyes, and you would prefer to use different colors. Please change the colors to the Oblivion color scheme."

    Changing the default colors presented similar problems. Quotes during this part of the test:

    • "Looking in Tools, not seeing it."
    • "Looking for a menu with a font."
    • "Looking for a toolbar to do this."
    • "Look for colors."

    The 3 testers who were able to set the font in the previous scenario also were able to change the colors, and one reported that it was easier because they saw the colors in the other exercise.

    Testers looked in all the menus to find the colors setting, but mostly in the File, Edit, View, and Tools menus - the same as in the font exercise.

    A preview of usability test results

    In my discussions with thought leaders about usability testing, I learned that you can often get a surprisingly accurate picture of usability in as few as 3 testers. It's one thing to hear that said; it's another to see it happen.

    This week, I worked with 7 usability testers in an informal lab setting: I set up a laptop in my office, with a USB keyboard and mouse, booted into a USB fob drive installation of Fedora 17 running the Gnome desktop. Each tester was logged into their own account, thus each tester started from the same "default" state.

    Testers ranged from about age 20 to mid-40s, both male (3) and female (4). Most testers (5) claimed "low" experience with computers, and almost all (6) used Windows as their primary desktop.

    I'll do a more detailed write-up of results soon, but here's a preview of the findings:

    Gedit

    Testers were easily able to accomplish the following tasks:

    • Write a simple note. The scenario asked them to type in a short note (about two paragraphs) so I could observe them recovering from typos and errors.
    • Basic editing. Replacing a single instance of a few words, and move a sentence from one paragraph to the other.
    • Replacing text. This asked testers to replace all occurrences of two names with new spellings of the same names.
    • Saving a file as a new copy of the file, with a new name.

    However, they experienced great difficulty in these tasks:

    • Changing the default font. Everyone agreed that an editor, even a simple editor like Windows Notepad (and Gedit is similar in function to Notepad), should let them change the text.
    • Changing the default colors. Most said that they would expect to change the colors, but either had trouble finding this function or were were unable to find it altogether.

    Firefox

    With a few one-off exceptions, testers had no problems using Firefox to achieve these tasks:

    • Search the web for a particular news site, and navigate to that site.
    • Set that website as the browser's default startup page.
    • Increase the font size.
    • Create a new tab or window, and navigate to a particular URL.
    • Save the page for off-line viewing.
    • Download an image from the website.
    • Make a bookmark to a website for later use, and give the bookmark a new name.

    Nautilus

    Testers were easily able to complete these tasks:

    • Create a new folder.
    • Move a folder to a new location.
    • Give a folder a new name.
    • Delete a file.

    However, testers experienced great difficulty with these tasks:

    • Create a new bookmark or shortcut for a folder.
    • Search for a file.


    I'll dig into these results further in another post.

    Usability tests this week

    This week, I conducted a usability test of three open source programs:
    1. Gedit, a text editor
    2. Firefox, a web browser
    3. Nautilus, a file manager
    These programs were selected based on your input of open source programs that feature good usability.

    I asked for volunteers to participate in a usability test. Since I work on a university campus, it was fairly straightforward to get volunteers for this usability test; I offered free candy and pop to anyone who sat in the test. In all, 10 volunteers stepped up. Unfortunately, a few were no-shows, leaving 7 who helped out (plus my wife who participated in a "dry run" of he usability test the weekend before - so technically there were 8 people). My ideal was about a dozen testers, but I said I would be happy with 5 or 6. So I'm good.

    At the start of each usability test, I gave the tester a brief context of the study, and why I am looking at open source usability. I explained that this was a usability test, so it was about the software. If the tester experienced problems, that's okay. Don't feed judged. I'm here to observe. And along the way, I'll be taking notes and looking at your screen.

    I also asked the testers to speak aloud what was going through their mind. For example, if they were looking for a Print button, just say "I'm looking for a Print button." And if they were looking for something on the screen, they move the mouse to where they were looking, so I could get an idea what they were searching for.

    During the usability test, the subject was presented with a number of scenarios. Each scenario was printed at 16 pt Garamond on a separate sheet of paper, and provided a brief context and an action they were asked to complete. For example: You are about to start working on a new project, and you would like to keep all your files in the same folder. Please create a new folder called My Project in the Documents folder.

    I listed the usability scenarios in a previous blog post.

    The usability tests went for about 30-40 minutes, depending on the tester. So I was pretty might right-on with my time estimates. I took a few minutes at the end of each program to ask some followup questions, and at the end of the test we did a quick "plus/delta" to wrap up.

    Wednesday, November 14, 2012

    Make FLOSS usable!

    "FLOSS" is the general term for "Free/Libre and Open Source Software", a more inclusive term than simply "open source". But the concept behind the OpenUsability project applies to all open source software: make FLOSS usable!

    From the About page:
    The idea of OpenUsability originated at the KDE developers conference in 2003. Jan Muehlig and his Berlin-based company relevantive had performed a comparative usability study of KDE 3.1 and Windows XP. The developers appreciated the work of the usability professionals, and a discussion came up how to involve usability feedback continuously with development. So far, it had been difficult for both sides to get in touch with each other: On the one side, self-acclaimed usability experts had reduced the developers' trust in the effects of usability, and on the other side trustful usability specialists often had problems to get in touch with the right people in a project.

    OpenUsability came to fill this gap. Since 2004, Open Source Software Projects can register to get usability support. A growing number of usability practitioners started to work under the label "OpenUsability", and provided the projects with usability feedback, attended developer conferences and fairs.
    (Note: the comparative study between KDE and Windows XP no longer exists on the relevantive.de site, so I have dropped the links.)

    They provide a list of projects that are specifically targeted to help certain open source programs: ArgoUML, Edyk, GCstar, ProjectPier, and so on. However, I don't see any general guidelines for how open source developers can apply usability practices into their own work, or successful ideas for good usability.

    Sunday, November 11, 2012

    Usability scenarios

    This weekend, my wife helped me with the study by agreeing to be a usability tester, by doing a "dry run" of the usability test scenarios. This helped me to gauge time and determine if the scenarios could be completed by a "typical" user with "average" knowledge. (Although she runs Linux on her laptop, my wife admits she is not very technical - she has "average" knowledge.)

    I previously shared a draft of the usability scenarios/exercises for Gedit, Firefox, and Nautilus. Here are the new questions after the dry run: (I also updated the previous blog posts, but this is for reference.)

    Gedit

    1. You need to type up a quick note for yourself, briefly describing an event that you want to remember later. You start the Gedit text editor (this has been done for you).

    Please type the following short paragraphs into the text editor:

    Note for myself:

    Jenny and I decided to throw a surprise party for Jeff, for Jeff's birthday. We'll get together at 5:30 on Friday after work, at Jeff's place. Jenny arranged for Jeff's roommate to keep him away until 7:00.
    We need to get the decorations up and music ready to go by 6:30, when people will start to arrive. We asked everyone to be there no later than 6:45.

    Save this note as party reminder.txt in the Documents folder.

    2. After typing the note, you realize that you got a few details wrong. Please make these edits:

    • In the first paragraph, change Friday to Thursday.
    • Change 5:30 to 5:00.
    • Move the entire sentence Jenny arranged for Jeff's roommate to keep him away until 7:00. to the end of the second paragraph, after no later than 6:45.

    When you are done, please save the file. You can use the same filename.

    3. Actually, Jeff prefers to go by Geoff, and Jenny prefers Ginny. Please replace every occurence of Jeff with Goeff, and all instances of Jenny with Ginny.

    When you are done, please save the file. You can use the same filename.

    4. You'd like to make a copy of the note, using a different name that you can find more easily later. Please save a copy of this note as Geoff surprise party.txt in the Documents folder.

    For the purposes of this exercise, you do not need to delete the original file.

    5. You decide the font in the editor is difficult to read, and you would prefer to use a different font. Please change the font to DejaVu Sans Mono, 12 point.

    6. You decide the black-on-white text is too bright for your eyes, and you would prefer to use different colors. Please change the colors to the Oblivion color scheme.

    Firefox

    1. You would like to see what's happening in the news. Your favorite news website is BBC News, but you don't remember the website's address.

    Search for the BBC News website, and bring up the site.

    2. You would like to bring up the BBC News website every time you start up the web browser. Please set the BBC News website as the default page.

    3. You don't have your glasses with you, so it's hard to read the text on the BBC News website. Please make the text bigger on the BBC News website.

    4. You now would like to see what's new with your favorite "open source" project, FreeDOS. The website's address is http://www.freedos.org/

    However, you want to keep an eye on the news while you are looking at the FreeDOS website. In a new site, navigate to the FreeDOS website.

    5. You would like to save a copy of the FreeDOS front page, so you can view it later on your laptop during your bus ride home tonight. Save a copy of the page to the Documents folder.

    6. You decide to download a copy of the FreeDOS program screenshot, the sample program image on the right side, under "Welcome to FreeDOS". This image might come in handy for a document you are working on (writing the document is not part of this exercise).

    Save a copy of this image as freedos.png in the Pictures folder.

    7. You would like to visit the FreeDOS website later, but don't want to keep typing in the address every time. Create a bookmark for the FreeDOS website.

    When you create the bookmark, name it simply FreeDOS.

    Nautilus

    1. You are about to start working on a new project, and you would like to keep all your files in the same folder.

    Please create a new folder called My Project in the Documents folder.

    2. Oops! You realize that the files in your new project will be photos, so you prefer to keep them under the Pictures folder instead of Documents.

    Please move the My Project folder from Documents to Pictures.

    3. Your project now has a new name, so you decide to rename the project folder to use the new name.

    Please rename the My Project folder to Space Photo Project.

    4. As you work on your project, you expect to go back to the folder frequently. A shortcut or bookmark to the folder would be handy, rather than having to navigate to it each time.

    Create a bookmark for Space Photo Project.

    5. Do you remember the party reminder that you created earlier, in the text editor exercise? In that exercise, you created a copy of party reminder.txt as Geoff surprise party.txt, but did not delete the first file.

    Please delete the party reminder.txt file in the Documents folder.

    6. Do you remember the image that you downloaded earlier, in the web browser exercise? The filename was freedos.png, but you don't remember where you saved it.

    Find the freedos.png file and move it to the Pictures folder, if it is not there already.

    Friday, November 9, 2012

    More interesting links

    Here are a few other interesting links shared with me by Chad Fennell, from the U of M:

    Ethnio, a web-based tool to intercept users on your website, so you can recruit them for UX live research.

    Regarding the Drupal tests, some of the resulting documentation - gives some sense of impact and adoption of the recommendations that resulted from the tests: Chad is also very involved in the Drupal project, working in the area of usability. Drupal is an interesting case, and appears to be fairly unique among open source projects for their community focus on usability.

    The last round of testing:
    Drupal usability
    The resources section contains links to participant videos, the issues cue matrix that resulted, etc.

    Here is where most ongoing usability discussions happen (outside of IRC):
    Drupal usability discussion

    A good place to ask more question: #drupal-usability on freenode.

    A few interesting links

    I wanted to share a few links, originally shared with me by Nick Rosencrans from the U of M, and Drupal:

    The case for open source design : a description of the way open source development works, with a bunch of helpful diagrams and such

    OS Alt : search engine for open source alternatives to commercial software

    Drupal usability : results from our usability test conducted for a Drupal team

    Wednesday, November 7, 2012

    Nautilus usability scenarios

    In the usability test for the Nautilus file manager, I will ask testers to do these scenarios:

    (Before beginning this phase of the usability test, I will start Nautilus for the tester. Note that Nautilus starts up in the Home folder.)

    1. You are about to start working on a new project, and you would like to keep all your files in the same folder.

    Please create a new folder called My Project in the Documents folder.

    2. Oops! You realize that the files in your new project will be photos, so you prefer to keep them under the Pictures folder instead of Documents.

    Please move the My Project folder from Documents to Pictures.

    3. Your project now has a new name, so you decide to rename the project folder to use the new name.

    Please rename the My Project folder to Space Photo Project.

    4. As you work on your project, you expect to go back to the folder frequently. A shortcut or bookmark to the folder would be handy, rather than having to navigate to it each time.

    Create a bookmark for Space Photo Project.

    5. Do you remember the party reminder that you created earlier, in the text editor exercise? In that exercise, you created a copy of party reminder.txt as Geoff surprise party.txt, but did not delete the first file.

    Please delete the party reminder.txt file in the Documents folder.

    6. Do you remember the image that you downloaded earlier, in the web browser exercise? The filename was freedos.png, but you don't remember where you saved it.

    Find the freedos.png file and move it to the Pictures folder, if it is not there already.

    7. You decide you are done working with the test program you wrote earlier, in the text editor exercise. Delete the test.sh program file in the Documents folder.

    (UPDATE 11/11/12) Since I've removed the program scenario from Gedit, I will drop scenario 7.

    Firefox usability scenarios

    For this part of the usability test, I will ask the testers to complete these scenarios:

    (Before beginning this phase of the test, I will start Firefox for the tester. Note that on Fedora Linux, Firefox starts up with a default home page from the Fedora project.)

    1. You would like to see what's happening in the news. Your favorite news website is BBC News, but you don't remember the website's address.

    Search for the BBC News website, and bring up the site.

    2. You would like to bring up the BBC News website every time you start up the web browser. Please set the BBC News website as the default page.

    3. You don't have your glasses with you, so it's hard to read the text on the BBC News website. Please make the text bigger on the BBC News website.

    4. You now would like to see what's new with your favorite "open source" project, FreeDOS. The website's address is http://www.freedos.org/

    However, you want to keep an eye on the news while you are looking at the FreeDOS website. In a new site, navigate to the FreeDOS website.

    5. You would like to save a copy of the FreeDOS front page, so you can view it later on your laptop during your bus ride home tonight. Save a copy of the page to the Documents folder.

    6. You decide to download a copy of the FreeDOS program screenshot, the sample program image on the right side, under "Welcome to FreeDOS". This image might come in handy for a document you are working on (writing the document is not part of this exercise).

    Save a copy of this image as freedos.png in the Pictures folder.

    7. You would like to visit the FreeDOS website later, but don't want to keep typing in the address every time. Create a bookmark for the FreeDOS website.

    When you create the bookmark, name it simply FreeDOS.

    8. You don't like looking at advertisements on websites. A friend said you can block ads in your web browser by installing something called Adblock Plus.

    Install Adblock Plus in your web browser.

    I'm not really sure the Adblock Plus scenario is a common usage example for "average" users with "typical" experience. What do you think?

    (UPDATE 11/8/12) After some discussion, I've decided to drop scenario 8 from the usability test. Installing Adblock Plus is not a task that "typical" users with "average" knowledge would be expected to do.

    * See, this is one of those times that user personas and usage scenarios would come in handy! We would be able to refer back to those to decide if Firefox's "typical" users would need to install add-ons like Adblock Plus.

    Gedit usability scenarios

    In asking testers to exercise the Gedit text editor, I will ask them to run through these scenarios:

    (Before beginning the Gedit usability test, I will start Gedit for the tester. Note that Gedit starts up with an empty document.)

    1. You need to type up a quick note for yourself, briefly describing an event that you want to remember later. You start the Gedit text editor (this has been done for you).

    Please type the following short paragraphs into the text editor:

    Note for myself:

    Jenny and I decided to throw a surprise party for Jeff, for Jeff's birthday. We'll get together at 5:30 on Friday after work, at Jeff's place. Jenny arranged for Jeff's roommate to keep him away until 7:00.

    We need to get the decorations up and music ready to go by 6:30, when people will start to arrive. We asked everyone to be there no later than 6:45.

    Save this note as party reminder.txt in the Documents folder.

    2. After typing the note, you realize that you got a few details wrong. Please make these edits:

    • In the first paragraph, change Friday to Thursday.
    • Change 5:30 to 5:00.
    • Move the entire sentence Jenny arranged for Jeff's roommate to keep him away until 7:00. to the end of the second paragraph, after no later than 6:45.

    When you are done, please save the file. You can use the same filename.

    3. Actually, Jeff prefers to go by Geoff, and Jenny prefers Ginny. Please replace every occurence of Jeff with Goeff, and all instances of Jenny with Ginny.

    When you are done, please save the file. You can use the same filename.

    4. You'd like to make a copy of the note, using a different name that you can find more easily later. Please save a copy of this note as Geoff surprise party.txt in the Documents folder.

    For the purposes of this exercise, you do not need to delete the original file.

    5. Gedit can also be used to write programs. For the purposes of this exercise, let's assume you are a programmer. Please open the test.sh program file in the Documents folder. It should look like this:

    #/bin/bash

    echo "Hello there!"
    echo -en "The date is: "
    date

    if [ "$COLS" -lt 80 ] ; then
        echo "this is a narrow terminal"
    fi

    if [ "$UID" -eq 0 ] ; then
        echo "you appear to be the administrator"
    fi

    You realize that the middle section that refers to COLS is not working, so as a debugging step you decide to comment out the three lines, starting with if and ending with fi.

    Comment out these lines by placing a # at the start of each of the three lines. When you are done, the file should look like this:

    #/bin/bash

    echo "Hello there!"
    echo -en "The date is: "
    date

    #if [ "$COLS" -lt 80 ] ; then
    #    echo "this is a narrow terminal"
    #fi

    if [ "$UID" -eq 0 ] ; then
        echo "you appear to be the administrator"
    fi

    When you are done, please save the file.

    (UPDATE 11/11/12) I've decided to drop scenario 5 from the usability test. This really isn't something that an "average" user with "typical" knowledge would do, and it would skew the results. I did a "dry run" usability test with family over the weekend, and it caused problems and added too much time to the test.

    6. You decide the font in the editor is difficult to read, and you would prefer to use a different font. Please change the font to DejaVu Sans Mono, 12 point.

    7. You decide the black-on-white text is too bright for your eyes, and you would prefer to use different colors. Please change the colors to the Oblivion color scheme.

    Scenarios in usability testing

    Usually when creating an information project, you would start by describing the people who are likely to use your product, and what they need to get out of your product. These are called user personas and are very helpful in generating a common understanding with everyone on the project about the users and their goals. To make the personas more concrete, you should include photos of people who look like the users you describe. (Stock photos are fine as long as they are casual, and always make sure you have permission to use the photo.) Photos put a human touch on the personas.

    The next step after personas is to describe actions that your users would take to accomplish their goals. These are called user scenarios. Scenarios can be very detailed, or they can be left a high level - but they always describe how your users do their work. There's nothing like being with your users when they try to accomplish a task, so you can watch them first-hand, but a good "second best" is to capture that behavior in a user scenario.

    In usability testing, you start not with the user personas, but with the scenarios. These usability scenarios are usually described at a high level, as you do not want to burden your usability tester with a preconceived notion for how they should accomplish a task, simply provide a description of the task or tasks they need to perform using the information product.

    In my example of usability testing, I will ask usability testers to complete several tasks, described as scenarios. The scenarios will start with a brief context, then a textual description of the task the tester is asked to perform. In the usability testing field, these are elaborated scenarios.

    I'll describe the scenarios in other posts here.

    Monday, November 5, 2012

    Slashdot question on open source usability

    A few weeks ago, the popular technology blog Slashdot polled the community for interview questoins to ask open source advocate Bruce Perens. I was glad to see Bruce answered my question about usability in open source software:

    Usability in open source software?
    by Jim Hall

    Bruce, I'm doing a study of usability in open source software - how user interfaces can be designed in Free / open source programs so the program is easy to use by real people. So my question is twofold: What Free / open source program really got it right with usability? What qualities make for good usability in Free / open source software?

    Bruce: Although I evangelize the issue of building our software for people outside of our community, I am no expert in usability at all. So, what that means is that there is room for others to evangelize best practices. I'd like to learn. I'll gladly pass your message along, if you build a compelling case.

    If I had to guess, I'd say that Mozilla has more expertise in this than many Free Software projects. I have been running the GNOME 3 desktop in Debian Sid for a month or so, and I am not sure GNOME 3 is quite there yet. Keep it up, folks!

    Unfortunately, that's a pretty short answer. I didn't expect a longer answer, though - usability in open source software seems to be an overlooked topic. Not a lot of research in this area.

    Interesting to note a comment I see repeated when I ask open source software folks about usability: "I'm not a usability expert." But you don't need to be a usability expert to apply usability to open source software. As Redish said to me in our interview, you can learn a lot just by sitting down with a user, and watching them use the software. You don't need to be a usability expert to grab a friend or family member, and ask them to try out a scenario for 15 minutes. (I'm doing the same with my family when I return from a trip next weekend - the start of my usability testing!)

    As of this writing, none of the comments on the Slashdot article discuss usability in open source software. But I wanted to share anyway.

    Sunday, November 4, 2012

    Usability testing

    (This post has been updated with a few improvements, based on suggestions. Edits are in blue.)

    Next week, I plan to start doing usability testing on the three programs: Gedit, Nautilus, Firefox.

    Since I work on a university campus, I'll start by inviting students, facutly, and staff to participate in the usability study. In most usability tests, you give your testers a small gratuity for their time. Since I don't have a budget ($$) to give out cash rewards, I'll instead offer free cans of pop to take with them, and as much candy as they can carry away with them. That's pretty inexpensive for me - and on a university campus, that should still garner a fair turnout.

    I'll also invite folks from the surrounding town (we're small, it's easy to get word around) to participate in the test too. And I'l try to get friends and family to submit to a usabilty test, as well.

    My goal is about a dozen testers from different age groups. From the discussions I've had with "thought leaders" in usability testing, I may not need as many as a dozen folks to get usability good results - so I'll be satisfied if I only attract half that (five or six).

    Each tester will start with a fresh instance to test against. This needn't be a separate install; it can be the same workstation, but different test accounts so that subsequent testers always get the same "vanilla" starting point as the first tester. My thought is to have the tester work from a bootable USB fob drive running Fedora 17, with separate test accounts for each tester: "test1", "test2", "test3", ... Since each test account can only modify it's own settings, everyone starts from the same "default" settings. This is important, especially since I'll likely ask testers to create or modify bookmarks (Firefox) or change a default font (Gedit).

    The goal will be to have the tester engaged in a usability test for no longer than 45 minutes - that's 15 minutes per program. I'm thinking about the following steps:

    Welcome (less than 5 minutes)
    • Brief context of the study, what is my study about (inform tester about the purpose of the study, and it's output)
    • Short explanation for what a "usability test" is - we are not testing you, but you are helping us examine software; there are no "wrong" answers
    • Quick rundown of what the test will cover: Gedit, Nautilus, Firefox. (This is a good opportunity to share comparisons. For example, Nautilus is a file manager, like Windows Explorer, or Finder on Mac. Having that shared reference can help set the stage for the tester, but I need to be careful about drawing too direct a comparison.)
    • Ask if they have participated in a usability test before. This is probably a new experience for many of the potential testers.
    • Introduction to how to do a usability test - please "speak aloud" when you are looking for something, etc

    Testing (30-45 minutes, depending on the tester)
    • Gedit (10-15 minutes)
    • Firefox (10-15 minutes)
    • Nautilus (10-15 minutes)
    For each program, start by giving the tester a set of written scenarios - one at a time, one scenario per page - and ask him/her to complete that scenario. Also, ask the tester if they have used this program before, or how they would expect to use the program. 
    When the tester has completed a scenario, move on to the next scenario: each scenario should be printed on a separate piece of paper. At the end of all scenarios for a program, take a short break before moving to the next program. This is where I do some basic Q & A by asking followup questions. Typical usability followup questions include: "What were you thinking when you tried to do X?" "You seemed lost when you were working on Y - what would have made it easier?" "What were you expecting to see when you were doing Z?" Also, a quick "plus/delta": what worked well, and what needs to be improved?

    Wrap-up (less than 5 minutes)
    • "Thank you for your time."
    • Provide them with a 1-page information sheet about the three programs, as a giveaway. (Testers may like to refer back to this afterwards, as a way to better connect with the test.)
    • Give parting gift, and send them on their way.

    Friday, November 2, 2012

    And then there were three

    Earlier this week, I wrote about the two candidates for good usability that I will use in my usability study. After some discussion, I've decided to add Firefox back on the list of programs to examine. This now means I'll do a usability study of three successful open source programs, instead of two.

    When I asked for suggestions of programs to include in the study, many folks recommended a web browser - specifically Chrome and Firefox. But I was concerned that the ubiquity of these programs on other platforms (Mac, Windows) could mean we wouldn't be testing the usability of these programs during the usability test, but rather how well the testers navigate these browsers on another platform.

    But there's a lot to be learned by exploring the usability of Firefox. Just because the usability testers might already be familiar with Firefox shouldn't be a reason to exclude a program that so many people recommended as a candidate for good usability. So it's in.

    As a reminder, the three programs that I will study:

    1. Gedit (a text editor)
    2. Nautilus (a file manager)
    3. Firefox (a web browser)

    We'll start usability testing next week. My next step is to craft scenarios for each, which we will use in the usability test.

    Tuesday, October 30, 2012

    Designing a usability test

    Another thing that came of my discussions with thought leaders is how to design a usability test. A few thoughts on that.

    When I first considered this project, I thought I could do an "open ended" field usability test, where I solicited users at large to submit to a usability test of a program. They would do this test on their own computers, according to a set of scenarios that I would design for them. I imagined that, while users would carry their own experiences with them, they would be able to execute the test by following a set of user personas for the program. Personas would describe general users for the program, including details that inform the tester of what the persona would be like.

    In discussing this option with thought leaders, I realize that is not the right way to do a test. There's a lot of value by doing an in person usability test. Bring the test subject into a lab (the lab can be formal or informal - even a desk in an office) and ask that person to execute a series of workflow tasks. For example, to test a word processor, we might ask testers to type a few short paragraphs of text (provided for them), start a new tab, search and replace text, print, change the font, and other typical tasks. The key is that these scenarios should demonstrate typical, average use of the program.

    The test scenarios do not (necessarily) need to exercise every function of the program, as long as it demonstrates how a typical user of average knowledge would use the program. Generally, those test results can be applied to other parts of the program, as well.

    In this study, I am interested in what makes for good usability in open source software. The usability test design for this need not be complicated: I will create a bootable USB "flash" drive that runs Fedora, a version of Linux. This flash drive can be easily re-written between tests, to restore the drive to a known, default state so the next usability tester has the same starting point as the first usability tester.

    The usability lab can be as simple as an office with a closed door (to prevent distractions). To start the test, I would provide some background information for the tester: why are we doing the test, what they will be expected to do (the test scenarios) and how they should act during the test (ignore the observer, please speak aloud what is going through your mind at each step, etc.) Usability testers should be drawn from a wide pool, including users that are similar to the target audience: general users. Since I work on a university, potential testers might include students, faculty, and staff. I may also draw on family and friends.

    After each test, I will ask the testers about their experience, and possibly explore areas that looked particularly interesting or challenging for them. What did they expect to see during this part of the test? What should this other screen have shown you, since you said you were doing X? To wrap up, engage the tester in a brief "plus/delta" exercise: What worked well in the program? What features were particularly easy to use? But what other features were more challenging to use? Where did you feel confused?

    The plus/delta will be important for the results. I will need to report both in the final publication, even though I intend to focus on the features that contributed to good usability, so that other open source developers might mimic those successful features in their own programs.

    Discussions with thought leaders

    Over the last few weeks, I've interviewed several thought leaders in usability and open source software. I'd like to take another opportunity to recognize Ginny Redish, Eric Raymond, Richard Stallman, Chad Fennell, Dave Rosen, Greg Laden, and others for their time, and for their input to my study.

    What did I learn?

    In general, all agree that open source software provides a unique challenge. To be sure, many programs (including "commercial" or "proprietary" programs) may view usability as a final phase effort - going through a usability study only at the end of a project, right before go-live, intending to "validate" design decisions made earlier. That's if they consider usability at all during the project.

    However, open source programmers tend not to think about usability. Open source programmers, to borrow and paraphrase from Raymond, tend to "scratch an itch," to create programs that are interesting to them or solve a particular problem. The functionality and features of the open source program are key. How the program looks is secondary.

    To put it in another context, writing an open source program is like baking a cake. The interesting part of the process is putting the ingredients together, mixing them, to assemble the cake. Once the cake (or open source program) is "baked," the look and feel is less important. Appearance is like putting the icing on the cake. It's window-dressing, and thus carries less value.

    But we also (generally) agree that usability is important. How a program is used and how it appears are both key items. Truly technically-minded folks may not mind that a program is difficult to use, as long as it does the job. (For example, many folks responded to my "what programs have good usability?" question by suggesting very specific programs, with the caveat that the program may be difficult to pick up, but once you figure out how to use the menus - or once you get past a particular, quirky design choice - the program is fairly straightforward.) However, the general user with average knowledge will not be able to accomplish tasks with the software if it is difficult to use. If menus are obscure, if the workflow is convoluted, or if other aspects of usability are poorly implemented, then these users will not adopt the program.

    Interestingly, thought leaders also suggest that there has been very little - if any - experiential study done on the usability of open source programs. This seems to be a largely unexplored area, save for a few high-profile projects (such as the Gnome desktop environment) where a key partner (such as Gnome's Sun Microsystems) invested effort to examine usability.

    With a few exceptions (Drupal is one), the open source community at large does not include usability in their development process. As mentioned earlier, these open source developers do not see the value in studying the usability of their programs. They have little interest - or indeed, little knowledge of how to go about it. Thought leaders (Raymond, etc) suggest that open source developers would be more likely to mimic successful usability from other programs, rather than apply the rigor of usability study to their own work.

    So that is the launching point for the rest of my project. I want to do a study of programs that demonstrate good usability and provide an analysis for why these programs "work" successfully. Of course, I've been advised by at least one thought leader that successful usability may not be very transportable to other programs - what works for one program may not be good usability in another. My results will need to carefully discuss what makes for good usability in each of the test candidates, and how this might be applied as "lessons learned" for open source developers to adopt - with the caveat to apply the lessons carefully to other projects.

    Monday, October 29, 2012

    Candidates for good usability

    I'd like to thank everyone who responded to my plea, asking what programs have good usability. This question was repeated elsewhere on other blogs and on twitter, and I got quite a lot of feedback. Together, you suggested almost 100 candidates for good usability in open source software. I've collated the results, and I'd like to share my results.

    I was looking for GUI programs, not command-line programs or programs that use text mode or "TUI" (i.e. full-screen text-mode programs you can run at a regular terminal). In the usability test, we'll ask participants to sit at a computer and run through several exercises that are typical for that program. For example, to test a word processor, we might ask testers to type a few short paragraphs of text (provided for them), start a new tab, search and replace text, print, change the font, and other typical workflow tasks.

    Picking the right open source program is a tricky thing. The ideal program should be not too big (for example, very complex menus can "lose" the audience in the details) but neither should it be too small (a trivial program will not provide as valuable of results). The program should be approachable by general users.

    Several folks suggested graphics suites such as Gimp or Inkscape. Both are fine graphics programs, and I personally agree that these seem to do a lot of things "right" in usability. However, I've decided to skip these types of programs, as they are intended for people in the graphics profession, and not the general user. Similarly, I decided to exclude disk image programs, PDF writers, operating system tools, and similar programs because they are too specific to one area, or too technical.

    So, what programs did I decide to include as candidates?

    Many folks listed Gedit as an open source program with good usability. At first, I assumed text editors would be too simple to include in a usability test. When I think of text editors, my mind jumps to trivial text editors such as Windows Notepad. However, Gedit is a great suggestion! It provides a powerful text editor with a simple interface. In fact, the simplicity of the interface belies the features contained in Gedit. To quote the website: Currently it features:


    • Full support for internationalized text (UTF-8)
    • Configurable syntax highlighting for various languages (C, C++, Java, HTML, XML, Python, Perl and many others)
    • Undo/Redo
    • Editing files from remote locations
    • File reverting
    • Print and print preview support
    • Clipboard support (cut/copy/paste)
    • Search and replace
    • Go to specific line
    • Auto indentation
    • Text wrapping
    • Line numbers
    • Right margin
    • Current line highlighting
    • Bracket matching
    • Backup files
    • Configurable fonts and colors
    • A complete online user manual


    In other words, Gedit successfully addresses the advanced user (programmers, etc.) and the general user. For more, view the screenshots.

    Ideally, I'd like to include more than one open source program in the usability test. Looking through the suggestions, possible program types include email clients, file managers, simple graphics ("paint") programs, music players, file viewers, and web browsers. There's a lot there. I am drawn to the web browsers (Chrome, Firefox) but I wonder if too many of our potential testers have used these same programs on other platforms (Windows, Mac) so will already be too familiar with them when they do the usability test. We won't really be testing the usability of these programs, but how well the user already navigates them on a different platform.

    In the interests of expanding the candidate pool, I will add the Nautilus file manager to the list. File management is often overlooked because it is such a "basic" part of a desktop operating system. However, the usability of this essential feature is key. It is also a different kind of program than Gedit, so would yield different results. Together, the usability tests for Gedit and Nautilus would provide hints to successful usability in open source programs.

    Tuesday, October 23, 2012

    What programs have good usability?

    I want to ask for your help in my study.

    For my study, I want to do a "deep dive" on usability in open source software. After speaking with several "thought leaders," my thinking now is that it's better to do a case study, a usability critical analysis on an open source software program that has good usability. The results will be a discussion about why that program has good usability, and what makes good usability, so that other open source programmers can mimic the good parts.

    I'll also discuss what features are not good usability examples, so programmers can avoid those mistakes. But the focus will be more on the good and less on the bad.

    Picking the right open source program is a tricky thing. The ideal program should be not too big (for example, very complex menus can "lose" the audience in the details) but neither should it be too small (a trivial program will not provide as valuable of results). The program should be approachable by general users.

    There's no reason the program needs to be a Linux program. However, I prefer that the case study be of an open source program. Many open source programs also exist for Windows and MacOSX.

    So, what open source program would you suggest for the case study? Leave your suggestions in the comments:

    Monday, October 22, 2012

    Making an impact

    Just wanted to share this item from the University of Minnesota Libraries, on research impact measures.

    However, my discussions with thought leaders lead me to think that Linux Journal would be an excellent place to share my results. LJ is a respected magazine in the open source community - and as the name implies, it is read primarily by Linux developers and users. While LJ doesn't need the same level of impact measures that, say, Science would require, it will help to review the research impact measures when writing about my results.

    Sunday, October 14, 2012

    More on critical analysis

    I wanted to share more about what I mean by a usability critical analysis. Who does that? What are the measures? What are the steps?

    By way of providing context, let me take a step back to another usability test I performed on the FreeDOS website earlier this spring:
    This was a complete redesign of the website, which had not previously undergone any kind of usability study. The old website was a hodgepodge of content by different editors, each with different agendas.

    To redesign the website, I started by creating user personas that represented our user base, and usage scenarios typical of those users. FreeDOS users typically fall into one of three types: casual users who want to use FreeDOS to play old DOS games, people who want to use FreeDOS to run a legacy application, and technical users who use FreeDOS in embedded systems. From that information base, I derived the new site by asking "what does each persona need to find?" and designing navigation and content areas that responded to those needs. Through a process of iteration, I arrived at a prototype of the new website, and invited the FreeDOS community to help test the new website.

    The usability evaluation was a typical prototype test: I asked each tester to exercise the new prototype as if they were one of the user personas, and according to the usage scenarios. Could the testers find the information they needed? At the end of the usability test, I asked the testers to respond to a formal questionnaire about using the new website. The questionnaire also included a section where testers identified what was working and what needed improvement with room to provide detailed suggestions.

    This first evaluation led to several improvements in the prototype, and another prototype test with questionnaire. Through repetition, I arrived at the FreeDOS website that you see today.

    The process of asking usability testers to evaluate a prototype and comment on it via a questionnaire was invaluable. In that usability evaluation, I was interested in both what was working and what needed improvement, or a plus-delta exercise without using the terms "plus" and "delta". The "plus" items helped me identify what features were good, and the "delta" items allowed me to focus on problem areas to improve for the next iteration of the prototype website.

    And frankly, the "delta" items would have been no use to me if I had not been interested in using that feedback to improve the website.
    Let's jump back to the topic of usability in open source software. Last week, I interviewed Eric S. Raymond (quotes below are from that discussion). He provided an interesting view into open source developers: the process of usability is antithetical to how open source software is created. Open source developers prefer functionality over appearance, and by extension put (typically) no emphasis on interface usability. While some open source projects have a maintainer with good taste who dictates that good taste on the interface, Raymond commented that most programmers view "Menus and icons are like the frosting on the cake after you've baked it" and that any results that try to provide correction to usability is "swimming against a strong cultural headwind."

    Our discussion left us both with the realization that if my study is to have a positive contribution to the open source community, it needs to "focus first on the good examples, rather than trying to fix the bad."

    Therefore, I cannot use diagnostic usability in my study. I had originally planned to do a case study on the usability of an open source software program. My unwritten assumption was that I would start with user personas and usage scenarios, working with the author of candidate program to understand the user base and their typical actions. And (I assumed) the study would generate output similar to my work on the FreeDOS web site: what is working and what needs improvement. I might even have created an animated, non-functioning prototype of an existing program, and ask usability testers to evaluate that prototype against the user scenarios.

    However, after my discussion with Raymond, I need to modify that assumption. Instead, I plan to study an open source program of a suitable size, one that was successful in addressing usability. The results of the case study wouldn't be a diagnostic analysis of the usability issues, but a summary of what works in open source usability: a usability critical analysis.

    The benefits to this kind of usability study are immediate to the open source community. In my experience, and that of Raymond, most open source programmers are more likely to imitate successful designs rather than apply the rigor of usability studies to their own programs. If my study is to benefit the open source community as a whole, I need to change how I approach the case study.

    That brings me back to the questions "Who does that?" and "What steps?"

    In this model, the usability testers can be any willing participant. When I tested the FreeDOS website, my testers were members of the open source software community - many directly from the FreeDOS Project. In this study, the usability testers can be anyone who is interested. They do not need to be members of that open source software project; they do not even need to be part of any open source software project. The minimum requirement for these critical analysis testers is a willingness to use the program. Some base level of familiarity with the program would be helpful, but in a usability test should not be required.

    The steps in this critical analysis would likely be the same that I applied in the FreeDOS website usability test: starting with user personas and scenarios, ask each tester to exercise the program according to those scenarios. At the end of the test, ask the testers to respond to a formal questionnaire about their experience, for each scenario in the test. But in this case, the focus of the questions would be what worked well rather than what needs improvement.

    And that leads directly to the output of the study: a publishable result that identifies what works well in software usability, in a format that allows other open source developers to mimic those successful aspects of the design in their own projects.

    Saturday, October 13, 2012

    Usability critical analysis

    In my other post, I shared a new kind of usability test that I might use in my study: the "Usability Analysis". Similar to a heuristic review, this other usability test would essentially be the first half of a "plus/delta" exercise, focused on what is working in the design rather than discussing what needs to be improved.

    I realize now that such a critique already has a name: the critical analysis. For example, a literary critical analysis is simply a discussion of a work in literature. The use of criticism doesn't imply disapproval or a negative review, but instead a full review of the work.

    A better term for my proposed usability study might be "Usability Critical Analysis."