This comment from user drag caught my attention. It says, in part:
Studies of this type are very expensive and time consuming. You have to hire or have people on your team that are experts in software usability.And:
Then when you do your testing you have to get people that are not that familiar with the environment and then observe them as they interact with the UI and record their observations as they go through various tasks.
This is not easy and it's not cheap. No open source desktop has the resources, as far as I know, to actually sustain this sort of testing.
Here is the claim I will actually make though:Both statements are wrong.
Programmers who develop software are one of the least capable people when it comes to designing UI for the software they use.
Firstly because they lack the expertise… programmers tend to be a 'foot wide, mile deep' when it comes to technical knowledge. They know a huge amount about the specific things they work on. Unfortunately when it comes to things outside their specific type of software they work on they tend to be a bit clueless. Not that they are stupid or shortsided, [sic] that doesn't have anything to do with it.
And, secondly and far more importantly, they have a huge number of expectations and assumptions how people are going to use their software, as well as deep intimate knowledge about the software they are writing. This makes it extremely difficult, if not impossible, to step back and see the software from the perspective of a person that doesn't give a shit about the software, but just wants to accomplish something specific.
This is one of the biggest and most difficult problems that has plagued open source desktops from the beginning. The programmers who devote their time is fantastic and so on and so forth, but at a certain point you really need to pay money to hire experts.
And certainly usability is not decided through consensus.
As I discuss in my capstone "Usability Themes in Open Source Software" and in my presentation at GUADEC, usability testing does not need to be done by experts. Usability tests don't need to be highly formalized, or conducted in a lab by professionals. You can learn a lot just by setting up a simple usability test, then watching actual people use the software. With each iteration, usability testing identifies a number of issues to resolve and uncovers additional issues that, when addressed, will further improve the program’s ease of use.
Usability testing does not rely on a single method, however. There are multiple approaches to implement usability practices, from interviews and focus groups to formal usability testing. Whatever method you use, the value of usability testing lies in performing the evaluation during development, not after the program enters functional testing when user interface changes become more difficult to implement. In open source software development, the community of developers must apply usability testing iteratively throughout development. Developers do not require extensive usability experience to apply these usability practices in open source software.
The value of a usability test is to find areas where the program is difficult to use, so developers can improve the interface in the next version. The "hot spots" in the heat map show tasks that were difficult for testers, and which need attention from the designers or developers.