Usability studies of open source software seem to be somewhat rare in academic literature. Most notably, David M. Nichols and Michael B. Twidale have authored several articles on the subject (for example, “Usability Processes in Open Source Projects”). Calum Benson, Matthias Müller-Prove, and Jiri Mzourek wrote similar reviews of usability tests in open source software (ACM: “Professional Usability in Open Source Projects: GNOME, OpenOffice.org, NetBeans”).
A 2009 PhD thesis by Paula M. Bach (College of Information Sciences and Technology, Pennsylvania State University) explored “Supporting the user experience in free/libre/open source software development.” Similarly, a 2011 bachelor’s thesis by Jan-Christoph Borchardt (information design, Stuttgart Media University) examined “Coordinating low-cost usability testing in independent free & open source software projects,” and is now an open document (CC BY-SA 3.0) summarizing “Usability in Free Software.”
Other research has typically been conducted informally, or sponsored by corporate interests. In a 2013 presentation at the GUADEC 2013 conference (GNOME Users’ And Developers’ European Conference, held annually in cities around Europe), Fabiana Simoes discussed issues surrounding the capture and reporting of UX issues in open source software.
I researched the literature surrounding usability studies of open source software. For a list of academic resources on this topic, see my previous comment listing some 8 academic articles on the usability of open source software. These are the most-often-referenced resources. While this is not a complete list of everything out there, we can assume a literature review is fairly complete with you start to get significant overlap: when multiple sources keep references the same other sources.
The names that keep coming up most often in this field (usability in open source software) are Benson et al, Frishberg et al, and Nichols & Twidale.
But doing a study of the usability of open source software is a pretty big field; open source software covers a lot of programs. You can't test everything. I see two ways forward: focus the study, so you investigate the usability of a specific open source software program, or you study a system of programs to discover common themes in the usability of open source software.
In a previous study, I examined the usability of several common applications, including GNOME's gedit, GNOME's file manager (Nautilus) and the Firefox web browser. In my new usability study, I would like to examine programs that let me expand the analysis - but at the same time, provide a deeper analysis of my previous results. That means including the same or similar programs in this study that I investigated last time.
So for this test, I'm keeping my focus on GNOME. And as suggested by Andersen (2013), academia and practice need to develop a reciprocal relationship. It's a cycle; academia needs to provide accessible, actionable research that is published in places visited by practice. That allows the practice to advance, which provides future research opportunity for academia. The cycle continues, and both academia and practice benefit. That's why I'm sticking with GNOME applications for this usability study: doing a usability test on several GNOME applications provides a reciprocal benefit to academia (my work) and practice (GNOME).