There's definitely an "art" to creating good scenario tasks. As Renata, Diana and Ciarrai wrote in their blogs this week, you need to be careful not to (accidentally) provide hints for how to complete the task. You need to balance enough information to make the scenario task clear, but not so much that the tester can only complete the task in one particular way. There are often multiple ways to do things in software. Let your testers find the way that works for them.
Diana wrote: "Phrase your scenario tasks without uncommon or unique words that appear on the screen. If you do, you turn the tasks into a simple game of word-finding."
Ciarrai wrote: "Language that doesn’t just mimic that which is used in the program will help avoid leading the user in performing the tasks. Testing whether the user can achieve the goal without direct word association can make a usability test more fruitful."
Renata wrote: "Avoid statements that may end up giving too much information. Instead of saying click“this”, you should leave the participant finish the task in order to see if that feature is intuitive. Also do not force the user to execute that task in a certain way, there are many ways to accomplish a task so let the user’s choose their way to use the interface."
In their research, Renata, Diana and Ciarrai also talked about how scenario tasks need to be realistic. There's no point in asking a tester to do something that a real person wouldn't do. That's why it was important to understand scenarios before we researched scenario tasks. Understanding how and why a real person would use the software to do real tasks is invaluable in creating realistic scenario tasks.
And scenario tasks need to be written using the language that your testers would normally use. Avoid using very technical words if your users wouldn't be technical. You might use technical words and phrases if you were building a usability test for a programmer's IDE and Debugger, but you wouldn't use technical words and phrases for a general desktop environment like GNOME. It's all about finding the right balance and "voice" in your scenario tasks.
If you haven't read their blog posts this week, you should read Diana's post for the Dilbert cartoon she included. This one reminds me of a quote from The Hitchhiker’s Guide to the Galaxy (radio program, TV mini-series, then books). One of the characters says this after stealing a spaceship that has a rather monochromatic color scheme:
‘It’s the wild colour scheme that freaks me out,’ said Zaphod, whose love affair with the ship had lasted almost three minutes into the flight. ‘Every time you try and operate these weird black controls that are labeled in black on a black background, a little black light lights up in black to let you know you’ve done it.’While the black-on-black color scheme might look impressive, the interface will be invisible in normal light (from Dilbert).
~Zaphod Beeblebrox, The Restaurant at the End of the Universe