刚刚传来的最新消息称，格鲁吉亚已经战败，俄罗斯全面控制了南奥赛梯地区，双方将开始外交谈判。两国在奥运会开幕几小时后发生了这场战争，完全违反了奥林匹克停火精神，着实让世人遗憾。初步估计已经有2000平民伤亡。
奥运会不管在哪里举行，都应该履行3000多年来的神圣使命：成为世界和平的象征。今日射击场上俄罗斯运动员和格鲁吉亚运动员相互拥抱亲吻，才是真正奥林匹克精神的体现，这是政客们永远不去理解，也不用去理解，更不该去理解的东西。这就是人类黑暗社会的特征。
RVL
2008年8月4日星期一
Software usability testing
专业写作一篇，博客存档
关键词: software testing, usability testing, usability verification
RVL
----
SOFTWARE USABILITY TESTING
1. Introduction:
As the standard for modern software testing is more and more advanced, the requirements for all aspects of software testing, both horizontally and longitudinally, are also raising. That’s why we often say that the potential of software testing development is bonded with general software market. Every single aspect in software testing is important. In such a fierce competition environment, customers won’t consider much about the detail source of the bugs in your product, 2-3 serious bugs are enough for them to turn to other products and leave you alone, though your product may be perfect in other 99.9% aspects. Thus, what the software designers should do, is to avoid these negative effects before the product is released. “0% bugs” is impossible, of course, but “0% fatal bugs” is basically possible. Vulgarly speaking, after a company released their “version 1.0”, how could they convince their customers again and again by telling “our version 1.01 would have no bugs”, “our version 1.02 would have no bugs”… ?
If everything is done before version 1.0, that’s the deal. But the detail tasks of software testing are not so easy to describe. Among those systematically connected aspects, software testing engineers would follow the schedule to finish every part. But there is one part , which starts from the beginning, and finish in the very end, being done time by time. Besides, software engineers are only supporting roles in this part, the protagonists are the customers, which may even not know the difference between .net and java, or the concept of “exception”.
So that comes out Software Usability Testing. What‘s the meaning? What’s the object of this testing? How?... That’s what I would talk about in this article.
2. Basic Concepts of Software Usability testing
Software usability testing is a complex concept. In order to summarize it out, It is necessary to introduce the two basic component of this concept at first: Software Usability, Usability Testing.
In the wikipedia, the definition of usability is as following:
Usability is a qualitative attribute that assesses how easy user interfaces are to use. The word "usability" also refers to methods for improving ease-of-use during the design process.
-- Wikipedia: Usability
To deepen this concept to the software designing field, Usability could also be defined as “Usability measures the quality of a user's experience when interacting with a product or system—whether a Web site, a software application, mobile technology, or any user-operated device.[1]” . This definition is more accurate for the Usability in software field.
And what’s Usability Testing? Usability testing is a technique used to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practise, since it gives direct input on how real users use the system [2]. And according wikipedia, “this is in contrast with usability inspection methods where experts use different methods to evaluate a user interface without involving users. – Wikipedia : usability testing”
Combining these concepts and the general idea of software testing, I could define Software Usability Testing as : “A systematic series of testing activities to evaluate the product’s quality on user’s experiences during interaction with this product. ”
3. Software Usability
3.1 Definitions and assortment of software usability
So what’s the standard to judge if a software product has GOOD usability? Or not so good? Of course, Software Usability does contain several parameters to evaluate it. Two experts on usability research, Jakob Nielsen and Ben Shneiderman have both agreed that there are five attributes in usability in their books separately. Their view is also the most authoritative and is recorded in wikipedia:
Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
Efficiency: Once users have learned the design, how quickly can they perform tasks?
Memorability: When users return to the design after a period of not using it, how easily can they re establish proficiency?
Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
Satisfaction: How pleasant is it to use the design?
Some other experts have different opinion, another popular assortment is “Easy to learn, Easy to use, Easy to satisfy” [3]. But if the topic is about testing, the characters of different attributes should be more clear in order to discuss. So this article will mainly based on the assortment of wikipedia.
3.2 Examples on software usability attribute
Back to the five attributes of software usability, as these concepts may seem too generalized, it would be easier to understand these concepts by several examples. Let’s suppose that a software company has been designing a product “soccer players stats analysis system (SPSAS)” for a soccer club “FC San Jose (FCSJ)”. So how to make the roles of these five attributes clear in this case?
Learnability: The users of SPSAS are the coaches and data analyzers of FCSJ. Maybe the data analyzers , who have professional computer backround, would be easy to handle this system on computers. But for those coaches, especially for the 62 years old head coach, Rex, who even have many difficulties on learning to send or receive emails, how could he deal with this system? Will it be easy to learn for Rex in his limited spare time?
Efficiency: Now let’s suppose that every necessary person in FCSJ has acquainted with this system. And during the pre-season training camp, the coaches need to acquire every player’s latest physical status as soon as possible, in order to work out the detail plan of pre-season training. So how could they analyze the data gathered from examinations and first trainings? And more important in this factor, how fast could they finish a well organized and systematic report on all players?
Memorability: And during the season, whenever the coaches want to use this system to evaluate a player’s recent performance, or working out a new short-term plan, they need to remember the basic functions and controls of this system, or at least easy to recollect.
Errors : During operating this system SPSAS, if there are bugs on analyzing players data, the coaches may have problem on fixing the problems and undo all works. To be even worse, if Rex selected a player with bad form according to the wrong data, and the player became a nightmare on the pitch, or injured. Afterwards if the coaches found out why, the software company will be in big trouble. So could this product avoid these bad situations?
Satisfaction : this seems to be more comprehensive than the four ones above. Will the FCSJ coaches want to use this system to help their work in the next season? Does Rex feel convenient? Are the results better than before? It’s very easy to summarize these questions : Are they satisfied with SPSAS?
3.3 Usability patterns and properties (arch-patterns)
As if we refer these software usability attributes during examination and further testing, it would be too vague and general. In fact, during practicing, there are still two levels below the usability attribute level: property level, and pattern level. Connecting with these two levels, the details of software usability would be shown to users, who are the main participant during software usability testing (which will be introduced later).
Usability properties are defined as the usability characteristics to be improved in a system. Usability properties can also be seen as the requirements of a software system for it to be usable (for example, provide feedback to the user, provide explicit user control, provide guidance to the user, etc). [4]
The second level was envisaged to identify specific mechanisms that might be incorporated into a software architecture to improve the usability of the final system. These mechanisms have been called usability patterns and they address some need specified by a usability property. Note that usability patterns do not provide any specific software solution to be incorporated into a software architecture, they just suggest some abstract mechanism that might be used to improve usability (for example, undos, alerts, command aggregations, wizards, etc.).[5]
As the table shows, the usability patterns are in charge for their parts of improving work in the application interface. And the relationships between these components and the property items are to provide solutions and mechanism to this software architecture. And during usability testing , these connections and functions would be more clear.
4. Software usability testing -- Goals
After introducing the knowledge structure of software usability, we can start the main topic of this paper : software usability Testing.
In the general assortment of software testing, software usability testing belongs to Quality assurance., as usability is one of the six important attributes of software quality. In the ISO 9126 standard, it provides six high-level quality attributes: functionality, reliability, efficiency, usability.[6]
4.1 Who are in charge for software usability testing (SUT)?
Basically, many experts consider users as the most effective testers on usability testing. Because whether the testing specialists have great level or not, whether their testing methods and technologies are advanced or not, they are not the ones who will use this product for long time. The most important factor in usability is user’s satisfaction. Thus, users should lead the SUT and do most of the work.
Is there any filter to these users? Of course. The company should select suited users to finish usability testing. They should be comprehensive, representative, and neutral with the company. The company should not cheat themselves with selecting users they “desire” to get a good result. If so, in the end the biggest loser would be the company itself, especially the testing department.
And besides users, testing specialists still should participate in SUT as important assistants. Users should do the testing under their planned direction in order to get the data they need. In some testing methods (such as monitor), testing specialists should stay with the users in the whole process. The details are on later chapters.
In order to filter the testers more effectively, the company should bring these questions while facing the testers.
n Are these testers able to complete the task scenarios successfully?
Keywords: basic ability, time.
n How fast would these testers finish each task?
Keywords: experience.
n How many clicks does it take to complete each task?
Keywords: testing plan.
n Do these testers perform well enough to meet the original objectives?
Keywords: evaluation
n How satisfied are the testers (users only) with the product?
Keywords: satisfaction
n What changes are needed to make the product to enable more users performing more successfully?
Keywords: improvement.
4.2 Objectives and Major focuses of software usability testing
In wikipedia, the “usability goals” defines these following objectives of usability testing :
“During usability testing, the aim is to observe people using the product to discover errors and areas of improvement. Usability testing generally involves measuring how well test subjects respond in four areas: efficiency, accuracy, recall, and emotional response. The results of the first test can be treated as a baseline or control measurement; all subsequent tests can then be compared to the baseline to indicate improvement.
Performance -- How much time, and how many steps, are required for people to complete basic tasks? (For example, find something to buy, create a new account, and order the item.)
Accuracy -- How many mistakes did people make? (And were they fatal or recoverable with the right information?)
Recall -- How much does the person remember afterwards or after periods of non-use?
Emotional response -- How does the person feel about the tasks completed? Is the person confident, stressed? Would the user recommend this system to a friend?”
And in software products, the usability testing is even more important than the ones in other fields. As software engineers need to use usability testing to find out the problems in their product and relative service as soon as possible, and if any phase of the product has usability problems, the whole product would be seriously affected.
During the software usability testing, all the testers, including the users who have been selected for SUT, and the leading testing engineers, will try to give out suggestions to improve the product in limited schedule and fund. This purpose is also the final objective for other testing phases in the whole design process.
For example, if the company is testing a web-based product (a web-based platform, or a commercial website, etc..)And during the process, testers would mainly focuses on following questions :
n Do testers click to pages or do they use search?
n What keywords do they use most when searching?
n Is the search textbox in a good location?
n Is the search textbox large enough for most of the words used?
n Do the search results provide leads to quick answers to the questions from users?
n If so, are these answers usually on the first page of results?
n Does the search do a good job of detecting and helping to resolve typing errors?
n Are the testers (users only) satisfied with the search engine in all?
4.3 When?
When will be fittest time to do the usability testing? The answer doesn’t contain any accurate time. The software usability testing is a process, starting in the very beginning of the design. As since started, designers need to collect suggestions from users to help them fix the goals and improve the design. During the whole designing process, this process is combined with the main process and executed time by time. Of course, it is still not the real process of software usability testing, but important foundation preparing for software usability testing. But generally, the process of collecting suggestions and opinions from users is the first phase in the SUT process.
Even a non-expert on software designing would know that designing a software product would be obviously related with the opinions of users. But they may don’t know the opinions are collected from different stages. If there are only suggestions around the end of designing, the product would definitely fail if they have been away from the users request since the beginning, as the relationship between the improvement the users could affect and the time used is inverse proportion.
So when should the testers do usability testing? To summarize it in one sentence: Test early; test often . Usability testing lets the design and development teams identify problems before they get "set in concrete." The earlier those problems are found and fixed, the less expensive the fixes are. As the project progresses, it becomes more and more difficult and expensive to make major design changes. The more you test and change based on what you learn, the more confident you can be that the site will meet your objectives and your users' needs when it is launched. [7]
5. Software usability verification methods
As it’s said in the above that survey (collecting suggestions from users) is important preparation for software usability testing, here comes a new question: what’s the real first step of software usability testing?
If we consider everything related with SUT as a whole process, the first phase is survey in the beginning (which will be executed also in next phases time by time), the second phase is verification, and the third phase is evaluation.
Now it’s time to introduce the basic knowledge of verification.
5.1 Concepts
After collecting results from the survey results analysis , requirements analysis, and brief evaluation of potential use-related hazard, the usability testers should develop usability specification which specifies the intended use and purpose of the product. The usability specification should be integrated into the design specification and provides a basis for the usability verification and validation. [8]
The usability verification checks the conformity of design results with the usability specifications. As soon as usability specification is finished, usability verification takes place, and provides an early usability evaluation. A lot of usability engineering techniques that involve experts or users could be applied for usability verification. Verification should be conducted with sufficient efforts in early design stages to avoid serious usability problems being brought up in later design stages.
Different levels of the design could be verified by using suitable usability engineering techniques. The conceptual ideas can be verified by checking descriptive material such as scribbles or drawings, task descriptions and interface mock-ups. Functional models and prototypes could be analyzed in a more profound and advanced method.
And about usability validation, it would take place in the final stage of the whole process, as it needs at least one functional prototype to use. The objective of usability validation is also to analyze if the design really fulfils the requirements of the specified applications. If sufficient usability verification efforts have been executed, the serious usability problems to be discovered by the final usability validation could be minor.[9]
Some methods consider verification as a part of testing methods, and define it as verification testing : a verification test is used to determine the overall usability of a product after implementation has taken place.
5.2 Criteria
As software usability verification is also a kind of verification activity, there is also a series of criteria for software usability. Both verification phases and evaluation phases will be authorized only if their usability testing is based on the criteria.
Now generally speaking, the most authorized criteria on usability is ISO 9241. The following passage is brief introduction of ISO 9241 on wikipedia.
“ISO 9241 is a multi-part standard covering a number of aspects for people working with computers. Although originally titled Ergonomic requirements for office work with visual display terminals (VDTs) it is being retitled to the more generic Ergonomics of Human System Interaction by ISO. As part of this change, ISO is renumbering the standard so that it can include many more topics. The first part to be renumbered was part 10 (now renumbered to part 110).
“Part 1 is a general introduction to the rest of the standard. Part 2 addresses task design for working with computer systems. Parts 3–9 deal with physical characteristics of computer equipment. Parts 110 and parts 11–19 deal with usability aspects of software, including Part 110 (a general set of usability heuristics for the design of different types of dialogue) and Part 11 (general guidance on the specification and measurement of usability).
“The 17 parts of the standard are:
Part 1: General introduction
Part 2: Guidance on task requirements
Part 3: Visual display requirements
Part 4: Keyboard requirements
Part 5: Workstation layout and postural requirements
Part 6: Guidance on the work environment
Part 7: Requirements for display with reflections
Part 8: Requirements for displayed colors
Part 9: Requirements for non-keyboard input devices
Part 10: Dialogue principles
Part 11: Guidance on usability
Part 12: Presentation of information
Part 13: User guidance
Part 14: Menu dialogues
Part 15: Command dialogues
Part 16: Direct manipulation dialogues
Part 17: Form filling dialogues ”
As it introduces, in ISO9241, from part 10 (now renamed to 110) to part 17 are the detail criteria of usability in ISO standard. Among them, part 110 and part 11 are the two most cited parts in the articles on usability. Part 110 deals with general ergonomic principles which apply to the design of dialogues between humans and information systems: suitability for the task, suitability for learning, suitability for individualization, conformity with user expectations, self descriptiveness, controllability, and error tolerance. And part 11 deals with detailed specifications on usability, including standards on effectiveness, efficiency and satisfaction. In the end of part 11, there are also some appendices on practical examples on background and usability testing, as well as differences with other criteria (such as ISO 9126, which also has a definition of usability)
Part 12- 17’s focuses are listed below: (from wikipedia)
ISO 9241-12
(1998) Presentation of information This part contains specific recommendations for presenting and representing information on visual displays. It includes guidance on ways of representing complex information using alphanumeric and graphical/symbolic codes, screen layout, and design as well as the use of windows.
ISO 9241-13
(1998) User guidance. This part provides recommendations for the design and evaluation of user guidance attributes of software user interfaces including Prompts, Feedback, Status, On-line Help and Error Management.
ISO 9241-14
(1997)Menu dialogues. This part provides recommendations for the ergonomic design of menus used in user-computer dialogues. The recommendations cover menu structure, navigation, option selection and execution, and menu presentation (by various techniques including windowing, panels, buttons, fields, etc.).
ISO 9241-15
(1998) Command language dialogues. This part provides recommendations for the ergonomic design of command languages used in user-computer dialogues. The recommendations cover command language structure and syntax, command representations, input and output considerations, and feedback and help.
ISO 9241-16
(1999) Direct manipulation dialogues. This part provides recommendations for the ergonomic design of direct manipulation dialogues, and includes the manipulation of objects, and the design of metaphors, objects and attributes. It covers those aspects of Graphical User Interfaces that are directly manipulated, and not covered by other parts of ISO 9241.
ISO 9241-17
(1998) Form-filling dialogues. This part provides recommendations for the ergonomic design of form filling dialogues. The recommendations cover form structure and output considerations, input considerations, and form navigation.
Actually design and process standards are different from these technical criteria. Technical standards tend to be written for a specific technology and to be very detailed in how to create a product that meets the standard. (such as ISO 9241-12 ~ 17).
" Process standards tell us how to work, which methods to use, and how to integrate with other processes. They describe methods and outline what should be done, in what order, to succeed in a project.
" Design standards set rules for good design. Corporate rules for the use of a logo are design standards, as are guidelines for use of color, layout, navigation or interaction.
Both are difficult to both write and enforce. Even when there is general agreement on a design principle, the diversity of industry practice and content has made it difficult to create strong standards to embody usability knowledge. Both have to find the line between vague principles and excessive detail. [10]
6. Software usability testing methods
6.1 Scope and definition
As the standard for usability verification is included in the criteria for the whole usability related activities, it is necessary to make clear the scope and basic definition for usability in this series of criteria. As there are many parts dealing with usability in ISO 9241, I choose part 11 as an example.
6.1.1 scope
ISO 9241-11 defines usability and explains how to identify the information which is necessary to take into account when specifying or evaluating usability of a visual display terminal in terms of measures of user performance and satisfaction. Guidance is given on how to describe the context of use of the product (hardware, software or service) and the relevant measures of usability in an explicit way. The guidance is given in the form of general principles and techniques, rather than in the form of requirements to use specific methods.
The guidance in ISO 9241-11 can be used in procurement, design, development, evaluation, and communication of information about usability .ISO 9241-11 includes guidance on how the usability of a product can be specified and evaluated. It applies both to products intended for general application and products being acquired for or being developed within a specific organization.
ISO 9241-11 also explains how measures of user performance and satisfaction can be used to measure how any component of a work system affects the whole work system in use.
The guidance includes procedures for measuring usability but does not detail all the activities to be undertaken. Specification of detailed user-based methods of measurement is beyond the scope of ISO 9241-11, but further information can be found in Annex B and the bibliography in Annex E.
ISO 9241-11 applies to office work with visual display terminals. It can also apply in other situations where a user is interacting with a product to achieve goals. ISO 9241 parts 12 to 17 provide conditional recommendations which are applicable in specific contexts of use. he guidance in this Part of ISO 9241 can be used in conjunction with ISO 9241 Parts 12 to 17 in order to help identify the applicability of individual recommendations.
ISO 9241-11 focuses on usability and does not provide comprehensive coverage of all objectives of ergonomic design referred to in ISO 6385. However, design for usability will contribute positively to ergonomic objectives, such as the reduction of possible adverse effects of use on human health, safety and performance.
ISO 9241-11 does not cover the processes of system development. Human-centered design processes for interactive systems are described in ISO 13407. [11]
6.1.2 Definitions
For the purposes of this part of ISO 9241, the following definitions apply:
Usability: Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. NOTE- See Annex D for other approaches to usability.
Effectiveness: Accuracy and completeness with which users achieve specified goals.
Efficiency: Resources expended in relation to the accuracy and completeness with which users achieve goals.
Satisfaction: Freedom from discomfort, and positive attitudes towards the use of the product.
Context of use: Users, tasks, equipment (hardware, software and materials), and the physical and social environments in which a product is used.
Work system: System, consisting of users, equipment, tasks and a physical and social environment, for the purpose of achieving particular goals. ( The context of use consists of those components of the work system which are treated as given when specifying or measuring usability. )
User: Person who interacts with the product.
Goal: Intended outcome.
Task: Activities required to achieve a goal. ( These activities can be physical or cognitive. Job responsibilities can determine goals and tasks. )
Product: Part of the equipment (hardware, software and materials) for which usability is to be specified or evaluated.
Measure (noun): Value resulting from measurement and the process used to obtain that value. [12]
6.2 Methods and techniques
Usability testing can be performed with developers, HCI experts, or representative end users. Some authors distinguish between "testing," which they limit to empirical end-user oriented methods, and "evaluation," which utilizes HCI professionals' expertise.[13]
6.2.1 Peer to peer testing
In this kind of software usability testing, there are a tester (presider or his/her assistant) and a target user. The target user will finish a series of tasks accompanied with the tester. This process will be shot if the target user agrees. The tester will observe the target user continuously, in order to acquaint with the user’s operations , thoughts and relative parameters such as mistakes, time cost, clicks per task…The tester needs to follow these actions , record the problems the user have faced and analyze them.
6.2.2 Heuristics evaluation
Heuristic evaluation is a usability testing method to find the usability problems in user interface design, in order to mark these problems in next stages to get improved. This method requires more users, according to Nielson’s theory, five evaluators would be enough to find out 75% usability problems. This would be the ideal number of target users under the proportion of market value and remaining balance.
So generally, heuristics evaluation will have 5 users (at most 8 users) to help the testers evaluating the human-computer interaction, find out the problems and suggest improvement plans under the usability criteria.
6.2.3 focus groups
This method is basically considered as an assistant method for usability testing. 6 – 12 people, including experts on this field, designers and users, group together and have a creative and active meeting. Under the presider’s direction, the group will discuss on specific topic and motivate each other for useful thoughts. There should be also at lease one staff to record useful messages and generate a meeting report.
6.2.4 User models
User models method is to simulate the human – computer interaction process, which is considered as the key solutions to the problems. Human is supposed to have their purpose while using software system, and a large purpose could be separated into several small purposes. To finish one small purpose, users have many different choices on actions and methods to take. Thus, each small process could cost some time which could be calculated, and the sum of these time costs, is the model time total cost. Then tester could foresee the total time the user could cost.
This method is very useful for some special products which could not invite users to attend usability testing (such as dangerous product or highly precise products). The most famous model in human computer interaction field is GOMS (Goals, Operators, Methods, Selections).
GOMS reduces a user's interaction with a computer to its elementary actions (these actions can be physical, cognitive or perceptual). Using these elementary actions as a framework an interface can be studied. There are several different GOMS variations which allow for different aspects of an interface to be accurately studied and predicted.
For all of the variants, the definitions of the major concepts are the same. Goals are what the user intends to accomplish. Operators are actions that are performed to get to the goal. Methods are sequences of operators that accomplish a goal. There can be more than one method available to accomplish a single goal, if this is the case then selection rules are used to describe when a user would select a certain method over the others. Selection rules are often ignored in typical GOMS analyses. There is some flexibility for the designers/analysts definition of all of these entities. For instance, one person's operator may be another’s goal. The level of granularity is adjusted to capture what the particular evaluator is examining. [14]
6.2.5 Survey
In chapter 4, I introduced the basic concepts of survey and when to do it , how to do it. In this chapter I will focus on its methods. Survey includes paper surveys and conversation surveys.
6.2.6 Thinking Aloud Protocol
Think aloud protocol is a method used to gather data in usability testing in product design and development, in psychology and a range of social sciences. The think-aloud method was developed by Clayton Lewis while he was at IBM, and is explained in C. Lewis and J. Rieman, "Task-Centered User Interface Design: A Practical Introduction".[15] The purpose of this method is to make explicit what is implicitly present in subjects who are able to perform a specific task.
Think aloud protocol includes a series of specified tasks for participants. Among them, the users are asked to tell whatever they are thinking, doing, looking at, and feeling during their tasks. And the testers, who play the roles “observers” in this method, would be able to see the process of task completion as early as possible. The test sessions are often audio and video taped so that developers can review what the participants did and reacted. This method could be used in any stage of development. So it’s a cheap way to get a lot of good qualitative feedback during testing.
6.2.7 Co-Discovery Method
In Co-discovery method , there would be two participants who would attempt to perform tasks together under observation. Thus, it would be very useful for testers to observe their interaction, as they could bring out more insights than a single participant does.
During this performance, tester should provide the product and a scenario of tasks to perform to the two users under observation. They should be asked to perform the tasks using the product and explain what they are thinking about while working with different parts of the interface. The tester should direct the two users to help each other in the same manner and working together to accomplish a common goal.
This technique can be used during any phase of development. It is ideal for Computer-Supported Cooperative Work (CSCW) products, groupware, and other products designed to be used by workers in team environments.
Think aloud protocol is a method used to gather data in usability testing in product design and development, in psychology and a range of social sciences. The think-aloud method was developed by Clayton Lewis while he was at IBM, and is explained in C. Lewis and J. Rieman, "Task-Centered User Interface Design: A Practical Introduction".[15] The purpose of this method is to make explicit what is implicitly present in subjects who are able to perform a specific task.
Think aloud protocol includes a series of specified tasks for participants. Among them, the users are asked to tell whatever they are thinking, doing, looking at, and feeling during their tasks. And the testers, who play the roles “observers” in this method, would be able to see the process of task completion as early as possible. The test sessions are often audio and video taped so that developers can review what the participants did and reacted. This method could be used in any stage of development. So it’s a cheap way to get a lot of good qualitative feedback during testing.
6.2.7 Co-Discovery Method
In Co-discovery method , there would be two participants who would attempt to perform tasks together under observation. Thus, it would be very useful for testers to observe their interaction, as they could bring out more insights than a single participant does.
During this performance, tester should provide the product and a scenario of tasks to perform to the two users under observation. They should be asked to perform the tasks using the product and explain what they are thinking about while working with different parts of the interface. The tester should direct the two users to help each other in the same manner and working together to accomplish a common goal.
This technique can be used during any phase of development. It is ideal for Computer-Supported Cooperative Work (CSCW) products, groupware, and other products designed to be used by workers in team environments.
Eye tracking is a complex usability testing technique which allows testers to identify what participants look at during the course of a usability test. It involves several kinds of equipments and different technologies. Some known technologies are marked contact lenses, image processing cameras, skin electrodes and reflector trackers.
The reflector trackers are most popular and useful in eye tracking technique. As it doesn’t require physical contact with the user’s eye , more users would be glad to accept to participate the test. It uses a beam of light to project onto the eye. a sophisticated camera picks up the difference between the pupil refection and known reference points to determine what the user is looking at.
But the eye tracking system equipment is too expensive to be popular in small companies, so a usability lab that purchased eye tracker would be helpful for those companies who cannot afford the maintenance.
Use this technique when you absolutely have to identify what a person looks at during a usability test. For most products, regular inspection or testing methods will identify enough usability problems to the point that eye tracking isn't necessary.
The reflector trackers are most popular and useful in eye tracking technique. As it doesn’t require physical contact with the user’s eye , more users would be glad to accept to participate the test. It uses a beam of light to project onto the eye. a sophisticated camera picks up the difference between the pupil refection and known reference points to determine what the user is looking at.
But the eye tracking system equipment is too expensive to be popular in small companies, so a usability lab that purchased eye tracker would be helpful for those companies who cannot afford the maintenance.
Use this technique when you absolutely have to identify what a person looks at during a usability test. For most products, regular inspection or testing methods will identify enough usability problems to the point that eye tracking isn't necessary.
6.3 Other concepts
6.3.1 Usability latitudes
Usability latitudes refers to a set of parameters in usability testing, including following ones :
n Probability of success of task operation.
n Task operation efficiency
n Task operation expectation (user)
n User estimate after task operation
n User satisfaction
n Probability of failure in each task
n Probability of success of task operation after repeat operation again
[19]
6.3.2 Usability documents
Here is some documents that could be used in software usability testing.
n User screener
n Testing schedule
n User profile
n User concert form & DNA
n Testing script
n Pre-test questionnaire
n Post-test questionnaire
n Task cards
n Checklists
n Reports
n DVD
[20]
7. Software usability testing organizations
Basically, most of usability testing organizations are very famous for their software usability testing techniques and authority. Here are some brief introductions of these organizations.
Usability Net - http://www.usabilitynet.org/home.htm
Long history, many basic knowledge on the website, good for new usability testers. Need proxy in China mainland.
Usability News – http://www.usabilitynews.com/UsabilityNews offers a facility for the publication of news articles, jobs, events and calls for papers for users involved within the HCI and usability community. Frequently update.
HFES-Human Factors and Ergonomics Society - http://www.hfes.org/ A professional human-computer engineering organization locates on UK. The Society's mission is to promote the discovery and exchange of knowledge concerning the characteristics of human beings that are applicable to the design of systems and devices of all kinds.
The Society furthers serious consideration of knowledge about the assignment of appropriate functions for humans and machines, whether people serve as operators, maintainers, or users in the system. And, it advocates systematic use of such knowledge to achieve compatibility in the design of interactive systems of people, machines, and environments to ensure their effectiveness, safety, and ease of performance.
The Society was founded in 1957. It has 23 technical groups and numerous local and student chapters. See our history page for a discussion of the Society's growth and development.
ACM SIGCHI - http://www.acm.org/sigchi/
ACM SIGCHI, the ACM's Special Interest Group on Computer-Human Interaction, brings together people working on the design, evaluation, implementation, and study of interactive computing systems for human use. ACM SIGCHI provides an international, interdisciplinary forum for the exchange of ideas about the field of human-computer interaction (HCI).
Usability.gov http://usability.gov/
The website maintained by the health and human service department of USA. Also includes some expertise information on usability.
Usability.gov is the primary government source for information on usability and user-centered design. It provides guidance and tools on how to make Web sites and other communication systems more usable and useful. Topics include:
· a step-by-step guide on how to plan, analyze, design, and usability test a highly usable Web site;
· quick access to the latest usability methods;
· an introduction to usability, how to get started, and what it costs;
· the latest research-based Web design guidelines;
· templates and examples for assessing audience needs, writing a usability test report;
· and much more.
8.Applications
6.3.1 Usability latitudes
Usability latitudes refers to a set of parameters in usability testing, including following ones :
n Probability of success of task operation.
n Task operation efficiency
n Task operation expectation (user)
n User estimate after task operation
n User satisfaction
n Probability of failure in each task
n Probability of success of task operation after repeat operation again
[19]
6.3.2 Usability documents
Here is some documents that could be used in software usability testing.
n User screener
n Testing schedule
n User profile
n User concert form & DNA
n Testing script
n Pre-test questionnaire
n Post-test questionnaire
n Task cards
n Checklists
n Reports
n DVD
[20]
7. Software usability testing organizations
Basically, most of usability testing organizations are very famous for their software usability testing techniques and authority. Here are some brief introductions of these organizations.
Usability Net - http://www.usabilitynet.org/home.htm
Long history, many basic knowledge on the website, good for new usability testers. Need proxy in China mainland.
Usability News – http://www.usabilitynews.com/UsabilityNews offers a facility for the publication of news articles, jobs, events and calls for papers for users involved within the HCI and usability community. Frequently update.
HFES-Human Factors and Ergonomics Society - http://www.hfes.org/ A professional human-computer engineering organization locates on UK. The Society's mission is to promote the discovery and exchange of knowledge concerning the characteristics of human beings that are applicable to the design of systems and devices of all kinds.
The Society furthers serious consideration of knowledge about the assignment of appropriate functions for humans and machines, whether people serve as operators, maintainers, or users in the system. And, it advocates systematic use of such knowledge to achieve compatibility in the design of interactive systems of people, machines, and environments to ensure their effectiveness, safety, and ease of performance.
The Society was founded in 1957. It has 23 technical groups and numerous local and student chapters. See our history page for a discussion of the Society's growth and development.
ACM SIGCHI - http://www.acm.org/sigchi/
ACM SIGCHI, the ACM's Special Interest Group on Computer-Human Interaction, brings together people working on the design, evaluation, implementation, and study of interactive computing systems for human use. ACM SIGCHI provides an international, interdisciplinary forum for the exchange of ideas about the field of human-computer interaction (HCI).
Usability.gov http://usability.gov/
The website maintained by the health and human service department of USA. Also includes some expertise information on usability.
Usability.gov is the primary government source for information on usability and user-centered design. It provides guidance and tools on how to make Web sites and other communication systems more usable and useful. Topics include:
· a step-by-step guide on how to plan, analyze, design, and usability test a highly usable Web site;
· quick access to the latest usability methods;
· an introduction to usability, how to get started, and what it costs;
· the latest research-based Web design guidelines;
· templates and examples for assessing audience needs, writing a usability test report;
· and much more.
8.Applications
By now, commercial software for professional software usability testing have already been applied and established mature market. Here are some good commercial tools on software usability testing. (Introductions are from their advertisements)
ErgoLight Usability Software (ErgoLight Usability Software Ltd.)
Comprehensive GUI quality solutions ? for the professional Windows application developer. ErgoLight offers solutions for developers of Windows applications for testing and evaluating their usability
WebMetrics Tool Suite National Institute of Standards and Technology
Tool Suite contains rapid, remote, and automated tools to help in producing usable web sites. The Web Static Analyzer Tool (WebSAT) checks the html of a web page against numerous usability guidelines. The output from WebSAT consists of identification of potential usability problems, which should be investigated further through user testing. The Web Category Analysis Tool (WebCAT) lets the usability engineer quickly construct and conduct a simple category analysis across the web.
Bobby Center for Applied Special Technology
Bobby is a web-based public service offered by CAST that analyzes web pages for their accessibility to people with disabilities as well as their compatibility with various browsers.
DRUM Serco Usability Services
DRUM is a software tool, which has been developed by close cooperation between Human Factors professionals and software engineers to provide a broad range of support for video-assisted observational studies.
Form Testing Suite Corporate Research and Advanced Development, Digital Equipment Corporation
Provides a test suite developed to test various web browsers. The test results section provides a description of the tests.
9. References
[1] Usability.gov , “learn more about usability testing ” http://www.usability.gov/refine/learnusa.html
[2] Nielsen, J. (1994). Usability Engineering, Academic Press Inc, p 165
[3] Microsoft Corporation (2000). The usability in software designing.
Jing Xiangdong, Meng Li. (2000). Brief talks on software usability. <>(2000.6, CHN) ISSN:1008-0775.0.2000-06-000
[4][5]Natalia Juristo, Marta Lopez, Ana M. Moreno, M. Isabel Sánchez (2003) “Improving software usability through architectural patterns” School of Computing - Universidad Carlos III de Madrid, Spain
[6] Jerry Zeyu Gao, H. -S. Jacob Tsao, Ye Wu. (2003) Testing and quality assurance for component-based software .(P 336) ISBN 1-58053-480-5.
[7] Usability.gov , “learn more about usability testing ” http://www.usability.gov/refine/learnusa.html
[8] Waldemar Karwowski. (2006) International Encyclopedia of Ergonomics and Human Factors. (P 1724). ISBN : 041530430X
[9] Waldemar Karwowski. (2006) International Encyclopedia of Ergonomics and Human Factors. (P 1725). ISBN : 041530430X
[10] Whitney Queensberry (2004) “Can usability standards help improve practice around the world?”
[11] ISO 9241-11:1998 “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability” Chapter 1
[12] ISO 9241-11:1998 “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability” Chapter 3
[13] Michael D. Levi and Frederick G. Conrad ” Usability Testing of World Wide Web Sites” http://stats.bls.gov/ore/htm_papers/st960150.htm
[14] Wikipedia, “GOMSl” http://en.wikipedia.org/wiki/GOMS
Page related references :
Dix, A., Finlay, J., Abowd, G., Beale, R., Human Computer Interaction , Prentice Hall, 1993
Lecture Notes, Abowd, G., CS6751, Georgia Institute of Technology, November-1997
Kieras, D., John, B., The GOMS Family of Analysis Techniques: Tools for Design and Evaluation, CMU-HCII-94-106, 1994
Kieras, D., John, B., Using GOMS for User Interface Design and Evaluation: Which Technique?, June-1996
Kieras, D., John, B., The GOMS Family of User Interface Analysis Techniques: Comparison and Contrast
Lecture Notes, CS5724, Virginia Tech, Fall 1996
[15] Wikipedia, “think aloud protocol” http://en.wikipedia.org/wiki/Think_aloud_protocol
Page related references :
Ericsson, K., & Simon, H. (May 1980). "Verbal reports as data". Psychological Review 87 (3): 215–251. doi:10.1037/0033-295X.87.3.215.
Ericsson, K., & Simon, H. (1987). "Verbal reports on thinking", in C. Faerch & G. Kasper (eds.): Introspection in Second Language Research. Clevedon, Avon: Multilingual Matters, 24–54.
Ericsson, K., & Simon, H. (1993). Protocol Analysis: Verbal Reports as Data, 2nd ed., Boston: MIT Press.
[16] Ronan Fitzpatrick , Strategies for Evaluating Software Usability
[17] Ronan Fitzpatrick , Strategies for Evaluating Software Usability
[18] Genise, Pauline. “Usability Evaluation: Methods and Techniques: Version 2.0” August 28, 2002. University of Texas.
[19] Pear @ studez.com, “UI – Usability testing” http://www.uml.org.cn/Test/200803054.asp
[20] Pear @ studez.com, “UI – Usability testing” http://www.uml.org.cn/Test/200803054.asp
ErgoLight Usability Software (ErgoLight Usability Software Ltd.)
Comprehensive GUI quality solutions ? for the professional Windows application developer. ErgoLight offers solutions for developers of Windows applications for testing and evaluating their usability
WebMetrics Tool Suite National Institute of Standards and Technology
Tool Suite contains rapid, remote, and automated tools to help in producing usable web sites. The Web Static Analyzer Tool (WebSAT) checks the html of a web page against numerous usability guidelines. The output from WebSAT consists of identification of potential usability problems, which should be investigated further through user testing. The Web Category Analysis Tool (WebCAT) lets the usability engineer quickly construct and conduct a simple category analysis across the web.
Bobby Center for Applied Special Technology
Bobby is a web-based public service offered by CAST that analyzes web pages for their accessibility to people with disabilities as well as their compatibility with various browsers.
DRUM Serco Usability Services
DRUM is a software tool, which has been developed by close cooperation between Human Factors professionals and software engineers to provide a broad range of support for video-assisted observational studies.
Form Testing Suite Corporate Research and Advanced Development, Digital Equipment Corporation
Provides a test suite developed to test various web browsers. The test results section provides a description of the tests.
9. References
[1] Usability.gov , “learn more about usability testing ” http://www.usability.gov/refine/learnusa.html
[2] Nielsen, J. (1994). Usability Engineering, Academic Press Inc, p 165
[3] Microsoft Corporation (2000). The usability in software designing.
Jing Xiangdong, Meng Li. (2000). Brief talks on software usability. <
[4][5]Natalia Juristo, Marta Lopez, Ana M. Moreno, M. Isabel Sánchez (2003) “Improving software usability through architectural patterns” School of Computing - Universidad Carlos III de Madrid, Spain
[6] Jerry Zeyu Gao, H. -S. Jacob Tsao, Ye Wu. (2003) Testing and quality assurance for component-based software .(P 336) ISBN 1-58053-480-5.
[7] Usability.gov , “learn more about usability testing ” http://www.usability.gov/refine/learnusa.html
[8] Waldemar Karwowski. (2006) International Encyclopedia of Ergonomics and Human Factors. (P 1724). ISBN : 041530430X
[9] Waldemar Karwowski. (2006) International Encyclopedia of Ergonomics and Human Factors. (P 1725). ISBN : 041530430X
[10] Whitney Queensberry (2004) “Can usability standards help improve practice around the world?”
[11] ISO 9241-11:1998 “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability” Chapter 1
[12] ISO 9241-11:1998 “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability” Chapter 3
[13] Michael D. Levi and Frederick G. Conrad ” Usability Testing of World Wide Web Sites” http://stats.bls.gov/ore/htm_papers/st960150.htm
[14] Wikipedia, “GOMSl” http://en.wikipedia.org/wiki/GOMS
Page related references :
Dix, A., Finlay, J., Abowd, G., Beale, R., Human Computer Interaction , Prentice Hall, 1993
Lecture Notes, Abowd, G., CS6751, Georgia Institute of Technology, November-1997
Kieras, D., John, B., The GOMS Family of Analysis Techniques: Tools for Design and Evaluation, CMU-HCII-94-106, 1994
Kieras, D., John, B., Using GOMS for User Interface Design and Evaluation: Which Technique?, June-1996
Kieras, D., John, B., The GOMS Family of User Interface Analysis Techniques: Comparison and Contrast
Lecture Notes, CS5724, Virginia Tech, Fall 1996
[15] Wikipedia, “think aloud protocol” http://en.wikipedia.org/wiki/Think_aloud_protocol
Page related references :
Ericsson, K., & Simon, H. (May 1980). "Verbal reports as data". Psychological Review 87 (3): 215–251. doi:10.1037/0033-295X.87.3.215.
Ericsson, K., & Simon, H. (1987). "Verbal reports on thinking", in C. Faerch & G. Kasper (eds.): Introspection in Second Language Research. Clevedon, Avon: Multilingual Matters, 24–54.
Ericsson, K., & Simon, H. (1993). Protocol Analysis: Verbal Reports as Data, 2nd ed., Boston: MIT Press.
[16] Ronan Fitzpatrick , Strategies for Evaluating Software Usability
[17] Ronan Fitzpatrick , Strategies for Evaluating Software Usability
[18] Genise, Pauline. “Usability Evaluation: Methods and Techniques: Version 2.0” August 28, 2002. University of Texas.
[19] Pear @ studez.com, “UI – Usability testing” http://www.uml.org.cn/Test/200803054.asp
[20] Pear @ studez.com, “UI – Usability testing” http://www.uml.org.cn/Test/200803054.asp
订阅：
博文 (Atom)