The Master in Computer Science is a joint program with

Carleton University, administered by the

Ottawa-Carleton Institute for Computer Science

by

Master in Computer Science

University of Ottawa

Spring 1999

(Back to the main page of theses supervised by Dr. Lethbridge

This research project explores effects of not having considered usability when assessing the functionality of a software system. Also, it experiments with the introduction of usability into latter stages of software projects in which developers are new to usability. A particular software project confronting that scenario is used to explore these issues. The selected project had created a program comprehension tool that had been given to a group of users for some time. For that reason, this research also explores challenges posed to studying the usability of program comprehension tools. A usability study is designed in order to perform the research. Three techniques are applied in the study: (a) heuristic evaluation, (b) user and task analysis and (c) thinking aloud usability testing. Heuristic evaluation is used to look for usability deficiencies in the tool design. This is achieved by asking a group of evaluators to judge the tool according to a list of usability guidelines and principles. Thinking aloud usability testing is used both (a) to look for problems that users experience when learning and using the tool, and also (b) to explore how well users who had been given the tool were capable of using its functionality. Both are achieved by observing a group of users performing some tasks typical of the tool's design intent. User and task analysis is used to find a right group of users and a good set of tasks for the thinking aloud usability testing technique. It is also used to obtain information about how the users had employed the tool. The study was very productive in finding information about all the above issues within the particular software project. It revealed that none of the users was capable yet to correctly use the tool functionality, and therefore it was impossible to assess the tool functionality at that moment. Not having considered usability had been one of the causes of this. Also, many issues were observed in relation to (a) having started considering usability at a late stage in the tool lifecycle and (b) the software project's reaction to being exposed to usability for the first time. Finally, certain particularities and difficulties were encountered with planning and studying the usability of the tool.

I want to thank: (a) Dr. Timothy Lethbridge for introducing me to this fascinating area as well as for all his support and help during the project, (b) the KBRE group, for their participation and help, (c) Gina (indeed, the best thing that has happened to my life), for her love and help throughout the project, and (d) the people of Mexico who through Conacyt provided the financial support.

With love to Gina, my Father, my Mother, Juan Arturo, Mauricio, Father in law and rest of the family. Also to Omar, Lorena, Armando, Lulú, Hans, Daniel and Pérez family, Quique and the rest of our friends.

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Antecedents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Motivation and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2. Review of Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 Usability of Software Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 User Centered Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.2 Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1.2.1 Discount Usability Engineering . . . . . . . . . . . . . . . . . . . . 18

2.1.2.2 Heuristic Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.1.2.3 Thinking Aloud Usability Testing . . . . . . . . . . . . . . . . . . 21

2.1.3 Acceptance of User Centered Design in Software Projects . . . 22

2.2 The Software Project and Program Comprehension Tool . . . . . . . . 23

2.2.1 The Software Project and Problem Addressed . . . . . . . . . . . . . 24

2.2.2 The Program Comprehension Tool . . . . . . . . . . . . . . . . . . . . . 28

2.2.3 Status of the Program Comprehension Tool and

Software Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

 

PAGE

3. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.1 Ideas for planing the Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.2 Study Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.2.1 Heuristic Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.2.2 User and Task Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.2.3 Thinking Aloud Usability Testing . . . . . . . . . . . . . . . . . . . . . . 37

4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.1 Heuristic Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.1.1 Procedure Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.1.2 Usability Insights obtained with the Heuristic Evaluation . . . . 41

4.1.3 Issues during the Heuristic Evaluation . . . . . . . . . . . . . . . . . . . 45

4.2 User and Task Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.2.1 Procedure Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.2.2 Insights obtained with the User and Task Analysis . . . . . . . . . 54

4.2.3 Issues during the User and Task Analysis . . . . . . . . . . . . . . . . 57

4.3 Thinking Aloud Usability Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.3.1 Procedure Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.3.2 Usability Insights obtained with the

Thinking Aloud Usability Testing . . . . . . . . . . . . . . . . . . . . . . 63

4.3.3 Issues during the Thinking Aloud Usability Testing . . . . . . . . 73

4.4 Other Insights and Issues independent of the Techniques . . . . . . . . 78

5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

1. The Tool before the project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
2. Usability Problems found in the Heuristic Evaluation . . . . . . . . . . . 102
3. Tasks used in the Thinking Aloud Usability Testing . . . . . . . . . . . . 128
4. The Tool before the Thinking Aloud Usability Testing . . . . . . . . . . 131
5. Usability Problems found in the

2.1 Dimensions in which users’ experience differ . . . . . . . . . . . . . . . . . . . . . . 9

2.2 Start life cycle for software development . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.3 System acceptability and usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.4 Learning curves of different systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.5 Patterns that SEs follow when exploring the systems . . . . . . . . . . . . . . . . 26

2.6 Main window of the Program Comprehension Tool . . . . . . . . . . . . . . . . . 29

2.7 Architecture of the Program Comprehension Tool . . . . . . . . . . . . . . . . . . 30

3.1 Classification of users of the tool (SEs) . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.1 Problems according to the usability guidelines . . . . . . . . . . . . . . . . . . . . . 79

4.2 Types of problems found using both techniques . . . . . . . . . . . . . . . . . . . . 79

4.3 Percentage of problems found by the techniques . . . . . . . . . . . . . . . . . . . . 82

4.1 Problems found in the Heuristic Evaluation according to the

usability guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.2 Problems found in the Heuristic Evaluation according to the

second classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.3 Second classification of the problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.4 Initial procedure for determining the tasks . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.5 Modified procedure for determining the tasks . . . . . . . . . . . . . . . . . . . . . . 60

4.6 Problems found in the Thinking Aloud Usability Testing

according to the usability guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.7 Problems found in the Thinking Aloud Usability Testing

according to the second classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.8 Answers given by participants in the questionnaire . . . . . . . . . . . . . . . . . . 69

One of the reasons for creating software systems is to make it possible for people to achieve their goals more efficiently. Systems seek to do this by providing functionality which supports people in tasks for accomplishing those goals. However, producing systems that can help people effectively and which are fully adopted by them is a complex process. Part of that process involves designing and implementing the system. Another part involves users learning, employing and finally adopting the system. Many times, a system must pass through these stages several times before it is really efficient and fully adopted. Good system design is key for a system seeking to help users effectively. Critical factors for creating systems with good design are:

1. The system must provide functionality that enables users do what they need. That is, the system must have good utility.
2. The system must be done in a way that facilitates users learning and efficiently exploiting its functionality. That is, the system must have good usability.

When a new system is created, requirement analysis activities are performed in order to discover possible functionality that can help users in their goals. That is, developers look for what might be good for the system to provide to the users. Once having identified certain functionality, developers build the system and give it to the users.

Nonetheless, it is common that developers want to confirm whether that functionality can in fact help users to achieve their goals more efficiently or whether it should be improved. To do that, developers give the system to a group of users for some time so that they can use it when working on their goals. Then, later, developers explore if the system has helped the users to achieve their goals more efficiently or not. If not, developers asked these users for information about any noticed deficiencies in the functionality as well as suggestions for improving it. In the rest of the document, we will refer to this as 'developers assessing the functionality of the system'.

Interest in assessing functionality becomes particularly strong in the case of systems seeking to help users in complex problems where it is unknown precisely how a system could help and what functionality it should provide. Typically, these are problems in which not even experts in the domain are able to suggest possible functionality for a system. In these cases, the truly efficiency of this type of functionality remains unknown until users employ the system for working on their goals.

Throughout this document, we will use the term "experimental functionality" to refer to functionality for which it is still unknown whether it can help users effectively and which developers want to assess.

Nowadays, a common situation in software projects is that a system is developed without considering its usability. Activities are undertaken to discover and implement certain functionality for the system, but nothing is done to ensure that users will be able to easily learn and efficiently employ that functionality. In these conditions, the system is given to a group of users for some time, and later, developers want to assess the functionality. That is, they want to see if the functionality has helped those users to achieve their goals more efficiently, as described in section 1.1. Here, it is important to notice that behind the idea of assessing the functionality, developers assume that the users will correctly use the system when working on their goals. In other words, that users will learn to correctly employ the functionality during the time the system is given to them.

This scenario motivated us to perform this research. We wanted to:

1. Explore effects of not having considered usability when assessing the functionality of a system.

We were interested in exploring effects in relation to (a) the usability of the system and (b) how users learn and employ the functionality during the time the system is given to them.

Another common situation is that people in those projects are not aware of usability. They don't know why it is important and how it can be considered when developing a system. Thus, we also wanted to:

2. Experiment with the introduction of user centered design and usability into the latter stages of software projects in which developers are new to these concepts.

To pursue the research, we selected a particular software project confronting the above scenario. The project we chose had developed a program comprehension tool providing experimental functionality. Based on this, we also wanted to:

3. Explore particular challenges posed to studying the usability of program comprehension tools.

In summary, the objective of the research was to look for evidence relating to these issues within such particular software project developing a program comprehension tool.

The results and conclusions of this research should be of general interest to several sources. They should be of interest to:

• Software projects that want to assess the functionality of a system in which usability has never been considered. Particularly to projects developing systems providing experimental functionality.
• Anybody intending to study the usability of a program comprehension tool.
• Anybody intending to perform usability studies of software systems in general.
• Anybody intending to introduce usability into the latter stages of software projects in which developers are new to usability.
• Finally, to anybody interested in pursuing research opportunities discovered here.

Another secondary but important contribution should be to the usability of the program comprehension tool that we experimented with. The results of this research should help developers to improve the usability of the tool, which in turn should help the software project to assess and improve its experimental functionality. We want to mention that it was not the intent of this research to actually make any improvements to the tool, rather to explore the process that might lead to those improvements.

All the issues we wanted to explore were related with usability of software systems. As mentioned in Chapter I, we pursued the research by experimenting with a particular software project. Hence, the literature search focused on two areas: (a) usability of software systems, and (b) details about the particular software project that was selected.

As mentioned earlier, the notion of usability is related to the way the functionality of a system is provided to the users. Usability in software systems has been traditionally considered within the scope of the user interface, and most times it is only addressed in detail by literature about user interface development. It has typically not been considered a central issue in software engineering. However, the success of a software system lies among other things in providing what users need in such a way that users can learn it and efficiently use it. In other words, creating a system with good utility and good usability.

Many different approaches for creating software systems in different fields of computer science currently exist, such as: The Spiral Model for Software Development and Enhancement [Boehm 1988], The Unified Software Development Process [Jacobson, Booch, Rumbaugh 1998], The Usability Engineering Model [Nielsen 1993a], The Usability Engineering Lifecycle [Mayhew 1999], etc. All discuss and address to a different extent the issues of utility (e.g. functionality) and usability of software systems. Some approaches address in detail the issue of utility but miss at fully addressing usability. The same is true vice-versa. Unfortunately, no complete approach fully addressing both issues currently exists.

Traditional approaches in the Software Engineering field describe development of functional and non-functional requirements for creating a system. They focus on capturing aspects of the utility of the system in such requirements and might mention capturing usability aspects of the system in the non-functional requirements. They provide methodologies for implementing those requirements into the system. However, these approaches can lead to creating a system with good utility but that does not necessarily lead to a system with good usability. They miss at providing some fundamental concepts and techniques that are well known in the Human-Computer Interaction field for ensuring that a system realizes good usability. For example, the unified software development process mentioned above describes a set of contextual task analysis activities to obtain information or ‘use cases’ from which the requirements for a system are derived. However, this approach does not explicitly refer to a ‘work environment analysis’. This important part of a contextual task analysis points to aspects of the users’ actual work environment that should drive user interface design decisions. Also, this method does not emphasize ‘contextual observations of work’, or even the need to tap into ‘current user task organization’ and ‘task sequence’ models in the design of the user interface. Consequently, while leading to create a system with good utility, this approach will not necessarily lead to a system with good usability [Mayhew 1999].

On the other hand, traditional approaches in the Human-Computer Interaction field focus on realizing good usability in a system but do not provide an in depth methodology addressing the utility of the system.

This phenomenon of incomplete approaches seems to be related to poor communication among groups of people supporting different approaches as well as different "cultures" or schools of thought. It is important to say that approaches in different "cultures" do not compete but rather potentially complement each other. Although no complete approach fully addressing utility and usability currently exists, some efforts are being undertaken [Rosson and Carroll 1995].

Among all the approaches, probably those that address best the issue of usability are those following the notion of User Centered Design [Norman and Draper 1986], which has the concept of usability at its center. Among other things, these approaches address in detail the issue of studying and evaluating the usability of software systems.

In this research, all the issues we wanted to explore involved studying the usability of a software system, therefore, an approach based on user centered design could provide concepts and techniques to achieve our objectives.

The notion of user centered design involves a whole set of concepts and principles seeking to create products that are highly useful to the users. Much research has been done in the last decade within user centered design with respect to software systems. Gould and Lewis [1985] give a good description of the main principles behind it:

• Establish an early focus on users and the tasks they perform to achieve their goals.
• Perform empirical measurement of the software’s usability using simulations and prototypes.
• Perform cycles of iterative design, where a cycle consisting of design, usability evaluation and redesign should be repeated as many times as necessary.

Many authors describe different approaches for creating software systems following the principles of user centered design [Hix, Hartson 1993; Nielsen 1993a; Shneiderman 1998; Mayhew 1999]. All embody the same principles although they differ in certain issues such as steps in the approach, notation, naming, etc. In order to be consistent, we decided to follow The Usability Engineering approach proposed by Nielsen [1993a]. In this approach, Nielsen describes a series of activities that should be followed by people developing a system. These activities can be divided into three stages:

1. A pre-design stage, which should provide essential concepts to be used through all the lifecycle of the software. This stage comprises the following activities:

1. User and task analysis.
2. Performing a competitive analysis.
3. Setting usability goals.

1. A design stage, which guides the design of the user interface. This stage comprises the following activities:

1. Performing parallel design.
2. Including users in design.
3. Coordinating the total interface of the system.
4. Applying guidelines and heuristic rules.
5. Prototyping.
6. Performing empirical testing.
7. Performing iterative design.

1. A post-design stage, in which feedback from the use of the system in the field is collected.

User and task analysis means analyzing the intended user population of the system in order to learn about them as well as the tasks with which the software is intended to help. At a minimum, developers should visit the users at their work place. User differences and variability in tasks are the two factors with the biggest impact on usability. The concept of users should include all people that will be affected by the software system. User experience is an important factor in user differences and that should be considered; Nielsen suggests that it differs in at least three dimensions, as Figure 2.1 shows:

Learning about the tasks should include observing the users' goals, needs, and how they approach tasks. Exceptional circumstances should be considered too. Also, it is necessary to analyze if users' tasks could be improved by the use of the system.

Task analysis centered around users generates concrete statements of representative user tasks that provide good coverage of the functionality that the system should provide and that can be used for usability testing later. This contrasts with other traditional approaches, which attempt to derive an abstract functional specification of the system [Glen 1995]. Also, task analysis centered around users emphasizes human factors more than traditional analysis by attempting to identify where users waste time or are made uncomfortable. Early focus on users and tasks is one of the most important activities in order to reach high usability.

Performing a competitive analysis refers to analyzing similar products according to established usability guidelines in order to notice their strengths and weaknesses. Some user testing with other products can also serve to see how the functionality of the other software supports the user tasks. This can provide ideas for the new software. Lewis and Rieman [1993] list some points in favor of using other product's ideas:

1. It is not easy for developers to come up with design ideas as good as those already implemented in high quality products.
2. Using ideas from other known products can improve the learning of the intended software because they are more likely to be already known by users.
3. It can save design time.

The intention is not to steal particular copyrighted ideas but rather to take general ideas and try to improve them.

Setting usability goals refers to establishing concrete goals that the system ideally has to comply with before it is released. Many times a trade off exists between different usability attributes, therefore the goals must be based on the results of the user and task analysis. Ideally, the intended system should perform better in these goals than other similar products. Also, Nielsen suggests performing a financial impact analysis to estimate the cost-benefits that will be provided by accomplishing these usability goals with the system.

Performing parallel design means to create different design ideas and then merge them all into one. In parallel design, the goal is to generate as much diversity as possible, therefore developers should work independently until they have completed the first drafts of the ideas, then all ideas are merged together into one design that comprises the best from all. Nielsen claims that parallel design can save development time by exploring many ideas at the same time.

Including users in design refers to have some users (a) criticizing developers' designs, (b) capturing problems with current developers’ concepts, and (c) contributing with other ideas. This step is usually good at capturing mismatches between users’ actual tasks and developers model of the tasks. Also, users seem to be very good at reacting to designs they don’t like or won’t work in practice. However as Nielsen describes, "users are not designers" therefore this step shouldn’t consist of asking users what they want, it is much better to show users paper mockups or some screen designs presenting the ideas.

Coordinating the total interface of the system means to review the design in order to provide consistency among all its parts. Consistency should apply to all parts of the current software project, including the user interface, documentation, training courses, etc. Developers should share and follow similar principles. Also here, the application of standards can help in achieving consistency.

Applying guidelines and heuristic rules has as its objective to implement well-known principles in the current system design in order to improve its usability. There are many well-known principles and guidelines that can provide improvements in the usability of a system. Standards, for instance, are one type of guideline.

A key step for developing usable systems is the development of an early prototype or simulation of the user interface. Prototypes can range from paper sketches to working programs. Any prototype should be quick to develop, as well as quick and easy to modify; therefore a good tool for prototyping plays an important role. Some authors [Hix and Hartson 1993; Lewis and Rieman 1993] suggest that a tool for prototyping should:

1. Be easy to learn and use.
2. Posses good visual capabilities.
3. Allow easy extensions of current prototypes.
4. Give full support for the type of interface being developed.
5. Support modular coding practices.
6. Provide support for turning the prototype into the final program.

Different types of prototypes vary according to the functionality they implement. A horizontal prototype is one where the complete user interface is presented but it has no underlying functionality. A vertical prototype includes in-depth functionality of only certain parts of the interface. A scenario prototype fully implements selected paths in the user interface. A scenario prototype works well for user testing but it has the problem that users must stay within its paths.

Once a system design has been decided, it is necessary to empirically test it in order to capture usability problems. Testing is usually done using prototypes, however systems already implemented should be also tested. Whitefield et al. [1991] provide a classification of the different testing possibilities: (a) whether or not the system has been implemented, and (b) whether or not real users are involved in the testing.

Prototyping and testing are key activities in user centered design and have several purposes:

1. Performing early usability evaluation of the final system.
2. Exploring design ideas to get the best final choice.
3. Making iterative refinements.
4. Making easier the communication about the usability status of the current design to the development team.

One reason behind prototyping and testing is the concept that it is too difficult to design a complex system and get it correct the first time [Hix and Hartson 1993]. Some of the reasons are (a) the current limitations in psychological theory that don't permit people yet to accurately predict users' behavior with the system, and (b) the difficulty for developers at the start of a project to have a complete understanding of the entire context in which a system will be used on the field.

Therefore, any development project should expect and plan for iterations where the system is refined or even redesigned. This leads to the concept of performing iterative design where the system goes through cycles of design and evaluation until it reaches pre-established goals.

It is likely that many design decisions will be made during iterations. In order to maintain control and organization, Nielsen recommends maintaining a design rationale document. This document should contain the rationale behind decisions made about the system including the user interface. Also the document (a) captures the process followed throughout design, (b) serves to communicate design to other people, and (c) serves as a guide to future modifications of the system. A suggested format to capture design rationales is the "QOC" notation [McKerlie and MacLean 1993]. QOC stands for Questions, Options, and Criteria. Design questions are linked to design options, which in turn are linked to the criteria that caused particular options to be accepted or rejected.

Finally, once the system is released it is necessary to keep collecting information from its use in the field. This information should be used to plan for future versions of the system or for other products as well.

In cases where resources or time are limited, Nielsen [1994c] suggests a Discount Usability Engineering approach. In this approach the techniques to perform are:

1. User and task observation.
2. Cheap prototyping and scenarios.
3. Simplified thinking aloud usability testing where users verbalize their thoughts as they use the prototypes.
4. Heuristic evaluation in which the prototypes are evaluated according to guidelines.

User centered design contrasts with traditional development methodologies that follow the waterfall model in that (a) prototyping, (b) usability testing and (c) iterative design are performed. In the waterfall model on the other hand, development tries to go in a series of sequential stages where each stage is completed before the next one starts. Therefore in this model, the system specification is completed before implementation starts. On the other hand, user centered design assumes that usability problems will appear in the first specification of the system and therefore a prototype is necessary to discover such problems.

Hix and Hartson [1993] describe user centered design as following a "star life cycle" (Figure 2.2) in which usability evaluation is at center of the star and the points of the star represent other activities during the design process like prototyping, for instance. The points in the star are not ordered and they can be repeated as necessary.

In one study [Nielsen 1993b], four case studies on iterative design were used. The interfaces passed through three to five iterations, with usability measured in the same way after each iteration. The study found average usability improvements of about 38% after every iteration.

Another study [Boehm et al. 1984] tried to compare the prototyping approach with the traditional software specification approach. Seven teams developed versions of the same software. Three teams used the prototyping approach and four the specification approach. The study found that better interfaces were developed by the prototyping approach.

As described above, usability testing is a key element in the user centered design approach of system development. Now, there are two important issues to explore: (a) what exactly usability is, and (b) how usability can be evaluated.

Usability is a narrow concern compared with the more general concept of system acceptability, which Nielsen describes as "the question of whether the system is good enough to satisfy all the needs and requirements of the users and other potential stakeholders". Figure 2.3 below illustrates the factors of system acceptability and its relationship with usability as defined by Nielsen.

In this model, utility is defined as the question of whether the functionality of the system can do what is needed, and usability, as the question of how well users can exploit that functionality. Therefore usability applies to all aspects of the system with which a human might interact. The model also describes usability as mainly composed by five usability attributes:

1. Learnability.
2. Efficiency of Use.
3. Memorability.
4. Error Handling and Prevention.
5. User Satisfaction.

Learnability refers to how easy a system is to learn. In some sense, this is the most fundamental usability attribute since most systems have to be learned, and learning is the first experience that most users have with a system. There are many ways in which learnability can be measured; however, the concept of learning a system should always be seen as users learning the system in order to achieve useful work.

Efficiency of use refers to how efficient and productive a system is for a user once he or she has learned to use it. Users may never finish learning a system, although their learning curves tend to flatten out after they have acquired some expertise, as Figure 2.4 shows. Therefore, a typical way to measure efficiency of use is to decide on some definition of expertise, getting some users with such expertise and ask them to perform some tasks with the system.

Memorability refers to how easy a system is to remember. Typically, memorability is associated with casual users. A casual user is somebody who has learned the system but has not used it for some period of time. Improvements in learnability usually produce improvements in memorability [Nielsen 1994a].

Error handling and prevention refers to (a) the rate and number of errors that users make while using the system and (b) the ease with which users recover from those errors. Typically, an error is defined as any action that does not accomplish the desired goal. However, not all incorrect actions should be considered as errors, since sometimes users recover almost immediately from them and there is not a big impact on user's performance. On the other hand, other errors are more catastrophic because they make it more difficult for users to recover from them. Both types of errors should be measured separately when considering this attribute.

User satisfaction refers to a measure of how pleasant to use the system is. A typical way to measure satisfaction is asking users about their opinion of the system, which is generally achieved through the use of questionnaires or interviews. However it is necessary to be cautious with users' opinions because they are often closely related to the peak difficulty that users experience with the system, and also, people are often too polite in the responses.

Much research has been done in the last decade about evaluating usability and many different methodologies currently exist. The most common of all has been usability testing involving users [Nielsen and Mack 1994].

Typical goals when measuring usability are to provide quantitative measures of some usability attributes. For example, how long users take on average to complete a particular task or how many errors on average users make while performing certain tasks with the system. Quantitative measures are normally used to set usability goals for a system, and to evaluate system status to see if it has reached predetermined objectives. However, qualitative measurements are also taken sometimes to measure users' satisfaction, like in the case of asking users to rate a system. The types of measurements to be taken depend on what the desired goal is as well as in what status within the lifecycle the system is.

There have been some criticisms of classic usability testing methods [Mack and Nielsen 1994a]. Some of those are that classic methods require professional usability experts and special laboratory equipment, also, obtaining large numbers of users so that the results can be statistically significant is very expensive, time consuming and difficult. Furthermore, the costs may out weight the benefits in small projects. As we mentioned before, alternative methodologies have been proposed, like those in the discount usability engineering approach proposed by Nielsen [1994c]. In this approach, the emphasis is to get qualitative information that can be communicated to developers. This information identifies concrete usability problems with the system and helps in generating solutions to them.

Nielsen claims that the methods in this approach are easy to learn, inexpensive, fast to apply, don’t require special equipment, and can be performed by software developers. He suggests that discount usability engineering can be used to supplement classical methods or when other more expensive techniques are not feasible. In fact, he says, in many practical situations the goal should be to identify usability problems and not necessarily to get quantitative measures about them.

Many different usability evaluation techniques exist and they can be divided on the basis of whether they use real users or not. Techniques that don't require users are usually referred as Usability Inspection Methods [Nielsen and Mack 1994]. In these techniques, evaluators are used to inspect the system, looking for usability problems that users might have. Evaluators inspect the system based on guidelines and their own judgement.

As we will discuss in Chapter III, two usability evaluation techniques were applied in the research. Both following Nielsen's discount usability engineering approach. A description of such techniques is provided below.

The first technique used in this research is a usability inspection methodology called Heuristic Evaluation. In this technique, evaluators receive a list of guidelines and are asked to explore the system's interface to look for things that violate any of the guidelines or that they consider as problems based on their experience and intuition. Evaluators are allowed to use any guidelines or principles they consider necessary. Each evaluator proceeds independently in his exploration and they are not allowed to communicate with each other until all explorations have finished. An observer may watch the evaluator's exploration to record notes and answer questions they have about the system.

Heuristic evaluation uses a short list of heuristic guidelines and few evaluators. A list of guidelines that has become very popular is the one developed by Nielsen [1992b]. This list contains the following ten usability guidelines that represent what any system with good usability is expected to have:

1. Simple and natural dialogue.
2. Speak users' language.
3. Minimize users’ memory load.
4. Consistency.
5. Good feedback.
6. Clearly marked exits.
7. Shortcuts or accelerators.
8. Good error messages.
9. Good error prevention.
10. Good help and documentation.

It has been observed in heuristic evaluations, that single evaluators miss most of the problems in their evaluation, although different evaluators find different problems. Therefore, better results are obtained by combining information from several evaluators. In a study, Nielsen [1993a] averaged results from six different projects that used heuristic evaluation. He found that single evaluators discovered 35% of all the problems detected and five evaluators discovered 75% of them. For many practical purposes, Nielsen recommends the use of three to five evaluators.

Also, it has been seen that evaluators’ expertise has some effect in the amount of problems detected. In another study [Nielsen 1992b], the same interface was subjected to heuristic evaluation by three different groups of evaluators: (a) novices, who had knowledge about computers but no usability expertise, (b) single experts, who were usability specialists but not specialized in the domain of the interface, and (c) double experts, who had expertise in both usability and the domain of the interface being evaluated. The study found that novices detected 22% of the problems in the interface, single experts 41% and double experts 60% of them. The study concluded that best results are obtained by using double experts as evaluators. However, the study recommends the use of single experts for many practical purposes and use of double experts just when optimal performance is necessary.

Supporters of heuristic evaluation claim that this technique is easy to learn, fast to apply and inexpensive. Some studies [Desurvire 1994; Jeffries et al. 1991] compared heuristic evaluation with other usability evaluation methods. They found that heuristic evaluation detects most of the usability problems discovered by the other techniques and it identifies the most serious ones too. They also confirmed that better results are obtained when evaluators have usability expertise.

The second technique used in this research is called Thinking Aloud Usability Testing. In this technique, a group of end users of the system or 'participants' is asked to verbalize their thoughts while performing some tasks with the system. The tasks must represent examples of the uses to which the system will be put in the field, and its solution must cover as much functionality of the system as possible. An observer may watch the participants to record notes, encourage them to keep talking and answer questions they have about the interface. However, the observer must be careful in not helping too much so that he doesn't alter participants' performance. The sessions might be recorded or videotaped for later analysis.

The theory behind speaking aloud while performing the tasks is that verbal comments provide insights into the user's mental process, therefore making it easier to understand how these interpret the system. In this way, it's possible to capture users' misinterpretations and problems with the system. However, care must be taken when listening to users, since data showing what they are doing when a problem is present has more validity than users' own theories about the problem. Thinking aloud provides qualitative information rather than quantitative information as in other classical usability techniques.

Supporters of thinking aloud usability testing, claim that this technique is easy to learn, fast to apply, inexpensive and can be performed by non usability experts. Lewis [1982] describes some advantages of thinking aloud usability testing:

1. Users' comments help to identify a problem as well as its causes.
2. Users discuss problems as they occur when details are fresh in the user's memory.
3. Minor problems that cause annoyance or confusion but don't affect task completion times are more likely to be detected by thinking aloud testing.
4. User's comments help reveal user's subjective attitudes towards the interface.
5. Thinking aloud testing can be used with incomplete prototypes or mockups, since it doesn't attempt to measure task completion times.

Many authors [Nielsen 1993a; Hackos 1998; Mayhew 1999] point out that good ethical principles must be followed in any study or technique involving users. For example, users must be properly informed about the study and their decisions and feelings must be respected. Also, adequate confidentiality must be kept on any information obtained about the users.

Nowadays, many software projects face the necessity to develop highly useful systems. However, many of those are new to the notions of usability and user centered design. The success of user centered design greatly depends on the acceptance and commitment of the software development project to it. Several authors describe the idea of stages of acceptance and commitment to user centered design in software projects [Bias & Mayhew 1994; Ehrlich & Rohn 1994]. They explain that software projects can be in one of the following stages:

1. Skepticism

This stage is typical of projects that have never been involved with user centered design. At this stage developers believe that user centered design will only lengthen the development of the system. Normally, this type of project is very concerned just with the functionality and features of the system and not in its usability. If user centered design activities are started, developers don't see what type of benefits can be brought to the system and project.

3. Curiosity

As projects moved from skepticism, developers become curious about the benefits of user centered design. They recognize the problems with the system and might admit not having the skills for knowing what improvements should be done. They are reluctant to give control over the design of the user interface and utility of the system to someone who is not implementing it.

4. Acceptance

At this stage, people in the project understand and rely on user centered design. They accept somebody in the project providing guidance in the design and evaluation of the system. They see usability as critical but the system might still be shipped containing usability problems.

5. Partnership

The same authors describe that the introduction of user centered design in software projects requires an evolutionary process in which the projects must gradually grow in their acceptance and commitment to user centered design.

As stated earlier, we performed the research by exploring on a particular software project that had created a program comprehension tool. Such tool provided experimental functionality that developers wanted to assess. In this section, we present details about (a) the software project and the problem it addressed, (b) the program comprehension tool, and (c) the status of the tool and software project at the time of this research.

The maintenance of large software systems has become a tremendous problem. Large software systems containing several millions of lines of source code have to be repaired and upgraded. Software Engineers (SEs) need to spend long periods of time exploring and trying to understand large bodies of source code in order to make the required changes. Also, it takes a long time for SEs who are novices to the systems to be able to start making the changes. Thus, the creation of effective tools that can help SEs in maintenance activities is a major issue today.

The Knowledge Based Reverse Engineering software project (KBRE) is a collaborative project between a large Telecommunications Company and the University of Ottawa, which has been looking for solutions to the mentioned problem.

One major goal of the KBRE project has been the creation of software tools that can effectively help SEs to perform maintenance activities on large software systems. Such tools should help SEs to learn about large software systems more quickly, and should also help SEs that are experts in a system to make changes more easily.

At the beginning of the KBRE project, the tasks and activities that SEs perform when maintaining the systems were not well understood. It was unknown how or what tasks software tools could provide help with and what functionality should be necessary to implement in such tools. Not even SEs expert in maintaining systems were able to accurately suggest possible functionality. This situation led people in the project to perform studies about SE work practices [Singer and Lethbridge 1997]. The goal was to generate ideas of possible functionality that could enhance the work practices and that could be implemented into the tools. The rationale was that creating tools that meshed and enhanced existing behavior should help SEs in maintaining the systems and should be more easily adopted. The lack of tool adoption has been a major problem in the area of tool design for SEs.

Several types of data about SEs work practices were collected during these studies, including:

1. A web questionnaire asking SEs about their work.
2. Observation of the work of one SE at the Company during 14 weeks.
3. Observations of eight SEs at the Company during one hour of their work.
4. Series of interviews with SEs.
5. Analysis of company-wide tool usage statistics.

Many insights and ideas were obtained from these studies. One of these was that searching is an important and key component of the real, day-to-day work of SEs. Therefore, improvements in searching should help SEs to do their work better [Singer, Lethbridge et al. 1997]. This made people in the KBRE project focus the efforts on designing tools with good search capabilities.

Another observation was that SEs spend a significant amount of time looking at the source code of the systems, which suggested the creation of tools with intelligent source code viewing capabilities. According to these studies, SEs need to explore the software systems with the goal of determining where modifications have to be done. Also, there are two categories of SEs performing this task (a) novice SEs, who are not familiar with the system and must learn about it, and (b) expert SEs, who know the system very well but can not maintain a complete-enough mental model of it. Novices are normally less focused than experts and spend more time studying things that are not relevant to the problem. All SEs repeatedly search for items of interest in the source code and navigate relationships among them. They never try to understand the system in its entirety but they are content to understand just enough to make the changes required. After working on a particular area they move to other parts of the system and forget details about the previous area, having to re-explore it again if they re-encounter it later. In conclusion, many SEs spend a large proportion of their working time trying to understand the source code prior to making changes to it. People who performed these studies call this approach Just in Time Comprehension [Singer and Lethbridge 1997].

Another important result obtained from these studies was a set of patterns that SEs follow when they explore software systems (Figure 2.5).

The results from the studies were used to generate a set of requirements for software tools seeking to help with the Just in Time Comprehension approach. This set of requirements consisted of three functional and seven non-functional requirements, listed below:

1. Provide search capabilities such that the user can search for, by exact name or by way of regular expression pattern-matching, any named item or group of named items that are semantically significant in the source code. The term semantically significant refers to exclude arbitrary sequences of characters in the source code text that have no meaning to SEs."
2. Provide capabilities to display all relevant attributes of the items retrieved in requirement F1, and all relationships among the items."
3. Provide capabilities to keep track of separate searches and problem-solving sessions, and allow the navigation of a persistent history."

1. Be able to automatically process a body of source code of very large size, i.e. consisting of at least several million lines of code. Many systems used by real industrial SEs can be large in size."
2. Respond to most queries without perceptible delay."
3. Process source code in a variety of programming languages." Studied SEs use at least two languages.
4. Wherever possible, be able to interoperate with other software engineering tools." Studied SEs make use of other tools.
5. Permit the independent development of user interfaces (clients)."
6. Be well integrated and incorporate all frequently used facilities and advantages of tools that SEs already commonly use."
7. Present the user with complete information, in a manner that facilitates the just-in-time comprehension task" [Lethbridge and Anquetil 1997].

Based on the above results, the KBRE project created a Program Comprehension Tool implementing those requirements in its functionality [Lethbridge and Anquetil 1997]. Since its creation, the following limitations to the tool were accepted:

1. The server component of the tool would run on only one particular platform."
2. The tool would not handle object oriented source code."
3. The tool would not deal with dynamic information, i.e. information about what occurs at run time."

Figure 2.6 below, shows the version of the tool used at the start of this research. The main features that fulfill the requirements of searching and displaying relevant attributes (F1 and F2) are implemented in the bottom two panes. The bottom left pane presents a hierarchy of items that can be incrementally expanded by asking to show attributes of such items and related items, or that can be searched for information about a given item. Selecting an item in the hierarchy presents information about it in the bottom right pane, from which the user can select any item of text and request information about it.

The main features that keep track of separate searches and allow a persistent history (F3) are in the top pane. Each element in this pane is a complete state of the bottom two panes. A hierarchy of these states is saved persistently, so that the user can work with the same explorations that he had from previous sessions.

The non-functional requirements are implemented within the architectural design of the system (Figure 2.7). This architecture includes a fast database, an interchange language for language-independent information about software, and a client-server mechanism allowing the incorporation of existing tools.

As mentioned earlier, the functionality of the tool arose from the study of SEs work practices, and the nature of the problem it seek to help meant it was an experimental functionality. Since its creation, the tool had been given to a group of SEs at the Company so that they could use it during maintenance activities. At the time of this research, the tool had been available for 1.5 years to that group of SEs, and developers wanted to assess its functionality.

On the other hand, the tool had been developed without considering its usability. It had not been evaluated and tested for usability. Also, the KBRE project had not consistently monitored many issues such as:

1. Number of SEs that had employed the tool.
2. How those had learned and employ the functionality.
3. Problems and difficulties when learning the functionality.
4. Problems and difficulties for efficiently exploiting the functionality.
5. Level of expertise reached by them.
6. Subjective satisfaction with the tool.

In addition, people in the KBRE project didn't know much about user centered design. Some developers had a basic idea about user interface design, but basically nobody knew about usability and usability evaluation.

In summary, the tool and KBRE project provided an excellent foundation in which we could perform the research. In the next chapters, we will describe the procedure followed to perform the research as well as the results and conclusions obtained from it.

According to the overall objective of the research (section 1.1), the goal was to look for evidence within the KBRE software project and its program comprehension tool about the issues in which we were interested. Looking for such evidence required performing an appropriate activity. This chapter presents a description of the rationale followed in planning that activity as well as a description of it.

According to the first research objective, we had to look for effects of not having considered the usability of the tool. A first interest was on effects in relation with the usability of the tool at the moment of this research. A second interest was on effects in relation to how SEs had learned and employed the functionality during the past 1.5 years.

As suggested in the literature, we cold look for effects of the first type by studying the usability of the tool with some usability evaluation techniques. In regards to looking for effects of the second type we could proceed as follows:

1. Identify a group of SEs who had used the tool during maintenance activities (category IV in Figure 3.1).
2. Explore how those SEs were capable of using the tool functionality. We could do that by asking them to complete some typical tasks in the best way they knew.

It is important to mention that the group of SEs who had been given the tool consisted of approximately twenty SEs, therefore it would be feasible to perform these investigations.

According to the second research objective, we had to experiment with introducing usability into the KBRE project. Hence, the same study for exploring the usability of the tool suggested in the previous issue could serve for this. Nonetheless, we could broaden the experimentation by doing a whole iteration of evaluation, redesign and re-evaluation of the tool applying different evaluation techniques. We might discover different things when the tool is evaluated, when it is redesigned and when the redesign is evaluated again. Also, the use of different techniques might reveal different things.

According to the third research objective, we had to look for particularities of studying the usability of the program comprehension tool. Again, the same study suggested in the previous issues could serve for this. But once more, we could broaden the search by using several types of usability evaluation techniques, such as one not involving users (e.g. usability inspection method) and another one including users. We might discover different particularities when using different types of techniques.

Based on the above ideas, we decided to do a usability study of the tool including the following three techniques:

1. Heuristic Evaluation
2. User and Task Analysis
3. Thinking Aloud Usability Testing.

Heuristic evaluation and thinking aloud usability testing would serve to explore the usability of the tool. Also, thinking aloud usability testing would be used to explore how a group of SEs who had used the tool during maintenance activities were capable of using the tool functionality. Because the idea in the research was to find qualitative information about the issues with which we were interested, both techniques would follow a discount usability engineering approach. Also, our interest was in usability issues preventing SEs from correctly using the functionality of the tool, therefore, the study would focus on searching for things about learning, efficiency of use and subjective satisfaction.

Heuristic evaluation would be the first technique applied. It would detect minor problems in the tool that developers would try to fix as much as possible before the thinking aloud usability testing.

The user and task analysis would be performed before the thinking aloud usability testing. This technique would serve to obtain a right group of SEs (e.g. participants) and a good set of tasks for the thinking aloud usability testing technique. Also, it would serve to gather information about how the tool had been used at the Company. Next, we provide specific details regarding the application of each one of these techniques.

This technique had the objective of looking for usability deficiencies in the tool by applying usability guidelines and principles. It would follow the approach described in Chapter II. That is, a group of evaluators would be asked to evaluate the tool according to the list of usability guidelines proposed by Nielsen. Three evaluators with some usability expertise would be used in order to get the best cost-benefit results. Each evaluator would proceed independently and might be accompanied by an observer.

The results would be summarized in a report and communicated to developers. Some sessions would be performed in which developers and the experimenter would review the problems together. Developers would work on solutions to the problems identified while the thinking aloud usability testing was prepared.

This technique had the objectives of (a) identifying a right group of participants, (b) generating a good set tasks for the thinking aloud usability testing, and (c) gathering information about how the tool had been used at the Company. To achieve this, a series of interviews and observations would be performed with (a) people who had conducted the initial studies about SEs work practices, (b) developers of the tool and (c) people at the Company.

Regarding the selection of participants, the following groups of SEs would be identified:

1. SEs who had not used the tool in maintenance activities but knew about the company systems (category III in Figure 3.1).
2. SEs who had used the tool in maintenance activities and knew about the company systems (category IV in Figure 3.1).

From each one of these groups, three to five SEs would be invited to participate in the study. In the rest of the document, we will refer to those SEs simply as the novice and expert participants. Both groups of participants would be used to explore the usability of the tool. Novices would help at looking for deficiencies in learning the functionality and experts at looking for deficiencies about efficiently using the functionality. Both groups would be asked for information about their satisfaction with the tool. In addition, experts would serve to explore how well they were capable of using the tool functionality.

In regards to the tasks, these would be real tasks of the type for which the tool was designed. The patterns that SEs follow when exploring software systems would be used to help develop the tasks. These patterns had been observed when the tool functionality was generated and could help at finding real sequences of using the features. Also, some tasks would be designed so that performing them optimally would involve the use of critical features that had not been altered by the solutions to the problems found in the heuristic evaluation (e.g. features that remained in the same place with same label). This was important in order to guarantee that the knowledge that expert participants had about those features were still valid in the test. In other words, to guarantee that if an expert didn't use one of those features during the test, it was because he really didn't know the feature and not because the feature had been altered. In summary, the tasks would comply with the following rules:

1. Involve exploration of a real Company system.
2. Be linked to a real scenario of a maintenance activity.
3. Be possible to perform within a certain time limit.
4. Its solution should cover the patterns as much as possible.
5. Its solution should cover as much functionality of the tool as possible.
6. Its solution should involve critical features not altered after the heuristic evaluation.

This technique had the objective of looking for usability deficiencies experienced by a group of SEs. Also, it would serve to explore how they had learned and used the tool. The participants and tasks obtained from the user and tasks analysis would be used for this technique. The technique would follow the approach described in Chapter II. That is, the participants would be asked to verbalize their thoughts while performing the tasks with the tool. There would be one session for each participant. The tasks would be given one at a time during the sessions. An observer would watch the participants and would take notes during the sessions. The sessions would be videotaped for later analysis. Once the participants had finished the tasks or the time limit had expired, they would be asked to complete a short questionnaire and debriefed for some minutes. The sessions would be performed at the Company facilities using computer equipment similar to that which the participants normally work with.

Like in the heuristic evaluation, the results would be summarized in a report and communicated to developers. Also, some sessions would be performed in which developers and experimenter would review the problems together. Some video clips of participants experiencing the problems would be presented.

Novice participants would receive a 15 to 20 minutes tutorial before the session. The tutorial would be given by a SE who was a member of the group responsible of supporting the tool within the Company.

As recommended in this type of technique, a pilot study would be performed before the sessions with the participants. Two users of the tool similar to the participants would be asked to perform the tasks. They would follow the same procedure as the participants except that these then would not complete the questionnaire and would not be debriefed. The pilot study would serve to capture possible problems with the tasks, investigate details to be careful of when performing the sessions, and other practical issues about the application of the technique.

Finally, as suggested in the literature, certain ethical considerations would be applied. The videotapes and information obtained about the participants would be kept confidential. Only people performing the study would have access to that information. Also, just small clips showing specific problems would be watched during the sessions with developers. These considerations would be informed in advanced to the participants, who would be asked for their consent.

The following chapter presents the results of our research obtained through the usability study performed to the program comprehension tool. We discuss usability deficiencies found in the tool and how those were related to not having considered usability during its development. We also discuss how usability had affected the way SEs had learned and used the functionality of the tool. We describe particular challenges encountered with studying the usability of the tool and finally, we describe issues detected with introducing usability into the KBRE project.

As mentioned in Chapter III, the study involved applying three different methodologies. Thus, we present the results chronologically, with sections about each one of the techniques. At the end of the chapter there is a section with other results that were independent of the techniques.

The first part of the study consisted of applying heuristic evaluation following the procedure described in Chapter III. That is, three evaluators with knowledge about designing user interfaces were used. All of them had some knowledge in usability evaluation although they were not usability experts; they had performed just a few heuristic evaluations before. Evaluator-1 and 2 had backgrounds in Computer Science and experience in exploring software systems (the domain of the tool). Evaluator-3 had a background in Psychology and some knowledge about programming languages, however he didn't have much experience in exploring software systems. On the other hand, evaluator-1 was a developer of the tool and therefore he was fully familiar with it. Evaluator-2 and 3 were unfamiliar with the tool and learned about it one week before the study.

The heuristic evaluation was performed as described in Chapter III. That is, the evaluators received a list of ten usability guidelines and were asked to evaluate the tool according to them. Each evaluator performed the evaluation independently from the others and proceeded as follows:

1. Evaluator-2 (the author) performed the evaluation alone and before the other two evaluators. He would serve as the observer of evaluator-3 and as the experimenter of the heuristic evaluation. This is, he was in charge of coordinating the technique and summarizing the results obtained from it. Therefore, it was important that he performed the evaluation before the other evaluators so that he wouldn't suffer any bias introduced by the results of the others. Also, evaluator-2 had to be careful when serving as the observer of evaluator-3 to not introduce any bias to this later one.
2. Evaluator-1 performed the evaluation alone and sent his results to the experimenter.
3. Evaluator-3 performed a two-hour evaluation session in which he dictated his results to an observer (evaluator-2). The observer recorded the problems and helped with questions that evaluator-3 had about the tool.

The results from the three evaluators were merged and summarized in a report. We noticed that evaluator-2 (the experimenter) had acquired many insights by observing the results of the other two evaluators. Therefore, thanks to the precaution of performing his evaluation before the others, his results were not biased and it was possible to compare them with the other two evaluators. Finally, some sessions were done so that developers and the experimenter could review the results together.

The heuristic evaluation supplied a lot of information about the usability of the tool. Most of the results were ‘possible’ usability problems that SEs could experience with the tool but there were also other insights like suggestions for improvements given by the evaluators. A total of one hundred and fourteen usability problems were identified and they can be consulted in Appendix 2. Seventy-one problems (62%) were about difficulties in learning the tool, sixty-five (57%) were about efficiency of use, and twenty-two (19%) were about both (notice that a problem can be about learning and efficiency of use at the same time). Three examples of these problems are:

• Problem: The tool had two windows that showed a hierarchy of items. Also, the tool had a menu called "Hierarchy" that only referred to the hierarchy of one of the windows. This could be confusing for SEs that don't know yet the relationship (Learning problem).

According to the usability guidelines, the problems were distributed as follows:

As we will describe later, we generated a second categorization of the problems. According to that classification the problems were distributed as follows:

According to the usability guidelines (Table 4.1) the biggest category of problems consisted of violations to the simple and natural dialogue principle. Its problems accounted for 42% of the problems found and most of them were related with (a) non-optimal functionality (30%), (b) possible users' confusion or misunderstanding with the tool (18%), and (c) bad graphical design in the user interface (16%) (Table 4.2). Some problems in this category were:

The second biggest category of problems consisted of violations to the speak users' language guideline, which accounted for 17% of the problems found. These problems were mainly related to possible user confusions or misunderstandings with the tool. A problem in this category was:

• Problem: The information at the top of one window appears editable but is not. SEs that know similar widgets from other systems could think the information is editable (Speak users' language problem).

The third biggest category of problems consisted of violations to the consistency guideline, which accounted for 13% of the problems found. There were many inconsistencies throughout the interface but major sources of inconsistencies were in (a) the design of the features and its behavior, (b) the treatment of feedback, and (c) the graphical design of the user interface. Three problems in this category were:

Together, the problems about the simple and natural dialogue and consistency guidelines accounted for 55% of all the problems found. By analyzing the problems, we saw that many of them were a consequence of the lack of systematic usability activities during the development of the tool. For example:

1. Many problems about simple and natural dialogue were deficiencies in the organization of the functionality of the tool. Many would have probably not existed if that arrangement had been designed by following tasks within real scenarios that SEs perform. Also, many were of the type of problems normally detected on a first evaluation. An example of these problems is:

• Problem: The feature for loading database files opened a pop-up window that showed all the files in the default directory, including files that were not valid databases. SEs could select wrong files and that could cause unexpected problems (Lack of evaluation problem).

1. Many consistency and graphical design problems would not have existed if the tool had been implemented by following a standard style definition. An example of these problems is:

It is important to say that the problems found were probably not all the existing problems in the tool. As we will mention in the next section, lack of domain knowledge and missing tool information hindered developers from identifying certain types of problems. Many of those problems would have belonged to the categories of (a) simple and natural dialogue, (b) good feedback, (c) good error prevention, and (d) clearly marked exits. Likewise, the tool didn't have online nor written help yet, therefore almost no issues regarding usability of the tool in combination with its help were obtained.

The heuristic evaluation was in general easy to plan and perform although there were certain things particularly more difficult or different than expected. We encountered the following issues when using this technique:

1. Although it had been relatively easy to find three evaluators with some knowledge about usability, we noticed that finding evaluators with good knowledge in usability and experience in heuristic evaluation was not easy in general. Finding a fourth evaluator would have been very difficult, for instance. We saw that people with experience in usability were not very easy to find at that moment.
2. The evaluators ended up relying more on their own intuition and experience with user interfaces than in the abstract usability guidelines when capturing problems. We noticed that little experience evaluating a system according to abstract principles was the main cause of this.
3. Insufficient information about the tool hindered some evaluators from identifying problems. As we mentioned earlier, evaluator-2 had little knowledge about the tool and evaluator-3 didn't know about the tool and its domain either. All this made them seek for whatever tool information available in order to be able to make certain judgements. For example:

• They looked for a complete list of tool requirements and specifications. They wanted to know how the functionality had been designed to address the SEs tasks. However, this information didn't exist completely and that meant they couldn’t always judge whether things in the functionality had or not usability deficiencies.

• They looked for information about the design of the user interface. They wanted to know why the functionality had been organized the way it was. Unfortunately, this information didn't exist either and that meant they couldn’t always judge whether things about such organization were good or bad designs. This happened, for instance, with some groupings of features, labels, keyboard shortcuts and graphical design issues.

We belief that, with this information, evaluators would have made better judgements about which were critical problems and would have probably identified a larger number of problems.

1. There were problems found by several evaluators and others just by one evaluator. Also, each evaluator captured a different number of problems. By analyzing the results, we noticed that different evaluators tended to find different categories of problems:

• Evaluator-1 found 28% of the total number of problems captured in the technique. He found many problems regarding 'bugs' and the utility of the tool, but tended to find fewer problems about user interface design than the other two evaluators. For example, he found many things about (a) the lack of certain tool functionality, (b) the correctness of the results given by the tool, and (c) the correctness and completeness of the feedback given by the tool. He was able to judge whether a feature was indeed necessary or not for exploring software systems and whether it was working correctly. On the other hand, his extensive experience with the tool made him unable to notice certain types of problems, such as inconsistencies, graphical design, feedback, etc. For example, he didn't find many things about (a) difficulties experienced by SEs learning the tool (like problems for finding certain features, for instance), or (b) inconsistencies and lack of standards in the user interface.

• Evaluator-3 found 72% of the total number of problems captured in the technique. He found, on the other hand, many problems about user interface design but less about the utility and correctness of the tool. For example, he was very good at noticing problems about (a) consistency and standards, (b) graphical design, (c) labels, (d) difficulties for learning the tool, and (e) difficulties understanding error messages.

• Evaluator-2 found 39% of the total number of problems captured in the technique. He was more balanced than the other two evaluators in finding deficiencies related with both: (a) the utility of the tool and (b) general user interface design. Although, overall, his results were more similar to evaluator-1; that is, he tended to find more problems with the utility and correct behavior of the tool and less about user interface design. We think this probably happened due to his background in Computer Science and experience exploring software systems (the tool domain). His insights revealed deficiencies about tool learnability and efficiency of use.

1. Merging and summarizing the problems required much time and energy. As stated in the plan, the problems would be communicated to developers so that they worked on solutions while the thinking aloud usability testing technique was prepared. But, solving the problems efficiently required that developers understood the problems very well. As suggested in the literature [Jeffries 1994], the problems had to be presented in the report in a way that facilitated their understanding by developers. However, the problems as given by the evaluators presented many difficulties so that these could be easily included in the report:

• Many didn't provide information about its location in the tool.
• Many were not stated in their most general way. Instead, they only referred to particular instances of the problem within the tool.
• Many didn't have explanations about why they could be in fact problems to the SEs.
• Many didn't have suggestions for improvements.
• Many didn't provide an estimation of its severity.
• Many had not been related to the usability guidelines.

At the end, many problems had to be located in the tool, verified for its existence, rephrased or rewritten before including them in the report. The time and effort for this task was much bigger than expected.

We noticed that little experience with the technique and with communicating usability problems to developers had been the main reasons why the evaluators had not reported the problems better.

1. Having evaluators with different levels of experience in (a) the tool domain and (b) using the tool, provided a broader and larger spectrum of problems. Regarding the first type, we noticed that the tool domain was so specific and complex that only evaluators 1 and 2 who had background in Computer Science and knowledge exploring software systems were able to see deficiencies in the functionality regarding how it helped conveniently at exploring software systems. On the other hand, little knowledge in the tool domain made evaluator-3 capture other types of problems not detected by the other two, such as screen design problems. With regard to the second type, we noticed that having a developer of the tool as evaluator-1 had been very helpful. He had extensive knowledge using the tool and that made him capture many insights about its correct behavior. On the other hand, evaluators-2 and 3 didn't know how to use the tool and that made them identify many problems with learning the tool.

As mentioned in Chapter II, some authors describe that best results are obtained by using double-experts as evaluators (e.g. evaluators with expertise in usability and the domain of the system). However, we obtained a broader spectrum of problems by having different types of evaluators as described above. That is, we had one single-expert and two double-experts. We also added the category of expertise using the system and we had one expert and two novices on using the tool.

We do not know if this combination can always be better in heuristic evaluations, particularly in evaluations of systems where the problem domain is very complex. But this is a topic for further research where more experimentation could be performed.

2. The number of problems was much bigger than expected and the project was overwhelmed with the results. Many of them represented difficult or costly things to repair and some were not possible to fix. The fact that the tool was already implemented imposed even more difficulties to fixing the problems. All these were causes of not having evaluated the tool before and lack of knowledge of what usability deficiencies meant.
3. Prioritizing and organizing the problems became critical to developers. According to the plan, developers should fix as many problems as possible while the thinking aloud usability testing was prepared. The large number of problems and limited time to fix them made it critical for developers to prioritize and organize the solutions. For that, developers immediately looked at the severity ratings provided in the report. Nevertheless, they ended up relying more on other factors such as time and resources available to prioritize the problems.

To organize the problems, developers tried to use the usability guidelines but these were very abstract for them and consequently not very useful. Based on that, we decided to create a second classification of the problems that they could understand and help them better at organizing the problems. This classification divided the problems in terms of the tool functionality and difficulties that SEs could have with it (Table 4.3).

1. Non optimal functionality

2. Possible confusion

3. Possible misinterpretation

4. Unnecessary functionality

5. Lack of functionality

6. Lack of consistency

7. Graphical design

8. Feedback

9. Lack of robustness

10. Wrong behavior

11. Bad labeling

12. Lack of labeling

TABLE 4.3 Second classification of the problems

4. Developers had difficulties identifying good solutions to the problems. We noticed it was difficult for them to understand the problems in their most general way. They required much time to see solutions that did not lead to other obvious usability problems. Also, they had difficulties comparing the effectiveness of several proposed solutions. We noticed that insufficient usability knowledge and experience solving usability problems was the main cause of this.

On the other side, insufficient tool information also affected developers ability to identify good solutions to the problems. Some developers were not in the KBRE project yet when certain decisions about the design of the tool had been made. Consequently, insufficient tool information meant that these developers didn’t know details about the organization of the functionality as well as the design rationale of the user interface. Hence, developers experienced difficulties judging the current design and finding solutions consistent with the rationale used. We noticed this issue made people in the project start thinking about generating some tool documentation.

5. The problems were not always fixed properly and with optimal solutions. Many times, they were fixed for the particular instances mentioned in the report. Also, certain solutions would very likely lead to other usability problems. We noticed that non-optimal understanding of the problems and its consequences, having the tool already implemented, limited time and non-optimal prioritization of the problems were the main causes of this.
6. The sessions between experimenter and developers were extremely helpful. We confirmed that it had been essential to have the report properly written. However, we saw that the report alone would have been far from enough to communicate all the information to developers and facilitate the generation of good solutions. Instead, the sessions help at:

• Reviewing the problems in its most general way.
• Reviewing implications of sub-optimal solutions.
• Evaluating different proposed solutions.
• Prioritizing the problems.
• Deciding about problems that should wait to be fixed. There were some problems for which it was not clear if they seriously affected SEs and therefore their solution should wait until they were confirmed in the thinking aloud usability testing.

We want to mention that the experimenter played a key role in these sessions. He had to actively help developers in all the above issues.

1. Certain problems provided by this technique were not always very convincing to developers. Even though developers accepted most of the problems as deficiencies in the tool, some developers were not very convinced that certain problems should be fixed. We noticed that the lack of belief from developers came mostly from insufficient understanding of how those problems could produce an effect on the SEs, particularly those that could produce a slow but strong impact on them.
2. After fixing several problems, developers started having some difficulties maintaining control over the source code of the tool. As one developer expressed: "this code is starting to suffer from the spaghetti syndrome, we are losing order and structure in it". We noticed this was a consequence that the tool was already implemented.
3. Knowledge in the tool domain had been essential for the experimenter. He had to help evaluator-3 with several issues about the domain. Also, he had to fully understand the problems, appropriately explain those in the report, communicate them to developers, and provide advice during the generation of solutions. We noticed that the tool domain was so specific and complex that insufficient knowledge in it would have seriously hindered the experimenter in these activities.
4. The heuristic evaluation was a good first step for introducing usability and user centered design into the KBRE project. It made developers become aware of what usability meant and the level of usability of the tool. It also meant that the project matured regarding user centered design. By the end of this technique, the project had passed from the skepticism stage to the curiosity stage, as described in Chapter II.

The next part of the study consisted of performing a user and task analysis as mentioned in Chapter III. That is, a series of interviews and observations were performed with (a) people who had conducted the initial studies about SEs work practices, (b) developers of the tool and (c) people at the Company.

The first step in the analysis consisted of some interviews with the people that had conducted the initial studies about SEs work practices and developers of the tool. We asked them about the KBRE project, the problem it addressed, the results from those studies, and the tool.

The next step consisted of field observations and interviews with some SEs and other people in the Company involved in the KBRE project. We wanted to gather more information about the project and the tool as well as obtaining a list of possible participants for the thinking aloud usability testing technique.

Once a list of possible participants had been obtained, the next step was to interview each one of them in order to select those who were the best according to the objectives. We had to identify a right group of novice and expert participants. It is important to mention that the experiment was not explained to them at this point.

Other SEs not in the list of possible participants were interviewed next. We asked them to give some examples of typical scenarios and tasks in the exploration of the Company systems during maintenance activities. Such examples would be used in the generation of the tasks. It was important that the possible participants didn't know about the tasks that they would be asked to perform and that's why we asked other SEs for these examples.

After the previous interviews, the manager of some SEs was also interviewed. He had been in involved with the KBRE project and was familiar with the tool. We asked him to verify all the information collected in the previous steps and contribute with missing information. We also asked him for some examples of typical scenarios and tasks in the exploration of the Company systems during maintenance activities.

At this moment, the study and its policies were explained to the possible participants and they were asked to participate in the experiment.

Once having obtained a group of participants, the rest of the analysis focused on generating the tasks. As described in Chapter III, the procedure for obtaining the tasks was performed. However, as it will be discussed below, certain difficulties arose with it as planned and it had to be modified.

The user and task analysis provided extremely useful information. It helped to find a right group of participants and a good set of tasks according to the objectives. It also provided many insights about (a) the target population of users of the tool, (b) their needs and tasks, as well as (c) how SEs had used tool during the past 1.5 years and how useful it had been for them. All sources of information were good and provided useful information. Frequently, different types of information were obtained at the same time. Overall, the selection of the participants was completed before the tasks had been fully determined.

One of the first things found was that the target population of users of the tool had been defined as: 'those SEs performing maintenance activities that followed the Just in Time Comprehension approach'. Also, that the tool had been designed mostly based on information from one particular group of SEs. However, the group of SEs who had been given the tool included (a) the SEs in this previous group as well as (b) other SEs in other groups that also performed maintenance activities and followed the Just in Time Comprehension approach.

It was found next that only four SEs had used the tool during maintenance activities so far, and those had been using the tool very differently. Some had employed the tool everyday and others just on specific days. Some had employed the tool for many of their tasks and others only for a few specific tasks. Also, three of them had been more recently hired at the Company. These commented on having used the tool in order to learn the Company systems. On the other hand, many SEs had not used the tool yet and a few had not even heard about it.

All this made us see that the group of SEs who had used the tool during maintenance activities was much smaller than developers imagined. Developers thought that the tool was more widely known and used by the SEs. It was clear that SEs had not been monitored much after the initial studies.

At this moment, we identified a group of SEs who had used and had not used the tool during maintenance activities. These were good candidates to be the expert and novice participants for the testing respectively.

Next, we noticed that SEs in distinct Company groups had differences that could affect their interaction with the tool. SEs in different groups (a) had different goals in mind when exploring the Company systems and (b) tended to have a very different knowledge about the Company systems. These differences meant that two SEs in different groups could:

1. Approach the same goal with different tasks.
2. Perform the same tasks but in a different order.

These, in turn, had an influence in:

1. When they explored a Company system.
2. How they explored a Company system. That is, what the starting and final points of the exploration were, which paths of exploration they followed, how deep within the Company system the exploration was, etc.

These differences, together with the fact that the tool had been mostly designed based on one particular group, made us conjecture whether the tool was equally useful or not to all SEs within the defined target population. In fact, as it will be discussed later, we confirmed this hypothesis later during the thinking aloud usability testing. The following is an observed example of all these issues:

Likewise, we noticed that SEs within the same Company group had also differences that could affect their interaction with the tool. SEs in the same group could have a completely different knowledge of the same Company system, and that could have the effect that for the same goal certain SE had to explore the Company system but another one not.

At this moment, the candidates were invited to participate in the test. Eight participants were obtained for the thinking aloud usability testing. Initially, as it will be explained below, four were identified as novices and four as expert participants (categories III and IV in Figure 3.1 respectively), but later, we discovered that one of the experts belonged in fact to the novice category (he had very recently started using the tool). Seven were in a same Company group and one of the experts (the one we discovered later was a novice) was from a different group. Some of them had participated in the initial studies about SEs work practices. It is important to mention that the group of experts consisted of the only three SEs who had used the tool during maintenance activities.

Finally, the tasks for the experiment were generated. As stated before, SEs were asked for examples of typical scenarios and tasks in the exploration of the Company systems during maintenance activities. Here, we noticed that SEs always provided examples that were extremely complex and focused on very high level goals. Many times, they had not been able to accurately describe what sub-tasks or steps would be necessary to perform for solving the examples, neither what explorations of the Company system had to be done. This made us see that SEs not always consciously realized many of these issues during maintenance activities.

The set of tasks obtained contained twenty-eight tasks complying with all the rules established; however, as it will be discussed below, these had to be modified two times later, one after the pilot study and another one after the sessions with the first two participants.

The user and task analysis was easy to plan but not very easy to perform. We encountered the following issues during this technique:

1. Efficiently employing theory and principles at the moment of conducting the technique was not very easy to do. Theory and principles about planning and conducting the technique were simple and straightforward to understand. But remembering all and using them efficiently while conducting the technique was not easy. We noticed that the nature and amount of information gathered created a "messiness" atmosphere in which it was difficult to recognize the best principles and the way to use them. This made us see that obtaining best results definitely required experience and maturity with the technique.
2. The tasks were particularly hard to generate. Summarizing, the original procedure planned for determining the tasks was as shown in Table 4.4 below.

Also, the tasks had to comply with the following rules:

• Involve exploration of a real Company system.
• Be linked to a real scenario of a maintenance activity.
• Be possible to perform in less than two hours.
• Its solution should cover the patterns as much as possible.
• Its solution should cover as much functionality of the tool as possible.
• Its solution should involve critical features not altered by the heuristic evaluation.

However, we encountered the following difficulties:

1. The examples of real scenarios and tasks given by the SEs were not useful for determining the tasks. As mentioned above, those examples were extremely complex and focused on very high level goals. Also, SEs had not been able to say what sub-tasks or steps would be necessary to perform for solving the examples, therefore:

• It was impossible for us to know if the participants would use the tool for solving those examples.
• It was almost impossible to see what patterns participants would follow when solving the examples.
• It was very difficult know what and how the tool features might be used.
• Finally, many of the examples were not possible to complete in less than two hours.

1. A second difficulty we encountered was that finding sections of the Company system that the tasks explored was hard. Based on our previous observations of SEs, the participants could be extremely familiar with particular parts of the Company system. So, we had to create tasks that involved exploring sections of the system that were not extremely known by all of them. Otherwise, it was possible that a certain participant didn't have to use the tool for solving a task simply because he knew the answer already.

• For tasks where the participant had to determine if something existed, we phrased them so they asked the participant to count the number of things found.

• We phrased all the tasks asking the participant whenever possible to see the answer(s) in the windows of the tool.

1. A final observation when generating the tasks was that some of the missing tool information would have been useful. The specifications, for instance, could have helped to understand better how the tool features supported SEs tasks, and therefore would have helped at identifying sequences of features for solving real SEs tasks.

1. Once generated, the tasks became very helpful for other purposes. Before the study, the KBRE project didn’t have any examples of real scenarios and tasks of what the tool was used for during maintenance activities and that is why we had to generate some for the test. But, once the tasks had been generated, we noticed these became very helpful to developers. They started using them for testing solutions to the problems found in the heuristic evaluation as well as testing new design ideas of future tool features.

Once having obtained the participants and tasks, the next part consisted of applying the thinking aloud usability testing technique following the procedure described in Chapter III. That is, the participants were asked to verbalize their thoughts while performing the tasks. The sessions were videotaped for later analysis. The tasks were given to them one at a time. An observer watched the participants and helped them when necessary. The observer took notes about usability insights noticed during the sessions. These notes were used later during the analysis of the videos. Once finishing the tasks, the participants were asked to fill a short questionnaire and debriefed for some minutes. The sessions were performed at the Company facilities using computer equipment similar to that with which the participants normally work. Novices received a 15-20 minutes tutorial before the session given by a SE member of the group supporting the tool within the Company.

A pilot study was performed before the sessions with the participants. Two developers of the tool participated in this test. They were asked to perform the tasks generated for the experiment. They had extensive knowledge of the tool and exploring software systems but no knowledge about the Company systems. Their sessions followed the same methodology described above but they didn't complete the questionnaire, were not debriefed and didn’t receive the tutorial. This test revealed the necessity to make some changes to the tasks and a second version of them was created. Some new tasks were included in order to exploit more tool functionality, and some tasks were rephrased due to observed difficulties with their understandability.

Next, the first two participants performed their sessions. These were one novice and one expert participant. These sessions revealed certain things not noticed with the pilot study that made it necessary to do certain changes to the tasks again. Thus, a third version of them was created. This was the final version of the tasks and can be consulted in Appendix 3.

At this moment, the rest of participants performed their sessions. They were asked to do the third version of the tasks. It is important to mention that although the first two sessions were done using the second version of the tasks, they were fruitful in capturing many usability insights about the tool and those were included in the final analysis.

Once finishing the sessions with all the participants, the next step was to merge and summarize the information in a report. The annotations, videos and questionnaires provided useful information that was included in the report.

Finally, like in the heuristic evaluation, some sessions were scheduled so that developers and experimenter reviewed the problems together. For some problems, short video clips about them were presented. The clips showed the participants experiencing the problems.

As stated in the plan and told to participants, the videos and other information about them were all kept confidential. Only people involved in the usability study had access to that information. Also, only clips showing specific problems were watched during the sessions with developers.

Like with heuristic evaluation, much information about the usability of the tool was found with thinking aloud usability testing. Most of that information was usability problems that participants experienced with the tool but there was also information about user satisfaction, possible improvements, and insights about the utility of the tool. Also, this technique provided information about how expert participants were capable of using the tool functionality.

A total of seventy-two usability problems were identified and those can be consulted in Appendix 5. Thirty-eight of them (53%) had already been identified in the heuristic evaluation and were confirmed in this technique. Thirty-eight of the problems (53%) were about difficulties in learning the tool, fifty-four (75%) were about efficiency of use, and twenty (28%) about both (notice that a problem can be about learning and efficiency of use at the same time).

Learning problems were mostly about difficulties for (a) perceiving that the tasks could be achieved by using the tool, (b) finding all the necessary features to complete a task, confusions due to (c) misleading information provided by the tool and (d) behavior of the tool different than expected. Two examples of these problems are:

• Problem: Much of the power of the tool resides in combining functionality between two of the windows. However, it was very hard for participants to learn how to combine that functionality in order to solve a task (Difficulty in noticing that goals can be achieved).

On the other hand, efficiency of use problems were mostly related with obstacles and difficulties for achieving the tasks in optimal ways. Many times, the participants were able to achieve the tasks but they made many unnecessary or non-optimal steps. One example of these problems is:

• Problem: The pop-up window of a searching feature often hid the matches found. Whenever that happened, participants had to move that window in order to see the match (Efficiency of use problem).

From the seventy-two problems, five (7%) were experienced by the eight participants, ten (14%) were experienced by more than six, and thirty-one (43%) were experienced by less than three. Two examples of these problems are:

• Problem: In one of the windows, different objects could look exactly the same. Whenever that happened, participants had to select each one of them in order to be able to see the differences (Problem experienced by all participants).

Novice participants experienced forty-five problems (62%) and expert participants experienced forty-eight problems (66%). Also 20% of the problems were only experienced by novices, 7% were only experienced by experts, and the rest (73%) were experienced by both. Most of the problems experienced by novices (75% of them) were about learning the tool. Most of the problems experienced by experts (81% of them) were about efficiency of use. Some examples of these problems are:

• Problem: Many important features were within middle mouse menus, but middle mouse menus were extremely hard to discover by participants (Learning problem affecting mainly novice participants).

• Problem: The Auto-grep feature serves to find occurrences of one object within another. It powerfully implements one of the most common SE tasks, which was noticed during the study of SEs work practices. However, the implementation was so non-intuitive and difficult to learn that none of the participants already knew how to use it and it hadn't been used yet (Learning problem affecting both participant groups).

• Problem: Nested procedures/functions in the source code were not clearly identified. Therefore participants had to find them manually by observing the code (Efficiency of use problem affecting mainly expert participants).

• Problem: The tool didn't have undo facilities and participants experienced situations where erroneous actions couldn't be reversed (Efficiency of use problem affecting both participant groups).

According to the same classifications used in the heuristic evaluation, the problems were distributed as shown in Tables 4.6 and 4.7 below. Like in the heuristic evaluation, the biggest category of problems consisted of violations to the simple and natural dialogue guideline (Table 4.6). Its problems accounted for 61% of the problems found and most of them (48%) were about non-optimal functionality. The second biggest category of problems (22%) consisted of violations to the good feedback guideline. Some examples of these problems are:

• Problem: Pop-up menus worked so that they stayed only if the mouse button remained pressed. Normally, participants required time to look and explore the features in those menus. Many times, participants were observing a menu and accidentally released the mouse button, which led them to lose the menu. Also, participants had difficulties with (a) keeping the mouse button pressed for a long time, and (b) keeping it pressed while moving the mouse in order to select something (Non-optimal functionality problem).

• Problem: The tool did not always presented feedback when there were no results from an action. Many participants didn't easily notice when an action with an empty result had finished. Sometimes they scrolled the windows trying to identify any possible changes to the information. Some executed the action again several times to convince themselves that the action had had an empty result. Participants started developing the concept of not trusting the feedback given by the tool (Feedback problem).

An analysis of the problems revealed that many were consequences of the lack of systematic usability activities during the development of the tool. For example:

1. Many problems were related with not efficiently considering the context in which SEs worked in a particular moment. We noticed that depending on the context participants performed different sequences of steps to accomplish the same task. However, the tool provided ways to solve tasks that were not context sensitive, consequently, the tool sometimes helped participants efficiently and others times not. It is possible that more knowledge about how SEs worked depending on the context could have enabled the creation of a context sensitive tool design.
2. In the tool, some features worked differently depending on the objects selected. Hence, some problems were because the tool did not inform participants efficiently about (a) the objects selected in a particular moment and (b) how certain features would work based on that selection. These problems would have been detected before if the tool had been tested with SEs.
3. Some problems were related with not providing support in all the steps involved in completing full SEs’ tasks of exploring the Company systems. It is likely that the use of scenarios of complete SEs goals for designing the tool would have avoided many of these problems. Furthermore, many of them had not been detected because, as we will discuss later, developers had been testing the tool with low-level SEs tasks only.

The questionnaire and debriefing were also productive. They revealed aspects about user satisfaction, utility of the tool and how it had been used so far. The number of participants was not big enough to generate a reliable statistical analysis. But the results were used to complement the other information and obtain a more accurate picture of the status of the tool. The questionnaire and average of the answers were as shown in Table 4.8 below. Among the most important insights were:

• Participants considered that the tool was easy to learn although learning took time (something we noticed SEs didn't have too much of).
• Participants considered that the tool was useful at exploring the software systems, but only once certain practice with it had been acquired.
• Participants considered that the user interface of the tool was easy to use although it was not extremely efficient.

In general, the opinions were very homogeneous (standard deviation "stdev", around 0.5). Other interesting insights were:

1. Novice participants considered that learning the tool was a bit difficult (1.8) but experts considered it easy (3). This was probably because experts had already learned the tool and they were not experiencing many learning difficulties anymore.
2. The opinions of whether the tool was helpful at typical sequences of actions for exploring the system were very consistent in novices (3.3, stdev of 0.5) and very inconsistent in experts (stdev of 1.3).
3. Opinions of novices and experts also differed about the efficiency of use of the user interface. Experts were consistent in their answers (2.5, stdev of 0.5) while novices did not agree on them (stdev of 1.1).

An interesting insight arose from one of the expert participants who had learned the tool since it was created. He considered that the version used in the testing was not as helpful as some previous versions. He had learned how to perform his tasks on the first versions of the tool, and differences in the version used in the test (the latest version) had forced him to relearn how to accomplish the tasks on it. Also, not all the tasks possible to do in the previous versions could be done in this version. This made us see that changes in the features of the tool had not been properly analyzed and had affected the SEs. Also, this was another indication that SEs had not been monitored.

All the tasks were completed in the ten sessions (including the pilot study). However, certain tasks were completed in very inefficient ways. That is, not in the best way possible to complete them with the available tool functionality. This indicated that novices were not able to discover the best resources of the functionality and experts did not know many of them yet. For example, some tasks required looking for all the occurrences of a particular string in many files. The tool provided a feature for solving such task in one step. However, none of the participants used that feature and, instead, completed the tasks by using another feature that found occurrences of a string one by one within one file only. So, not knowing about the more powerful feature made that participants completed the tasks using the less powerful feature many times.

These observations along with information obtained from the debriefs with the expert participants, made us see that none of the experts knew all the tool functionality yet and no one had used many of the most advanced features in it. Therefore, because these participants were all the SEs who had used the tool during maintenance activities, that meant it was impossible for the KBRE project to assess the functionality of the tool at that moment. In other words, nobody in the group of SEs who were given the tool had used all the power of the functionality during maintenance activities yet.

The most common reasons why SEs had not used all the functionality were:

1. They had not discovered many features.
2. They had not understood the meaning of certain features by looking at its label.
3. They had not understood the way certain features had to be used. Certain features required to be used in a peculiar way or order. So, each time that SEs had tried those features they had been unlucky and had employed them in an incorrect way.

An interesting observation was that many things intended for experienced users of the tool, such as keyboard shortcuts, were rarely used (that’s why there were not many problems with them). This was another indication that expert participants had not used all the functionality yet.

All these made us see how not having considered usability had caused poor learnability in the tool design, which in turn had caused that SEs never learned and used all the functionality during the past 1.5 years in which the tool had been available. Thus, making impossible for people in the KBRE project to see whether the experimental functionality of the tool could help SEs to achieve their goals more efficiently or not. In other words, making it impossible to assess the functionality.

On the other hand, participants expressed a positive attitude towards the tool. Novice participants commented on advantages they noticed between having the tool and manually exploring the Company systems. Expert participants commented on having received some benefits from the tool, particularly those who had used the tool for learning the Company systems. However, based on our observations about SEs needs we believe that if expert participants had known all the benefits possible to get from the functionality, they would have probably had a much better opinion about the utility of the tool. None of them had been aware of all the possible benefits so far.

We want to mention that at the end of each session, we instructed participants on how they could have achieved the tasks very efficiently. We saw that they easily learned the features involved in solving the tasks and made comments such as: "if at least somebody had told me about that feature before, I would have been using the power of this tool since long time ago". This made us realize that besides good learnability, other means for helping SEs at learning the system would have been very helpful for the KBRE project.

In general, the application of this technique greatly helped participants at learning the tool. Solving the problems posed by the tasks and instructing them on efficiently using the functionality for achieving those tasks made that they learned the core functionality of the tool and realized great benefits possible to get.

During the study, we saw that participants encouraged other SEs to start using the tool. This situation was very well received by the KBRE project, which had always been seeking to have SEs using the tool. This made us see that good usability could have generated positive attitudes in SEs learning the tool and those would have encouraged other SEs to learn it and use it, which in turn would have helped in the assessment of the functionality.

Finally, because nobody knew well all the tool functionality, it is likely that many usability deficiencies were not captured, particularly about efficiency of use. Finding more of this type of deficiencies required testing with SEs who knew the functionality better. This made us see that having the same participants from this test in other future tests could reveal more insights about efficiency of use.

The thinking aloud usability testing was in general easy to plan and perform but like in the other techniques, there were certain things particularly more difficult or different than expected. We encountered the following issues during this technique:

1. Planning and applying this technique took much longer and was more difficult than the heuristic evaluation. There were many more practical issues and details to deal with than in the heuristic evaluation. Many were in regards to using the video and audio equipment, performing and controlling the sessions, and analyzing the information. We noticed that insufficient experience with the technique was the main cause of this.
2. The pilot study was essential and very productive. It revealed some problems with the tasks and many other practical issues about the application of the technique, such as the use of the video equipment, difficulties with testing at the normal Company facilities, etc.
3. There were some interesting insights during the sessions with the first two participants.

• Participants had much more knowledge about the Company systems than developers and that made them see several different ways for solving the tasks, including some in which the tool was not necessary. Developers, on the other hand, found the tool as the only way to solve the tasks. This made us confirm our observations that in order to force participants to use the tool, the tasks had to be cautiously selected and phrased, otherwise, they could have solved the tasks without using the tool. We found that the tricks used for writing the tasks had worked well, but as we will discuss next, certain adjustments to them were done.

• Participants were harder to control and it was more difficult to make them focus on the tasks than developers. They asked many more questions during the sessions than developers. When participants found the answer of a task, they tended to turn and look at the observer, express the answer to him and then wait for a gesture from him. Consequently, we realized that the tasks had to be phrased so that the participants could talk while performing them but not requiring (as much as possible) help from the observer. In other words, phrasing them in a way that the communication between observer and participants was kept at a minimum. Based on this we adjusted the tricks for writing the tasks as follows:

• For tasks where the participant had to determine if something existed, we phrased them asking the participant to write down the number of things found and sometimes its names (depending on how long the names were). This included writing "none" in case of not finding anything.

• For tasks where the participant had to reach certain conclusion, we phrased them asking the participant to write down his conclusion (whatever that was).

• We phrased all the tasks asking the participant whenever possible to see the answer(s) in the windows of the tool.

These two previous insights (a and b) made us see that the real target population of users of the tool was not 'those SEs performing maintenance activities that followed the Just in Time Comprehension approach' as it had been originally defined. But rather, those SEs that performed exactly the type of maintenance activities of the particular group of SEs from which the tool had been mostly based.

1. Two hours was about the maximum time for good sessions. The average times of the sessions were 1:32 hrs for novices (not including the tutorial) and 1:13 hrs for experts including the questionnaire and debriefing. We noticed that longer sessions would have been uncomfortable for the participants.

On the other hand, the tasks obtained with the modified procedure worked very well. All the tasks could be completed in less that two hours and, as we had seen, many tasks typical of what the tool was designed for could have taken much longer than that.

2. All sources of information used in the technique provided useful and different types of data. The annotations, questionnaire and debriefing served to capture usability problems with the tool but were particularly good for capturing comments from the participants about how much the tool had been used so far and how useful it had been for them. It was possible to relate usability problems with participants' opinions. These comments revealed aspects about:

1. Participants' expertise with the tool.
2. Participants' attitudes towards the tool.
3. Participants' problems beyond the scope of the tool that had affected its use.
4. Strengths and weaknesses of the tool according to real goals of participants.
5. Utility of the tool according to participants' Company group.

The videotapes, on the other side, were extremely helpful at confirming aspects captured on the annotations and revealing many usability problems not observed during the sessions.

1. We noticed several issues regarding videotaping:

1. Having the videotapes was extremely helpful. We noticed that usability sessions can be very difficult to conduct by only one person and especially if that person is not yet a usability expert. In our case, videotaping released some pressure in capturing usability problems and gave us more resources for controlling the sessions.
2. On the other hand, the analysis of the videos was tiring and very time-consuming. We had to spend about 3 to 5 times the duration of a video in order to analyze it. Videotapes of novices lasted an average of 1:14 hrs and videotapes of experts an average of 0:56 hrs, however, the total time spent analyzing the tapes for all the eight sessions was about forty hours.

1. Like in the heuristic evaluation, the number of problems was big again and that increased the concern of people in the KBRE project about the usability of the tool and its consequences.
2. Like in the heuristic evaluation, the sessions between experimenter and developers were extremely helpful. Again, we confirmed that having the report properly written had been essential. However, the report alone would have not been enough to communicate all the information to developers. Instead, the sessions served to:

• Watch video clips of the participants struggling with the tool. We found this was extremely effective. Developers easily understood the problems and developed commitment to solve them.
• Understand the problems and prioritize its solutions.
• Experiment with different procedures for presenting the problems and video clips to developers. The one we found most effective was:

1. Ask developers to read a problem.
2. Establish a short discussion about it.
3. Show the video clip(s) presenting the problem.

Again, the experimenter played a key role in these sessions. He had to actively help developers in many issues.

1. Following the policies of the study about confidentially created a feeling of professionalism and commitment in everybody around it, including developers, participants and other people in the Company. We noticed that explaining the procedures and policies to everybody before the study, and then following them carefully had been essential.
2. The thinking aloud usability testing was an excellent step after the heuristic evaluation to continue introducing usability and user centered design into the KBRE project. It increased developers’ awareness about the usability status of the tool as well as their knowledge in usability. Like in the heuristic evaluation, we saw that the KBRE project matured more in its attitude towards user centered design. By the end of this technique, it had passed from the curiosity stage to the acceptance stage, as described in Chapter II.

Up to now we have presented the particular results obtained with and during the different techniques applied in the study. Next, we present other results independent of the techniques.

1. In total, the tool presented a large number of deficiencies. One hundred and forty-eight different usability problems were found in the tool. According to the usability guidelines, the problems in both techniques were distributed as shown in Figure 4.1 below.

Most problems were related to (a) simple and natural dialogue, (b) speak users' language, (c) consistency, and (d) good feedback. Many of these were effects of having designed the tool with insufficient information about complete SEs' tasks and working environment as well as user interface guidelines and standards. Many problems were the type of problems that normally appear on a first usability evaluation. We found that many could have been easily avoided with the application of certain user centered design activities, such as having used complete scenarios and tasks to drive design, having evaluated tool with prototypes and having followed user interface standards and guidelines more systematically.



FIGURE 4.1 Problems according to the usability guidelines

 

Ninety-three problems (62%) were about deficiencies in learning the tool, eighty-seven (59%) about efficiency of use, and thirty-two (21%) about both (Figure 4.2).



FIGURE 4.2 Types of problems found using both techniques

These problems indicated that the tool design did not facilitate learning and using efficiently its functionality. In fact, as stated earlier, those problems had already affected SEs. None of them was capable yet to efficiently exploit all the functionality.

2. Not all the problems meant that something in the tool had to be fixed. The results indicated that good tool learnability and other means for helping SEs to learn the tool were key elements for assessing the functionality. Many problems existed simply because participants didn't know certain information, but once they knew it, the problems didn’t exist anymore. We noticed that developers realized that some learning problems did not necessarily mean that something in the tool had to be fixed, instead, including some information in a tutorial or training course, or in a manual or on-line help could possibly solve the problem. Hence, people in the KBRE project started thinking on other possible means for helping SEs to learn the tool besides improving its learnability, such as training the SEs with some courses or practical examples.
3. Testing the tool without the intention of solving high level SEs goals had led to not detecting certain deficiencies in the tool. We noticed that developers tested the tool with typical tasks that SEs do for exploring the Company systems, however, when they performed such tasks they never had the intention of solving some high level goal that SEs normally had. This phenomenon was also observed with the SE who gave the tutorials to novices. In the sessions, we had observed that with different goals in mind, participants (e.g. SEs) solved the same task using different sequences of features. Consequently, not having high level goals in mind had made developers always complete a task using the same sequence of features, thus, not exploring other possible paths of using the features that SEs could perform, and ultimately, not detecting problems with them. In fact, we noticed that the sequences of features used by developers to solve the tasks tended to encounter many fewer 'bugs' than others used by participants to solve the same tasks. This certainly showed that sequences used by developers had been already cleaned of errors.

On the other hand, not having a high level goal in mind when solving the tasks during the demos had only shown a partial power of the functionality. That is, demos had only shown how the tool could be useful under particular circumstances.

4. Insufficient communication between the KBRE project and SEs had been a cause seriously affecting assessing the functionality. The tool had been given to a group of SEs so that they used it during maintenance activities. On one side, poor usability in the tool had affected some of those SEs from exploiting the functionality efficiently. But, on the other side, a large number of SEs had not used the tool at all. These last SEs had obviously not been affected by poor tool usability but had not used the tool for other reasons, such as ignorance that it existed or was available, low motivation or limited time for trying it, etc. On the other hand, the project had not monitored the SEs much, and therefore, they never had known the status exactly.

We noticed that a permanent channel of communication between the project and SEs was key for having a group of SEs using the tool. The project had not established such channel for several reasons, such as, ignorance about its importance, lack of people involved in it, insufficient time and resources, etc. We found that the usability study worked extremely well for providing that channel.

5. Combining heuristic evaluation and thinking aloud usability testing worked excellent for obtaining information about the usability of the tool. One hundred and fourteen problems were found with the heuristic evaluation and seventy-two with the thinking aloud usability testing. Thirty-eight of the problems found with the heuristic evaluation (33%) were confirmed with the thinking aloud usability testing (Figure 4.3).

Both techniques complemented each other very well. Each one provided different types of problems. Also, by using both, we could confirm certain problems and obtain a more reliable severity of them.



FIGURE 4.3 Percentage of problems found by the techniques

6. The study, as performed, worked very well for introducing usability and user centered design into the KBRE project. We noticed that applying two evaluation techniques took longer but produced a much better effect in the project. Both usability evaluation techniques worked well but in particular thinking aloud usability testing.

Although late in the tool lifecycle, having introduced usability at that time was worthwhile. The study revealed problems and issues unfeasible to solve, but it helped the project seeing how to proceed for assessing the functionality. Before the study, the project was facing the problem that SEs had not used the tool much and it was not known exactly why and what to do.

By the end of the study the project had passed from the skepticism stage to the acceptance stage, as described in Chapter II. We noticed that people in the KBRE project became aware of the following issues:

• Assessing the functionality at that time was not possible yet.
• It would be necessary to have a group of SEs efficiently using the functionality in order to assess it. Therefore, certain activities should be planned for helping SEs to learn the functionality. They realized that usability studies could be one such activity. Also, it would be important to monitor SEs more closely.
• Solving the problems in the tool would indeed improve its usability but that would be only a first step towards achieving good usability. They realized that (a) problems captured this time were probably not all the existing ones and (b) solutions to the problems found should be tested with SEs. They saw it would be necessary to have more iteration of evaluation and redesign.
• Any new functionality should be prototyped and evaluated before being implemented.
• The scenarios and tasks generated for the study should be kept and used for testing solutions to usability problems, new design ideas and in any other future usability studies.
• It would be necessary to have people in the project helping with usability. They realized they didn’t have the skills nor the time for that.

The present chapter presents the main conclusions and lessons learned from our research done through the usability study of the tool. These conclusions are organized according to the issues we were interested in the research. We provide as well a series of recommendations based on such conclusions and other insights from the research.

Following are the conclusions according to the three issues we were interested in the research (section 1.2).

In regards to possible effects of not having considered usability when assessing the functionality of a system, we found the following things:

1. Not considering usability can affect for the assessment of the functionality. The usability study revealed that there was not a group of SEs that could use all the power of the functionality of the tool at that moment, and therefore, it was impossible for the KBRE project to assess such functionality. In other words, it was impossible to determine if the functionality was very good already for helping SEs at maintaining the Company systems more efficiently or if it should be improved. This group of SEs didn't exist for several reasons. On one side, problems such as, ignorance that the tool existed, low motivation, limited time, etc., had caused that many SEs did not use the tool at all. But, on the other hand, not having considered usability had meant that the tool presented a large number of usability deficiencies, which in turn, had made that those SEs who had used the tool were not capable of using all the power of the functionality. In other words, bad usability in the tool meant it was not possible to assess the functionality at that moment.
2. Helping users learn the system is essential for assessing its functionality and good usability plays a key role, particularly good learnability. As we mentioned above, there was not a group of SEs that could use all the power of the functionality at that moment. It was clear that just having left the tool available to SEs during 1.5 years had not been enough so that such a group of SEs emerged. Also, not having monitored how SEs used the tool had meant that the KBRE project never realized causes affecting the use of the tool. In summary, it was clear that in order to have a group of SEs capable of using all the power of the functionality it would have been necessary to help the SEs at learning the functionality. For that, good tool learnability would have been key.
3. Usability testing can be an excellent activity for helping users learn the system. We saw that the usability study served as an excellent communication channel between developers and SEs. On one side, it meant that more SEs learned and used the tool better. Participants learned to exploit the power of the functionality by doing the tasks of the test, and they encouraged other SEs to learn and use the tool as well. On the other side, the study helped the KBRE project at discovering causes affecting the use of the tool as well as how to proceed in ensuring that a group of SEs could be capable of using all the power of the functionality.
4. Ideally, usability should be considered from the beginning of a project, but considering it late is worthwhile and much better than never considering it. Despite all the problems and effects that not having considered usability had caused, having started usability activities at the time of this research produced immediate benefits. The KBRE project was facing the problem that not many SEs had used the tool and it was unknown exactly why and what to do. As we mentioned before, the usability study exposed many of the causes of that problem and helped at seeing how to proceed next. After the study, developers had a much clearer idea of what kind of things they should pay attention to in order to improve the tool. It is important to mention that the advanced status of the tool within its lifecycle meant that not all the problems and things revealed by the study could be solved or improved.

 

5.1.2 Objective Two

With regard to experimenting with the introduction of user centered design and usability into the latter stages of software projects in which developers are new to these concepts, we made the following observations:

5. Usability must be gradually introduced by following a process. Throughout the study, we confirmed that working towards good tool usability definitely required commitment and participation of many people in the KBRE project, including developers, SEs and managers of both. Also, it required certain level of developers' maturity with usability, its effects and how to address it. But we saw as well that people in the project required time and being exposed to different types of activities in order to acquire that maturity and commitment to usability. In summary, we confirmed that it was necessary to use a gradual process including different types of activities to introduce usability into the KBRE project.
6. Initial activities must focus at creating usability awareness in the software project (maybe sacrificing obtaining the best possible usability results). As we mentioned above, working towards good tool usability required developers' maturity and commitment to usability. After the study, we saw that performing the best techniques to obtain the best usability insights was not as important before the study as developing such commitment and maturity. We found that making people in the project become aware of (a) what usability was, (b) the usability status of the tool, and (c) how it had affected the SEs, generated the most powerful effect for developing maturity and commitment to the usability of the tool. The best activities had been evaluating the tool, fixing tool usability problems and testing the tool with SEs. However as well, insufficient maturity of the project at that time meant that these activities not always produced the best effects with regard to the usability of the tool. At that moment, developers were only becoming aware of what usability meant and how difficult achieving good tool usability was.
7. Creation of scenarios and tasks of system usage is very helpful. We confirmed that examples of scenarios and tasks reflecting the tool’s design intent produced many benefits. They served to perform the testing but also greatly supported developers in testing the solutions of the problems and testing designs of future features.
8. System requirements and specifications as well as user interface information are very important. We saw that knowledge on (a) the tool domain, (b) how it addressed the problem, (c) the rationale of its design, and (d) how it had to be used, was essential for developers when trying to improve the tool. Not having means to completely acquire that knowledge seriously affected new developers that had recently come into the KBRE project. We saw that system specifications and user interface information would have been particularly helpful.
9. Experience introducing usability is an important requirement for the usability specialist, particularly when developers are new to usability. Insufficient knowledge about usability meant that developers of the tool had difficulties:

• Understanding usability problems in its most general way.
• Seeing implications of not efficient solutions.
• Evaluating effectiveness of different solutions.
• Prioritizing the problems.
• Deciding how to proceed on fixing the problems.

Therefore, the person performing the study had to support developers in all these issues. We found that this type of support was not always easy to provide; producing the best results definitely required experience dealing with the above issues.

1. Developers have difficulties generating good solutions the first time they solve usability problems. The above difficulties meant that developers not always devised good solutions to the problems. Sometimes, the solutions would lead to other obvious usability problems. This situation along with the state of development of the tool, limited time and insufficient resources meant that developers did not always implement good solutions.
2. Once exposed to usability, software projects can quickly mature about accepting usability. During the study, the KBRE project passed from the skeptical stage to the acceptance stage, as described in Chapter II. In all the study, we never noticed anything that could have caused the project to reject or disagree with usability. Basically the only reason why usability had not been considered was insufficient knowledge. Every time people in the project learned something new, they were more convinced and committed to usability.

 

5.1.3 Objective Three

In regards to exploring particular challenges of studying the usability of program comprehension tools, we found the following things:

3. Domain knowledge is an important requirement for the usability specialist. The tool domain was so particular and complex that knowledge in it was important for all the usability specialists involved in the study. On one hand, evaluators in the heuristic evaluation required this knowledge to be able to make accurate judgments about the tool. In fact, insufficient domain knowledge posed difficulties for one evaluator. On the other hand, the experimenter in the study had to provide support to this previous evaluator on several issues about the domain. Also, he had to fully understand the problems, appropriately explain those in the report, communicate them to developers, and provide advice during the generation of solutions. All these required domain knowledge on the part of the experimenter.
4. Testing with real users is essential. It is well known that that the more similar the participants in a usability study are to the users of the system the more accurate the results of the study will be. As we mentioned above, the tool domain was so particular and complex that the population of users of the tool (SEs) was very specific. In the study, we saw that people with very similar characteristics to the SEs behaved very differently to those when approaching the problems that the tool was designed for, and therefore they used the tool very differently. We noticed that testing with real users of the tool (SEs) was even important for the pilot study.
5. Task determination becomes particularly difficult and complex. Like in the previous issue, the complexity of the tool domain meant that SEs could approach the problems very differently and in very unpredictable ways. This implied that finding suitable tasks for the study that made them use the tool functionality in a more or less predictable way was very difficult.

 

5.1.4 Other Conclusions

We also found the following things:

6. In regards to heuristic evaluation:

• Using evaluators with usability knowledge is important. This knowledge was important so that they knew what to do and look for in the evaluation. However, we noticed that people with this knowledge were not easy to find.

• Better results can be obtained by combining evaluators with different levels of domain knowledge and system expertise. We saw that each type of evaluator captured different types of problems and this provided a broader spectrum of problems. We don’t know if this combination is always better in general. But this is a topic for further research where more experimentation could be performed.

1. In regards to thinking aloud usability testing:

1. This technique is more difficult than heuristic evaluation. Applying this technique required dealing with many more issues during its planning and application.
2. It is an excellent way to introduce usability into a software project. It worked excellently in creating awareness and commitment to usability in the KBRE project.

1. Combining both techniques is an excellent approach. Applying both techniques took us longer but produced a much better effect on introducing the KBRE project to usability and user centered design. Using the two techniques exposed the project more to usability, which made it mature and become more committed to it. Also, both techniques complemented each other very well. Each one provided different types of problems. By using both, we could confirm certain problems and obtain a more reliable severity of them.

This research led to some recommendations. The following set of recommendations is for software projects developing a system in which its functionality will be assessed. They were derived from the conclusions discussed in section 5.1.1 above.

1. Pay attention to both the utility and the usability of the system. Plan for resources and activities to look over both. Not paying attention to one could affect the assessment of the functionality and consequently reduce chances for improving the system.
2. Create a plan for ensuring that the group of users who is given the system correctly learn and use the system. Establish a permanent channel of communication between them and developers. Plan for activities to help them at learning and using the system. Monitor how they employ the system (see conclusions in section 5.1.1 above).

The following set of recommendations is offered for software projects that have started without usability activities but that want to initiate activities to look at and improve the usability of a system. They were derived from the conclusions discussed in section 5.1.2 above.

3. Establish a plan of action. If time and resources are limited, use a discount usability engineering approach. Important activities to include in the plan are:

1. A user and task analysis. Clearly identify the target population of users of the system. Identify characteristics of those users that can affect how they use the system. Identify the tasks that the system is trying to help them with. Identify possible groups of users within the target population and their characteristics. Document all identified information.
2. Identify and maintain information about the system such as system requirements, specifications and design rationale of the user interface. Emphasis should be on information that can help people at learning the system and its domain.
3. Create a set of typical scenarios and tasks that reflect the system’s design intent. Use identified information as help.
4. Establish usability goals for the system. Consider pre-established objectives of the system and use identified information as help.
5. Plan for iterations of evaluation and redesign. Iterate until reaching the usability goals.

1. As much as possible maintain a usability "champion" in the project.
2. Instruct developers on usability before starting the project. Also, plan for activities to continually improve the usability knowledge of people in the project.
3. If using heuristic evaluation:

1. Try to get evaluators with as much knowledge in the domain of the system as possible.
2. Provide support to evaluators with information about the system such as, requirements, specifications and design rationale of the user interface.
3. Plan for sufficient time and resources to summarize problems found by evaluators.
4. Plan for some sessions with developers to review the problems.
5. Integrate evaluators into a pool of usability specialists for future evaluations of this or other systems.

1. If using thinking aloud usability testing:

1. Use system information such as requirements, specifications and design rationale of the user interface as help for determining the tasks.
2. Phrase the tasks in a way that the communication between observer and participants is kept at a minimum. Participants must able to talk while performing the tasks but should not require (as much as possible) help from the observer. This could be accomplished by asking participants to count something, write down the answers, or observe something on the screens of the system.
3. Plan for at least three times the duration of videos for their analysis.
4. Plan for some sessions with developers in order to review the problems.
5. Integrate participants into a pool of participants for future tests of this or other systems.

The research project presented here investigated several issues. First, it explored effects of not having considered usability when assessing the functionality of a particular program comprehension tool. This tool had been created to provide certain experimental functionality that its developers wanted to assess. For that, the tool had been given to a group of users for some time. Second, the research experimented with introducing usability into the software project that was creating the tool at a late stage of the lifecycle. Finally, the research explored challenges posed to studying the usability of the program comprehension tool.

A usability study was designed and performed to pursue the research. Three techniques were applied in the study: (a) heuristic evaluation, (b) user and task analysis and (c) thinking aloud usability testing. Heuristic evaluation served to look for usability deficiencies in the tool design. For this, a group of evaluators judged the tool according to a list of usability guidelines and principles. Thinking aloud usability testing served both (a) to look for problems that users experienced when learning and using the tool, and also (b) to explore how users who had been given the tool were capable of using its functionality. Both were achieved by observing a group of users performing some tasks typical of the tool's design intent. User and task analysis served to obtain a group of users and a set of tasks for the thinking aloud usability testing technique. It also served to gather information about how the users had employed the tool. The study followed a discount usability engineering approach.

The study was in general easy to plan and perform although certain things were more difficult than expected. Most difficulties were related to practical problems in planning and applying the techniques. It was difficult to find within the usability literature advice or examples of certain practical issues.

The study was very productive in finding information about all the objectives in the research within the particular software project. In summary, it revealed that none of the users who had been given the tool was capable yet of correctly using its functionality, and therefore, it was impossible to assess the tool functionality at that moment. Not having considered usability had been one of the causes of this. Also, the study exposed many issues in relation to (a) having started considered usability at a late stage in the tool lifecycle and (b) the software project's reaction to being exposed to usability for the first time. Finally, the study exhibited certain particularities and difficulties with planning and studying the usability of the program comprehension tool. In summary, the research project was satisfactory and accomplished all its objectives.

Bachman, D. (1989). A methodology for comparing the software interfaces of competitive products. Proceedings of the Human Factors Society 33^rd annual meeting 1989, pages 1214-1217.

Bias, R., Mayhew, J. (1994). Cost-Justifying Usability. Academic Press.

Boehm, B., Gray, T., Seewaldt, T. (1984). Prototyping versus Specifying: A Multiproject Experiment. IEEE Transactions on Software Engineering 10, 3, pages 290-303.

Boehm, B. (1988). A Spiral Model for Software Development and Enhancement. Computer, May 1988, pages 61-72.

Ericsson, K., Simon, H. (1984). Protocol analysis: Verbal reports as data. MIT Press.

Denning, S., Hoiem, D., Simpsom, M., Sullivan, K. (1990). The value of thinking-aloud protocols in industry: A case study at Microsoft Corporation. Proc. Human Factors Society 34^th annual meeting, pages 1285-1289.

Desurvire, H. (1994). Faster, Cheaper!! Are Usability Inspection Methods as Effective as Empirical Testing?. In Usability Inspection Methods. J. Nielsen and R. L. Mack Eds. John Wiley & Sons, New York, pages 173-202.

Ehrlich, K., Rohn, A. (1994). Cost Justification of Usability Engineering: A Vendor's Perspective. In Cost-Justifying Usability, D. J. Mayhew and R. G. Bias Eds. Academic Press, New York, pages 73-110.

Gaylin, B. (1986). How are windows used ? Some notes on creating an empirically-based windowing benchmark task. Proc. ACM CHI '86 Conf., pages 96-100.

Glen, David (1995). A case study on moving from a character-based full screen user interface to a graphical user interface. Master's thesis, California State University, Chico.

Gould, J. D., Lewis, C. (1985). Designing for Usability: Key Principles and What Designers Think. Communications of the ACM 28, 3, pages 300-311.

Hackos, J., Redish, J. (1998). User and task analysis for interface design. John Wiley & Sons.

Hix, D., Hartson, H. (1993). Developing User Interfaces: Ensuring Usability Through Product & Process. John Wiley & Sons, New York.

Jacobson, I., Booch, G., Rumbaugh, J. (1998). The Unified Software Development Process. Addison Wesley.

Jeffries, R., Miller, J., Wharton, C., and Uyeda, K. (1991). User Interface Evaluation in the Real World: A Comparison of Four Techniques. In Proceedings of the CHI '91 Conference on Human Factors in Computing Systems. ACM Press, New York, pages 145-151.

Jeffries, R. (1994). Usability Problem Reports: Helping Evaluators Communicate Effectively with Developers. In Usability Inspection Methods. J. Nielsen and R. L. Mack Eds. John Wiley & Sons, New York, pages 273-294.

Lewis, C., Rieman, J. (1993). Task-Centered User Interface Design: A Practical Introduction. University of Colorado, Boulder, Colorado.

Lethbridge, T. (1997). User Interface Design and Implementation SEG 3120 -course notes-; University of Ottawa.

Lethbridge, T., Anquetil, N. (1997). Architecture of a source code exploration tools: A software engineering case study. University of Ottawa. Computer Science Technical Report TR-97-07.

Lethbridge, T., Singer J., (1997). Understanding Software Maintenance Tools: Some Empirical Research. Workshop on Empirical Studies of Software Maintenance (WESS 97). Bari, Italy, October, 1997.

Lewis, C. (1982). Using the 'Thinking-Aloud' Method in Cognitive Interface Design. Research Report RC9265. IBM T. J. Watson Research Center, Yorktown Heights, New York.

Mayhew, J. (1991). Principles and Guidelines in Software User Interface Design. Prentice Hall.

Mayhew, J. (1999). The Usability Engineering Lifecycle. Morgan Kaufmann Publishers.

McKerlie, D., MacLean, A. (1993). Experience with QOC Design Rationale. In Adjunct Proceedings of the INTERCHI '93 Conference on Human Factors in Computing Systems, pages 213-214.

Nielsen, J. (1992a). Evaluating the thinking-aloud technique for use by Computer Scientists. In Hartson, H.R., and Hix, D. (eds.), Advances in Human-Computer interaction, Vol. 3, Ablex, Norwood, NJ., pages 69-82.

Nielsen, J. (1992b). Finding usability problems through Heuristic Evaluation. Proc. ACM CHI '92 Conf., pages 373-380.

Nielsen, J. (1993a). Usability engineering. AP Professional.

Nielsen, J. (1993b). Iterative User Interface Design. IEEE Computer 26, 11, pages 32-41.

Nielsen, J. (1994a). Enhancing the explanatory power of Usability Heuristics. Proc. ACM CHI'94 Conf., pages 152-158.

Nielsen, J. (1994b). Usability Laboratories: A 1994 Survey. Behaviour & Information Technology 13, 1&2, pages 3-8.

Nielsen, J. (1994c). Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier. In Cost-Justifying Usability, D. J. Mayhew and R. G. Bias Eds. Academic Press, New York, pages 245-272.

Nielsen, J. (1994d). Heuristic Evaluation. In Usability Inspection Methods. J. Nielsen and R. L. Mack Eds. John Wiley & Sons, New York, pages 25-62.

Nielsen, J., Mack, R. (1994). Usability Inspection Methods. John Wiley & Sons.

Norman, D., Draper, S. (1986). User Centered System Design: New Perspectives on Human-Computer Interaction. Hillsdale, NJ: Erlbaum Associates.

Norman, D. (1990). The design of everyday things. Currency Doubleday.

Preece, J., Rogers, Y., Sharp, H., Benyon, D. (1994). Human-Computer Interaction. Addison Wesley.

Rosson, M., Carroll, J. (1995). Integrating Task and Software Development for Object-Oriented Applications. CHI’ 95 Proceedings, pages 377-384.

Shneiderman, B. (1998). Designing the user interface, Strategies for Effective Human-Computer Interaction. Addison Wesley. 3^rd edition.

Singer, J., Lethbridge, T. (1997). Work practices as an alternative method to assist tool design in software engineering. University of Ottawa. Computer Science Technical Report TR-97-08.

Singer, J., Lethbridge, T., Vinson, N. and Anquetil, N. (1997). An Examination of Software Engineering Work Practices. CASCON '97, Toronto, October, pages 209-223.

Sommerville, I. (1997). Software engineering. Addison Wesley. 5^th edition.

Storey, M.-A.D., Wong, K., Fong, P., Hooper, D., Hopkins, K., Müller, H.A. (1996). On Designing an Experiment to Evaluate a Reverse Engineering Tool. Proceedings of the 3rd Working Conference on Reverse Engineering, (WCRE'96), 1996, pages 31-40.

Storey, M.-A.D., Wong, K., Müller, H.A. (1997). How Do Program Understanding Tools Affect How Programmers Understand Programs. Proceedings of WCRE'97, 1997, pages 12-21.

Whitefield, A., Wilson, F., and Dowell, J. (1991). A framework for human factors evaluation. Behaviour & Information Technology 10, 1 (January-February), pages 65-79.

______________________________________________________________________

______________________________________________________________________

______________________________________________________________________

1. The first field indicates the number of the problem and its location in the user interface according to the codes below. It also contains a letter "v" in case the problem was Verified during the thinking aloud usability testing.

OW: Overall in the Interface

DW: about Dialog Windows opened by interface features

HW: related to the History Window

HiW: related to the Hierarchy Window

SW: related to the Source Window

TS: related with the Tool bar and Shortcuts

PW: related with the Preferences dialog Window

GG: related with the Global Grep feature and its window

2. The second field indicates the usability guideline violated by the problem.
3. The third field indicates whether the problem was about Learning (letter "L"), Efficiency of use (letter "E"), or both. It also classifies the problems according to the following categories:

1. Non optimal functionality
2. Possible confusion
3. Possible misinterpretation
4. Unnecessary functionality
5. Lack of functionality
6. Lack of consistency
7. Graphical design
8. Feedback
9. Lack of robustness
10. Wrong behavior
11. Bad labeling
12. Lack of labeling

1. The fourth field indicates an approximate Usability Severity (US) of the problem according to the codes below. The severity values were established when reviewing the problems in the sessions with developers.

1. OVERALL IN THE INTERFACE

1. RELATED WITH THE HISTORY WINDOW

2. RELATED WITH THE HiERARCHY WINDOW