Heuristic Analysis of the Ottawa Hospital Resident Practice Profile (RPP) Application

By Carl Sonnen

Introduction

In this study, heuristic analysis was performed using a modified set of Nielsen’s heuristics adapted specifically for healthcare applications—the Nielsen–Shneiderman heuristics—proposed by Zhang [5] in 2003 (see Table 1). The domain experts in this case are physician residents at the Ottawa Hospital who use the Resident Practice Profile application to record patient visit information for later statistically accurate recall on the types of patients and kinds of medical conditions they are encountering. The RPP application is found at http://quickforms3.eecs.uottawa.ca/rpp/. The login credentials may be sought with permission from Professor Liam Peyton form the University of Ottawa (lpeyton@uottawa.ca). A fair amount of research in the area of heuristic analysis of health care and medical services applications has already been done. A heuristic analysis reveals a high number of issues in very short time, depending on whether one or more evaluators are involved and depending on the skill of the evaluator(s). It is seen as low cost and requires little preparation other than providing a list of heuristics to the evaluator(s) and access to the application itself. For large, complex enterprise applications, sometimes an additional task list needs to be provided. RPP is a small application so the analyst can evaluate every major piece of functionality in a short time.

Table 1  Nielsen-Schneiderman Heuristics [5]

Nielsen–Shneiderman Heuristics	Description
1 Consistency and standards	Users should not have to wonder whether different words, situations, or actions mean the same thing. Standards and conventions in product design should be followed.
2 Visibility of system state	Users should be informed about what is going on with the system through appropriate feedback and display of information.
3 Match between system and world	The image of the system perceived by users should match the model the users have about the system
4 Minimalist	Any extraneous information is a distraction and a slow-down
5 Minimize memory load	Users should not be required to memorize a lot of information to carry out tasks. Memory load reduces users' capacity to carry out the main tasks
6 Informative feedback	Users should be given prompt and informative feedback about their actions
7 Flexibility and efficiency	Users always learn and users are always different. Give users the flexibility of creating customization and shortcuts to accelerate their performance
8 Good error messages	The messages should be informative enough such that users can understand the nature of errors, learn from errors, and recover from errors
9 Prevent errors	It is always better to design interfaces that prevent errors from happening in the first place
10 Clear closure	Every task has a beginning and an end. Users should be clearly notified about the completion of a task
11 Reversible actions	Users should be allowed to recover from errors. Reversible actions also encourage exploratory learning
12 Use users' language	The language should be always presented in a form understandable by the intended users
13 Users in control	Do not give users the impression that they are controlled by the systems
14 Help and documentation	Always provide help when needed, ideally context-sensitive help

Background

Jaspers [1] cites that heuristic analysis by double experts (experts in both the specific healthcare domain and in usability testing) can provide an excellent benefit-cost ratio in that up to 75% of all application issues can be uncovered by the analysis of just 2 – 3 of these double experts. Using non-experts to assess the application heuristically would require 14 or more analysts to uncover the same number of problems. Jaspers further states that up to half of the problems uncovered in a heuristic analysis will not be discovered during user testing. Two limiting issues are brought to light by Jaspers: 1) The availability of usability experts and double experts in particular may be limited and 2) The proliferation of different sets of heuristics currently available to use.

The RPP application has three main sections: 1) Visits, where a physician can add, edit and keep track of all patient visits and the visit details logged while using the application, 2) Reports section where physicians may run canned reports and 3) Team Members or account management section.

Usability Issues

A sample of the 15 violations uncovered and their severity rating are provided below.

1.   Main Form Missing Feedback

Only 1 major heuristic violation was uncovered. This was regarding the feedback provided to the user after a visit record is added—the Clear Closure heuristic. In Figure 1 below there is no real indication that a record is successfully added after pressing ‘Submit’ on the previous sub form were all the cumulative information has been entered.

 Figure 1  Main Application form missing feedback to physician user that a New Visit Data record has been added

Suggested Fix:

Add the following message with some specific visit details to indicate the visit information has been saved to the bottom section in Figure 1 above: ‘Patient visit in Patient’s Home having Fever on 2016-02-22 10:30:23 has been successfully added’

2.   Too much irrelevant data displayed in some reports

In some reports there are pages and pages of data displayed for a single report which is particularly onerous to view in a mobile device. Every specific medical diagnosis is displayed amounting to hundreds of records when really only those conditions the physician has actually encountered during visits/rounds needs to be listed (see Figure 2 below). This violates the Minimalist  principle/heuristic. It was rated as a minor violation.

Figure 2  End of Life Detail Report including records with 0 # of Visits

Suggested Fix:

Filter records to display only those records with number of visits that are one (1) or more (i.e. not zero)

3.   Drop down selection icons inconsistent

On the main form a plus (‘+’) icon is used to open the Tracking, Demographics and Assessment sub forms (See Figure 3). In all other forms sub forms are referenced with an accordion icon (‘˅’)—see Figure 4. This violates the Consistency and standards heuristic. It is a minor severity issue.

 Figure 3  Main Add Visits form with ‘+’ expansion signs

Figure 4  Demographics sub form with accordions (‘˅’) expansion signs

Suggested Fix:

Make drop down sub form selection icons all consistent. Make them all accordion (‘˅’) icons

Summary of remaining violations found:

Table 2  Results after applying Nielsen-Schneiderman Heuristic Analysis

Heuristic	Issue	Severity
1 Consistency and standards		2 - Minor
	On some sub forms arrow keys can be used to scroll down/up the list. On the main pages you use the tab key to move around—inconsistent. Note: This is probably not an issue on a touch screen mobile device.
2 Visibility of system state	Submit button is always blue. Should change to Green (i.e. Go) but only when page is updated, indicating data has been changed and is ready for update	2 - Minor
3 Match between system and world	Application is devoid of colour. Unlike real-world. Colour helps us identify and distinguish differences. Adding 2 or 3 subtle colours would make application easier to identify controls and sections	1 - Cosmetic
5 Minimize memory load	When creating new accounts users type in the location instead of choosing it from a selection list. Possible data inconsistency Should use the same lookup data used when selecting location when creating a visit	2 - Minor
6 Informative feedback	Submit button should ideally change from Grey to Green (i.e. implying ready to Save) when page updated	1 - Cosmetic
7 Flexibility and efficiency	No shortcut keys discernable (Probably not required on a touch screen mobile device and only on a desktop).	2 - Minor
	Too many canned reports even though they are somewhat filterable.  Allow users to create custom reports through their own query builder page
8 Good error messages	N/A (see heuristic 9)	N/A
9 Prevent errors	N/A Excellent job preventing all errors	N/A
11 Reversible actions	If user selects a number of items in a sub form, then the only way to undo multiple selections is to click them ‘off’ again. No Clear or Cancel button	2 - Minor
12 Use users' language	Presuming clinical language is correct. Responsibility of domain experts	N/A
13 Users in control	User mostly selecting data from lookups. Filtering features on visit summary comprehensive. Filtering on reports adequate. May want to allow customizable reports through a query builder to allow more user control	2 - Minor
14 Help and documentation	Only a very limited problem submission form. Email address should be auto-populated from account information.  Form should have a general  type issue selection lists	2 - Minor
	No online user’s guide

Conclusion

A small number of minor or cosmetic violations were uncovered. Overall the RPP application was simple, robust, self-evident and easy to use. No catastrophic problems were encountered and only one problem classified as major was uncovered. All the other problems encountered were cosmetic or minor in nature.

Suggestions for future research on the usability of the RPP application involve conducting more heuristic evaluations and merging the results with this study and also performing a user study, preferably with resident physicians, which might include observation sessions and/or questionnaires.

References

[1]     Jaspers, M. 2008. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics. Elsevier. 78 (2009), 340-353. DOI= 10.1016/j.ijmedinf.2008.10.002

[2]     Saparova, D., Basic, J., Lu, Y., Kibaru, F., Ma, Y. and Yadamsuren, B. 2013. Usability Problems in Patient- and Clinician-Oriented Health Information Systems: What Are They and How Do They Differ? DHM/HCII, Springer. Part I, LNCS 8025, 276–285

[3]     van Engen-Verheula, M., Peutea, L., de Keizera, N.,  Peeka, N. and Jaspers, M. 2015. Optimizing the user interface of a data entry module for an electronic patient record for cardiac rehabilitation: A mixed method usability approach. International Journal of Medical Informatics. Elsevier.  87 (2016), 15–26. DOI= http://dx.doi.org/10.1016/j.ijmedinf.2015.12.007

[4]     Wu, P.H., Chen C.H., Chen H.T., Shu, C.H., Lin, F.S., Wang, Y.W., Li, H.J., Wu, Y.T. and Lai, F. 2010. User Inspection of National Taiwan University Hospital’s Telehealth Care Information System. 32nd Annual International Conference of the IEEE EMBS, IEEE. 2010, 4542-4545

[5]     Zhang, J., Johnson, T., Patel, V., Paige, D., and Kubose, T. Using usability heuristics to evaluate patient safety of medical devices. 2003. Journal of Biomedical Informatics. Elsevier. 36 (2003), 23–30. DOI=10.1016/S1532-0464(03)00060-1