A Heuristic Evaluation of LeetCode

by Xuejing Ma

What is LeetCode?

LeetCode is a famous online programming website. It helps software engineers, computer science students, and aspiring engineers improve programming skills, prepare for technical interviews and get their dream jobs.

LeetCode provides more than 1000 coding problems. Users can use the embedded programming interface to write a solution code for each problem. After submitting your solution, the system will automatically judge your answer. If the answer is correct and accepted, the system will give you a feedback and it will also compare the efficiency of your answer to others from all over the world.

LeetCode also provides a discussion board for users to discuss all kinds of topics related to programming, technical interviews, careers, etc. It also holds a online contest each week. Users can participate the contest and earn points by solving problems online.

Figure 1 demonstrates a user interface of LeetCode. The menu bar shown in the green box provides many useful sections we have mentioned above. The Problems section is the most used one. Users can practice plenty of problems by programming online in an embedded coding editor. The Articles section provides many excellent solution explanations for most of the problems. Discuss is an area where users can communicate with each other by posting and commenting. Other sections such as Explore, which provides online courses, Contest, which holds coding matches, etc. are also very popular among the users.

Figure 1. The user interface of LeetCode

The icons shown in the red box indicate other features of LeetCode. By clicking on the avatar, users can access their account features. The console-like icon is a coding playground for experiments. The bell-shaped icon is for notification. Finally, if you subscribe and become a premium member, you can access more advanced features. The most popular one is unlocking the company tab. Thus, a user can check the previous interview questions of the company with which he/she will have an interview.

Our evaluation will cover most components and focus on Problems, Articles, Discuss and some features under the user account.

What is a heuristic evaluation?

A heuristic evaluation is an inspection approach to identify usability issues in a piece of software. It involves experts examining the software and judging its compliance with recognized usability principles which are called the heuristics. We used Nielsen's 1990 heuristics in our evaluation. This set is first introduced by Jakob Nielsen and Rolf Molich in their paper Heuristic evaluation of user interfaces.

What are the usability issues of LeetCode?

In our evaluation, we found five significant usability issues. Each subsection below describes one issue with a short demonstration video. We also point out which of the heuristics the issue violates. Finally, we provide some improvement suggestions for each issue.

A. Search results do not prioritize exact matches

Searching is heavily used by users of LeetCode. In video 1, we searched for a problem called Binary Tree Preorder Traversal under the Problems section. Surprisingly, we found the best match result occurs in the third place. Then, we searched for an article containing keywords Two Sum under the Articles section. Again, the best match did not appear at the top. During our evaluation, we found that, in many cases, users need to scan through the results and find the best match manually.

This issue violates heuristic 2 - speak the user’s language, because the order of search results is not intuitive and understandable for most users. Therefore, users need extra time and effort to locate the best result.

Video 1. Illustration of how an exact match does not appear as the top search result

Improvement Suggestion

1. A quick fix is to highlight the matching keywords in each result. In fact, it has already been done in the article search shown in the video. Thus, the result with the most matching words will be easier to find.
2.  The developers need to improve their searching algorithm and make the order of search results more straightforward and understandable.

B. Difficulty in navigating around a list of pages

Page navigation is another common functionality of a website. In video 2, we tried to find the last submission record page which contains the earliest record. Unfortunately, the system only provides two buttons, newer and older. If the submission record history is long, users must click older many times to flip page by page and get the oldest record. In the video, we used a trick, changing the page number in the URL, to show the last page. However, most users do not know the trick. Even if they know it, they need to guess multiple times because the total page number is not available. The same issue also exists in the Articles section. In the global ranking page, we found some improvement. Designers give us two shortcuts for the first and last page.

This issue violates heuristic 7 - provide shortcuts, because users do not have efficient shortcuts to jump among different pages.

Video 2. Illustration of the difficulty in accessing the last page of the submission record

Improvement Suggestion

1. The developers can refer to the design from GeeksforGeeks, another interview preparation website. As shown in figure 2, this design gives two shortcuts for the first and last page. In addition, users can fast forward and backward by clicking the largest and smallest page number. It also shows the total page number so that the user can modify the page number in the URL without any error.
2.  A better version for the design in figure 2 is to change the immutable current page number display (shown in the red box) to be mutable. Thus, the user can jump to any page without changing the URL.

Figure 2. A better page navigation design

C. Difficulty in interpreting distribution charts

In video 3, we submitted an answer to the problem Binary Tree Preorder Traversal. After that, the system showed us general performance information which includes runtime and memory usage. Next, we clicked the detail link and checked the performance distribution charts. At first look, we could not find our position in the charts because there was no highlighting nor label to indicate it. However, after careful observing, we noticed a grey area which indicates our position. But it is too light for users to discover.

We also found that if we use the mini chart below to zoom in a specific area, and if this area is correct, then a label will appear in the main chart to indicate our position. Again, it is not straightforward because most users will not zoom in the chart and they will never find the label.

Another problem we found is related to unit consistency. The memory usage information uses MB as the unit while the chart uses kb. Users need to do the conversion and we don’t know whether to use 1MB = 1024kb or 1MB = 1000kb since both are used in different systems.

The issue first violates heuristic 3 - minimize memory load. Users need to memorize the performance data to locate their position if there is no label and they do not notice the grey area. It also violates heuristic 4 - be consistent. The inconsistency of different memory usage units confuses users and requires them to give extra effort in unit conversion.

Video 3. Illustration of the difficulty in interpreting the distribution charts

Improvement Suggestion

1. The designers should use more eye-catching color such as yellow or orange to highlight the user’s position in the chart.
2. The chart should display the indication label by default, not after zooming in.
3. The system should use a consistent unit for memory usage size.

D. No efficient approach to access another user's profile

In video 4, we tried to find a friend user whose ID is davidluoyes. LeetCode does not provide any search box to find a user by ID directly. Therefore, we need to find the ID davidluoyes first, and then, click the ID and enter the profile page. We tried to first find any post or article written by davidluoyes. Unfortunately, we found nothing. And then, we tried to search in the global ranking list which contains a lot of IDs. However, after searching for a few pages, we gave up because there are more than 3000 pages! We searched for the whole system and could not find any way to get the ID.

Since there is no other choice, we used the magic trick again, modifying the ID in our own profile page URL to davidluoyes. Finally, we entered his page and opened his most recent post. Obviously, he wrote a post, but why we did not find it at first? We will answer this question in the next section.

This issue violates heuristic 7 - provide shortcuts. The designers failed to give the user an access shortcut to another user’s page given the ID. Therefore, it is almost impossible to check another user’s information unless you know the URL trick.

Video 4. Illustration of no proper approach to access another user's profile

Improvement Suggestion

1. The system should provide a search box for a user to find another user by the ID.
2. There should also be a friend list for a user to access a friend’s profile quickly without searching.

E. Different discussion sections are confusing

In video 5, we tried to find a comment from davidluoyes under a post called Kadane's Algorithm - Since no one has mentioned about this so far :) (In case if interviewer twists the input). Just like we did in video 4, we searched the whole Discuss section and found nothing. We even searched in the Articles section and could not find any useful information. Then, we thought there might be comment history in davidluoyes’s profile page. Unfortunately, there is only post history.

After the first attempt failed, we had to use another piece of information that the post is related to a problem Best Time to Buy and Sell Stock. We searched the problem name in Discuss and found something similar. Unfortunately, it is not the post. Finally, we gave the correct steps:

• Go to the problem page Best Time to Buy and Sell Stock under the Problems section
• Click the Discuss tab under the problem (not the Discuss in the menu bar) and enter an independent discussion area only related to the problem
• Search the post and enter the post page
• Search davidluoyes for his comment page by page

But why can’t we find anything under the Discuss section? In the video, we showed that the Discuss under the problem has a URL like leetcode.com/problems/… The Discuss section in the menu bar has a URL like leetcode.com/discuss/… They are totally different components of the system. That is the reason we cannot find any post related to a specific problem under the Discuss in the menu bar. The two discussion areas have the same name and their interfaces are very similar to each other. So, most users will be confused by this design and have difficulties to find a post under a problem.

This issue violates heuristic 2 - minimize memory load because it requires the user to remember both post and problem names in order to find a post related to that problem. Regarding comments, it also requires users to memorize the post name that they commented so they can view the comment in the future. The issue also violates heuristic 7 - provide shortcuts because a user cannot search a comment by user ID under a post.

Video 5. Illustration of the confusing discussion section design

Improvement Suggestion

1. The designers should add a section called Comment History in the user’s profile page. It will provide a shortcut for each comment without memorizing the post name.
2. The designers should also change the Recent Posts section in the profile page to Post History. The user should be able to access a full history of posts and comments.
3. The designers should add a search box under each post for users to search for comments.
4. Regarding the confusing Discuss design, the designers can remain the current problem interface. However, they should link the problem Discuss tab to the Discuss section instead of creating a separate discussion area. Under Discuss, they can add a Coding Problem button. After clicking it, a coding problem list displays below. The users can pick a problem and enter its discussion area. In addition, if a user searches a post, the system should do a global search within Discuss and return all matching results from all categories.

Conclusion

LeetCode is a very powerful but complex website. Although most users of LeetCode are proficient in computer science and software development, the designers still need to improve the usability to attract more users who are new to programming. The usability issues should not be their obstacles.

As we have seen above, many usability issues can be solved by some URL tricks. However, the website designers should not expect all users to have deep knowledge of computer science or website development. Therefore, the interaction style should be more friendly and straightforward so that the users do not need to use magic charms. Finally, the system architects should carefully review and modify the system model and make it more consistent and understandable.