From Writing Aids to Creative Collaboration: An UX Evaluation of ChatGPT Across User Groups

By Beril Borali

How does the educational background shape user experience with conversational AI?

Our study, inspired by the study of Skjuve and others, set out to explore how users from varied educational backgrounds engage with ChatGPT (version 3.5), focusing on their behavior and satisfaction levels. The participant pool consisted of 9 university students, representing a range of disciplines including Computer Science and Engineering (CSE) and other disciplines (non-CSE) included Social Sciences, Health, and Professional Studies.

Two User Groups, One Tool

In this study, participants were grouped to engage in different variations of writing and editing activities using ChatGPT, as well as a collaborative writing task. Our study used a method that blends structured surveys with live, interactive sessions, providing a comprehensive view of how people from various academic backgrounds use and respond to ChatGPT's capabilities in a series of writing-focused exercises.

Study Results 

Both CSE and other disciplines participants displayed an average of intermediate proficiency in using ChatGPT.

ChatGPT as a Writing and Editing Assistant:

• Satisfaction levels increased after refining initial drafts, reaching similar levels for both groups.

• CSE participants showed a notable increase in satisfaction with final essay versions.

• Non-CSE participants rated final essays higher in quality and trustworthiness.

• Interaction ease with ChatGPT rated similarly across tasks and groups.

• Non-CSE users more satisfied with ChatGPT responses; error frequency varied between groups.

Collaborative Writing with ChatGPT

• High engagement across all groups, slightly more in non-CSE disciplines.

• Co-writing dynamics: CSE participants found ChatGPT more complementary.

• Creativity ratings higher among non-CSE participants.

• Story development satisfaction varied slightly, with overall experience rated higher by non-CSE groups.

• Future interest in collaborative writing with ChatGPT was mixed, generally higher in non-CSE groups.

Addressing ChatGPT's Content Policy Challenges in Creative and Professional Settings

In our recent exploration of ChatGPT's usability, we uncovered a notable challenge that significantly affects both creative and professional users. This issue centers around the AI's content policy, which can unintendedly hinder the creative process and disrupt professional tasks.

Creative Limitations for Writers

A example of this issue emerged during a co-writing session (presented in the photo below), where a participant's narrative involving mature themes triggered ChatGPT's content filters. 

This resulted in the AI generating error messages and halting the creative flow, leading to user frustration and disengagement. 

Creative Limitations for Writers

The censorship or flagging of ChatGPT could potentially be a problem for fiction writers. As it is apparent, a large number of adult books will deal with mature subject matter, such as profane language, violence or sex. However, since ChatGPT will flag or censor this type of content, this heavily limits its creative use to mostly children books. If ChatGPT wants to be seen as a tool for writers targeting at adult audiences, there will be a need for a model that is able to deal with classic literary tropes such as a detective solving a murder or a cowboy shooting down enemies.

Professional Implications in Legal Contexts

However, the censorship issue doesn’t just limit itself to creativity, it can also be problematic in professional fields, such as law for example. Taking the example of the field of law related to domestic abuse, a statistic from Canadian Centre for Justice and Community Safety Statistics said that there were 358, 244 victims of police-reported violence in the country last 2019. Assuming that a large number of these domestic abuse incidents in this statistic resulted in a court case, this means that lawyers working on any of these potentially hundreds of thousands domestic abuse court cases that year are unable to use ChatGPT to help them since they will most likely include triggering material which could result in flagging or censorship.

Tailoring ChatGPT for Diverse Content Sensitivities 

It is important to note that there are some use cases for a ChatGPT that doesn't deal with mature content, such as a help desk ChatGPT or a elementary school tutor chatbot. However, as it was previously explained, there are many instances where a ChatGPT that can deal with potentially triggering material is necessary. Hence, OpenAI should be able to provide various types of ChatGPT with different sensitivities to mature themes depending on the use case and audience.

Impact of gamification on user experience

By Thomas O. Ajayi

Introduction

The Entertainment Software Association(ESA) reported in 2016 that 65% of US households are home to at least one person who plays more than 3 hours of video games every week.
The success of digital games in the entertainment industry has made product owners and designers of non-game digital products turn to learning from digital games in order to understand what makes them engage the players and success in order to apply the same knowledge to improve not only improve the user experience of their products but also to make them enjoyable.

Interestingly, academicians and user experience design practitioners alike have researched what makes games successful and how to implement them to motivate and engage users of digital products that are non-games to interact more with the products and improve overall user experience. This concept of applying game-like elements  to non-game situations, with the goal of motivating and engaging users is now popularly referred to as “Gamification”.Popular game-like elements used in gamification are leaderboard, avatar, points and badges.

Surprisingly, my recent research into understanding the impact of gamification on user experience showed both the positive and negative influence of gamification on users as opposed to the assumption that gamification is only full of positive rewards. Below are the highlights of the positives and negatives effects of gamification on user experience and how to apply gamification to support more positive user experience in digital products  based on the insights from academic researches of Hammedi et al. , Koivisto & Hamari and Vanroy & Zaman and user experience design practitioners

The positive effects of gamification on user experience

1. It enhances users' motivation to interact with products:

   
   

Vanroy & Zaman researched the effects of rewards with  game elements of badges, points and leaderboard on university students in Belgium based on their participation in a Google+ community created for a course. Analysis of the students' response to online surveys and focus groups confirmed that the rewards they get based on their participation either as individual or group on the Google+ community motivated them to interact with the Online community more.

2. It boosts users' engagement and focus:

   
   

Hammedi et al  found out that gamified health care helps patients focus all their attention on the activity that they are performing. A practical example was the case of when patients used the Vr4Child app (a game-like application designed to relax children to ensure that they will remain calm during medical procedures), their engagement helped them to not focus on the nurses’ and therapists’ actions. Thereby helping the medical practitioners to perform their procedures without the interruption of the kids(patients).

2. It helps transform boring/tedious activities into more exciting and entertaining ones: 
   

   
   

Through the implementation of Wii Fit Plus- an innovative technology that allows patients to improve their balance through challenges such as a snowboarding simulation or walking on a tight, patients with various medical problems, such as the amputation of a limb, a stroke were report by Hammedi et al. reported to continually improve their physical exercise and to work harder to reach their limits. Without this gamified solution, such patients could find exercise which supports their healing process boring.

The negative effects of gamification on user experience

1. It can encourage users to cheat to get rewards: This was seen in the study by Vanroy & Zaman as students were reported to repeatedly perform the behaviour for which they were already rewarded in order to receive more badges.

2. The enjoyment and usefulness of gamification can decline with time Thereby leading to abandonment of the products where gamification is implemented: This negative effects is shown in the study of Koivisto & Hamari on the users perceived benefits of fitocracy where the younger user demography found the online fitness coach marketplace useful when they start tracking their daily fitness achievement using fitocracy application but after some times, they do not find the rewards they get for tracking their fitness motivating again. 

3. Gamified platforms can become addition for users: Lilith & Tamm reported some digital products especially social media e.g Facebook, youtube or Twitter implemented  gamification using Pull-to-refresh feature to see new posts continuously and the users might be attached to the social media because of the gamification experience. Thereby spending their productive time on the applications.

How digital product owners and user  designers can implement gamification to enhance positive user experience

The negative effects of gamification on user experience highlighted above has shown that integration of game elements in digital elements is not always motivating, engaging or sustainable. The question is now how do product owners and or user experience designers implement gamification to support positive effects while the negative effects are reduced or removed. Scholars and design practitioners proposed that using the Yu-kai Chou behavioural framework named “Octalysis” shown in the Figure 1 below could help in designing gamification for good.

Figure 1:  Yu-kai Chou behavioural framework. Source

The proposition is that one can easily understand what drives users' motivation using this framework and that subsequently will guide the game elements used in the gamification process.

Below are the highlights of what the 8 drivers of human motivation in the behavioural  framework stands for.
Meaning: means that users believe that the meaning of what they do is more important than the thing itself, giving a sense of self-mission.
Accomplishment: means the progress and new skills gained by users when they complete things.

Empowerment: providing creative channels for users to take initiative.
Ownership: means users’ ability to own and control things. 

Social influence: users’ interconnection with other users.
Scarcity: means that some things can be owned by only some users. 

Unpredictability: means mobilising users’ curiosity by showing limited things.
Avoidance: means showing the disadvantages of bad events, and users will choose to avoid losses. More details about this can be found here.

In addition to using the above framework as a guide while designing gamification to tailor the game elements based on the motivation factor(s) of users, designers it could also be helpful for designers to monitor products for side effects( i.e choosing which behaviours to encourage, figuring out how to measure those behaviors, and ensuring that the metric being tracked reliably correlates to the behavior that is wanted) and Set engagement limits. For example a mobile app can be designed to  automatically disable  notifications if a user is driving or remind the user to take screen to avoid distractions that can lead to accidents.

Conclusion
Considering the benefits of the gamification in digital products highlighted in this article and the quest for businesses and user experience designers to compete in the digital era to gain users attention and get them glued to their products,  one can say that  the strategy of gamification has come to stay.
A cue can be taken by product owners and designers alike from the successful implementations of gamification or the proposed fireworks from academics and industry
However, it is important to note that no one size fits all framework or guidelines to implement gamification successfully. Instead more research into who the users are, what the users want to achieve with the use of the product and the game elements that match the users types are needed in order to design digital products where gamification enhances positive user experience.