Analysis Of User Experience Of ChatGPT And Gemini Users Using The User Experience Quistionnaire (UEQ) For Education

Ilham Nasrul; Angraini Angraini; Muhammad Luthfi Hamzah; Eki Saputra

doi:10.32736/sisfokom.v13i3.2250

Authors

Ilham Nasrul Department of Information System Faculty of Science and Technology, Universitas Islam Negeri Sultan Syarif Kasim Riau
Angraini Angraini Department of Information System Faculty of Science and Technology, Universitas Islam Negeri Sultan Syarif Kasim Riau
Muhammad Luthfi Hamzah Department of Information System Faculty of Science and Technology, Universitas Islam Negeri Sultan Syarif Kasim Riau
Eki Saputra Department of Information System Faculty of Science and Technology, Universitas Islam Negeri Sultan Syarif Kasim Riau

DOI:

https://doi.org/10.32736/sisfokom.v13i3.2250

Keywords:

Artificiall Intelligence, ChatGPT, Gemini, User Experience, User Experience Questionnaire

Abstract

AI is becoming more and more crucial in the digital age to support kids in overcoming obstacles to learning and succeeding academically. The use of chatbots is one example of AI progress. Two well-known chatbots are Gemini and ChatGPT. Because they are useful and support a variety of learning tasks, including answering questions, producing articles, expanding knowledge, and other academic activities, both applications are highly well-liked and preferred by students. By using a case study on the Facebook community with the number of samples needed in this study as many as 377 respondents based on the Krejcie and Morgan formula, The purpose of this study was to determine whether user experiences with different applications differed. User experience measurement was carried out using the User Experience Questionnaire (UEQ) approach on the variables of Efficiency, Novelty, Attractiveness, Stimulation, Perspicuity, and Dependability. The results of the study show that all user experience variables for the ChatGPT and Gemini applications received poor ratings, and there were no significant differences in any of these variables. However, based on UEQ measurements, it was found that both applications received better scores on the stimulation and novelty variables, while the attractiveness, clarity, efficiency, and accuracy variables received poor results. To improve user experience in the ChatGPT and Gemini applications, the quality of all variables needs to be enhanced.

References

F. Fauzi, L. Tuhuteru, F. Sampe, A. M. A. Ausat, dan H. R. Hatta, “Analysing the Role of ChatGPT in Improving Student Productivity in Higher Education,” J. Educ., vol. 5, no. 4, hal. 14886–14891, 2023, doi: 10.31004/joe.v5i4.2563.

V. A. Putri, K. Carissa, A. Sotyawardani, dan R. A. Rafael, “Peran Artificial Intelligence dalam Proses Pembelajaran Mahasiswa di Universitas Negeri Surabaya,” Pros. Semin. Nas., hal. 615–630, 2023.

S. Pokkakillath dan J. Suleri, “ChatGPT and its impact on education,” Res. Hosp. Manag., vol. 13, no. 1, hal. 31–34, 2023, doi: 10.1080/22243534.2023.2239579.

M. R. Karaman dan İ. Göksu, “Are Lesson Plans Created by ChatGPT More Effective? An Experimental Study,” Int. J. Technol. Educ., vol. 7, no. 1, hal. 107–127, 2024, doi: 10.46328/ijte.607.

R. N. Darma, A. Syahid, Fatma, Ayu Putriana Lestari, Ahmad Basahil, dan Rini Listiya Ningrum, “The Student’s Perception of Using ChatGPT for EFL Students,” J. Ilmu Pendidik. Nas., vol. 1, no. 3, hal. 143–146, 2023, doi: 10.59435/jipnas.v1i3.193.

J. Naher, M. H. Maruf, S. T. Bakht, dan S. S. Sadaf, “A Survey to Understand the experience of ChatGPT Usage among Engineering University Students in Bangladesh,” Tech. Rom. J. Appl. Sci. Technol., vol. 15, hal. 1–8, 2023, doi: 10.47577/technium.v15i.9660.

C. K. Lo, “What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature,” Educ. Sci., vol. 13, no. 4, 2023, doi: 10.3390/educsci13040410.

E. L. Amalia dan D. W. Wibowo, “Rancang Bangun Chatbot Untuk Meningkatkan Performa Bisnis,” J. Ilm. Teknol. Inf. Asia, vol. 13, no. 2, hal. 137, 2019, doi: 10.32815/jitika.v13i2.410.

A. R. Ellis dan E. Slade, “A New Era of Learning: Considerations for ChatGPT as a Tool to Enhance Statistics and Data Science Education,” J. Stat. Data Sci. Educ., vol. 31, no. 2, hal. 128–133, 2023, doi: 10.1080/26939169.2023.2223609.

L. S. Balhorn, J. M. Weber, S. Buijsman, dan ..., “What does ChatGPT know about natural science and engineering?,” arXiv Prepr. arXiv …, hal. 1–18, 2023, [Daring]. Tersedia pada: https://arxiv.org/abs/2309.10048%0Ahttps://arxiv.org/pdf/2309.10048

C. M. H. Chan et al., “Perception of Harms and Benefits of Electronic Cigarettes Among Adult Malaysian Men: A Comparison by Electronic Cigarette Use and Smoking Status,” Asia-Pacific J. Public Heal., vol. 31, no. 7_suppl, hal. 32S-43S, 2019, doi: 10.1177/1010539519860730.

W. Suharmawan, “Pemanfaatan Chat GPT Dalam Dunia Pendidikan,” Educ. J. J. Educ. Res. Dev., vol. 7, no. 2, hal. 158–166, 2023, doi: 10.31537/ej.v7i2.1248.

R. G. Jurado, T. Pettersson, dan M. Zwierewicz, “STUDENTS ’ ATTITUDES TO THE USE OF ARTIFICIAL INTELLIGENCE Universidade Alto Vale do Rio do Peixe ( BRAZIL )”.

Y. Heryanto, F. Farahdinna, dan S. Wijanarko, “Evaluasi Responsivitas dan Akurasi : Perbandingan Kinerja ChatGPT dan Google BARD dalam Menjawab Pertanyaan seputar Python,” vol. 9, hal. 248–256, 2024.

A. N. Aini, R. Rinofah, dan A. Maulida, “Efektifitas Pengembangan Artificial Intelligence (AI) pada Chatbot MbakPia,” JIIP - J. Ilm. Ilmu Pendidik., vol. 6, no. 10, hal. 8224–8228, 2023, doi: 10.54371/jiip.v6i10.3047.

B. Karyadi, “Pemanfaatan Kecerdasan Buatan Dalam Mendukung Pembelajaran Mandiri,” J. Teknol. Pendidik., vol. 8, no. 2, hal. 253–258, 2023, doi: 10.32832/educate.v8i02.14843.

S. Sugiono dan I. Agency, “Pemanfaatan Chatbot Pada Masa Pandemi Covid-19 Kajian Fenomena Society,” no. November, 2022.

D. A. N. O. Pso, “INTI NUSA MANDIRI,” vol. 18, no. 1, hal. 59–64, 2023.

B. Ram dan V. Pratima, “Artificial intelligence AI-based chatbot study of ChatGPT, Google AI Bard and Baidu AI,” World J. Adv. Eng. Technol. Sci., vol. 8, no. 1, hal. 258–261, 2023.

Carolus Borromeus Mulyatno, “Analisis User Experience Game Android : Amazepong Bali 1.0 Sigit,” J. Pendidik. dan Konseling Univ. Pahlawan Tuanku Tambusai, vol. 4, hal. 1349–1358, 2022, [Daring]. Tersedia pada: https://journal.universitaspahlawan.ac.id/index.php/jpdk/article/view/11165

R. Umar, A. Z. Ifani, F. I. Ammatulloh, dan M. Anggriani, “Analisis Sistem Informasi Web Lsp Uad Menggunakan User Experience Questionnaire (Ueq),” METHOMIKA J. Manaj. Inform. dan Komputerisasi Akunt., vol. 4, no. 2, hal. 173–178, 2021, doi: 10.46880/jmika.vol4no2.pp173-178.

M. Schrepp, “Buku Pegangan Kuesioner,” Marhaeni, vol. 59, hal. 1–15, 2020.

M. S. Rahaman, M. M. T. Ahsan, N. Anjum, M. M. Rahman, dan M. N. Rahman, “The AI Race is on! Google’s Bard and Openai’s Chatgpt Head to Head: An Opinion Article,” SSRN Electron. J., 2023, doi: 10.2139/ssrn.4351785.

D. Khuntari, “Analisis Pengalaman Pengguna Aplikasi Gojek dan Grab dengan Pendekatan User Experience Questionnaire,” J. Tek. Inform. dan Sist. Inf., vol. 8, no. 1, hal. 275–286, 2022, doi: 10.28932/jutisi.v8i1.4499.

A. N. Hidayat, B. T. Hanggara, dan B. S. Prakoso, “Analisis Perbandingan Pengalaman Pengguna Pada Aplikasi Dompet Digital (Studi Kasus Pada OVO dan DANA),” … Teknol. Inf. dan Ilmu …, vol. 5, no. 6, hal. 2181–2187, 2021.

M. A. Kresnanto, B. T. Hanggara, dan B. S. Prakoso, “Analisis Pengalaman Pengguna pada Aplikasi Mobile Booking Hotel dengan menggunakan Metode User Experience Questionnaire ( UEQ ) ( Studi pada RedDoorz dan Airy ),” J. Pengemb. Teknol. Inf. dan Ilmu Komput., vol. 4, no. 10, hal. 3637–3646, 2020.

S. Rasio Henim dan R. Perdana Sari, “Evaluasi User Experience Sistem Informasi Akademik Mahasiswa pada Perguruan Tinggi Menggunakan User Experience Questionnaire,” J. Komput. Terap., vol. 6, no. 1, hal. 69–78, 2020, [Daring]. Tersedia pada: https://jurnal.pcr.ac.id/index.php/jkt/

E. Haerani dan A. Rahmatulloh, “Analisis User Experience Aplikasi Peduli Lindungi untuk Menunjang Proses Bisnis Berkelanjutan,” SATIN - Sains dan Teknol. Inf., vol. 7, no. 2, hal. 01–10, 2021, doi: 10.33372/stn.v7i1.762.

A. Noor dan E. L. Hadisaputro, “Analisis Pengalaman Pengguna Pada Aplikasi TIX ID Menggunakan Metode User Experience Questionnaire,” J. Inf. Syst. Res., vol. 3, no. 4, hal. 672–677, 2022, doi: 10.47065/josh.v3i4.1881.

R. A. Ningsih dan K. D. Tania, “Comparison of Tiket.com and Pegipegi User Experience Using the UEQ Method,” J. Tek. Inform. dan Sist. Inf., vol. 9, no. 2, hal. 177–188, 2023, doi: 10.28932/jutisi.v9i2.6079.