By Regan A.R. Gurung, Associate Vice Provost and Executive Director of the Center for Teaching and Learning and Professor of Psychological Science, Oregon State University
It seems like the world is divided into two main groups of people. There are those who think a lot about artificial intelligence (AI) and grapple with how best to have their students use it while they use it themselves. There are also those who have heard enough about AI to want to ensure students never use it. There are still several academics in both camps who have not played with AI (anecdotal data suggests they are disproportionately in the second group). While no one should be a fan of wanton academic dishonesty or flagrant unsanctioned AI aid on assignments, fears of AI use for cheating distract us from bigger issues.
Yes, the ethical use of AI is important, but there are others as well. I have an overarching framework for thinking about AI which serves as my guide for AI use just as my teaching philosophy guides my course design and instructional methods. Incorporating many key considerations and student learning, I try and get a F.E.A.L for AI. This Faster, Ethical, Accurate, Learning (F.E.A.L., Gurung, 2023) model is one easy way for me to assess whether AI should be used for an assignment or activity. I ask four main questions. Will AI make the task faster? Would AI use be ethical? Is the output accurate? Will it influence learning? A recent article provides some interesting answers to the perhaps the most important question, how does AI use influence learning?
What was done?
Wu and Yu (2024) conducted a meta-analysis of research on Chatbots, statistically combined the results from different studies. AI chatbots or conversational agents such as ChatGPT, interact with the user incorporating a range of AI techniques such as natural language processing, machine learning, and neural networks. The chatbot can save your input and questions (prompts) and learn from previous input to provide better output. The article examined various databases and found studies where chatbots were used to improve learning, defined as the extent to which student gain and apply valuable skills (p.13). They found 1387 potential documents and after removing those that did not have experimental designs, lacked statistical information, did not measure learning, were written in a language other than English, were irrelevant or incomplete, focused on 24 articles. A sample experimental design from one of the studies used (Lee et al., 2022) is shown below (Figure 1). Students interacting with the AI are shown in Figure 2.
What did they find?
There was a statistically significant effect of using chatbots on learning in many realms. Using AI improved learning performance, motivation to learn, the sense of being able to succeed as a learning (i.e., self-efficacy), interest, and the perceived value of learning. Chatbot use also relieved learner’s anxiety. Statistically, effect sizes, a measure of how strong a relationship is, can range from 0 onwards with numbers above .40 signifying a stronger effect. In this meta-analysis some effects were as high as 1.40 (value of learning) and 1.03 (performance). These positive effects were stronger for college-based studies, and when use was under 10 weeks in duration (effect size = 1.18).
What does this mean for us?
We cannot ignore the positive gains of Chatbot use. The strong effects of using chatbots were evident across many domains regarding learning. In contrast to past studies showing inconsistent effects of AI chatbots on learning, the results of this paper suggest students using chatbots could learn better. Given the effects on psychological variables such as motivation and self-efficacy, this paper urges us to look beyond just the ethical implications of AI use, but also to how non-cognitive psychological factors can be influenced by AI use. If using Chatbots can increase interest in learning as demonstrated here, it implies a need for research to be conducted on several potential mediators and moderators of the relationship between AI use and learning outcomes.
A word of caution. Meta-analyses are notorious for obfuscating critical differences in design. There are a lot of devils in the details. As much as I am enthused by this paper, I note it focuses on only 24 studies. Furthermore, a range of learning outcomes were used (e.g., writing skills, test scores), many of the studies had small sample sizes, and most of the studies included focused on learning language. These factors make generalization difficult.
The bigger point is this:
There is no denying the fact that AI can be used in many ways, even to cheat on assignments (Bubaš, & Čižmešija, 2023) but Wu and Yu’s (2024) meta-analysis provide strong evidence of educational gains from chatbot use. The effects on learning of AI use were large. These findings add to research showing that AI can greatly benefit teachers and students (Rahman & Watanobe, 2023) and increase student skills and motivation (Wollny et al., 2021). This recent piece shows that AI chatbot use even leads to improvements in learning. Faculty need to reflect on the best ways to harness the benefits of this tool.
References
Bubaš, G., & Čižmešija, A. (2023). A critical analysis of students’ cheating in online assessment in higher education: Post-COVID-19 Issues and Challenges Related to Conversational Artificial Intelligence. MIPRO ICT and Electronics Convention (MIPRO), 905-910. https://doi.org10.23919/MIPRO57284.2023.10159826
Lee, Y.-F., Hwang, G.-J., & Chen, P.-Y. (2022). Impacts of an AI-based chabot on college students’ after-class review, academic performance, self-efficacy, learning attitude, and motivation. Educational Technology Research and Development, 70(5), 1843–1865. https://doi.org/10.1007/s11423-022-10142-8
Rahman, M. M., & Watanobe, Y. (2023) ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences. 2023; 13(9):5783. https://doi.org/10.3390/app13095783
Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are We There Yet? – A Systematic Literature Review on Chatbots in Education. Frontiers in artificial intelligence, 4, 654924. https://doi.org/10.3389/frai.2021.654924
Wu, R., & Yu, Z. (2024). Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology, 55, 10–33. https://doi.org/10.1111/bjet.13334
Figure 1 Experimental procedure (Lee et al., 2022)
Figure 2 Students of the experimental group doing after-class review with the chatbot (Lee et al., 2022)
About the author: Regan A. R. Gurung, Ph.D. is Associate Vice Provost and Executive Director of the Center for Teaching and Learning at Oregon State University and Professor of Psychological Science. This is part of our series of Research Advancing Pedagogy (RAP) blogs, designed to share the latest pedagogical research from across the disciplines in a pragmatic format.
Image generated with Microsoft Copilot
Leave a Reply