Lesson 1 – Benefits, Limits and Risks of Generative AI
As a Math teacher, Moira is noticing the impact of student reliance on ChatGPT – even when used to check work. Often, students will complete a math problem and have ChatGPT check it. But sometimes, ChatGPT is wrong. Since they are not confident in their own work, students will ask Moira why their own answer is wrong when, in fact, it is correct. Moira is concerned that in over-trusting Gen AI tools, her students will not learn how to trust their own skills.
The conversational nature of Generative AI (Gen AI) can provide a false sense of its intelligence. In reality, it generates responses to prompts based on statistical associations – not thinking.
It often makes errors and reflects the biases of its training data. Yet, it is very easy to trust since their presentation style is so confident and convincingly human.
To set the stage for use of Gen AI with your students, it is important to consider its benefits, limits, and risks.
Benefits
Helpful Assistant
Can help you to complete certain tasks more quickly and effectively. It is very good for initial research, summarizing, editing, and providing general feedback on your work.
Personalized Learning
Can prompt it to engage in tutoring, simulation, and debates. With custom chatbots, you can query either a specific set of information (such as course information and learning content) to get immediate responses.
Enhanced Creativity
Can help you to brainstorm ideas, create analogies, draft fictional case studies, and suggest ideas for activities.
Limits
Flawed Results
It can “hallucinate” or generate wrong information (e.g. create research references that do not exist, math problems that are not solvable etc.).
Bias and Training Data
Gen AI models can perpetuate biases that pre-exist in the training data.
Dependent on Quality of Tool and Data
Gen AI requires a lot of computer processing resources. Free tools have more processing and output limits which can impede quality.
Risks
Privacy Concerns
Gen AI tools can collect information about its users. In particular, free versions of tools may use or share this information.
Overreliance
Outsourcing too much thinking to Gen AI can hinder foundational knowledge and skill development. It can also limit independent learning and building of confidence.
Ethical Considerations
- Training Data Includes Copyrighted Materials
Gen AI training data is comprised of information scraped from the Internet. This includes copyrighted content that was collected without creator permission.
- EquityFree versions of tools may become less robust over time than paid ones. Students with paid versions could have advantage over those limited to free ones.
Reflection
Gen AI can be useful, but it isn’t neutral.
These questions invite you to think about how your assessment design helps students build judgment, awareness, and care as they learn to work with Gen AI.
- What do you want students to learn and how can Gen AI support or hinder this?
- What does safe, responsible use of Gen AI look like in your industry or subject area, and how do students learn this?
- What strategies could help students learn to spot bias or gaps in Gen AI outputs and respond thoughtfully?
Explore Further
Generative AI in a Nutshell – How to Survive and Thrive in the Age of AI (video), Henrik Kniberg
Watch some or all of this video to understand how Gen AI works.
You’re Not Behind (Yet): How to Learn AI in 29 Minutes (video), Futurepedia
This longer video includes sections on agentic AI and “vibe coding”.
Strengths and Weaknesses of Gen AI, University of Leeds
A good summary that also considers weakness such as AI’s carbon footprint, harms to workers involved in reviewing toxic content during the training model refining process, and the potential for model collapse.
Limitations and Risks, Generative Artificial Intelligence and University Study, University of Reading
A thorough overview of limitations and risks of Gen AI related to trustworthiness of generated information, ethics including reinforcing stereotypes and magnifying inequities, cognitive and behavioural impacts, and safety.