Researchers Created Chatbot to Teach Law Class in Uni, But AI Kept Messing Up

A significant revelation was the sheer effort required to get the chatbot working effectively in tests.

- Armin Alimardani and Emma A. Jane

"AI tutors" have been hyped as a way to revolutionise education. The idea is generative artificial intelligence (AI) tools (such as ChatGPT) could adapt to any teaching style set by a teacher. The AI could guide students step-by-step through problems and offer hints without giving away answers. It could then deliver precise, immediate feedback tailored to the student's individual learning gaps.

Despite the enthusiasm, there is limited research testing how well AI performs in teaching environments, especially within structured university courses.

In our new study, we developed our own AI tool for a university law class. We wanted to know, can it genuinely support personalised learning or are we expecting too much?

OUR STUDY

In 2022, we developed SmartTest, a customisable educational chatbot, as part of a broader project to democratise access to AI tools in education.

Unlike generic chatbots, SmartTest is purpose-built for educators, allowing them to embed questions, model answers and prompts. This means the chatbot can ask relevant questions, deliver accurate and consistent feedback and minimise hallucinations (or mistakes). SmartTest is also instructed to use the Socratic method, encouraging students to think, rather than spoon-feeding them answers.

We trialled SmartTest over five test cycles in a criminal law course (that one of us was coordinating) at the University of Wollongong in 2023.

Each cycle introduced varying degrees of complexity. The first three cycles used short hypothetical criminal law scenarios (for example, is the accused guilty of theft in this scenario?). The last two cycles used simple short-answer questions (for example, what's the maximum sentencing discount for a guilty plea?).