Library Guides
- Library
- Library Guides
- AI for T&L in Education
- AI and assessment design
AI and assessment design: A multi-layered approach
The nearly ubiquitous access to artificial intelligence for text generation that gained momentum in 2023 has shaken up the education sector largely due to concerns about academic integrity. There were some who thought AI detectors were going to be the answer (spoiler alert: they're not and are biased against non-native English users) and others who thought banning AI was the best approach.
Approaches to mitigate AI misuse that include considering the shortcomings of AI are ultimately doomed to reducing an academic to a Sisyphusian game of whack-a-mole, as the technology and good practice in using it become more sophisticated.
Mitigating AI misuse is a multi-layered approach that includes government, university administration, program-level, course-level and assessment-level considerations, as is illustrated in Professor Phil Dawson's "anti-cheating approaches tier table" below.
Ranked from Most Effective at the top to Least Effective, Professor Dawson makes the case (Dawson, 2022) that it is not just ONE of these approaches that will mitigate the misuse of AI for academic dishonesty. Rather, it is an integrated, "Swiss cheese" model of risk management that involves multiple stakeholders and decision-makers and communication with students that will make the difference.
Inclusive Assessment Practices
In our pursuit of educational excellence, it's essential to recognise and address the diverse needs of all students, including those with disabilities. Inclusive assessment practices ensure that every student has an equitable opportunity to demonstrate their knowledge and skills.
Highlighting the Impact of Disabilities on Assessment:
Incorporating Accessibility and Accommodations:
Guidance for Implementing Inclusive Practices:
Call to Action:
Assessment design considerations
TEQSA, the Tertiary Education Quality Standards Agency, has released the final version of a guidance document on assessment design in the age of AI, available for download from the TEQSA website.
Developed through a process of collaboration and consultation with the higher education community, the document emphasises the recommendations work best as an integrated whole that involves work at the qualification or program level and not just on a course or unit basis.
There are two principles that serve to underpin five ways in which assessment can be implemented to mitigate misuse of AI while leveraging the benefits AI can offer and helping students develop the skills to use AI tools effectively and ethically in their academic, professional and personal lives. Some of these are just good practice in assessment design that hopefully are already built into the way we assess students. Each comes with a rationale and, in the case of the implementation propositions, with scenarios of the principles being enacted in a course or program.
As illustrated above, the two principles are
Principle 1: Equip students to ethically use AI and understand its limitations and impacts
Principle 2: Provide multiple, inclusive, contextualised assessments for trustworthy judgments about student learning
The five "propositions" about assessment include approaches academics can influence and approaches that need to be discussed at the program level. They state that assessment must emphasise:
Security at meaningful points across a program
So, how does one mitigate the use of AI for academic dishonesty at the course level? As per the Dawson chart, there are several implementation and design strategies as well as communication and pedagogical choices that can work together.
Firstly, being explicit about how AI CAN be used in your course and how students should not only cite its use but describe methodology will provide students with a baseline understanding of your expectations. Are students going to need AI skills for their future work? What skills will they need? How can you help them to be mindful, effective and ethical users of AI in their academic and professional lives? Some example course-level AI statements, thought-starters and rationales can be found on the Stanford Univeristy Teaching Commons AI course policy page.
For assessment-level expectation statements, consider the guidelines that academic journals set out in terms of citation and methodology statements related to AI.
To what extent you allow its use will depend on your course. As with any course-level design, start with the learning objectives.
What are the learning objectives they need to achieve? Where are students in their studies? What discipline? What academic skills do they have and what do they need to develop? How are you going to provide students with opportunities throughout the learning modules, weeks and overall course to develop mastery of the knowledge and skills they'll need to achieve the objectives?
Consider what you are assessing in terms of learning. Are you assessing throughput - the process of learning? Or are you simply marking an end product? Your assessment marking guide or rubric should reflect what you think is important. We all know that students are strategic about how they spend their time and effort when undertaking study - as are we when undertaking our work. If you want students to develop critical thinking and evaluative judgment, is a report or an essay or a discussion forum the best possible vehicle?
Could you provide students with choices - as per the Universal Design for Learning Framework - that would allow them to express and enact their learning in modes that are meaningful and useful to them in their context? And consider if and how you could use techniques involving direct interaction with students, such as various oral assessments or scenario-based interactive oral assessments, as described in this Griffith University resource (PDF, 488 KB). Interactive orals have been scaled up to large cohorts by marking students as the assessment occurs, meaning that the time taken to mark is the same as that allotted to a text-based assessment.
Next, consider the evidence base you'll want them to use to support their thinking. Is it content you'll be supplying or is it content you want them to select or a combination? You can ask that their sources include your presentations, academic texts from the reading list you provide, and sources that represent diverse points of view such as those by Indigenous scholars and academics from other countries.
If much of this seems like a re-visit of the basic principles of good assessment design, it's because it is. Like AI that is so competent that people come to trust it too much (Dell'Acqua, et. al. 2023) - it is worth asking ourselves if plagiarism detection software has made us complacent in the assessments we rely upon.
Where possible, ask a peer who is AI-savvy to test drive your assessment or test it yourself, using the prompt engineering techniques contained in this guide. Feed in your rubric and marking guide and ask the AI to provide feedback on how well your work achieved the stated expectations. Doing so may lead you to revise not only your assessment but your marking guide or rubric.
In the section on AI basics, a simple definition of intelligence as the ability to solve complex problems was used to set the stage for understanding artificial intelligence. However, what about human intelligence? What do humans have that machines don't?
Researcher and educator Dr Rose Luckin's work on human intelligences considers what machines can help us understand assessment design that considers not just mastery of learning objectives, but the human skills that set us apart.
The video below sets out what Luckin has identified in her research as seven elements of human intelligence. It should start playing at minute 19:08 (if not, that is where the relevant portion begins) and continue watching until 23:48.
Bringing in research from Korteling, et. al. (2018) and Luckin's work with Margaret Bearman (Bearman & Luckin, 2020), the five core human intelligences can be summarised as:
The book chapter authored by Luckin and Bearman in the book "Re-imagining University Assessment in a Digital World", lays out what the researchers feel are the differences between the way AI processes data and how humans think. It starts with a brief introduction to AI, where it is highlighted that there are different types of AI, some can learn and some only perform within the parameters of what they were programmed to do. The authors then lay out what they consider to be elements of human intelligence (drawing on the research of co-author Rose Luckin, the most important of which they consider to be evaluative judgment, the ability to define “quality” and the conscious development of a personal epistemology.
They then provide an exemplar programmatic assessment strategy for first, second, and third-year students where the sophistication of the task increases to help students to develop the two aforementioned elements of intelligence. They then go on to describe the underlying pedagogical and learning design elements that support the assessment, including asking students to consider which elements of the assessment could have been completed successfully with AI and where the human element is more desirable.
The authors acknowledge that AI is better and faster at some tasks than humans but make the case that having a competent human in the loop and thinking of AI as a tool.
They finish by restating the complexity and nuanced nature of human intelligence and suggest that given the existence of AI not only for cheating but the need to learn how to use it means assessment needs to go beyond only assessing academic intelligence. This is an important book chapter for researchers, academics, and learning designers considering how to design both for and with AI as it not only considers how to prevent the misuse of AI, but how to consciously design for human intelligences.
Integrating AI in the Classroom and Assessments
As we navigate the evolving landscape of education technology, AI presents transformative possibilities for teaching and assessment. Integrating AI into the classroom can personalise learning experiences, provide real-time feedback, and support diverse educational needs.
Personalised Learning Paths:
Interactive Learning Environments:
Automated Grading and Feedback:
Adaptive Assessments:
Accessibility Features:
Bias and Ethical Considerations:
Training and Professional Development:
Collaboration and Sharing Best Practices: