AI for T&L in Education: AI and assessment design

AI and assessment design: A multi-layered approach

The nearly ubiquitous access to artificial intelligence for text generation that gained momentum in 2023 has shaken up the education sector largely due to concerns about academic integrity. There were some who thought AI detectors were going to be the answer (spoiler alert: they're not and are biased against non-native English users) and others who thought banning AI was the best approach.

Approaches to mitigate AI misuse that include considering the shortcomings of AI are ultimately doomed to reducing an academic to a Sisyphusian game of whack-a-mole, as the technology and good practice in using it become more sophisticated. 

Mitigating AI misuse is a multi-layered approach that includes government, university administration, program-level, course-level and assessment-level considerations, as is illustrated in Professor Phil Dawson's "anti-cheating approaches tier table" below.  

Ranked from Most Effective at the top to Least Effective, Professor Dawson makes the case (Dawson, 2022) that it is not just ONE of these approaches that will mitigate the misuse of AI for academic dishonesty. Rather, it is an integrated, "Swiss cheese" model of risk management that involves multiple stakeholders and decision-makers and communication with students that will make the difference. 

  • Tier 'S' includes: 'Swiss cheese', 'Central teams', 'Amnesty/self-report', and 'Programmatic assessment'.
  • Tier 'A' features: 'Tasks students want to do', 'Vivas', 'Stylometry', 'Document properties', and 'Staff training'.
  • Tier 'B', there's: 'Learning outcomes', 'Proctoring/lockdown', 'Open book', 'Content-matching', 'Better exam design', and 'F2F exams'.
  • Tier 'C' has: 'Academic integrity modules', 'Honour codes', 'Reflective practice', and 'Legislation'.
  • Tier 'D' comprises of: 'Site blocking' and 'Authentic assessment'.
  • Finally, tier 'F' includes: 'Unsupervised MCQ', 'Bans, e.g., essays', and 'Reusing tasks'."

Inclusive Assessment Practices

In our pursuit of educational excellence, it's essential to recognise and address the diverse needs of all students, including those with disabilities. Inclusive assessment practices ensure that every student has an equitable opportunity to demonstrate their knowledge and skills.

Highlighting the Impact of Disabilities on Assessment:

  • Certain traditional assessment methods, such as face-to-face exams, can inadvertently pose challenges for students with specific disabilities. For instance, individuals with low working memory, dyslexia, or ADHD may find these environments more stressful or distracting, which can affect their performance.

Incorporating Accessibility and Accommodations:

  • To foster an inclusive learning environment, it's important to offer a variety of assessment methods and accommodations. This might include alternative formats, additional time, or the use of technology that supports their learning needs. By doing so, we honour the diverse ways in which students process information and express their understanding.

Guidance for Implementing Inclusive Practices:

  • Educators are encouraged to work closely with disability support services to understand the accommodations that can be made for assessments. This collaboration ensures that all students have access to the resources they need to succeed academically.

Call to Action:

  • Embracing inclusive assessment practices is a step toward dismantling barriers in education. We invite all faculty to consider how they can modify their assessment strategies to accommodate the varied needs of their students, creating a more equitable and supportive learning environment.

Assessment design considerations

TEQSA guidelines

TEQSA, the Tertiary Education Quality Standards Agency, has released the final version of a guidance document on assessment design in the age of AI, available for download from the TEQSA website.

Developed through a process of collaboration and consultation with the higher education community, the document emphasises the recommendations work best as an integrated whole that involves work at the qualification or program level and not just on a course or unit basis.

There are two principles that serve to underpin five ways in which assessment can be implemented to mitigate misuse of AI while leveraging the benefits AI can offer and helping students develop the skills to use AI tools effectively and ethically in their academic, professional and personal lives. Some of these are just good practice in assessment design that hopefully are already built into the way we assess students. Each comes with a rationale and, in the case of the implementation propositions, with scenarios of the principles being enacted in a course or program.

As illustrated above, the two principles are

Principle 1: Equip students to ethically use AI and understand its limitations and impacts

Principle 2: Provide multiple, inclusive, contextualised assessments for trustworthy judgments about student learning 

The five "propositions" about assessment include approaches academics can influence and approaches that need to be discussed at the program level. They state that assessment must emphasise:

  1. Appropriate, authentic engagement with AI
  2. A programmatic/systemic approach to assessment aligned with discipline and qualification values
  3. The process of learning (rather than the output)
  4. Opportunities for students to work with AI and each other, and 

Security at meaningful points across a program

Mitigating misuse

So, how does one mitigate the use of AI for academic dishonesty at the course level? As per the Dawson chart, there are several implementation and design strategies as well as communication and pedagogical choices that can work together.

Set expectations

Firstly, being explicit about how AI CAN be used in your course and how students should not only cite its use but describe methodology will provide students with a baseline understanding of your expectations. Are students going to need AI skills for their future work? What skills will they need? How can you help them to be mindful, effective and ethical users of AI in their academic and professional lives? Some example course-level AI statements, thought-starters and rationales can be found on the Stanford Univeristy Teaching Commons AI course policy page.

For assessment-level expectation statements, consider the guidelines that academic journals set out in terms of citation and methodology statements related to AI. 

Consider the Learning Objectives

To what extent you allow its use will depend on your course. As with any course-level design, start with the learning objectives.

What are the learning objectives they need to achieve? Where are students in their studies? What discipline? What academic skills do they have and what do they need to develop? How are you going to provide students with opportunities throughout the learning modules, weeks and overall course to develop mastery of the knowledge and skills they'll need to achieve the objectives? 

Assess process, not just output

Consider what you are assessing in terms of learning. Are you assessing throughput - the process of learning? Or are you simply marking an end product? Your assessment marking guide or rubric should reflect what you think is important. We all know that students are strategic about how they spend their time and effort when undertaking study - as are we when undertaking our work. If you want students to develop critical thinking and evaluative judgment, is a report or an essay or a discussion forum the best possible vehicle?

Ask for multiple modes, including direct interactions

Could you provide students with choices - as per the Universal Design for Learning Framework  - that would allow them to express and enact their learning in modes that are meaningful and useful to them in their context? And consider if and how you could use techniques involving direct interaction with students, such as various oral assessments or scenario-based interactive oral assessments, as described in this Griffith University resource (PDF, 488 KB). Interactive orals have been scaled up to large cohorts by marking students as the assessment occurs, meaning that the time taken to mark is the same as that allotted to a text-based assessment.

Require referencing of your key points

Next, consider the evidence base you'll want them to use to support their thinking. Is it content you'll be supplying or is it content you want them to select or a combination? You can ask that their sources include your presentations, academic texts from the reading list you provide, and sources that represent diverse points of view such as those by Indigenous scholars and academics from other countries.

If much of this seems like a re-visit of the basic principles of good assessment design, it's because it is. Like AI that is so competent that people come to trust it too much (Dell'Acqua, et. al. 2023) - it is worth asking ourselves if plagiarism detection software has made us complacent in the assessments we rely upon. 

Test your assessment and marking guide

Where possible, ask a peer who is AI-savvy to test drive your assessment or test it yourself, using the prompt engineering techniques contained in this guide. Feed in your rubric and marking guide and ask the AI to provide feedback on how well your work achieved the stated expectations. Doing so may lead you to revise not only your assessment but your marking guide or rubric.  

Human intelligences

In the section on AI basics, a simple definition of intelligence as the ability to solve complex problems was used to set the stage for understanding artificial intelligence. However, what about human intelligence? What do humans have that machines don't?

Researcher and educator Dr Rose Luckin's work on human intelligences considers what machines can help us understand assessment design that considers not just mastery of learning objectives, but the human skills that set us apart.

The video below sets out what Luckin has identified in her research as seven elements of human intelligence. It should start playing at minute 19:08 (if not, that is where the relevant portion begins) and continue watching until 23:48.

Bringing in research from Korteling, et. al. (2018) and Luckin's work with Margaret Bearman (Bearman & Luckin, 2020), the five core human intelligences can be summarised as:

  • Metacognition – reflect on thinking, change mind as we grow
  • Metasubjectivity – reflect on our emotional IQ and that of others
  • Metacontextualism – apply existing knowledge to new situations, adapt, be resilient
  • Evaluative judgement – what “quality” looks like
  • Self-efficacy – what do I need to know/practice to do X

The book chapter authored by Luckin and Bearman in the book "Re-imagining University Assessment in a Digital World", lays out what the researchers feel are the differences between the way AI processes data and how humans think. It starts with a brief introduction to AI, where it is highlighted that there are different types of AI, some can learn and some only perform within the parameters of what they were programmed to do. The authors then lay out what they consider to be elements of human intelligence (drawing on the research of co-author Rose Luckin, the most important of which they consider to be evaluative judgment, the ability to define “quality” and the conscious development of a personal epistemology. 

They then provide an exemplar programmatic assessment strategy for first, second, and third-year students where the sophistication of the task increases to help students to develop the two aforementioned elements of intelligence. They then go on to describe the underlying pedagogical and learning design elements that support the assessment, including asking students to consider which elements of the assessment could have been completed successfully with AI and where the human element is more desirable.

The authors acknowledge that AI is better and faster at some tasks than humans but make the case that having a competent human in the loop and thinking of AI as a tool.

They finish by restating the complexity and nuanced nature of human intelligence and suggest that given the existence of AI not only for cheating but the need to learn how to use it means assessment needs to go beyond only assessing academic intelligence. This is an important book chapter for researchers, academics, and learning designers considering how to design both for and with AI as it not only considers how to prevent the misuse of AI, but how to consciously design for human intelligences. 

Integrating AI in the Classroom and Assessments

As we navigate the evolving landscape of education technology, AI presents transformative possibilities for teaching and assessment. Integrating AI into the classroom can personalise learning experiences, provide real-time feedback, and support diverse educational needs.

AI-Enhanced Teaching Strategies:

Personalised Learning Paths:

  • AI can analyse students' learning styles, strengths, and areas for improvement, offering personalised resources and assignments. This approach ensures that learning is tailored to each student's needs, enhancing their understanding and engagement.

Interactive Learning Environments:

  • Utilise AI-driven tools to create dynamic learning environments. For example, simulations, educational games, and virtual labs powered by AI can offer hands-on learning experiences that are both engaging and informative.


AI in Assessments:

Automated Grading and Feedback:

  • AI can assist in grading assignments and providing immediate, personalised feedback to students. This not only reduces the workload for educators but also accelerates the feedback loop, helping students to learn and improve more effectively.

Adaptive Assessments:

  • Incorporate AI to develop adaptive assessments that adjust in difficulty based on the student's performance. This ensures that tests are fair and appropriately challenging, catering to the individual's learning level.


Promoting Equity and Inclusion with AI:

Accessibility Features:

  • AI tools can offer accessibility features such as text-to-speech, speech-to-text, and language translation, making learning materials more accessible to all students, including those with disabilities.

Bias and Ethical Considerations:

  • It's crucial to select AI tools that are developed with an emphasis on fairness and inclusivity, avoiding biases that could affect assessment outcomes. Educators should be vigilant and advocate for ethical AI use in the classroom.


Getting Started with AI Integration:

Training and Professional Development:

  • Encourage educators to participate in professional development opportunities focused on AI and its application in education. Understanding the capabilities and limitations of AI is key to integrating it effectively.

Collaboration and Sharing Best Practices:

  • Foster a community of practice among educators to share experiences, strategies, and resources for integrating AI in education. Collaboration can lead to innovative approaches and enhance the quality of teaching and learning.