The project pioneers an innovative approach to evaluating High-Impact Practices (HIPs) by leveraging advanced large language models (LLMs) to automate the grading of student critical thinking reflections. Building on research demonstrating LLMs’ capacity to achieve human-level accuracy in grading complex, multi-step explanations (Chen and Wan, 2024), this initiative addresses scalability challenges as the college expands its Experiential Learning and Critical Thinking (EXACT) Plan—a Quality Enhancement Plan initiative embedding experiential learning and critical thinking into 50+ courses by 2028.

The project will develop an AI algorithm trained to assess student reflections using validated rubrics for integrative learning and critical thinking, mirroring the interrater reliability of faculty committees. Unlike traditional automated grading tools, this system integrates two innovative strategies from recent LLM research:
1. Self-Consistency Voting: The AI grades each artifact five times and selects the most frequent outcome, mitigating randomness and improving accuracy. (Chen and Wan, 2024)
2. Prompt Engineering: Rubric items are augmented with detailed descriptions of each skill assessed to reduce ambiguity. This approach, shown to boost human-AI agreement to 70–80%, ensures the AI interprets student skills contextually rather than relying on keyword matching.

The project also builds on studies showing that LLMs excel at partial credit grading when rubrics emphasize reasoning (Chen & Wan, 2024). In practice, the focus will be on evaluating students’ self-reflections on critical thinking and problem-solving logic. To address variability in student responses, the AI will apply a multi-item rubric that dissects critical thinking into 1) effective communication, 2) evaluation & interpretation of information, 3) problem solving, and 4) alternative considerations, with each criterion defined by four levels. 

Anticipated challenges include initial AI-human discrepancies, such as overly strict interpretations of rubric terms, which will be addressed through rubric refinement cycles. To build faculty trust, workshops co-led by the SST EXACT Faculty Fellow and ITEC developer will demonstrate the AI’s alignment with committee standards using side-by-side comparisons of human and AI gradings.  Another challenge occurs within the realm of security of academic sensitive student data resulting from sharing this type of data with third party vendors.  This requires a cross-functional collaborative committee between the Office of Instructional Technology, Legal, and the EXACT Plan.

Expected outcomes include full automation of artifact assessment by Spring 2026, eliminating current sampling limitations that restrict capacity. During Spring 2025, the faculty assessors will access 318 discrete artifacts distributed proportionally between two courses (ITEC 1001 and HIST 1112) over 2 days. The 318 artifacts represent approximately an 18% sampling. This will be the first time that we will ask faculty assessors to evaluate this number of artifacts, so it is not known if this statistically necessary sample is achievable by the number of faculty assessors. Thus, the development of an AI algorithm potentially may enable us to achieve the statistically significant sample size more efficiently. The project will also produce a resource toolkit featuring adaptable rubrics, entropy-based auditing protocols, and feedback templates, shareable across institutions. Additionally, it will contribute research data on LLM performance in non-STEM contexts, such as humanities reflections, informing broader HIPs assessment practices.

By synthesizing advancements in prompt engineering and self-consistency voting, this project positions GGC as a leader in scalable, equitable HIPs evaluation. The AI’s ability to generate actionable feedback offers a cost-effective model for institutions striving to balance rigor with resource constraints. Deliverables will be disseminated via General Space (Digital Archives) through the Kaufman Library services, fostering cross-institutional adoption of AI-enhanced assessment frameworks. This digital archive is open access and will be linked to the GGC EXACT Plan website for easy access to the public.   

The outcomes of this work will be:
• A methodology for AI algorithm for evaluating critical thinking skillset writing reflection artifacts
• A resource toolkit featuring adaptable rubrics, entropy-based auditing protocols, and feedback templates, shareable across institutions
• Contribution to research data on LLM performance in non-STEM contexts, such as humanities reflections, informing broader HIPs assessment practices.

References:
1. Chen, Z., & Wan, T. (2024). Using Large Language Models to Assign Partial Credits to Students' Problem-Solving Process: Grade at Human Level Accuracy with Grading Confidence Index and Personalized Student-facing Feedback. arXiv preprint arXiv:2412.06910.