Gradebot: An AI Grading Assistant for Faculty

Vidyanand Choudhary, Amanda Jones, Arvind Sathi, Neena Sathi

Introduction

Artificial intelligence (AI) continues to evolve, offering innovative solutions to complex problems across various domains. One such application is Gradebot, a tool designed to assist teachers and professors with grading student submissions for assignments. Grading is a time-consuming and subjective process, often influenced by the grader's interpretation of the rubric and individual biases. Gradebot leverages artificial intelligence to streamline the grading process, offering a solution that is both efficient and unbiased. This tool represents a significant advancement in educational technology, addressing the challenges of subjective evaluation and time-consuming feedback in academic grading. Gradebot addresses these challenges by incorporating a detailed rubric that reflects the professor’s grading criteria, ensuring a more consistent and objective evaluation process.

While it is easy to grade multiple-choice responses, they often do not fully test students’ creativity, knowledge and writing skills.  Teachers often resort to essay style questions for this reason.  While multiple-choice questions can be graded quickly and objectively, essay questions require subjective judgment. Evaluating writing style, presentation skills, and depth of analysis can be time-consuming and inconsistent.

Teaching assistants often struggle with providing the detailed feedback necessary to help students improve, especially in identifying the boundary between good and excellent work.  Teachers often use a rubric to publish examination expectations and seek their teaching assistants to use the rubric for grading.  What if we could use generative AI to generate a draft of the grading feedback and the score for each student? The Teaching Assistant can then review and edit the scores and the feedback before sharing with the students.

What is Gradebot

Gradebot addresses these challenges by automating the grading process using AI by evaluating assignments based on a rubric. The rubric can be provided by the instructor or developed interactively by Gradebot. The features of Gradebot include:

·      Automated AI grading based on detailed rubrics.

·      Objective and consistent evaluation of student assignments.

·      Detailed feedback generation for students, enhancing the learning experience.

·      Instructor / TA can modify the grading rubric at any time and they should review and  edit the scores and feedback generated by Gradebot.

·      Interactively generate the grading rubric and test it.

Gradebot requires several inputs to function effectively:

1. A rubric file explaining how an assignment needs to be graded

2. The student’s assignment file

Gradebot Demonstration

Let's demonstrate Gradebot's capabilities through an example. Here is the assignment that was given to a class of executive education students on the topic of Generative AI:

Assignment: Select and Document Your Chatbot Use Case

In this assignment, you will choose and document a use case for your chatbot project. Please document your use case by answering the following questions:

1.        Use Case Name: What is the name of your selected use case?

2.        User Personas/Stakeholders: Who are the key user personas or stakeholders impacted by your use case?

3.        Potential User Count: Estimate the number of potential users who may be affected by this use case.

4.        Query Types: What types of queries or questions do you want your chatbot to be able to answer within this use case?

5.        Data Sources: Identify the key data sources required to implement this use case effectively.

6.        Benefits: Explain the potential benefits of implementing this use case for your chosen user personas or impacted stakeholders.

The instructor developed the following grading rubric, outlining 6 key grading criteria for evaluating student projects:

1.        Presentation of the User Persona: This criterion looks for a detailed persona with name, business functions, geography, or social characteristics. A comprehensive persona earns the full score. As shown in table below, partial answers will receive a partial score.

2.        Quality of the Prompt: This criterion looks for clear, precise, and engaging prompts that facilitate interactive dialogues with the bot. This area often has room for improvement even in the best responses.

3.        Classification Table: Students who effectively utilize terminology or examples within their tables earn full score.

4.        Engagement with Custom Documents: This involves loading and generating responses through interactions with these documents, demonstrating an understanding of the required analysis criteria using Generative AI Technology called RAG.

5.        Key Learnings: Students must illustrate their grasp of the project's concepts and clearly articulate their main takeaways.

6.        Quality of the Presentation: Judged on clarity and ease of understanding. An easy-to-follow presentation that clearly conveys its message meets the standards for excellence.

This rubric is provided to Gradebot in the following format, allowing Gradebot to understand what the thresholds are for high score vs low score:

Rubric Table

Building upon the foundation of its rubric-based analysis, Gradebot further demonstrates its utility by offering constructive feedback. To see the kind of feedback and score that Gradebot can generate, we input an assignment from a student named Victor Smartypants. Victor’s project focused on resume consultation, included the creation of multiple user personas and their interactions with the bot, and was presented in a comprehensive 14-page document.

Victor Assignment

Gradebot effectively graded this project according to the established rubric, providing both a score and constructive feedback. This evaluation will showcase how Gradebot examines an essay submission, highlighting good performance and areas for improvement.

Assignment Feedback

Victor Smartypants’s overall score was a commendable 90. Here's how Gradebot applied the rubric criteria:

- User Persona: Victor’s persona was well-defined, earning high marks, though he missed points for not including more detailed social characteristics or geography.

- Prompts: His prompts were interactive and well-crafted but lost points for a lack of connectivity.

- Classification Table: His table demonstrated a clear understanding and application of terminology, earning full points.

- Engagement with Custom Documents: Victor’s project showcased detailed engagement with the bot, effectively demonstrating the required analysis criteria.

- Key Learnings: He clearly articulated his grasp of the project concepts, earning full points.

- Presentation Quality: His presentation was clear and easy to understand, though suggestions were made for better story connectivity.

Gradebot exemplifies how AI-powered solutions can transform traditional processes, such as grading, by ensuring consistency, saving time, and providing detailed feedback. The broader applications of AI assistants demonstrate their potential to revolutionize various aspects of daily life and professional tasks.  Here’s how Gradebot can reliably provide grading functionality:

1. Rubric Integration: The professor provides a rubric or generates one using Gradebot. The rubric outline the criteria for grading assignments and provides specific expectations for different grade levels.

2. Automated Evaluation: Gradebot uses the rubric to evaluate student submissions. By referencing the taxonomy, it can identify key elements that distinguish high-quality responses from subpar ones.

3. Accurate Grading: The system assigns grades based on the taxonomy’s criteria, ensuring consistency with the professor’s expectations. This reduces the subjectivity inherent in manual grading.

4. Constructive Feedback: In addition to grading, Gradebot provides students with detailed feedback. The feedback is aligned with the taxonomy in the rubric, offering specific suggestions for improvement.

5. Continuous Improvement: Professors can refine the rubric over time, incorporating new insights and adjustments. This iterative process ensures that the grading criteria remain relevant and effective.

Conclusions

Gradebot illustrates how Generative AI can transform the grading process, providing consistent, objective evaluations and valuable feedback for students. As AI continues to evolve, the implementation of Gradebot will play a crucial role in harnessing expert knowledge to solve complex problems more effectively. Gradebot represents a transformative step forward in educational assessment, marrying the vast knowledge base of artificial intelligence with the nuanced requirements of academic grading. It not only saves time but also elevates the quality of feedback, empowering educators and students alike.

Disclaimer:

The product is being tested, and a beta version is available. The production ready version has not yet been released. We welcome feedback from all users. Please contact us at support@gradebot.ai