Summary

In project-based game development courses, structured feedback can be difficult with diverse projects involved. Meanwhile, reflection writing and play-testing peer reviews can be important sources of learning feedback for students. We study the use of LLMs to generate feedback and enhance the quality of student reflections and game reviews so as to improve learning outcomes in game development courses.

We have 3 sections:

1. Enhancing Reflective Learning
2. Enhancing Automated Feedback
3. Enhancing Play-Testing

In addition, we note the use of our web application tool iReflect in our game development courses in facilitating these reflective and play-testing processes.

iReflect: A Web Application tool designed for Game Development courses

iReflect is a play-testing and reflective learning tool designed for creative media and game development courses, developed by an alumni student. It is a web application that helps educators facilitate critical peer review, discussions over peer reviews and individual reflections over multiple milestones. In the game development courses that we performed the study in, iReflect is used by our lecturer to facilitate the above activities for the course. Our automated feedback system was integrated into iReflect for use by the students.

Enhancing Reflective Learning

Reflective learning is a key process for learners to convert their experiences into knowledge and insights for the future. In project-based game development courses where the projects are open-ended, it may be difficult to provide structured feedback that accommodates to the wide variety of game projects. Hence, reflective learning is another important way for game development students to internalise their experiences working on the project independently.

In this project, we look into integrating rubrics for critical reflections and prompt engineering to develop an automated feedback system for student reflections using OpenAI's GPT-4 model. We then study the effectiveness of this feedback system by putting it to use in one of our university's game development courses: CS4350 Game Development Project.

To study the effectiveness, we compare the initial version of the students' reflection when using the feedback system and the final version of the reflection for submission. After both versions were graded by our TAs using the reflective learning rubrics mentioned earlier, we find that the score has indeed improved for students that used the feedback system.

In addition, we also conducted a survey among the students regarding their opinions on integrating reflections into the game development course and found majority of the students agree that the reflections have improved their learning experience.

Enhancing Automated Feedback

Generating feedback using LLMs are known to have limitations, such as consistency and accuracy of feedback. We look into improving the consistency and accuracy of feedback through repeated querying and in-context learning. Additionally, we explore ways to generate constructive feedback, through more detailed prompting, to assist students in further improving their reflections.

By experimenting repeated querying on a set of graded reflections, we noticed that repeating the query 3 times and taking the average score for each reflection, helps to better align the LLM-generated scores with scores assigned by TAs.

In-context learning is incorporated by providing samples of graded reflections to the LLM, and results show that this further improves the accuracy of the LLM-generated scores.

To enhance feedback quality, we defined specific criteria for 'high-quality' feedback in the prompt, which led to noticeable improvements in the generated feedback, including an increase in constructive suggestions and thought-provoking questions.

Enhancing Play-Testing

Play-testing and conducting peer reviews are an important process in project-based game development courses as a source of feedback for the developers. We are currently looking into developing an automated feedback system for writing better play-testing peer reviews based on LLMs. We attempt to build a knowledge graph of topics related to game reviews (such as mechanics, genres, etc.), which shall serve as the knowledge base that can be used through retrieval-augmented generation (RAG) to help evaluate play-testing reviews.