Experiences with using an LLM-based Chatbot for a Multicultural Engineering Program Orientation (Experience)

Given the pace with which AI systems are being developed and used, there is a growing need for more guidance around the ethical use of AI. Due to the prominence of artificially intelligent systems, future engineers need to be able to analyze the available AI models and make responsible choices critically. In the Fall of 2024, The Human in Computing and Cognition (THiCC) Lab collaborated with the Multicultural Engineering Program Orientation (MEPO) at Penn State to teach incoming engineering students about the responsible use of AI systems with the help of an interactive Large Language Model (LLM) based chatbot. The MEPO is a four-day program designed to welcome incoming first-year undergraduate engineering students, primarily from racially and ethnically minoritized groups, by fostering connections with upper-division student mentors, academic success resources, and professionals in the field while also exposing them to typical elements of the engineering curriculum such as teamwork and innovation. One exciting component of MEPO is an engineering design competition, where students are asked to design prototypes to solve their assigned problem, for which this year’s theme was “decades.” Students were assigned to one of 14 groups; the groups were then assigned to one of 4 decades (1910s, 1930s, 1970s, 1980s). Each decade had an accompanying disaster that the students would be responsible for helping to resolve–the students assigned to the 1910s, for example, were tasked with designing a context-appropriate technological solution to help mitigate the Spanish Flu. The final objective was to create a prototype and a presentation regarding their findings and solutions to their assigned problem. The chatbot was meant to aid specifically the students as they brainstormed different ideas and solutions, allowing them to think critically about these intelligent systems as they used the chatbot. Before the four-day MEPO event, our team at the THiCC lab spent some time building the chatbot for the students to use. For the chatbot, we chose LLaMA-2 because of its reliable text generation and open-source nature, which includes transparency about the data sources used to train the system. We focused on transparency, as we wanted to highlight the importance of data sources and the significance of community-engaged open-source development. Additionally, we integrated Retrieval-Augmented Generation (RAG) to allow the chatbot to pull specific information, like historical data and disaster scenarios, from a custom pamphlet prepared by the MEPO team. This ensured the chatbot gave factually correct answers tied directly to the decades they worked on, which was later hosted on Huggingface spaces. On the first day of the MEPO, the THiCC lab team directed a lesson to introduce the students to the chatbot and its utility. The first half of the lesson was spent educating the students on the dangers and potential considerations of using LLMs and AI. The second half was spent showing the students certain variations in the usage of chatbots and the differences in the answers they provide. The variations included using a chatbot with pre-trained data (vanilla version), using the RAG version to retrieve factually correct answers from the pamphlet, and adding context to the RAG version to retrieve more nuanced answers. After the lesson, the students could use the various versions of the chatbot to help them in their design challenge and understand the difference in responses while using it for a given problem statement. During the four days, students presented a range of questions and feedback, from technical questions on how to access the chatbot to questions about motives and why they needed to use the chatbot. On the final day of the competition, students presented their designs and were able to thoughtfully consider the chatbot as an imperfect yet valuable tool in their competition. This report utilizes a structured survey and mixed-methods analysis to evaluate the educational impact of the chatbot and related activities on students’ comprehension of AI ethics and their overall learning experience.