The intersection of AI and education is rapidly growing, particularly through the integration of AI into Learning Management Systems (LMS). Proprietary models, such as OpenAI, are often the first choice for many companies and studies due to their accessibility and performance. However, due to many restrictions and concerns that come with third-party solutions, companies such as Skillhabit are considering alternatives. This thesis investigates whether or not an open-source Large Language Model can be used as an alternative to proprietary models. The aim was to develop a scalable and cost-efficient pipeline that could integrate into a Learning Management System for the company Skillhabit, that currently use OpenAI as their model. An opensource model was selected that fit the requirements, which was then fine-tuned on synthetic data generated using GPT-3.5 and connected using FastAPI. The prototype is available on the Hugging Face hub for easy access to the model of choice and the later fine-tuned version of it. It was developed entirely using Google Colab Pro +, allowing access to multiple stronger graphics processing units than the hardware available. The thesis also discusses the ethical and social considerations related to the topic, such as environmental sustainability, data privacy concerns and lastly fairness and bias in model behavior. There were limitations that include non-unified benchmarking, manual evaluation of datasets and due to time constraints, a full integration with the company’s Learning Management System was not completed. The model was evaluated with standardized automatic benchmarks. The benchmark results on the finetuned model indicate that the performance decreased on most tasks. The largest drop was on BigBenchHard with –3.67 percentage points compared to the original non-fine-tuned model on the same benchmark, and smaller decreases on HellaSwag (–1.75 percentage points), CommonsenseQA (–1.64 percentage points), Winogrande (–1.42 percentage points), and MMLU (– 0.32 percentage points). In contrast, performance improved on TriviaQAWikipedia (+1.55 percentage points) and IFEval, where three out of four metrics showed improvement (up to +1.3 percentage points). The decline in performance is suspected to be caused by overfitting, hallucinations, and more. The main takeaway of this thesis is that an open-source large language model can be used as an alternative to proprietary models, but a human evaluation such as A/B-testing is required for a more accurate and correct evaluation of the model.
Patranika et al. (Wed,) studied this question.