A Language Model for Pediatric Occupational Therapy Documentation: Model Development and Pilot Study

Abstract Background In occupational therapy, progress notes and other client-related administrative tasks are essential for providing treatment but are time-consuming. Therapists spend at least as much time on these tasks as providing care, which contributes to growing waitlists. Objective This study aimed to create a custom large language model to make the process of writing progress notes more efficient by converting point-form scratch notes from pediatric occupational therapy treatment sessions into draft documentation in subjective-objective-assessment-plan format. Methods Using a dataset of redacted historical progress notes, various training methods, including domain-adaptive pretraining and low-rank adaptation fine-tuning, were applied to train Llama 2 and 3 models. Since the historical notes lacked corresponding scratch notes, few-shot prompting with human-in-the-loop evaluations was used to generate synthetic scratch notes. This pairing of historical notes and generated scratch notes enabled effective fine-tuning of the Llama models on the desired task. The final model, a fine-tuned Llama 3 8B Instruct model, was piloted in a pediatric rehabilitation center and compared with Microsoft Copilot. Ten therapists used both models for 3 weeks each. Results The custom model notes scored higher than manually written notes on clarity, completeness, relevance, and organization ( P <.001) , and similarly on conciseness. They scored higher than those from Copilot on conciseness ( P <.001). However, in this small pilot, a significant reduction in time spent on documentation when using the custom model versus manual notes was not detected. Follow-up investigation revealed that time savings were observed only when therapists were coached to write sparse scratch notes; however, they tended to revert to detailed notes after coaching for which the model was not shown to improve efficiency. Conclusions The model had the capacity to save time when therapists provided brief input to the model. However, in practice, therapists preferred to provide detailed input. Used in this way, the model improved note quality rather than saving time.