What question did this study set out to answer?

This study aims to analyze the topic structure of clinical teaching activities using BERTopic amidst residency and medical internship training.

June 2, 20260 citationsOpen Access

BERTopic Mapping of Clinical Teaching Activities in Residency and Medical Internship Training

CCChujie Chen JLJun Li

Key Points

This study aims to analyze the topic structure of clinical teaching activities using BERTopic amidst residency and medical internship training.
Analyzed 4,811 de-identified records from a single-center teaching hospital between 2022-2025.
Implemented BERTopic for topic modeling and subjected it to multi-LLM checks for reproducibility.
Characterized topic robustness using coherence metrics, diversity measures, and clustering techniques.
Reduced models achieved coherence scores between 0.505–0.548 and topic diversity levels of 0.86–0.90.
Inter-LLM Cohen’s κ values were 0.709 and 0.878, indicating moderate to strong encoding reproducibility.
Seventeen of twenty-four residency topics and fifteen of twenty-four internship topics showed significant year trends.

Abstract

Introduction: Electronic teaching platforms record thousands of free-text descriptions of clinical teaching activities, but rarely at scale. We applied BERTopic to discover the topic structure of teaching narratives in two parallel training systems and subjected the taxonomy to a multi-LLM zero-shot encoding-reproducibility check using independent commercial large language models (LLMs). Methods: We analyzed 4, 811 de-identified records (2, 890 residency, 1, 921 medical internship) from the CCMTV platform at a single-center tertiary teaching hospital (2022–2025). After jieba tokenization and 46, 313 PHI placeholder substitutions, BERTopic with BAAI/bge-base-zh-v1. 5 embeddings was fit per system and reduced to 24 topics each. Coherence (cᵥ, NPMI), topic diversity, 5-seed bootstrap (ARI/NMI), and a minclusterₛize grid characterized robustness. Two LLMs (Codex GPT-5. 2; Gemini 3) annotated 200 stratified-sampled records under identical zero-shot prompts; inter-LLM Cohen’s κ quantified encoding reproducibility. Cross-system topic correspondence used cosine similarity of topic centroids with Hungarian matching. Results: Reduced models reached cᵥ 0. 505–0. 548, diversity 0. 86–0. 90, outlier rates 14–23%, ARI 0. 60–0. 67, NMI ≥ 0. 90. Inter-LLM Cohen’s κ = 0. 709 95% CI 0. 64–0. 78; 0. 878 0. 82–0. 92 excluding -1 residuals. Cross-system Hungarian matches yielded cosine 0. 783±0. 132; 17 of 24 pairs (71%) reached ≥ 0. 70, but a 1000-permutation merged-relabel null showed this count was not significantly above chance (p = 0. 76), supporting descriptive interpretation only. Per-topic binomial GLM (year predictor, BH-FDR over 24 topics per system) identified significant year trends in 17 of 24 residency and 15 of 24 internship topics. Conclusions: BERTopic with multi-LLM inter-LLM encoding-reproducibility checking offers a scalable, descriptive framework for monitoring the topic structure of clinical teaching and screening for potential topic-share changes, though population-level overlap claims should be read cautiously given the permutation-null result.

问 AI

Bookmark

View Full Paper