Abstract Purpose: Synthetic video generation offers substantial promise for AI development in robot-assisted minimally invasive surgery (RMIS), addressing the limited availability of annotated datasets. We present a novel approach for realistic and controllable RMIS video generation, leveraging robot kinematics to constrain the spatial position of the surgical tools in the generated frames. Methods: Our architecture uses a Blob Generation module that transforms 3D kinematic data into rigid 2D blob representations for spatial conditioning of a Frame Generation module using an encoder–decoder with temporal retention. Results: Quantitative and qualitative evaluations on the suturing task of the JIGSAWS benchmark dataset demonstrate that our approach enhances both video fidelity and positional accuracy of the surgical tools over long sequences. Blob spatial accuracy against pseudo-ground-truth labels yields average bounding box and mask Dice scores of 0. 78 and 0. 71, respectively, while video synthesis improves on average by 2% and 5% for SSIM and 4% and 9% for PSNR, while LPIPS reduces by 9% and 20%, compared to baseline state-of-the-art auto-regressive approaches for frames generated at t+10 t + 10 and t+20 t + 20. Conclusion: These results validate the potential of blob representation for position-aware, realistic video generation in RMIS. Synthesized data can augment available datasets, benefiting downstream tasks such as instrument tracking, workflow/skill analysis and robotic automation.
Mennillo et al. (Mon,) studied this question.