Neural ordinary differential equations provide a principled continuous depth formulation, yet their practical use is often limited by slow, numerically fragile, and computationally expensive training caused by the weaknesses of standard ODEs solvers. We introduce extended stability Runge Kutta methods, which are explicit fixed step solvers designed to remain stable at much larger step sizes than classical schemes such as RK4 or adaptive methods such as Dormand Prince. By enlarging the stability region of the real axis, these solvers cross the integration horizon with far fewer steps, providing deterministic computation, greatly reduced function evaluation cost, and smoother optimization dynamics. Across CIFAR 10, CIFAR 100, and Tiny ImageNet, extended stability solvers match the final accuracy of higher-order methods, usually within one to two percent, while producing significantly more stable gradients. On CIFAR 10, they reduce gradient clipping rates to between zero and twenty-five percent depending on integration horizon, compared with sixty to one hundred percent for standard solvers, and they maintain reduced spectral amplification in moderately stiff regimes, while preserving coherent gradient flow in more extreme stiffness settings. The 15-stage variant achieves a fourteen-fold speedup in wall time relative to Dormand Prince 5 and a two-fold speedup relative to ResNet 20, while requiring eighteen times and twelve times fewer function evaluations, with comparable floating point operations to RK4. These gains require no additional learned parameters, no regularization, no architectural changes, and no adaptive tolerance tuning. They also allow successful optimization in regimes where classical explicit solvers become unstable or diverge. Overall, the results show that the geometry of the stability region, rather than formal order or adaptivity, is the key factor that governs gradient flow conditioning under a fixed integration horizon. Higher-order methods can reach similar accuracy but suffer from unstable and oscillatory gradients. Extended stability methods maintain coherent gradient flow while providing substantial practical speedups, making stability region geometry an effective and simple tool for accelerating and stabilizing the training of neural ordinary differential equation models. • Introduce extended-stability Runge–Kutta (ESRK) solvers to Neural ODE. • Demonstrate ESRK acts as a numerical preconditioner for gradient flow. • ESRK-15 achieves stable training with h = 30 at fixed compute cost. • ESRK-21 further smooths spectral dynamics in high-gain regimes ( Δ T = 20 –30). • Stability-region size, rather than solver order alone, governs gradient-flow conditioning in Neural ODEs training. • Provides FLOPs and wall-time scaling showing ≤ 1.4 × overhead vs RK4. • Provided detailed Resnet-20 comparison benchmarks across three datasets, along with geometric properties analysis.
Goodship et al. (Sun,) studied this question.