Subspace Regularization for Trace Estimation of Non-Normal Operators: Resolvent Bounds and Minimax Complexity

This work develops a unified framework for the estimation of traces of functions of large, potentially non-normal linear operators under limited matrix-vector (MatVec) access. We study the fundamental limitations of randomized trace estimators in the presence of non-normality and establish minimax lower bounds that explicitly scale with resolvent growth measures such as the Kreiss constant. To overcome these intrinsic instability barriers, we introduce a subspace regularization framework based on pseudo-invariant projections. This allows us to isolate an effectively stable component of the operator while controlling spectral leakage in a quantifiable way. We further construct a computable surrogate for resolvent growth, enabling practical certification of stability from randomized Gaussian probing. On the computational side, we combine randomized sketching with Krylov subspace methods and numerical range (field-of-values) analysis to obtain sharp convergence guarantees for GMRES-based evaluations of projected operator functions. The resulting pipeline yields an end-to-end estimator whose variance and computational complexity can be explicitly characterized in terms of effective numerical rank and surrogate stability measures. Our main result is a compositional closure theorem that links geometry, randomness, and Krylov computation into a unified bound, showing that near-minimax optimal performance is achievable if and only if the sketch captures the pseudo-invariant dominant structure of the operator. This establishes a principled bridge between operator theory, randomized numerical linear algebra, and large-scale computational trace estimation.