Finding Service Mesh Bugs with Service Flow-Aware End-to-End Testing

We present MeshTest, the first end-to-end testing framework for traffic management of service mesh. The key idea of MeshTest is to automatically generate input configurations with end-to-end service flows, and then create real test request suites for each input based on detailed service flow modeling. There are two technical challenges. First, the input space of service mesh configurations is large and complex. The input configurations should be carefully orchestrated to form end-to-end service flow paths. Second, the abstract output network behavior cannot be directly checked for correctness, and we need to generate a set of real requests that are capable of checking possible behaviors. To address these challenges, we model the service flows of traffic management in service mesh, and propose a novel Service Flow Exploration technique to enumerate all possible configuration resources and interactions between them in the input configuration. We design and implement MeshTest, which contains an automatic input configuration generator based on Service Flow Exploration and a Service Mesh Oracle which leverages formal methods to generate test request suites. MeshTest has found 25 new bugs (20 confirmed and 10 fixed) in three popular service mesh systems, Istio, Linkerd and Cilium.