A triple-branch multi-scale network for real-time semantic segmentation

At present, multi-scale learning is a popular and effective method to improve the accuracy of real-time semantic segmentation. However, these multi-scale methods do not consider the influence of CNN receptive field on the feature discriminability. They still suffer from the small receptive field and insufficient feature extraction, which limits the accuracy of real-time segmentation. To solve the problem, we propose a novel Triple-Branch Multi-Scale Network (TBMSNet) for real-time semantic segmentation. Specifically, in initial feature extraction stage, we propose the Simple Inverted Residual (SIR) module with reducing the number of input channels, using two successive SIR modules to initially extract features, which can enhance the feature extraction capability of lightweight backbone network. Subsequently, we design a new multi-scale triple-branch structure to parse detail, semantic, and boundary information respectively. In triple-branch structure stage, we propose the Dilation-wise Residual (DWR) module in Semantic branch, which combines the multi-scale detail and boundary branches down-sample semantic feature maps to 1/64. The design can extend the valid receptive field and improving the capture efficiency of multi-scale information. Besides, we also design a Multi-scale Semantic Aggregation Pyramid Pooling (MSAPP) module in semantic branch, which connects multi-scale pooling maps before the convolutional layer to form local and global context representations for further enriching the semantic information, and extract multi-scale features more efficiently. Experiments show that our TBMSNet achieves an accuracy of 80.5% mIoU on Cityscapes and an inference speed of 50.4 FPS, achieving the best trade-off between inference speed and accuracy.