This paper presents a deterministic system that combines textual semantic data from panoramic images with LiDAR point clouds in a mobile mapping setup. Urban scenes often include textual elements, such as signs and business names, that provide key details typically missing from LiDAR-based urban digital twins. The presented method uses deep learning-based OCR to extract text from street panoramas and then categorizes it into urban types using a rule-based classifier. Text regions are geometrically projected into the LiDAR environment by converting image coordinates into viewing rays that intersect LiDAR surfaces, such as facades. Data from multiple panoramas are merged with confidence-weighted spatial clustering to produce consistent semantic markers for urban features. Extracted business names enable text-based searches of the LiDAR point cloud, allowing facility location by category, keyword, or brand. Tests on datasets from European and U.S. cities support plausible facade-level localization and demonstrate the framework’s ability to enhance LiDAR point clouds with searchable semantic information. The main contribution is not a new standalone OCR or LiDAR-processing algorithm, but a deterministic multimodal integration framework that combines deep-learning OCR, geometric back-projection, and cross-view spatial fusion to convert street-level textual cues into reliable, queryable 3D semantic markers within mobile-mapping LiDAR data.
Jasim et al. (Tue,) studied this question.