Doodle Your 3D with Materials: Creating 3D Model with Relightable Appearance from Single Freehand Sketches and Text Guidance

The rapid growth of augmented and virtual reality (AR/VR) has driven demand for efficient 3D content creation. Traditional CAD modeling remains time-consuming and difficult for beginners, while text-based 3D generation lacks geometric precision. Sketches, with their intuitiveness and expressiveness, offer a promising alternative, yet existing sketch-based methods often produce texture-less meshes and suffer from sparsity and ambiguity. To address these issues, we propose Deep3DSketch-PA, a sketch-based 3D modeling framework enabling fast and high-quality creation from a single free-hand sketch and optional text description. Our method employs a two stage of mesh generation: a custom network first extracts point-level and local geometric features, followed by a test-time optimization module that refines mesh structure. We further introduce a material graph representation to create consistent, relightable, and photorealistic appearances, overcoming the “dirty texture” limitations of prior methods. Experiments on synthetic and real-world datasets demonstrate that Deep3DSketch-PA achieves state-of-the-art results, outperforming existing approaches. By combining intuitive interaction with production-quality output, Deep3DSketch-PA offers an accessible and powerful tool for 3D content creation in AR/VR and related applications.