Abstract: Objectives: As an essential component of urban renewable energy, the assessment of rooftop photovoltaic (PV) potential depends on the accurate identification of structural characteristics. In particular, determining rooftop area and orientation is crucial for reliable PV capacity evaluation. Current mainstream approaches employ deep learning models to extract rooftop area and orientation from aerial imagery for PV resource assessment. However, variations in solar azimuth angles during image acquisition leads to differences in rooftop shadow distribution, resulting in significant apparent discrepancies among rooftops with the same orientation. Consequently, deep learning models may misclassify rooftop structural features. Methods: To address this core limitation, this study proposes a novel framework for robust solar energy potential estimation. Firstly, the advanced semantic segmentation architecture DeepLabV3+ is employed to simultaneously delineate rooftop boundaries and classify each rooftop region into primary orientation categories from input aerial imagery. Secondly, crucially recognizing the susceptibility of visualbased orientation classification to shadow-induced errors, the framework incorporates a refinement stage. This integrates Photovoltaic Geographical Information System (PVGIS) data – a validated source for location-specific historical and modeled solar radiation data. A multi-orientation quantitative integration strategy uses PVGIS to calculate theoretical annual solar irradiation values by weighting and synthesizing these PVGIS-calculated values, which can effectively mitigate errors caused by visual misclassification. Results: The results on the RID dataset show that DeepLabV3+ outperforms other models, which can improve the performance of the proposed framework. The quantitative integration strategy narrows the relative error in solar potential estimation. For 4, 8, and 16 orientation categories, the proposed framework reduces estimation errors to 0.26%, 0.25%, and 1.07%, respectively. By excluding disturbances of flat roofs, relative errors are further reduced to 0.14%, 0.10%, and 0.68%, respectively, which demonstrates the effectiveness of our framework. Conclusions: By combining high-resolution rooftop segmentation/orientation with geospatial irradiation modeling and a quantitative mechanism to resolve orientation ambiguity, the proposed framework effectively mitigates a primary source of error in aerial image-based solar potential assessment. This significantly enhances estimation accuracy, providing a more reliable tool for urban-scale renewable energy planning and investment decisions.