Abstract: The paper presents a general paradigm of semiautomatic building extraction from aerial stereo image pair.In the semiautomatic extraction system,the building model is defined by selected roof type input through human-machine interface and input points of rough position of the building by operator.With the geometrical constraints of the model,low-level and mid-level processing including edge detection,straight line segments extraction and line segments grouping are to establish the initial geometrical model of the roof-top.However,the initial geometrical model is not so accurate in geometry.To attain accurate results,straight-line extraction and positioning by least squares matching with object-space based geometrical constraints rectifies the initial geometrical model.The adjustment model integrates the straight edge pattern and 3D constraints together and is a well-studied optimal method.The edge model is based on strict zero-crossing definition,and the strict geometrical constraints of the corners of the extracted building as 'knowledge' that is clearly described by the adjustment model are fused with image feature,such as edge and straight line.The model uses all of the information and constraints of the building borders to attain an optimal result,so this is a model driven strategy that matches the roof model with the image pair depending on the criteria of least squares.In principle,the method does not need feature detection,so there are no threshold problemes.After gaining proper initial values,the adjustment model can flexibly process extraction of kinds of roof types by changing or assembling the geometrical constraints in object-space. On the other hand,the paper also discusses the scheme of practical system of semiautomatic extraction.To build a practical semiautomatic extraction system,correctness,accuracy,robustness,speed and interactivity,all these factors should be taken into account.In the paper,a robust algorithm of extracting main direction of the house is addressed.By detecting edge,tracing the edge chain vectors and extracting straight-line segments,a Weighted Histogram of Angles (WHoA) is generated by the angles of extracted line segments.With the histogram,the main direction of the building can be easily detected by searching the max value of the histogram.Histogram method is a statistical manner like Hough transformation; it can effectively remove the influences of gaps between collinear segments and racket segments.The experiments show that that main direction and outer border detection by the WHoA method is robust.The line segments extraction and confirming the main direction of the house could save the operator's input action and offer good initial values of the corners for the consequent adjustment evaluation.