Informal settlements in many Brazilian urban centers are growing up quickly. Remote Sensing techniques provide a cost-effective mean to measure such an expansion. In cities with rough topography, like the city of Rio de Janeiro, the initial expansion, predominantly horizontal, is gradually being shifted to vertical, as new floors are being built on the existing buildings. To estimate the changing of such type of buildings, 3D measuring techniques are required. This research proposes a method for generating Digital Surface Model (DSM) from digital photogrammetry techniques. The method takes an initial DSM calculated from a pair of stereoscopic images as input. These models have no knowledge of image scenes semantics. The input DSM is refined taking into account the information about the geometry of buildings identified by a process of segmentation and interpretation applied to both images of the stereo pair. The method is based on a priori knowledge that buildings from informal settlements in large urban centers in Brazil generally have their roof tops at approximately constant height (flat roofs). The process involves three main steps: firstly the tops of buildings are identified in each pair of stereoscopic images. Then, the regions corresponding to the top of the buildings are matched. The third step involves the 3D extraction of those buildings. Finally, the method generates a more accurate DSM plus an image label with the semantics of the identified buildings. The results obtained in the experiments on airborne imagery of a test area in the city of Rio de Janeiro showed a significant improvement in the original DSM, as it takes into account the semantics of the 3D reconstructed buildings.