HIGH PERFORMANCE PHOTOGRAMMETRIC PROCESSING ON COMPUTER CLUSTERS

Most cpu consuming tasks in photogrammetric processing can be done in parallel. The algorithms take independent bits as input and produce independent bits as output. The independence of bits comes from the nature of such algorithms since images, stereopairs or small image blocks parts can be processed independently. Many photogrammetric algorithms are fully automatic and do not require human interference. Photogrammetric workstations can perform tie points measurements, DTM calculations, orthophoto construction, mosaicing and many other service operations in parallel using distributed calculations. Distributed calculations save time reducing several days calculations to several hours calculations. Modern trends in computer technology show the increase of cpu cores in workstations, speed increase in local networks, and as a result dropping the price of the supercomputers or computer clusters that can contain hundreds or even thousands of computing nodes. Common distributed processing in DPW is usually targeted for interactive work with a limited number of cpu cores and is not optimized for centralized administration. The bottleneck of common distributed computing in photogrammetry can be in the limited lan throughput and storage performance, since the processing of huge amounts of large raster images is needed.


INTRODUCTION
Many operations in processing remote sensing data can be split into numerous tasks that can be easily done in parallel.The original images are similar in nature and can be processed independently to build ortho mosaics.This is due to the fact that cutlines calculation, orthorectification, image format conversion, pansharpening, many service opertations performed on one image or on few images and are independent on other images.Parallel processing of data can reduce the computer time needed to get the final result considerably.The current trends in computer hardware are the increasing number of cores (identical computer units) on the off-the-shelf CPUs, larger and faster computer networks and more and more affordable supercomputers as clusters of thousands of computer nodes.Conventional digital photogrammetric workstation (DPWs) use distributed processing that is based on interactive work and uses the ideologymany workstations and one server.
Figure 2. Conventional approach: many workstations and one server This approach is not efficient when modern computer clusters are used.

DISTRIBUTED
PROCESSING ON COMUTER CLUSTERS Special adjustments of software algorithms are needed for computer clusters.Such algorithms predict data workflow and split the computing tasks into several independent processes.The approach is different from the conventional one: the main ideology is: one workstation and multiple servers Figure 3. Cluster environment: one workstation and multiple servers Many service operations, image orthorectification and orthomosaic creation (including calculation of linecuts) on a specialized computer cluster can be done very effectively.We have run first tests in 2010 on KazNTU (Kazakh National Technical University named after K.I.Satpaev) computing cluster, with peak productivity of 10.9 teraflops, NetApp storage system running Windows HPS Server OS.Orthomosaic creation (scale 1:2000) using precomputed DTM (DEM, refined by structured lines) for the area of 1289 sq.km.took 3 hours.40 computing nodes were used with the average cpu load of 85%.The aerial block had 3558 original images, taken by UltraCam XP, with average GSD of 15 cm.The total amount of output orthomosaic was 156Gb.

VHR AND HR SATELLITE IMAGES CLUSTER PROCESSING
Current VHR and HR satellites (QuickBird, WorldView-1 WorldView-2, IKONOS, GeoEye-1, Kompsat-2, EROS-B, Cartosat-2 ) productivity is more than 2,500,000 sq.km per day.The overall productivity of VHR sensors will increase twice in a couple of years with launching of the new sattelites.It is now impossible to process this amount of data in real time on a single DPW.The accuracy of current sensors is sufficient to build ortho mosaics with the corresponding accuracy of scale 1:5000 using only RPC coefficients and without ground control points.
The algorithms of processing such images to produce ortho mosaics are fairly simple assuming one has DEM for the needed area.The images are preprocessed one by one, if needed, orthorectified one by one using RPC and a given DEM.Cutlines and radiometric balancing are computed using neighbouring images.At the last step the final ortho mosaic is created and saved in the predefined standard sheets.All steps can be run in parallel on the computer cluster.
Special software plus software computer cluster complex were developed in Racurs company (Russia).It creates photogrammetric project from an arbitary set of satellite images, performs bundle adjustment using RPC, computes orthoimages using existing DEM, excludes cloudy images or cloud areas, builds cutlines, computes tie point for better stitching of orthoimages and performs final seamless orthomosaic computation with color balancing.The efficiency of this fully automatic complex is 1.000.000sq.km. per day for GeoEye, DigitalGlobe or Alos images.Our tests show good efficiency and good scalability of the developed hardware-software solution.
Figure 5. Computer cluster hardware

ALGORITHMS USED
Most of processing steps described in the previous section are straightforward and described in many papers.We will give more details on some algorithms we used to increase the quality of the ortho mosaics and to automate the processing.The automatic cut lines are build as follows.We use the Voronoy diagram to build of preliminary lines: the whole area of interest is divided into the set of nonoverlapping image patches around images centers.After that the diagram's edges are replaced by polylines providing the best junction of the images.It's achieved with a help of penalty function that includes evaluation of conformity of images, their heterogeneity, intensity and gradient lines.For each edge the optimal polyline is calculated using dynamic programming algorithms.If M(L) is the maximum of the penalty function for the polyline L, the algorithm is to find a polyline L 0 such that M(L 0 )= min M(L) among all the polylines L from the starting point to the end vertex of the edge.The method can be described as finding the best route by "water flood".The following screenshot demonstrates the algorithm.The distributed computations are based on parallel considering of polylines.One of the unique algorithms we use to make ortho mosaics look better is the usage of tie points that are calculated along cut lines using standard cross-correlation algorithm.The images can be stitched by a limited predefined value using these tie points to eliminate discrepancies along seams (cut lines) as illustrated on the following picture.

CONCLUSION
Orthophoto production from VHR and HR sensors using RPC and precomputed DEM can be effectively run on specialized computer clusters with good scalability in fully automatic mode.A special adaptation of algorithms and software is needed for cluster environment.These algorithms are implemented in PHOTOMOD HPC Edition DPW.The developed algorithms are now extended for distributed GPU processing in cluster environments, for DTM computing and filtering, for special control function allowing to share the cluster between several workgroups.The software module for integration with GIS systems, databases and geoportals is also developed.

Figure 1 .
Figure 1.Building orthomosaic from a set of images

Figure 4 .
Figure 4. Aerial block and cpu load on a computer cluster

Figure 6 .
Figure 6.Automatic cut lines creationIf the images contain clouds a special module is used for their automatic detection.This algorithm includes a successive filtering of pixelsby intensity, deviation from the grey color, dispersion and heterogeneity.The last function measures the difference between changing of intensity along each axis and its variation.At the last step the remaining pixels are composed in components and their boundaries are calculated as polygons.

Figure 7 .
Figure 7.A Sample block of 4 images and cloud detection

Figure 9 .
Figure 9. Tie points to eliminate residual discrepancies along cut lines