A METHOD TO GENERATE FLOOD MAPS IN 3D USING DEM and DEEP LEARNING

High-resolution remote sensing imagery has been increasingly used for flood applications. Different methods have been proposed for flood extent mapping from creating water index to image classification from high-resolution data. Among these methods, deep learning methods have shown promising results for flood extent extraction; however, these two-dimensional (2D) image classification methods cannot directly provide water level measurements. This paper presents an integrated approach to extract the flood extent in threedimensional (3D) from UAV data by integrating 2D deep learning-based flood map and 3D cloud point extracted from a Structure from Motion (SFM) method. We fine-tuned a pretrained Visual Geometry Group 16 (VGG-16) based fully convolutional model to create a 2D inundation map. The 2D classified map was overlaid on the SfM-based 3D point cloud to create a 3D flood map. The floodwater depth was estimated by subtracting a pre-flood Digital Elevation Model (DEM) from the SfM-based DEM. The results show that the proposed method is efficient in creating a 3D flood extent map to support emergency response and recovery activates during a flood event.


INTRODUCTION
Accurate information about the flood inundation extent and water depth is essential for relief activities. Flood risk maps can protect human lives and property damages by providing timely damage assessments to plan relief work efficiently. The use of remote sensing data to assess flood hazards in near real-time has been popular over the last few decades (Huang et al., 2018). Unmanned Aerial Vehicles (UAVs) have been rapidly developed as tools for inundation mapping tasks. UAVs survey offers a viable alternative to conventional systems because they deliver highresolution data at a lower cost from complex urban landscapes, inaccessible areas, and hazardous environments (Feng et al., 2015, Boccardo et al., 2015. Several methods have been proposed to create a 3D inundation map from the remote sensing imagery. Since the pioneering work of Miller et al. (1958), DEMs have been widely analyzed for flood extent mapping. DEMs can be produced from field survey data, satellite and aerial images, and Light Detection and Ranging (LiDAR) data. Generating DEM utilizing conventional field surveys and GPS needs more skilled manpower and is timeconsuming and expensive to execute over a large area. LiDAR has been widely used to create DEMs since it can map large spatial areas by less manpower. LIDAR is a remote sensing technology that uses light in the form of a pulsed laser to measure ranges or variable distances to the Earth. LiDAR has a minimum human dependency, weather/ light independence, is a fast acquisition and processing approach for DEM creation. However, the LiDAR instrument cannot record elevation data from flooded areas since the lidar lights absorbed by water. SfMs solve multicamera viewing problem that generates high-density point cloud from high-resolution overlapping images (Caroti et al., 2015). SfM is the process of estimating the 3D structure of objects from the series of 2D images. The success of SfMs governed by the quality of the image resolution, degree of image overlaps, the relative motion of the camera with respect to the scene, and the presence of sharp edges (Watts et al, 2012, Remondino et al, 2014. While SfM photogrammetry is increasingly used for mapping in various applications, only a few researchers have used SfM for flood mapping, and its potential has not yet been fully explored by the remote sensing community. Meesuk, et al. (2012) investigated the efficiency of SfM approaches and achieved promising results in terms of accuracy for mapping flood extents. In their another work Meesuk, et al. (2015), they used the SfM technique to detect hidden urban features from a sequence of ground-view images that significantly improved the accuracy of flood simulation results. Hashemi-Beni et al. (2018) provided an overview of the opportunities and challenges of 3D flood extent mapping using UAV and SfM. The main challenges include (1) poor environmental and weather conditions (such as wind) during a flooding event ; (2) availability and visibility of ground control points (GCPs); (3) insufficient tie-points for image calibration due to the homogenous appearance of the water surface.; (4) needs for reclassification of noisy points cloud generated from SfM to determine the extent of the flood. Based on this context, we proposed an integrated method (SfM and Deep learning) to classify a 3D flood point cloud and generate a 3D flood map.
Deep neural networks such as Convolutional Neural Networks (CNNs) have been applied to many vision-related tasks for automatically extracting targets and demonstrated outstanding performance. CNNs applied to image classification (Ciregan et al., 2012), object detection (Girshick et al., 2014), scene labeling (Farabet et al., 2012), and object recognition (Taigman et al., 2014). The most notable advantage of CNNs is that they can construct high-level features from a raw dataset in an incremental manner. CNNs are widely used for remote sensing tasks because of their ability to successfully handle large training data sets, often yielding higher classification accuracy. Many research studies used CNN for automatically extracting a 2D flood extent (Gebrehiwot et al., 2019, Sarker et al., 2019. Gebrehiwot et al. (2019) modified VGG-based FCNs to extract the flooded extent from UAV images. The CNN-based model was trained using 100 manually labeled images and achieved more than 95% accuracy on extracting inundated areas. Sarker et al. (2019) proposed an FCN model for mapping flood extent in 2D from Landsat satellite images. A new satellite dataset from flooded areas across Australia was used to evaluate the performance of the model. A comparison of their proposed model to the conventional support vector machines (SVM) classifier shows that the FCN is more accurate (76.7%) compared to SVM (45.27%) for the classification task. These deep learning-based studies are limited to extract 2D flooded areas and cannot provide water level measurements from 2D imagery. To the best of our knowledge, the potential of CNNs for 3D flood extent mapping has not been explored and verified.
This research attempts to fill this gap in the literature by exploring and verifying the potential of an integrated CNN and SfM approach for 3D inundation mapping. In this study, the 2D flooded areas will be automatically extracted using a deep learning method. The extracted flooded areas will be overlaid onto the 3D map generated by the SfM approach to determine the flood extent. The floodwater level will be estimated by subtracting the SfM-based DEM (water surface) with a pre-flood Lidar-based DEM.
The paper is organized as follows. In Section 2, the study areas and the data used for the research are described. Section 3 explains the approaches used to create a 3D inundation map. The results and discussion are presented in Section 4. Finally, we conclude by summarizing our results in Section 5.

STUDY AREAS AND DATA
Two flood-prone areas, Lumberton and Princeville, in North Carolina, were chosen for this study. The city of Lumberton is located in Robeson County on the Lumber River, and the Town of Princeville is located along the Tar River in Edgecombe County. These locations have been highly affected by several flood events, including Hurricane Matthew and Hurricane Florence.
The data used for the research include the following: • UAV high-resolution imagery collected during hurricane Matthew (2016) over the study areas. These data were acquired by the North Carolina Emergency Management (NCEM). The size of each image is 4,000 x 4,000, with a resolution of 2.6 cm.

•
The pre-flood LiDAR data acquired over the town of Princeville. This Quality Level 2 (QL2) LiDAR with two pulses per square meter (pls/m2) with an accuracy of 9.25cm RMSE was collected by the North Carolina Emergency Management in 2014. This data was used to estimate the water level measurements.

Creating 3D flood map using SfM
SfM is a photogrammetric method commonly used for creating 3D models of a feature from overlapping 2D photographs taken from many locations and orientations. SfM works based on stereoscopic photogrammetry principles. That means triangulation is needed to calculate the 3D positions (x,y,z,) of objects from stereo pairs. In this study, we generated a 3D map from 80% overlapping UAV images. First, the pixels in the input images automatically identified matching features to create a point cloud model. These features can be corners or line segments and tracked from image to image to estimates the camera locations and orientations and the coordinates of the features. Based on the estimated camera positions, the locations of those points calculated and visualized as a 3D point cloud, in which each matchable pixels got its XYZ location in 3D space. Second, each set of three adjacent points connected into a triangular face to create a polygonal mesh model. Finally, the original images combined into a texture map and wrapped around the mesh to get a photo-realistic model.

Flooded Area Extraction Using FCN-8s
The second stage of the method involves extracting a 2D flood extent to classify the point cloud extracted from the SFM to water and non-water. For this task, a fully convolutional neural network (FCN) with a stride of 8 (FCN-8s) model was fine-tuned. The FCN-8s is composed of locally connected layers, such as convolution, pooling, and upsampling, without having any dense layer. This allows reducing the number of parameters and computation time. Given that all connections are local, FCN-8s can work on any image size. The FCN was proposed by Long et al. (2015) to train an end-to-end for semantic segmentation. In these models, VGG16 fully connected based classification layers were replaced by convolutional layers to maintain the 2-D structure of images. VGG-16 is a CNN architecture proposed by Simonyan et al. (2014) to investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. The network is sketched in Figure 1. To extract the 2D flooded area, the method consisted of annotating, training, classification, and evaluation stages. In the annotating stage, 150 UAV images were labeled manually using a MATLAB labeler tool. The network is trained using Stochastic Gradient Descent (SGD) for 6 epochs with a learning rate of 0.001, and a maximum batch size of 2. During training, 512-by-512 pixels size of 32 patches were randomly cropped and rotated in each batch size to increase the diversity of the training samples. We used a 10-fold cross-validation technique to evaluate the potential of the FCN-8s model. The importance of using 10-fold cross-validation is to give a less biased estimate of the FCN-8s model on unseen data. The training stage ended after 230,000 iterations for the experiments. In the classification stage, the performance of the network was tested using the unseen testing images. In the accuracy assessment stage, a confusion matrix was calculated to analyze the performance of the FCN-8s.

SfM and FCN-8s Results
The FCN-8s approach for 2D flood mapping was implemented in the MATLAB 2019b, and Agisoft Methashape and ARCGIS software were used for creating cloud point using SFM and implemented an integrated approach. The computer was configured with 32 GB memory, an Intel(R) Xeon(R) ES-2620 v3 @ 2.40GHz ×2 processors memory, and a single NVIDIA Quadro M4000 GPU.

Figure 2. Point cloud model created by SfM
Our test area in Princeville (yellow box in Figure 2) did not contain any GCPs for georeferencing purposes. Thus, the images of larger areas were processed (3000 UAV images) for using the available GCPs in that area in the triangulation and image-toimage georeferencing. All analyses were done only for the study area.  Figure 3d shows the 2D flood extent extracted by FCN-8s. The FCN-8s based classification achieved an accuracy of over 97% on distinguishing flooded areas from non-flooded. Once the FCN-8s model fine-tuned and trained with the dataset, the average processing time to extracted flooded areas was less than 2 seconds for a 4000x4000 pixels UAV image.

UAV and Lidar based DEMs results
The DEMs of the study area were generated using UAV data (figure 4a) and the preflooded Lidar data (figure 4b) to estimate the flood water level. As it is shown in Figure 5a, it is difficult to identify the flooded areas from the SfM based 3D map directly. Figure 5c shows the result of the integrated approach (deep learning and SfM) obtained by overlaying the deep learning-based flood areas on the SFM-based DEM. The result shows precisely the flooded areas. The depth of the floodwater estimated by subtracting the UAV and LiDAR based DEMs. The DEM created using LiDAR considered as a benchmark to estimate the floodwater depth since the LiDAR data was acquired before the flood event. The minimum and maximum floodwater depth values calculated in the flooded areas are 0.32 ft and 1.24 ft, respectively. The estimated floodwater depth values give additional information that can be used for rescue and damage assessment tasks.

CONCLUSION
Flooding is a severe natural disaster that causes economic damage and loss of life and property. Accurate 3D mapping during flooding events is vital for effective planning and rescue activities. The research proposed a method to create a 3D flood map by integrating SfM and deep learning. The results indicated that the deep learning-based method, such as FCN-8s, could accurately extract flooded areas with a classification accuracy of 97%. By overlaying the deep learning-based 2D flood map on the SfM-based DEM provides water level information. The 3D flood extent map can be used to support emergency response, and recovery activates during a flood event.