3 D RECONSTRUCTION FROM UAV-BASED HYPERSPECTRAL IMAGES

Reconstructing the 3D profile from a set of UAV-based images can obtain hyperspectral information, as well as the 3D coordinate of any point on the profile. Our images are captured from the Cubert UHD185 (UHD) hyperspectral camera, which is a new type of high-speed onboard imaging spectrometer. And it can get both hyperspectral image and panchromatic image simultaneously. The panchromatic image have a higher spatial resolution than hyperspectral image, but each hyperspectral image provides considerable information on the spatial spectral distribution of the object. Thus there is an opportunity to derive a high quality 3D point cloud from panchromatic image and considerable spectral information from hyperspectral image. The purpose of this paper is to introduce our processing chain that derives a database which can provide hyperspectral information and 3D position of each point. First, We adopt a free and open-source software, Visual SFM which is based on structure from motion (SFM) algorithm, to recover 3D point cloud from panchromatic image. And then get spectral information of each point from hyperspectral image by a self-developed program written in MATLAB. The production can be used to support further research and applications.


INSTRUCTION
Hyperspectral images can be used for finding objects, identifying materials, or detecting spatial processes.Compared to RGB images, which only contain three bands of the visual part of the electromagnetic spectrum (red, green and blue light).The hyperspectral image contains continuous hundreds bands of the electromagnetic spectrum in the sensor's wavelength range.And it can provide considerable information on the spatial spectral distribution of the object.
Traditionally, Hyperspectral images are acquired with satellites (Clark, 2016), airborne sensors (Quemada et al., 2014)or field spectrometers (Numata et al., 2008).Recently, with the size and weight of hyperspectral sensors have been shrinking and have been carried onboard of UAVs.Such as a new type of hyperspectral cameras like the UHD has been used for UAV remote sensing.
In computer vision, 3D reconstruction is the process of recovering the shape and profile of real objects.This process can be accomplished either by active or passive methods.Active methods derive the depth map using methods of structure light, laser range finder and radiometric rangefinders, then reconstruct the 3D profile by numerical approximation approach and establish the object in the scene based on model.Although this method can get high quality data, it also requires expensive equipment.In contrast, the development of passive methods provides the opportunity for very low-cost three-dimensional data acquisition.Passive methods measure the radiance reflected or emitted by the object's surface to reconstruct the 3D profile from a set of digital images.Passive methods include structure from motion (SFM), photometric stereo, shape-from-shading etc.
The SFM techniques can be used for estimating 3D structure from a series of overlapping 2D images.Each hyperspectral image records a data cube with one spectral information and additionally two dimensions of spatial information, and thus there is an opportunity to derive 3D information from hyperspectral image.From a different perspective, hyperspectral images have shown great potential in computer vision than RGB images, especially for the application that requires fine analysis of the spectral responses of object.Spectral data can used to vegetation monitoring (Glenn et al., 2012) (Aasen et al., 2015)tect environmental stress or plant diseases (Behmann et al., 2016) (Lowe et al., 2017) (Thomas et al., 2018).
Reconstructing the 3D profile from a set of hyperspectral images, which can obtain hyperspectral information, as well as the 3D coordinate of any point on the profile.So far, Very limited study gets 3D information of objects from hyperspectral images by passive methods directly.Most of the researches focus on the integration of spectral data into 3D information, while the 3D profile are generated by active methods, such as Lidar.The combination of hyperspectral information and 3D data has shown advantages for several applications: improve the accuracy of vegetation mapping (Zhang, 2014), the estimation of deterioration of artwork (Graneromontagud et al., 2013) and material detection (Brook et al., 2010).Hyperspectral information and 3D information were co-registered from different sources.However, this is a challenging problem when integrating data from different sensors, at different times, or possibly mounted on different platforms.
Recently, some studies also tried to derive 3D model from hyperspectral images directly.Due to 3D models reconstructed from different spectral bands demonstrate different properties, there is also a challenging that the co-registration from different bands 3D models (Zia et al., 2015).
Given these considerations, this research intends to explore the feasibility of reconstructing 3D profile which contains both 3D position and hyperspectral information without co-registration.
First, we get a series of images by using UHD camera which can simultaneously capture both hyperspectral images and a panchromatic image.Then in order to get the 3D model, we adopt a freely 3D reconstruction software to process the panchromatic images.Finally, we output our database with 3D and hyperspectral information.
The scope of this contribution is to introduce an innovative way to create a database containing hyperspectral and 3D information from images captured by a novel hyperspectral camera based on a SFM algorithms without co-registration both of 3D and hyperspectral information and 3D model of different bands.The rest of paper is organised as follows.Section2 introduces the proposed the method of get 3D and hyperspectral information.Section 3 presents the experimental results and discussion.And section 4 concludes this paper.

Image acquisition from the Cubert UHD185
Images of the study area were collected at the location, which is located in Shiping Town, Gulin County, Ganzhou City, Sichuan Province (N28 • 00 55 -N28 • 03 26 , E105 • 59 32 -E106 • 02 13 ).The entire image acquisition system is consisting of the camera, single board computer and a separate battery.
The UHD is a hyperspectral full frame camera which contemporaneously captures 138 spectral bands with a sampling interval of 4 nm.According to the cameras manufacturer recommendation, we only use 125 bands from 450 and 950 nm.Moreover, UHD can simultaneously capture both hyperspectral image with a resolution of 50 by 50 pixels and a panchromatic image with a resolution of 1000 by 1000 pixels.Using the software of camera, the resolution of hyperspectral image can be sharpened to panchromatic image resolution.It is used in our study.
The camera is controlled by the single board computer.It is connected to the camera via two gigabit ethernet cables and runs a server application for camera and data storage that is responsible for communication with the camera.Configuration can be done remotely via WiFi from a different control application running the computer.Within the WiFi range, live views can be sent to control applications and measurements can be triggered manually.In addition, a series of images can be recorded with a determined frequency and duration.
In this study, we use a lens with a focal length of 23 mm.The ground resolution at 30 m flying height is about 16 cm for the hyperspectral pixels and about 0.8 cm for the panchromatic image.It is based on the following generative model: As shown in Figure 1, SFM operates under the same basic principle as stereoscopic photogrammetry, namely that 3D structure can be reconstructed from a series of overlapping images.SFM is an emerging low-cost photogrammetric method for 3D topographic reconstruction.There are a variety of SFM tools now available.Generally, SFM processing software can be divided into two categories, the one is free and open source software, such as Bundler (Snavely, 2006), which is commonly used.And the other is closed source software, such as Agisoft Photoscan, Pix4Dmapper and Visual SFM provide the benefit of automated workflows.Nevertheless, a distinction can be made between them, Agisoft Photoscan and Pix4Dmapper are commercial software, while Visual SFM is free.It takes the input images and produce 3D point clouds, which can be further processed with additional software, such as PMVS2 (Furukawa and Ponce, 2010).
Although there are a lot of SFM programs that range from free software to expensive professional software.In general, the SFM workflow typically identical.The Visual SFM modeling software is capable of generating the 3D digital surface model in three primary stages, (1) Feature detection and matching; (2) Recover camera orientation and position; and (3) Construct the 3D geometry.
1. Feature detection and matching Feature detection and matching are an essential component of many computer vision applications.The key problem that SFM addresses is the reconstruction of the 3D struction of matching features in multiple photographs which taken from different angles.The Scale Invariant Feature Transform (SIFT) object recognition system, which is published byDavid Lowein 1999.It is a popular solution to the identification of features in individual images which can be used for image correspondence.In this study, we take a set of panchromatic as input images for feature detection and matching.

Recover camera orientation and position
Once the correspondences are established by using algorithms to automatically detect invariant features which overlap in the images, we can create a system of geometrical projective matrices and determine the position and orientation of each camera position.
3. Construct the 3D geometry A 3D point cloud is created by triangulation which is used to estimate the 3D point positions and incrementally reconstruct scene structure, fixed into a relative coordinate system.
Visual SFM was chosen because of the free cost, particularly the output production, capability to determine a set of correspondences between images and point, and can be further used to additional applications.

EXPERIMENTS AND DISCUSSION
In this study, we take part of the area as our dataset, which contains toally 42 images.These images are captured from the UHD hyperspectral camera, we can export hyperspectral images and panchromatic image simultaneously with a resolution of 1000 × 1000.As shown in Figure 2.
Reconstructing the 3D point cloud from the panchromatic image by Visual SFM software, the dataset comprised of 2 × 10 4 points approximately.In order to verify the accuracy of the reconstructed point cloud, we compared point clouds at different angles.We can see the difference in height between the point clouds.This is in line with the actual features.For example, the height of the roof should be higher than that of the surrounding features.As shown in Figure 3.
In this study we describe a novel method to derive hyperspectral information and 3D information from images captured by UHD hyperspectral camera based on SFM.UHD hyperspectral cameras provide two key advantages for 3D reconstruction from hyperspectral images compared to traditional method: with current methods the time, the sensor and the platform of hyperspectral images and panchromatic image are identical, without coregistration.The second advantage is that the full frame camera captures 2D spatial information.Therefore, a single image band can be used to reconstruct 3D information with SFM algorithm.Compared with the reconstruction from hyperspectral images this significantly reduces complexity.
With the SFM approach 3D information can be derived from panchromatic images.Compared to RGB images, which have three bands of the visual part of the electromagnetic spectrum.
The panchromatic image only have a single band of the camera's wavelength range.And thus, there is a drawback in feature extraction and matching.Within the study area, the types of features can be roughly divided into four categories: vegetation, road, roof, and bare land.Spectral curves shown in Figure 4.
Another bias in our study is introduced by the difference of spec-  As shown above, We can conclude that due to the mismatching of feature points which provide the spectral spatial position and the differences in spectral information between different images, there is an error in spectral information extraction.Thus, future studies should aim to improve the accuracy of feature matching and find more accurate ways to fit point spectral information.

CONCLUSIONS
In this study we introduced a method to derive a novel database with a lightweight UAV hyperspectral camera.The resulting database combines the location of the 3D surface with hyperspectral information emitted and reflected by the objects covered by the surface.The 3D point cloud are derived from a set of panchromatic images, and then using a self-developed program written in MATLAB to get spectral information of each point from hyperspectral image.The database can be used to support further research and applications.
Figure 1.Structure from motion After reconstruction, the Visual SFM saves SFM workspaces into NVM files, which contain input image paths and multiple 3D models.Below is the format description: NVM files: Each reconstructed model contains the following N umber of cameras List of cameras N umber of 3D points List of points The cameras and 3D points are saved in the following format Camera = F ile name f ocal length quaternion W XY Z camera center radial distortion 0 P oint = XY Z RGB number of measurements List of M easurements M easurement = Image index F eature Index xy 2.3 Intergration hyperspectral information into 3D point cloud Once the correspondences are established via NVM files, there is an opportunity to derive the hyperspectral information of each point from the ouput.With the simple correspondences between the position of feature point in image and the position of point, to derive hyperspectral information.Extracting spectral information of each point from hyperspectral image by a self-developed program written in MATLAB.The median filter is used to reduce noise in spectral information of a point.Then intergration from hyperspectral information into 3D point cloud.

Figure
Figure 2. RGB image of the study area

Figure 4 .
Figure 4.The spectral curves of the features are obtained from a single image, there is a difference in distinct features Figure 5. (a) RGB display of the study area.(b) RGB display of the new point cloud.