CONSTRUCTION OF A 3 D MEASURABLE VIRTUAL REALITY ENVIRONMENT BASED ON GROUND PANORAMIC IMAGES AND ORBITAL IMAGERY FOR PLANETARY EXPLORATION APPLICATIONS

This paper presents a method of constructing a measurable virtual reality environment based on ground (lander/rover) panoramic images and orbital imagery. Ground panoramic images acquired by a lander/rover at different azimuth and elevation angles are automatically registered, seamlessly mosaicked and projected onto a cylindrical surface. A specific function is developed for inverse calculation from the panorama back to the original images so that the 3D information associated with the original stereo images can be retrieved or computed. The three-dimensional measurable panorama is integrated into a globe viewer based on NASA World Wind. The techniques developed in this research can be used in visualization of and measuring the orbital and ground images for planetary exploration missions, especially rover missions.


INTRODUCTION
Planetary exploration usually consists of orbital and lander/rover missions.Orbital images map the planet globally and provide geometrical and spectral information for landingsite selection, precision landing, and various scientific researches.Ground (lander/rover) images provide more detailed information of the landing site for daily mission operations such as close-up investigation of scientific targets.Visualization of these orbital and ground images and their derived products is essential to support mission operations and scientific researches, as well as education and outreach.
In earth observation and GIS fields, virtual reality (VR) and virtual geographic environments (VGE) technologies have been utilized for various applications (Lin and Gong, 2001).The concept of measurable virtual reality (MVR) was proposed and implemented based on seamless stereo-orthoimage database with digital stereo-orthophoto partner technique (Li et al., 2002).The concept and application of digital measurable images (DMI) in digital earth and geospatial information service have also been exploited (Li et al., 2008(Li et al., , 2010)).GIS technologies have also been used in planetary exploration researches and operations, e.g., the Planetary Interactive GIS on-the-Web Analyzable Database (PIGWAD) developed by U.S. Geological Survey (Hare and Tanaka, 2000) and the OSU Mars WebGIS developed by the Ohio State University (Li et al., 2006).Overall, VR and MVR technologies have not fully studied and applied in planetary exploration applications.
As one of the most popular earth viewers, Google Earth (http://www.google.com/earth/)has included Moon and Mars orbital data, which enables users to fly anywhere on the Moon and Mars and view them from any view angles with multiple resolutions.Simple measuring functions such as length and heading are also provided.Panoramic images can be linked to the orbital images at locations where the panoramas were taken by lunar astronauts or Mars rovers.NASA World Wind (http://worldwind.arc.nasa.gov/),an open source earth viewer, has similar functions and includes more planets in the solar system.Overall, these viewers have very powerful and easy-touse functions of viewing and browsing, but the measuring functions are limited, in particular, no measuring function is provided for panoramic images.
In this paper, we present a method of constructing a Measurable Virtual reality Environment (MVE) based on ground (lander/rover) panoramic images and orbital imagery.The MVE is implemented using ground panoramic images acquired by Spirit rover of the Mars Exploration Rover (MER) mission and HiRISE (High Resolution Imaging Science Experiment) orbital images of Mars.The technical details of the MVE construction method are given in the following sections.images through registration, mosaicking and projection onto a cylindrical surface.In particular, in order to enable 3D measurement functionalities in the panorama viewer, a specific function is developed for inverse calculation from the panorama back to the original images.The globe viewer is a platform for visualization and measuring of DEMs and orthophotos derived from ground and orbital images through 3D mapping procedures.Rover paths are also included in the globe viewer.

Concept of MVE
The two viewers are interlinked and can be switched from one to the other.The MVE allows a user to visualize the orbital and ground images and maps from overhead view or the rover's view and measure 3D information easily.

Data
In this study, we use ground images taken by Spirit rover's Navcam (Navigation Camera) and Pancam (Panoramic Camera) and HiRISE orbital images covering the landing site of Spirit rover.The Navcam is a panchromatic stereo pair of engineering camera with 20 cm baseline separation (Alexander et al., 2006).
The Pancam is a multispectral stereo pair of science cameras with 30 cm baseline separation.Both Navcam and Pancam are mounted on the camera bar, which can rotate ±90 degrees of elevation and 360 degrees of azimuth, enabling acquisition of panoramic images.The rover images have been pre-processed by the Jet Propulsion Laboratory (JPL) Multi-mission Image Processing Laboratory (MIPL) and products such as mosaics, linearized (epipolar) images, 3D coordinate data and range maps are also provided (Alexander et al., 2006).The derived 3D data sets, e.g., 3D coordinate data (XYL files) and range map (RNL files) are stored with the index of the left images of Figure 2. A stereo pair of Pancam images acquired by Spirit rover the stereo pairs.The images and the associated 3D data can be downloaded from the MER Analyst's Notebook website (http://an.rsl.wustl.edu/mer/mera/mera.htm),which is part of the Geosciences Node of the Planetary Data System (PDS).
Figure 2 shows a pairs of linearized Pancam images.
HiRISE is a pushbroom sensor on board the Mars Reconnaissance Orbiter.It has an unprecedented high spatial resolution of up to 0.25 m/pixel, resolving objects below a meter across (McEwen et al., 2007).HiRISE images can be downloaded from the HIRISE webpage (http://hirise.lpl.arizona.edu/)or from the PDS website (http://pds.jpl.nasa.gov/).

PANORAMA MOSAICKING
Automatic construction of large, high-resolution image mosaics is an active area of research in the fields of computer vision, image processing, and computer graphics (Szeliski and Shum, 1997).Though some panoramic mosaics of rover images are provided in PDS, we build our own panoramic mosaic from linearized Navcam and Pancam images so as to enable 3D measurement functionalities.

Image pre-processing
In many cases, the rover traversed and stopped on slopes.This resulted in slanted horizons in the images.As a pre-processing step, horizon correction is performed using the images' exterior orientation parameters with the following equation:

Cylindrical projection
There are three commonly used projections for panorama construction: cylindrical projection, spherical projection and cube projection.In this paper, we focus on the cylindrical panoramas because of their ease of construction and good geometric characteristics.If the camera focal length or field of view is known, each perspective image can be warped into cylindrical coordinates easily (Chen, 1995).(Wikipedia, 2011a).The principle is using scaleinvariant feature transform (Lowe, 2004) to detect feature points in the overlapping images taken from the same location and delete mismatched points by a RANSAC algorithm.Based on the matched feature points (tie points), the images are aligned and blended together to form a larger image.In order to ensure high geometric accuracy for seamless mosaicking, bundle adjustment is performed to find the optimal overall registration.Based on the bundle adjustment result, the panorama can be fully stitched, transformed and saved in a standard image format.

VISUALIZATION AND MEASUREING IN MVE Panorama visualization
The mosaicked panorama is a large image with cylindrical projection.A panorama viewer is developed to visualize the panorama based on Microsoft DirectX technology.A virtual camera is set at the middle of the cylinder; rotation around the center and vertical displacement can be controlled to achieve panoramic browsing.The viewer transforms the cylindrical projection of the panorama in the viewing window to perspective projection on the fly.Similar to QuickTime VR (Chen, 1995), the panorama viewer has the basic functions such as rotating and tilting the view, zooming in and out.

Measuring in panorama
The capability of measuring 3D information in the panorama viewer is one of our innovations.Since the 3D information is associated with the original rover images, the key technique to achieve a 3D measurable panorama is the inverse calculation from the panorama back to the original images.This process is divided into the following steps: mapping from a point on the panorama viewer to the two-dimensional panoramic image, then to the cylindrically projected image (single image), finally to the original image.
Calculation of projected coordinates: First, the horizontal FOV of each pixel in the panorama is calculated by where W is the width of panorama, HFOV is the horizontal FOV of panorama which is usually 360 degrees.Suppose ( , )

3D Measuring in panorama:
As mentioned earlier, along with the original stereo images of the MER rovers, PDS also provides range data, slope data and 3D coordinates data.
The data are stored corresponding to the linearized left images' pixel coordinates.In the panorama viewer, the 3D information (such as ground location, elevation, distance to the rover, azimuth angle, slope etc.) of each pixel can be measured by clicking on the pixel and inverse computation.Not only a single point, but also a line can be measured in the panorama viewer.
As show in Figures 6 and 7, once two endpoints of a line are determined, the elevation and slope profiles will be plotted.In a similar way, more 3D information can be retrieved and plotted.For those stereo images without 3D information available, the panorama viewer will locate the stereo pair and invoke a stereo measurement window to computer the 3D information through image matching and space intersection in real time.Figure 8 shows the stereo image matching result invoked by clicking on the pixel in the panorama viewer.This on-the-fly 3D measuring function is realized by inverse calculation from the panorama to the original stereo images, stereo image matching with cross correlation and least squares matching, and space intersection for 3D position calculation.Similarly, elevation and slope profiles can be generated on-the-fly.Figure 9 shows the interface of the globe viewer.With the integrated panorama viewer and globe viewer, orbital and ground images and derived 3D information can be visualized and measured from different view angles seamlessly.

SUMMARY
This paper presented a method of constructing a measurable virtual reality environment based on ground panoramic images and orbital imagery.An innovative measurable panorama technique was developed and the panorama viewer integrated into a globe viewer to form the MVE, which allows for not only visualization but also 3D measuring of planetary images in a virtual environment.
More Mars data and lunar data will be processed and put into the MVE system for test and validation; relevant functions will be further improved.The techniques developed in this research can be used in visualization of and measuring the orbital and ground images for planetary exploration missions, especially rover missions.

Figure 1 .
Figure 1.Conceptual diagram of MVE , y) are the image coordinates of the original images, (x′, y′) are the coordinates of the horizon corrected image, κ is the rotation angle around the optical axis.Figure3shows an example image before and after horizon correction.

Figure
Figure 3. Horizon correction

Figure 4 .
Figure 4. Cylindrical projection Figure 4 is an illustration of cylindrical projection, which allows 360 degree view of the environment.The cylindrical projection has the limitation of no representation at north pole and south pole.This doesn't matter much for rover panoramic images because the images usually do not cover zenith and nadir.3.3 Panorama stitching Panorama stitching is based on the open source program Hugin with our new developments incorporate the images' exterior orientation parameters.Hugin is a cross-platform panorama photo stitching program developed by Pablo d'Angelo and others (http://hugin.sourceforge.net/).It is a GUI front-end for Helmut Dersch's Panorama Tools and Andrew Mihal's Enblend and Enfuse(Wikipedia, 2011a).The principle is using scaleinvariant feature transform(Lowe, 2004) to detect feature points in the overlapping images taken from the same location and delete mismatched points by a RANSAC algorithm.Based on the matched feature points (tie points), the images are aligned and blended together to form a larger image.In order to ensure high geometric accuracy for seamless mosaicking, bundle adjustment is performed to find the optimal overall registration.Based on the bundle adjustment result, the panorama can be fully stitched, transformed and saved in a standard image format.

Figure 5 .
Figure 5. Mosaicked panorama Figure 5 shows the panorama mosaicked by 81 single band linearized images of Spirit rover.The original images were acquired by Spirit's Pancam from Sol 672 to 677 at three different elevation angles.
is accomplished, the following parameters should be recorded in a project file to prepare for the inverse calculation from the panorama back to the original images: (1) Original image file path, image format, size, focal length, horizontal FOV, lens type; (2) Panoramic image size, image format, projection type; (3) Tie points in the original images.
is a point in the panorama, the corresp- onding point in the cylindrically projected single image

Figure 6 .
Figure 6.Measuring a profile in the panorama

Figure 8 .
Figure 8. Stereo measurement window associated with the panorama viewer