IMPLEMENTATION STRATEGY OF VISIBLE AND NEAR-INFRARED IMAGING SPECTROMETER ON YUTU-2 ROVER BASED ON VISION MEASUREMENT TECHNOLOGY

The Chang'e-4 successfully landed on the far side of the moon in January 2019. By the 12 lunar day, its Yutu-2 rover had achieved a breakthrough travel distance of greater than 300 m. A visible and near-infrared imaging spectrometer (VNIS), consisting of a visible and near-infrared (VNIR) imaging spectrometer and a shortwave infrared (SWIR) spectrometer was used for detecting mineralogical compositions of lunar-surface materials. Because VNIS is fixed on the front of the rover, and the field-of-view (FOV) of VNIR and SWIR are small (8.5° and 3.6° respectively), approaching and accurately pointing at the specific science target depend completely on the precise control of the moving rover. In this paper, a successful method of VNIS target detection based on vision measurement is proposed. First, the accurate position of the target is calculated via navigation camera imaging. Then, the moving path is planned by considering the terrain environment, illumination, communication condition, and other constraints. After the rover moves to the designed position, the binocular imaging of the hazard-avoidance cameras are activated, the detection direction and forward distance are calculated according to the images, and the FOV trajectory of the VINS is predicted while moving. Finally, by choosing the required moving control parameters, the imaging field of the VINS accurately cover the detected targets visually. These methods have been verified many times, and the results show that they are effective and feasible. The research results based on the VNIS data have successfully revealed the material composition on the far side of the moon and have deepened human understanding of its formation and evolution. * Corresponding author.


INTRODUCTION
The Chang'e-4 (CE-4) probe successfully landed in the Von Ká rmá n crater inside the South Pole-Aitken basin of the earth's moon at 10:26 am on 3 January 2019 . With the support of the relay communication satellite, "Queqiao", the lander and its Yutu-2 rover carried out in suit exploration and patrol surveys, achieving a series of important scientific findings.
By the end of the first 12 lunar days, the Yutu-2 rover reached a total travel distance of more than 300 m, achieving the "double three hundred" breakthrough (CLEP, 2019). Apart from the safe and efficient traversing, it is extremely important for the rover to accurately approach its designated science target and acquire scientific data using its science payloads (Liu et al., 2020). The visible and near-infrared imaging spectrometer (VNIS), consisting of a visible and near-infrared (VNIR) imaging spectrometer and a shortwave infrared (SWIR) spectrometer, is the major science instruments used to investigate the mineralogical composition of lunar-surface minerals (Jia et al., 2018;Li et al., 2019). As shown in Figure 1, the VNIS is mounted on the front of the Yutu-2 rover at a 45° visual angle to obtain VNIR image and SWIR data at a height of 0.69 m . Yutu-2 rover .
Because VNIS is fixed on the front of the rover, and the fields of view (FOV) of VNIR and SWIR are small (8.5° and 3.6°, respectively) , approaching and pointing at the specific science target depends completely on the precise control of the rover's driving path. In this paper, a method of VNIS detection based on vision measurement is proposed for the Yutu-2 rover, which was successfully applied during the CE-4 mission.

MAIN VISION SYSTEM
The Yutu-2 rover carries three pairs of stereo cameras: panoramic camera (Pancam), navigation camera (Navcam), and hazard avoidance camera (Hazcam) (Liu et al., 2014). The Pancam and Navcam are mounted atop the rover's mast, which can be rotated in both yaw and pitch directions. The Hazcam is fixed at its front panel. Among the cameras, the Navcam and Hazcam are engineering payloads and have been used for the guidance, navigation, and control system (Chen et al., 2014). The Pancam is part of the scientific payloads, used for remote scene investigation, topographical analysis and geological studies.
The decision-support method was based on vision measurement using images captured by Navcam and Hazcam, whose parameters are shown in Table 1. In its ground teleoperation mode, the lunar rover usually stopped after very short distance to capture a sequence of Navcam images at every 20° in the yaw direction at a fixed pitch angle to generate 0.02-m digital elevation model (DEM), digital orthophoto map (DOM) and relevant products for next-path planning. The overlap angle between adjacent images was about 26.6°. The Hazcam applied fisheye lenses to provide a wide FOV of the area starting from the front wheel of the rover. Before the probe launch, the intrinsic and structural parameters of both Navcam and Hazcam were been precisely calibrated.

METHODS
The vision-based measurement method for VNIS detection is as follows. First, the lunar rover is controlled to move closer to its detection target by relying on the Navcam. Then, the azimuth is calculated based on plane geometry, terrain, illumination, communications conditions, and other constraints. The rover is then turned to reach the azimuth. Next, a new azimuth is fineturned according to the prediction of the VNIS FOV. Finally, a short-distance straight-line approach is used to gradually advance toward the detection target relying on the Hazcam.
In detail, it includes the following four steps. Firstly, the accurate position of the detected target is calculated using Navcam images. Then the moving path is reasonably planned by considering the terrain environment, illumination, communication condition and other constraints. Because the VNIS is a passive detection device, it can only detect the visible part of the light, so it has strict requirements on the sun orientation. And it is fixed in front of the lunar rover, so we can only adjust the direction of the lunar rover to detect the light constraints. For Yutu-2 rover, according to the changing principles of solar altitude and azimuth, the detection azimuths in the morning and afternoon are set to northwest and southwest respectively, so as to avoid acquiring shaded and overexposure images. After the rover moves to the designed position, Hazcam images are acquired for terrain reconstruction so that the FOV trajectory of the VNIS during the moving process can be predicted. Finally, by choosing the required moving control parameters, the imaging field of the VNIS can accurately cover the detected target. Figure 2 shows the overall flowchart, with our work introduced in this paper highlighted. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2020, 2020 XXIV ISPRS Congress (2020 edition)

Positioning of the Detection Target
By controlling the binocular Navcams to create stereo images of the science target, the position information can be accurately calculated by photogrammetric techniaues. The details are as follows.

Stereo Matching:
The science target points are selected from the left image. Then, the correlation coefficient method and the least-squares method are used to match the target in the right image. As a result, the corresponding point in the right image can be obtained. Using this method, we can attain subpixel matching precision. The image coordinates of the scientific detection point in left and right images are denoted as ( , ) ll uv and ( , ) r r uv, respectively.

Distortion Correction:
Distortion correction can be obtained according to Navcam model and its calibration parameters, as follows:

Calculation of Exterior Orientation:
Because the Navcams are mounted atop the rover's mast, they can be adjusted to pre-set positions and attitudes by controlling the mast drives for deployment, pitch, and yaw. Position and attitude information of the mast can be obtained from the rover. Accordingly, the exterior orientation parameters of the Navcam images can be obtained in the rover frame, and they are represented by ( , , , , , )

Forward Intersection
The position of the science target in the rover coordinate frame is recorded as ( , , ) p p p X Y Z , which becomes the quantity to be solved.

Coordinate System Transformation:
Coordinate transformation is carried out to obtain the position of the science target using the given working coordinate system. See Eq. (5) for details: designates the rover's position, and rover R is the attitude matrix.

Path Planning
To enable accurate position detection by the VNIS, the rover's path planning must consider the following three items: environment modelling, determining the end position and azimuth, and path searching. The final azimuth must meet the requirement of detection implementation, such that the center FOV of the VINS points to the target to be detected.

Environment Modelling:
Along the rover path, stereo images are acquired at every waypoint using Navcam (sometimes Hazcam). Local DEM and DOM of 0.02-m grid spacing are routinely generated at each waypoint after the images are downlinked. The DEM provides the fundamental information for obstacle analysis and path planning, which usually covers a range of 16 m from the rover (Wang, 2020).
Based on the generated DEM, detailed terrain features are analysed through window-based calculation of several statistical metrics: slope, roughness and elevation change. Among these, the slop reflects the degree of surface steepness, see Eq. (6). Roughness reflects the extent of terrain fluctuation in the window area. The elevation change is the difference between the minimum and maximum height inside the window. By setting reasonable thresholds according to the rover's capabilities of obstacle-crossing and slope climbing ability, etc., these non-time-varying factors are integrated to generate an obstacle map. The obstacle map is then combined with illumination and communication conditions, which are timevarying factors, and is integrated into a comprehensive cost map for automatic path searching. 22 arctan ab slope c   The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2020, 2020 XXIV ISPRS Congress (2020 edition) Where 0 ax by cz d     .

Determining the End Position and Azimuth:
Considering that the rover may need to adjust its azimuth several times along the planned path, three conditions should be considered during position selection. The distance from the detection target should not be too far away (generally 1<d<3), the surrounding terrain should be relatively flat (0<slope<3.0), and there should be no obvious rocks in the way. Figure 3 shows the plane position relationship among the terminal point, the azimuth, and the science target point.  As shown in Figure 3, the distance between S1 and D can be calculated as: Then, the angle  can be solved according to the trigonometric function: Where y indicates the FOV of the VNIS in the y direction. Then, The azimuth is set to an angle in the horizontal plane, starting from the north direction, clockwise to the forward direction of the rover. We then obtain angle  , as shown in Figure 3. This can be accomplished using Eq. (10): Then, the moving distance can be calculated as the horizontal distance between points S1 and S2, as shown in Figure 3: Where x is the FOV of the VNIS in the x direction.

Path Searching:
Based on the established comprehensive cost map, searching an optimal path from a starting position to an end position requires methods such as A * or SD * lite (Li et al., 2017). Additionally, it is necessary to account for the mechanical capabilities of the rover during path searching, such as climbing ability (<30º ), obstacle crossing capability (<200 mm), and maximum speed (200 m/h) (Shen et al., 2015). The automatically searched path is described by coordinates of start, turning, and end points of the path, including the azimuth of the starting point and the curvature of each segment between these points. Here, their coordinates represent the center of mass of the rover.

FOV prediction
After the lunar rover completes its movement and turns toward the detection azimuth, it controls the binocular imaging of the Hazcam and predicts the FOV trajectory of the VINS during the moving process. See Figure 4. Next, prediction of the FOV using projection strategy is elaborated in detail.
Hazcam is used for close-up stereo imaging of the target because of their fish-eye lens of large FOVs. Through Hazcam images, we can see and measure the close-range scene of the rover more clearly. Secondly, the FOV of the Navcam cannot adequately cover the lunar surface at very short distance. The specific procedure is as follows. Firstly, a DEM is constructed based on the obtained binocular Hazcam images. Then, with known position and attitude information of the rover and the installation position of VNIS and its FOV, the imaging area of the infrared camera in the DEM can be calculated. Because the image size of the infrared camera is known, discrete points can be sampled at fixed intervals along the edge of the FOV. For each discrete point, a ray from the optical center of the left Hazcam to the discrete point is established, and the intersection of the line and the DEM is solved. After the intersections of all rays are calculated, they are connected to form the predicted imaging area of the DEM's infrared camera. Finally, the prediction area on the DEM is re-projected to the original Hazcam image.
According to the prediction results, it is determined whether the azimuth meets the detection implementation requirements. It is also determined whether the center of the FOV of the VNIS points to the detection target. If not, the course is fine-tuned via small left or right movement. This method adjusts the predicted FOV of the VNIS in a more intuitive way, which can be used to evaluate the rover's orientation. It further plays an important role in the completion of the overall mission.

Generation of Control Strategy
According to the movement distance calculated in Section 3.2 (Eq. 11), considering the geometric structure parameters of the rover, the wheel-train's motion parameters and mechanism, and its own running ability, the corresponding motion control instructions are generated. Considering the safety of the lunar rover in the complex environment, we use an integrated electronic system to control its movement and to directly input the desired speed and time requirements. To achieve precise control, we adopt a "short-distance straight line" step-by-step approach. Generally, this approach can be divided into the three actions: integer, direct, and path-point-tracking commands.

Integer command:
This command synthesizes smooth curves according to the sequence of path points, decomposes the fitted curves segment-by-segment according to their curvatures, and obtains multiple arcs connecting them in terms of curvature, length, and other information to generate combined motion speed, curvature, and mileage instructions.

Direct command:
Based on the kinematic model of the rover, the direct command is used to establish and solve the rover's driving kinematic equation. The entire command is decomposed into the expected rotation speed of the driving wheel and the expected rotation angle of the steering wheel.

Path point tracking command:
Here, coordinates are selected from the path sequence as tracking points, so that the rover can autonomously control its motion, speed, curvature, and mileage. Thus, it can dynamically track its designated waypoints.

ENGINEERING APPLICATIONS IN CE-4 MISSION
By the end of the first 12 lunar days, the Yutu-2 lunar rover successfully carried out three scientific explorations on the farside of the moon (i.e., the 3 rd , 9 th , and 12 th lunar days). Next, we describe the applications of the developed method in the 9 th day When the Yutu-2 rover maneuvered to waypoint LE00803, the teleoperator engineer noticed a strange phenomenon in the waypoint's Pancam mosaic image. There were some mysterious shiny substances in the interior of a small impact crater, and they displayed different morphological characteristics from the surrounding soil. The Pancam color mode was adjusted to create a close-up clear image, as shown in Figures 5 and 6. After collaboration, the scientific team decided to carry out detailed investigation of these unknown minerals by using VNIS to obtain corresponding spectral data.

Comprehensive Cost Map Generation and Coordinate Location
At waypoint LE00803, the Navcam was powered on to capture a sequence of stereo images. DEM and DOM with 0.02-m grid spacing, were automatically generated as the basis for obstacle analysis and path searching. Figure 7 shows the DEM and DOM, which were automatically generated from seven pairs of Navcam images captured at a fixed pitch angle. The mapped area covers about 1.6 m to 16 m from the front of the rover. Moreover, based on the techniques described in Section 3.1, we obtained the accurate location information of the detection target, as shown in red in the figure. The comprehensive cost map calculated based on the generated DEM, is shown in Figure 8 with a transparency of 0.4, where the red and yellow areas represent obstacles and high mobile cost areas, respectively. As can be seen from the figure, the crater, having a diameter of ~ 2.1 m, was northwest of the rover. Based on this information and other conditions described in Section 3.2, the path from waypoint LE00905 to LE00906 was searched, as shown on the comprehensive cost map of Figure 8.

Evaluation and Path Decision
After path searching, vision-based evaluation and decision was applied. This section discusses the process of evaluating the searched paths from waypoint LE00905 to LE00906.
In Figure 9, the searched path is overlain on DOM. As described, the traditional visualization of the searched path for decision support requires a center line, wheels trace, and course lines, as seen on the DOM in Figure 9. In addition to this overhead view, more evaluation work was done in the horizontal (perspective) view shown in Figure 10. This was achieved by back-projection of the path onto the image mosaic stitched from Navcam images. Because the resolution of the mosaic was at least 2 to 3 times higher than the DOM, more details were provided. Owing to the limited FOV of single Navcam imagery, we projected the path lines onto the image mosaic so that to carefully observe the path details of the start and end areas, As can be seen from the figure, the searched path well-avoided the two small stones marked with arrows 1 and 2, and their measured heights were about 2 cm. During the moving process, the outer edge of the left wheel also avoided stone 1, annotated by arrow 1. The rover was about 2.8 m away from the detection target ,and the slope of the end point was relatively small,. To minimize the number of times that the rover turns in place, we adopted the following moving strategy. First, we controlled the rover to move at a small curvature to LE00906 waypoint. Then, we controlled it to turn left in place to make a head aim at the detecion target.
After the rover moved to its end positon, we continued to control the Hazcam to aim at the detection target. Then, we used the method described in Section 3.2.2 to accurately calculate the detection azimuth. Then, we used the integrated electronic system to turn the rover in place towards its direction. Finally, we projected a short straight line so that the rover could gradually approaching the detection target.

Prediction the FOVs of VNIS
First, we used a direct command to control the lunar rover to move 20-s forward to the LE00907 waypoint. The VNIS FOV prediction results are shown in Figure 11. The green and red boxes in the figure represented the FOVs of the VNIR and SWIR, respectively. It can be clearly seen from the figure that the current FOV was far away from the detection target which is approximately at the center of the crater, and the lunar rover needed to continue to move forward a distance.
Subsequently, we controlled the movement of the lunar rover four times at short distances. The FOV prediction of the VNIS for this period is shown in Figures 12 and 13. Finally, the VNIS FOV covered the detection target, indicating success of this experiment.

CONCLUSIONS
This paper reported the successful method of the VNIS detection of the Yutu-2 rover based on vision measurement. With vision based measurement technologies, the driving path of the Yutu-2 rover was accurately controlled to ensure that the FOV of the VNIS focusing on the detected target. During the Yutu-2 rover lunar traversing, the above method was verified many times, and the results show that our proposed strategy was feasible and effective. The research results based on the detection data successfully uncovered the material composition on the far-side of the moon and deepened human understanding of its formation and evolution (Gou et al., 2019).
In the future, we plan to carry out research to estimate wheelterrain interaction parameters based on the measurements of wheel tracks in the stereo images to enhance the precise control of the lunar rover. Meanwhile, error analysis of the FOV of the VNIS prediction results will be carried out.