INFLUENCE OF ILLUMINATION CHANGES ON IMAGE-BASED 3D SURFACE RECONSTRUCTION

The paper investigates the influence of lighting conditions on image-based 3D surface reconstruction, with particular focus on periodic photogrammetric surveys for monitoring and 3D mapping applications. The analyses focus on the accuracy and completeness of each DSM and the daily and hourly repeatability of repeated photogrammetric surveys. Three test sites with rock slopes with a different orientation to the sun and different slope characteristics (slope, pattern, amount of outcropping elements that cast shadows) have been considered to ensure that results can give a general indication of the behaviours in different light conditions. In addition, a simulated virtual test site is included in the study to allow controlled image acquisition and evaluate the effect of the sun’s inclination on the DSM accuracy without influence of other weather conditions. The results show that, although there is an optimal time for the acquisitions, if particularly unfavourable light conditions are excluded, the accuracy reduction with time variation is always below 30%. The repeatability analyses by day and by time highlight a good consistence between DEMs belonging to the same day but acquired at different times and, also, between DEMs acquired at the same time but on different days. This suggests that reliable results can be obtained during continuous monitoring of, for instance, rock faces to identify rockfalls.


INTRODUCTION
Periodic photogrammetric surveys are commonly used for monitoring and 3D mapping applications. Affordable hardware and low operational costs, simplicity of components and high scalability, have made photogrammetric systems particularly suitable for long lasting or frequent acquisitions. In recent years, their application for on-site permanent installations of cameras have gained a lot of momentum as shown by several examples that can be found in the scientific literature (Roncella et al., 2014;Eltner et al., 2017;Kromer et al., 2019;Parente et al., 2019;Giacomini et al., 2020;Blanch et al. 2021;James and Robson, 2014;Mallalieu et al., 2017;Bruno et al., 2020). The use of fixed camera systems that acquire images repeatedly throughout the day exposes the survey to different and variable environmental conditions (e.g. weather, lighting) during the acquisition. It is well known that the accuracy and completeness of a Digital Surface Model (DSM) generated using photogrammetry are strictly influenced by lighting conditions (Gómez-Gutiérrez et al., 2014). Some authors recommend performing the acquisitions over a short period, for which the illumination can be considered constant (Bemis et al., 2014), others recommend planning the surveys during overcast but bright conditions to avoid strong shadows or glared surfaces which can negatively affect image matching (James and Robson, 2012). (Gómez-Gutiérrez et al., 2014) proposed identifying the best illumination condition in the area to be surveyed (considered as the one with less shadows) by estimating hillshade digital models that simulate the real environmental conditions at different periods. Despite the limitation associated with the need * Corresponding author to have an initial 3D model (i.e. a preliminary survey) to calculate the hillshade model, the method can provide valuable indications when single or low-frequency repeated surveys are performed. Nevertheless, using fixed systems that acquire images repeatedly throughout the day at very high frequencies (e.g. hourly or even more frequent), the survey is often exposed to variable lighting conditions (e.g. sun inclination, brightness and amount of shadows). A proper setting of the camera exposure parameters guarantees a correct global exposure of the scene but does not permit to fully overcome issues related to different local illuminations such as strong and changing shadows. These issues have a significant effect on change detection applications where completeness and accuracy of the entire DSM is required. The present work investigates the influence of different lighting conditions on the accuracy and completeness of DSM reconstruction related to the time of image acquisition. The work is not intended to find the most suitable time to perform the survey, but it aims at analysing the repeatability of the DSMs obtained throughout the day using fixed photogrammetric systems and at verifying the variability of precision with respect to the theoretical one estimated during the installation. In fact, when using fixed photogrammetric monitoring systems, it is important to achieve constant accuracy in the reconstruction of the DSMs, even between two successive acquisitions (e.g. for near real-time alerts) to provide reliable and accurate results for change detection analyses. A second objective of the present study is to provide an estimate of the variability of the accuracy that is likely to be expected in single surveys carried out under non-optimal lighting conditions.

Description of test sites
Three different test sites located in the Hunter Valley (New South Wales, Australia) have been selected for the study since the work is framed within a collaborative research project between the University of Newcastle (Australia) and University of Parma (Italy). The research project focuses on the monitoring of subvertical rock walls in coal mines, so-called highwalls. The selected test sites are particularly suitable to evaluate the influence of illumination on the survey, since they are free of any vegetation. The absence of vegetation and related cast shadows allows considering the influence of light in relation to the object characteristics only. There are no shadow-casting elements that can introduce additional variables in the results. The first test site is referred to as TS A. The surveyed rock wall covers an area of approximately 75 x 40 m 2 and contains a top and bottom section with a bench in-between ( Figure 1a). The top section (~10 m high) is extremely weathered and its texture is extensively degraded. Its slope angle is about 45°. The bottom section consists of several layer of siltstone and sandstone and a coal seam of about 1.4 m. This section has a slope angle of about 74°. The rock wall has an average dip direction of 255° i.e., it is facing North-Northwest (NNW). The data collection at TS A was conducted between February and March 2018. The rock wall of the second test site, referred to as TS B, also consists of a top and bottom section with a bench in-between ( Figure 1b). Both sections are about 30 m high and have an average slope angle of 72°. The total area covered is about 80 x 60 m 2 . Both sections are composed of horizontally bedded layers of sandstone, siltstone, mudstone and coal. The wall has an average dip direction of about 230°, i.e. it is facing Northwest (NW). The survey was conducted during September 2018. The area of the rock wall surveyed at the third test site, referred to as TS C, is approximately 60 x 47 m 2 (Figure 1c). The wall has no bench and its slope angle is about 72°. It also consists of horizontally bedded layers of sandstone, siltstone, mudstone and coal. The wall has a dip direction of 160°, i.e. it is facing West-Southwest (WSW). The survey was conducted during March 2019. The images considered in this investigation were acquired every two hours between 6.00 AM and 6.00 PM for each test site. The choice of limiting the analysis to periods lasting a maximum of one month was done to privilege the illumination variability within the day over its variability within the year and to ensure comparable lighting conditions corresponding to the same time. The images were mostly acquired on sunny days but also during  cloudy and hazy days. Therefore, the variability of the lighting conditions corresponding to the same time was mainly influenced by the weather conditions. In addition to the three test sites that consider physical rock walls, a simulated virtual test site (VTS) was included in the study. This allowed acquiring images in a more controlled environment and evaluating the effect of the sun's inclination on the DSM accuracy without influence of other weather conditions. The setup in the simulated virtual test reflects the one from the real test sites. Hence, the high-resolution 3D model of TS C has been textured and imported into the modelling software Autodesk 3DStudio Max 2020 and, then, rendered. Its orientation was set so that it faces South (S), to ensure good illumination throughout the day. In order to explore the influence of the sun inclination, different set-ups corresponding to maximum, minimum and medium sun elevations (solstices and equinoxes) have been tested, varying the elevation of the sun above the horizon (some sample images are provided in Figure 2). Rendered images depict the object at different times of the day with a frequency of one image every 30 minutes between sunrise and sunset.

Image acquisition and processing
In all the test sites, images have been acquired using a fixed stereo-photogrammetric monitoring system specifically designed for monitoring purposes in mining environments. The availability of such a system ensured high control and repeatability of the acquisitions, overcoming model coregistration issues. In fact, as described in detail in , the system acquires images simultaneously at scheduled times through two stand-alone fixed units, composed of a camera box, a battery box and a solar panel for power supply.
The system is continuously connected to the internet and controlled remotely by an operator. Once collected, the images are automatically uploaded onto a remote FTP server and made available for offline image processing. After the first system installation and calibration, during which interior orientation (IO) and exterior orientation (EO) parameters are accurately computed through camera calibration and bundle block adjustment (BBA) using ground control points (GCP), all the subsequently acquired stereo-pairs are automatically processed on the basis of these fixed orientation parameters. The main part of image processing, including dense matching and 3D DSM reconstruction, is performed using the commercial software Agisoft Metashape (Agisoft, 2021) and its Python scripting interface. The set-up of the acquisition system was quite similar in all the three sites. The two camera units have been located at a significant distance from the rock wall (87-108 m) with a base to distance ratio equal to 1:3. The cameras have been oriented in a slightly convergent pose to assure maximum overlap. Each camera unit consists of a full format Nikon D810 (7360 × 4912 pixel resolution) equipped with a fixed 50 mm focal length optics (AF-S Nikkor 50 mm f/1.8G Lens). Table 2 reports the geometric characteristics of each test site and the expected precision, computed according to the equation for stereo restitution in the normal case (Kraus, 2011): where Z is the distance from the object, c is the principal distance of the optical system, B is the base length and represents the expected precision of the parallax between two homologous points. In the current study, the precision of single image point measures was considered equal to ±1 pixel (see e.g. Luhman et al., 2013) and the parallax precision was estimated using the error propagation law. In VTS two rendering cameras have been positioned in front of the virtual model simulating the real acquisition geometry (Z=89 m and B=26.6 m) in a slightly convergent pose, symmetrical with respect to the central part of the wall. The cameras have been set with a focal length equal to 50 mm and the renderings were produced with the same resolution of the real images (7360 × 4912 pixel), using the Arnold renderer integrated in the software package Autodesk 3D Studio Max, which supports image based lighting, including photometric physical sky. The 3D surface has been textured with a high-resolution texture obtained from the real images of TS C acquired in an overcast but bright day, therefore without self-shadows. In addition, the reflectivity model provided by the software for stone materials has been applied to simulate realistic light reflections.

Site Z [m] B [m] GSD [mm] σz [mm]
Num. Both real and simulated image pairs have been automatically processed. The documentation on the internal algorithms used by Metashape is scarce due to commercial constraints. Nevertheless, from available information on the Agisoft user forum (Agisoft Forum, 2021) it is possible to affirm that, to initialize the matching, Metashape performs a feature matching with an approach similar to the SIFT algorithm. During this step, it detects points that are stable under viewpoint and lighting variations, on the base of which it is able to detect correspondences across the images and, then, to compute camera IO and EO parameters. In these analyses, images have been processed without downscaling to ensure the best accuracy, and IO and EO parameters have been estimated during the installation phase of each site and kept fixed in all the successive elaborations. Then, the dense matching procedure of Metashape, apparently a stereo Semi-Global matching method (Remondino et al., 2014), has been applied. In this step, images were processed after a preliminary image size downscaling by a factor of 4 ("medium quality" in Metashape terminology) to reduce computing times and resulting file sizes.

DSMs
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) The resulting 3D point clouds have then been exported and both accuracy and repeatability assessment have been performed comparing it automatically with a reference model using CloudCompare (CloudCompare, 2021). The former analysis aims at evaluating if and how much the different times of acquisition (resulting in different brightness of the scene and variable amount and darkness of shadows) can influence the correct reconstruction of the 3D surface. This assessment has been performed by comparing each DSM with a benchmark mesh model obtained from terrestrial laser scanner (TLS) acquisitions. In detail, each 3D point cloud produced by Metashape has been imported in CloudCompare, where an Iterative Closest Point (ICP) algorithm has been applied to optimise registration and eliminate possible systematic co-registration errors. The Cloudto-Mesh distance algorithm has been applied to compute the distance between the two models, considering a comparison range of 0.2 m (i.e. points located at a distance greater than 20 cm were considered gross matching errors and excluded from the comparison). In addition, model completeness has been computed as the ratio between the number of actual reconstructed points and the total number of occlusion-free points, obtained by projecting the DSM (no-gap) surface on the stereo-pair images.
The aim of the repeatability analysis was twofold. On the one hand, hourly repeatability was examined to assess how much the choice of different reference times for daily scheduled monitoring might affect the change detection accuracy. On the other hand, the repeatability of models within the same day was investigated, in order to assess the reliability of high-frequency monitoring and distinguish the uncertainty of model reconstruction due to different illuminations from real changes in the monitored rock wall. In fact, these analyses are designed mainly for monitoring purposes, where change detection is performed by comparing models obtained at different periods, so repeatability of model reconstruction becomes crucial to be able to measure small changes with confidence. The repeatability assessment was carried out on raster Digital Elevation Models (DEMs). To avoid systematic errors and make all DEMs comparable, each 3D point cloud was firstly finely registered with the reference TLS mesh model by an ICP procedure. Then, it was projected to a plane passing through the central part of the rock wall, and rasterized using a cell-size resolution equal to 4 cm (i.e., approximately 4 times the GSD values, in agreement with the image size downscaling by factor of 4 operated during the dense matching process). For the hourly repeatability analysis, a reference DEM was computed for each hour by averaging the DEMs referring to the same time. The same was done for the daily analysis, considering the average of DEMs grouped by day.

Accuracy assessment for the simulated test case
The results of the simulated test case (Figure 3, the results are grouped by time) show very good agreement between the original digital model and all the reconstructed DSMs, with an average RMS (Root Means Square) error equal to 1.2 cm. Illumination changes due to the different period of the year do not seem to significantly affect the results, i.e. different elevations of the sun at the same time do not affect the model accuracy. The best results have been obtained at the highest sun elevation, but the worst RMS (experienced during winter solstice) is only 2% higher.
The changes in DSM reconstruction accuracy are more evident if considered at different times of the day. The blue line in the box plot in Figure 3 refers to the median RMS value at each full hour, the light-blue boxes represent the RMS variability between the 25 th and the 75 th percentile, while the black error bars highlight the minimum and the maximum RMS values. The grey line refers to model completeness. The horizontal axis indicates the different times of image acquisition. In particular, in the central hours of the day (from 11 AM to 2 PM), the increase of general brightness and shadow's intensity results in lower accuracy of the models (+19.8% of RMS value).
Since the wall faces south, the trend is symmetrical throughout the day, with a peak at noon. Variability (range between minimum and maximum RMS) also reaches its maximum level in the central hours of the day.
Since the stereo-pair geometry is constant throughout the experiment it could be expected that a more accurate DEM reconstruction should also be more complete. In these simulated tests, against our expectation, the completeness increases as the RMS increases, reaching a maximum of 99.8% at noon. Real tests, on the contrary, seem to confirm the expected behaviour (see Section 3.2). Currently we cannot find a valid explanation and further investigations are required.

Accuracy assessment for the real test sites
The results of the real test cases show that general accuracies are in line with the expected precisions and remain consistent over time. The analysis of the data grouped by day shows an overall stability of accuracy and does not evidence a correlation with different daily weather conditions (sunny, cloudy, partially cloudy day). The increase in RMS is quite small and turns in rather negligible absolute values for TS A and TS B. In TS C, however, in a single day only, lighting issues resulted in maximum increase in RMS equal to 39%. However, when excluding this day, the RMS average variability is around 9.25%.      Table 3. Summary of average RMS by site and the maximum (average in parenthesis) RMS increase by day, calculated with respect to the day that provides the minimum RMS. Table 3 summarizes the average RMS obtained for each site considering all the DSMs and the maximum RMS increment by day (relative and absolute).

Test Site C (wall facing WSW) -Accuracy by time
As highlighted for the simulated test case, the analysis of data grouped by time produces more variable results and is affected by the orientation of the rock face, the contrast (determined by the presence of sharp shadows) and the general brightness of the scene at different times. All these factors are strongly sitespecific, therefore, in the following, results will be presented by site. TS A (Figure 4a) is facing North-Northwest and, at the time of the survey, the sunrise was at 06:40 AM, so the 6:00 AM images were taken just before sunrise and, in most cases, are under exposed. The images taken at 8:00 AM are illuminated by diffuse light, the images at 10:00 AM present the greatest contrast with the highest presence of sharp shadows. At later times, the shadows decrease to a minimum at 2:00 PM and then begin to lengthen again, but never reach the contrast levels of 10:00 AM. The maximum RMS values can be observed with images acquired at 6:00 AM. Then, the RMS progressively decreases during the day as the brightness of the scene increases. The greatest variability of results, considering only the data ranging from 25 th to 75 th percentiles, occurs at 10:00 AM when shadows are very sharp. The best results (lower RMS and lower variability without outliers) are obtained at 2:00 PM, when shadows are minimal. The maximum percentage RMS increase with respect to the most accurate result (at 2:00 PM) is 15.4% and occurs at 6:00 AM. As expected, the completeness decreases with higher RMS, with the exception of models referring to 6:00 AM, which are the less accurate but the most complete. TS B (Figure 4b) is facing Northwest and unlike the previous site, for the time of year when surveys were carried out, the images acquired at 6:00 AM were taken just after sunrise. The slightly different exposure of the site results in a different wall illumination compared to the one previously observed. Images taken at 6:00 AM present diffuse light but are still a bit under exposed. At 8:00 AM the wall is still all in shadow and the pictures are barely backlit. Again, the 10:00 AM images present the largest and sharpest shadows. Minimum shadows are observed at 12:00 PM and 2:00 PM, but then increase again. At 6:00 PM the light conditions are poorer due to proximity to sunset. The highest RMS value occurs at 6:00 AM, and also at 10:00 AM and 6:00 PM. When the rock wall has many shadows or is dimly lit, the accuracy is lower. The variability of the results is instead very limited for all the times. Some maxima are observed at 6:00 AM. The maximum RMS percentage increase is equal to 22.3% and occurs at 6:00 AM. The completeness, also in this case, is inversely proportional to the DEM reconstruction accuracy. TS C (Figure 4c) is facing West-Southwest, so the survey is particularly challenging in the morning because the images are always backlit. For this site, the first image (6:00 AM) is often very underexposed because the acquisition occurred before sunrise. The best lighting conditions are towards the end of the day, when the wall, due to its exposure, is in favour of illumination. The RMS trend is clearly affected by lighting conditions. The highest RMS is at 6:00 AM, when some photos are particularly dark. Here the completeness is the lowest (50%). Fairly high RMS values are also present at 8:00 AM and 10:00 AM, when many images are backlit, and at 2:00 PM when the wall is covered by very sharp shadows. Completeness is confirmed to be inversely proportional to the RMS, while the greatest data variance is at 6:00 AM, 10:00 AM and 2:00 PM, i.e. with unfavourable lighting conditions. Percentage variation of RMS is very high at 6:00 AM, when it doubles (+118.6 %). On the contrary, in the last part of the day with good illumination, the variability is lower (30% between 2:00 PM and 6:00 PM).

Repeatability assessment by time
The repeatability analyses by time confirm the behaviour observed from the accuracy assessment. Figure 5 shows the trend for each site. For all the sites, the repeatability of the results matches the RMS presented in Figure 4. The times when the models are most accurate provide the best repeatability.
In TS A, the internal consistency of photogrammetric models acquired at the same time but on different days is lower than their accuracy. In fact, globally, the average RMS is 17% higher than the average accuracy observed with the TLS comparison (see Table 3). The lowest repeatability (albeit with small percentage variations, as indicated by the labels in the graph) is observed at 6:00 AM, when the illumination of the rock wall is very low, and, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B2-2021XXIV ISPRS Congress (2021 unexpectedly, at 2:00 PM, when lighting conditions seem to be the best, as confirmed by the accuracy analysis. In TS B, on the contrary, the average RMS provided by the repeatability analysis is 44% lower than the average accuracy provided by the TLS comparisons, demonstrating a good consistency of the measurements, beyond absolute accuracy. The trend over time ( Figure 5b) retraces generally with what is observed in the accuracy assessment ( Figure 4b), with peaks at 6:00 AM and 10:00 AM in alignment with under exposed images and sharp shadows. An exception occurs at 4:00 PM and 6:00 PM, where the behaviour is opposite to that noted for the accuracy. The percentage variations of RMS over time are slightly greater as in TS A and reach +28%.
In TS C, the internal consistency of the observations is greater than the absolute accuracy. The average repeatability RMS is, in fact, 30% lower. The general trend in Figure 5c is similar to the one shown in Figure 4c, but the variability of the data acquired at 10:00 AM (backlit images) is emphasized. Although the DSMs acquired at 10:00 AM are globally accurate (Figure 4c), they are scarcely consistent. Models acquired at 6:00 AM (dimly lit rock face) are confirmed to be highly variable as well as inaccurate and incomplete. The percentage variation of RMS is higher than in the other two sites and, until 2:00 PM, is always over 20%. This may be related to the different orientations of the rock faces: TS A and TS B are facing NNW and NW respectively. Therefore, the shadows have a fairly homogeneous and symmetrical variation during the day. TS C instead is facing WSW and therefore has good lighting conditions during the second part of the day only, while in the morning the luminosity of the scene and the camera exposition for image acquisition are very unfavourable.
In VTS, the hourly repeatability agrees with the standard deviation of the RMS obtained from the accuracy assessment. The average RMS is quite low (8 mm) showing a good consistency of the models. Compared to the real world test sites, where the percent RMS increase over time was quite low (except for certain periods when lighting conditions were unfavourable), here the variability over time during the day is more noticeable and is not symmetrical throughout the day, as should be expected. One explanation might relate to the actual time during the year when the images were acquired. For the real test sites the survey timespan was limited purposely to one month, to grant approximately the same lighting condition. Therefore, the variability of the RMS is limited to those periods of the day that have, by their nature, unfavourable lighting conditions. On the contrary, the virtual case includes images acquired throughout the entire year, so the same time might correspond to different (to some extent) lighting conditions, depending on elevation of the sun above the horizon and on sunrise and sunset time. This affects all the acquisitions almost with the same extent, regardless of the time of the day, and might justify the observed RMS variability over time.

Repeatability assessment by day
The repeatability analysis by day highlights a good consistence between DEMs belonging to the same day but acquired at different times. The average RMS is equal to 1.64 cm for TS A, 1.39 cm for TS B and 3.21 cm for TS C, with a standard deviation of 0.9 mm (TS A), 1.1 mm (TS B) and 9.8 mm (TS C). Therefore, the variability of the models acquired during the same day is largely contained within the expected precision of the survey. The variability of RMS between consecutive days is quite low and contained below 20% for TS A and TS B, while TS C registers RMS increases up to +120% in some particular days, always caused by the bad lighting conditions in the early morning.
As already pointed out in Section 3.1, for repeatability it was not possible to establish a clear correlation between weather conditions and RMS. The graph in Figure 7 shows the average RMS obtained computing all the models and the RMS obtained analysing the sunny or cloudy days only for each real test site. In TS A, the sunny days seem to produce less consistent DEMs than the cloudy ones, while in TS C the behaviour is the opposite. In TS B, this analysis was not performed because almost all of the acquisitions had occurred on sunny days, thus a meaningful comparison for cloudy days could not be extrapolated.

CONCLUSIONS
The paper investigated the influence of lighting conditions on image-based 3D surface reconstruction, with particular reference to periodic photogrammetric surveys aiming at monitoring and 3D mapping applications. The analyses concerned the accuracy and completeness of each DSM and the daily and hourly repeatability and were performed on three different rock walls and on a virtual case study.

Virtual Test Site -Repeatability by time
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B2-2021XXIV ISPRS Congress (2021 during the day cause the greatest differences. As expected, very sharp shadows or unfavourable image acquisition conditions (e.g. backlighting) worsen the results. According to the orientation of the rock wall with respect to North and the period of the year in which the survey is carried out (i.e. the elevation of the sun above the horizon), a best time within the day can be identified to obtain the most accurate DSM, which generally corresponds to the minimum amount of shadows. With the exception of the virtual test site, which shows the opposite behaviour, generally the most accurate models are also the most complete. Although there is an optimal time for the acquisitions, if particularly unfavourable light conditions are excluded, the accuracy reduction with time variation is always below 30%. The daily average RMS is reasonably constant on all sites. Noteworthy is the different average accuracy obtained for TS A and TS B, despite the acquisition geometry and expected precision being very similar. The analyses did not show apparent correlations with light variability, thus, the higher RMS obtained in TS A could be due to problems in the image block orientation. From the repeatability analysis, it is possible to infer that for daily monitoring, the time that guarantees the highest accuracy generally guarantees the highest repeatability as well. In TS B and TS C, for the same time of day, the RMS obtained from the repeatability analyses is lower than the one obtained from the TLS comparisons. This means that systematic errors resulting from unfavourable illuminations are absorbed and, hence, do not greatly affect comparisons between photogrammetric models. This extends the possibility of monitoring even for challenging lighting conditions, as changes can be estimated with confidence with respect to previous photogrammetric DSM acquired. In addition, with the exception of TS A, the RMS obtained are much lower than the expected precision, indicating that assuming the precision of single image point measures equal to ±1 pixel was probably very conservative. Finally, the repeatability analysis within a single day has shown that it is possible to carry out and compare surveys made at different times of day, for instance, for identifying a rapid evolving phenomenon (e.g. a rockfall event), with reliable results. The overall consistency of the results demonstrates the reliability of such monitoring systems over time.