INTEGRATION OF KALMAN FILTERING OF NEAR-CONTINUOUS SURFACE CHANGE TIME SERIES INTO THE EXTRACTION OF 4D OBJECTS-BY-CHANGE

Automatic extraction of surface activity from near-continuous 3D time series is essential for geographic monitoring of natural scenes. Recent change analysis methods leverage the temporal domain to improve the detection in time and the spatial delineation of surface changes, which occur with highly variable spatial and temporal properties. 4D objects-by-change (4D-OBCs) are specifically designed to extract individual surface activities which may occur in the same area, both consecutively or simultaneously. In this paper, we investigate how the extraction of 4D-OBCs can improve by considering uncertainties associated to change magnitudes using Kalman filtering of surface change time series. Based on the change rate contained in the Kalman state vector, the method automatically detects timespans of accumulation and erosion processes. This renders change detection independent from a globally fixed minimum detectable change value. Considering uncertainties associated to change allows detecting and classifying more occurrences of relevant surface activity, depending on the change rate and magnitude. We compare the Kalman-based seed detection to a regression-based method using a three-month tri-hourly terrestrial laser scanning time series (763 epochs) acquired of mass movements at a highmountain slope in Austria. The Kalman-based method successfully identifies all relevant changes at the example location for the extraction of 4D-OBCs, without requiring the definition of a global minimum change magnitude. In the future, we will further investigate which kind of change detection method is best suited for which types of surface activity.


INTRODUCTION
Change processes in natural scenes occur at a large range of spatial and temporal scales, leading to dynamic shaping of the surface. Local landscape dynamics can be captured at high spatial and temporal resolution using near-continuous terrestrial laser scanning (TLS). With permanent setups of such near-continuous TLS, point clouds are acquired at cm-scale measurement accuracy and spatial resolution, and at sub-daily temporal resolution over periods of months to years (Eitel et al., 2016). Current geoscientific applications of near-continuous TLS include monitoring of landslides, sandy beaches, and rockfallaffected areas (e.g., Kromer et al., 2017;Williams et al., 2018;Vos et al., 2017). Deriving change information from these data with thousands of epochs requires automatic methods, which are able (i) to detect changes confidently at small magnitudes, i.e. with low associated uncertainties, and (ii) to extract surface activity as individual objects during the timespan and within the local area of their occurrence.
Change between repeat point cloud acquisitions can be detected and quantified via bitemporal point cloud distance computation by comparing pairs of epochs, respectively. Point cloud distances are commonly derived using the established multiscale model-tomodel cloud comparison (M3C2) algorithm (Lague et al., 2013), which reduces associated uncertainties of changes via spatial averaging. By additionally leveraging the full temporal information of near-continuous time series data, the uncertainty associated with change can be further reduced. Kromer et al. * Corresponding author (2015) achieve this with a method considering spatial as well as temporal neighbours for averaging of point cloud distances.
Smoothing of time series using a median average in a userdefined temporal window was further used by Eltner et al. (2017) and Anders et al. (2019) to filter out noise from near-continuous time series of surface changes. To fully consider spatially and temporally variable uncertainties through error propagation, we recently proposed to use uncertainty values as Bayesian priors in Kalman filtering of surface change time series . After bitemporal change quantification, relevant, i.e. significant, change can be detected as the point in time when the change value exceeds the associated uncertainty. A statistical test is commonly used at a confidence level of 95 % to determine the significance of changes regarding the so-called level of detection (cf. Lague et al., 2013).
Once detected in the time series, the extraction of individual surface activities from 3D time series is challenging if multiple changes occur at the same location (i.e. with spatial overlap), both simultaneously or consecutively. Methods of change extraction are then required to separately delineate surface activity in space and time. Time series clustering (Kuschnerus et al., 2021a;Winiwarter et al, 2022) is an approach which yields novel layers of change information based on temporal properties of surface behaviour. Clustering is based on the full time series data, though, and therefore does not separate individual occurrences of surface activity in their respective timespans and spatial extents. For the spatial and/or temporal extraction and separation of overlapping changes, we have developed the concept of 4D objects-by-change (4D-OBCs). 4D-OBCs are defined by similar change histories within their area and timespan, representing single occurrences of surface activity. Via spatiotemporal segmentation, changes detected in the temporal domain are spatially delineated by considering the entire history during the timespan of a change based on a metric of time series similarity in seeded region growing .
Seed detection in the time series for spatiotemporal segmentation of surface changes is so far based on a globally defined minimum detectable change and has so far not considered spatially and temporally variable uncertainties. In this paper, we therefore integrate change detection using Kalman filtering of surface change time series, following Winiwarter et al. (2022). We investigate how this influences the result of change analysis. Therein, we aim for an increase in the detection of significant changes. In the current 4D-OBC method, some occurrences of surface activity may be excluded from the analysis because a global minimum detectable change value is used for all epochs and locations in the scene. The other way around, 4D-OBCs may be falsely extracted, i.e. not representing actual surface activity. This occurs if change values exceed the global minimum detectable change but the changes are a result of unquantified noise. This may occur, for example, during poor atmospheric conditions, such as high moisture or precipitation, for laser scanning acquisitions (e.g., Kuschnerus et al., 2021b). To account for this, we examine the full consideration of uncertainties using Kalman filtering of surface change time series compared to the regression-based change detection.
In this study, we investigate an alternative method of seed detection for the extraction of 4D-OBCs by analysing a tri-hourly TLS time series. The dataset was acquired over a period of three months at a high-mountain slope in Tyrol, Austria (763 epochs). Observed changes include removal and deposition of rockfall debris due to anthropogenic works, erosion induced by gravitational mass movement at the surface of debris, as well as snow cover changes due to snow fall in the final weeks of acquisition. We evaluate the results of Kalman-based 4D-OBCs compared to results obtained when applying the regression-based seed detection on either smoothed or unsmoothed input data. After seed detection, by either of the two change detection methods, region growing is performed in the time series of surface changes to determine the spatial extent of 4D-OBCs. The evaluation hence regards (i) the change detection and determination of timespans by considering uncertainties in the Kalman-filtered time series of surface changes, and (ii) the correctness of subsequently extracted 4D-OBCs. 4D-OBC extraction has the overall benefit that it can be identified if local areas in the scene experience multiple surface activities. Thereby, the method may more confidently detect surface activity with apriori unknown properties as individual occurrences even when there is spatial and/or temporal overlap throughout an observation period.

METHODS
We investigate the extraction of 4D-OBCs for three approaches of seed detection. The first uses linear regression to detect surface activity of accumulation and erosion based on a globally defined minimum detectable change value in completely unsmoothed input data of surface changes. The second uses the same regression-based seed detection, but data is smoothed in the temporal domain using median averaging to reduce uncertainty in surface changes. The third approach detects seeds using Kalman filtering by fully considering measurement and alignment uncertainties to identify relevant change occurrences.
The detected seeds are timespans of surface activity which are used for subsequent spatiotemporal segmentation of 4D-OBCs. This region growing segmentation is performed on the unsmoothed and smoothed data for the regression-based seed detection, respectively. For the Kalman-based seed detection, region growing is performed on the unfiltered time series of surface changes, i.e. on unsmoothed input data. An overview of 4D-OBC extraction is illustrated in Figure 1, with the new Kalman-based method visualized for the detection of seeds (Fig. 1b).
In the following, we introduce the study site and dataset (Section 2.1). Subsequently, we explain the main steps of change analysis with bitemporal change quantification (Section 2.2) and 4D-OBC extraction using the different methods of seed detection (Section 2.3). The evaluation of results is outlined in Section 2.4.

Study Site and Data
We use a tri-hourly time series of TLS point clouds acquired from 28 July 2021 to 15 November 2021 (763 epochs) in Vals in the Austrian Alps (47°02'48" N 11°32'08" E). The target scene is a valley slope featuring a large debris cone caused by a rockfall event in December 2017 (cf. Hartl, 2019). Surface activity at present mainly regards anthropogenic works continuing to move material at the lower parts of the debris cone and small gravitational mass movements on the slope surface. Additionally, snow fall occurred during the acquisition period analysed in this paper.
The permanent TLS setup is an extension of the initial measurement series presented by Schröder and Nowacki (2021). A RIEGL VZ-2000i TLS was used for point cloud acquisition with an angular resolution of 0.015° at around 800 m measurement range, resulting in a point spacing of 10 to 20 cm in the target area of change observation. No georeferencing is performed in our study. Since we are interested in surface activity within the scene, the data is analysed in the local Cartesian coordinate system with the sensor as origin. Due to the fixed position of the laser scanner, acquired point clouds are already coarsely aligned. To improve the co-registration accuracy between point cloud epochs for surface change analysis, we align each epoch in the time series to the first point cloud (2021-07-28 12:00) as global null epoch. Alignment is performed by deriving a rigid transformation matrix using an iterative closest point (ICP; Besl and McKay, 1992) method on stable surfaces within the scene. The stable surfaces are manually selected as centroids with specified radii (between 0.75 and 2.5 m). They are located mainly on roofs and facades of buildings adjacent to the target area on the slope. We use the ICP method implemented in the software OPALS (Glira et al., 2015;OPALS Development Team, 2018).
After fine alignment, we derive terrain points from the point clouds for surface change analysis by removing outliers and vegetation using a statistical outlier filter (Rusu et al., 2008; settings: number of nearest neighbours = 8, standard deviation multiplier = 10.0), the SMRF filter (Pingel et al., 2013; settings: cell size = 0.5 m, slope parameter = 2) and a filter on the waveform deviation (≤ 50). We apply the filters using the implementation in PDAL (PDAL Contributors, 2018). These preprocessing steps provide the final, aligned point clouds used for all change analyses methods applied subsequently.

Bitemporal Change Quantification
Bitemporal change quantification is the common step for all change analysis methods in this paper. We derive bitemporal changes for each epoch to the global null epoch using a variant of the M3C2 algorithm (Lague et al., 2013) with error propagation (M3C2-EP; Winiwarter et al., 2021). The M3C2 derives the distance of the surfaces from a reference to a compared epoch by averaging the 3D position in the reference point cloud at so-called core point locations. Change is then measured in normal direction of a plane fit to the neighbourhood points in a defined radius around the core point. This establishes an oriented cylinder with a separately defined radius, and the distance to the compared point cloud is determined at the average position of points that are intersected by this cylinder. We determine change at a subset of locations, so-called core points. These core points are obtained by subsampling the reference point cloud ( The uncertainty of quantified changes for each core point and epoch is derived with the M3C2-EP by including knowledge on the measurement accuracy of the sensor (according to the manufacturer specification) and the alignment accuracy provided by the ICP method (cf. Section 2.1). Uncertainties of each individual point measurement are thereby fully propagated into the change derived at each core point per compared epoch. We obtain spatially and temporally variable uncertainty values associated to quantified changes, as each core point holds one uncertainty value for each epoch. These uncertainties are considered for change detection in the time series using the Kalman-based seed detection, which is presented in Section 2.3.2.
For comparison of the regression-based method (Section 2.3.1) using smoothed input data, we reduce the uncertainty of change values following the approach by Kromer et al. (2015). Therein, we smooth surface changes to filter out noise in the time series by setting each epoch to the median change value in a temporal window of 24 h (corresponding to a temporal window of eight epochs for our data). The filtering is only applied in the temporal domain, as spatial averaging is already performed within the M3C2 point cloud distance computation (cf. Anders et al., 2019).

Extraction of 4D Objects-By-Change (4D-OBCs)
Spatiotemporal segmentation of 4D-OBCs is performed as seeded region growing of surface changes based on their time series similarity during the timespan of occurrence .

Seed Detection using Linear Regression
The regression-based seed detection follows the change detection method presented in Anders et al. (2022). Therein, timespans of erosion and accumulation are identified using piecewise linear regression on the time series of change values. A least-squares line fit is applied to groups of change values with similar gradients. To determine if relevant change occurred, i.e. if a detected change should be used as seed candidate, the method requires to specify a threshold value representing the minimum detectable change. This is a globally fixed value, which is solely based on the average alignment uncertainty of the 3D time series. Accordingly, we set this value to 0.05 m for the dataset in this paper. A lower value would cause a large number of 4D-OBCs to be segmented which do not represent relevant changes, but stem from alignment uncertainty and to an assumingly lower degree from measurement uncertainty. At the same time, as spatially and temporally variable uncertainty is not considered, actual surface changes with magnitudes below 0.05 m cannot be detected and are lost to the analysis. As seeds for the extraction of 4D-OBCs, we require that the timespans of surface activity are detected with a start and end time. The start of a surface activity is detected at the point in time when the estimated change rate exceeds its level of detection, and the end when the change rate falls below its associated level of detection (cf. Fig. 1b). If the corresponding change value does not surpass the level of detection in the extracted timespan, the surface activity is disregarded. After full change detection in the time series at all core point locations, the seed candidates are sorted by decreasing change magnitude, which is derived as the difference between highest absolute change value and change value at the start time of a detected activity. This final list of seed candidates is then used as input to the spatiotemporal segmentation to perform region growing of 4D-OBCs, which is described in the following section.

Spatial delineation of 4D-OBCs
After seed detection, the spatial extent of 4D-OBCs is determined via region growing in the time series of surface changes. The regression-based method uses the median-averaged change values in one variant (cf. Section 2.2), and completely unfiltered change values in the second variant. The Kalman-based method uses only the completely unfiltered change values as input for 4D-OBC extraction.
The timespan of 4D-OBCs is represented by the temporally detected change with given start and end time. Seeds are used if they have not been segmented into a previous 4D-OBC, i.e. they are used if there is neither spatial nor temporal overlap. The homogeneity criterion for region growing is time series similarity which is derived as inverse of Dynamic Time Warping (DTW) distance (Berndt and Clifford, 1994). The DTW distance threshold is automatically determined for each 4D-OBC based on the results of multiple segment versions computed for a range of thresholds in parallel (here: 0.5 to 0.9). The rationale is to maximize the object extent whereas avoiding strong overestimation which occurs as soon as thresholds become so loose that a large area outside the stricter segment version would be added. For all details on the method of automatic spatiotemporal segmentation, the reader is referred to .

Evaluation
We evaluate the results of change analysis by comparing the change detection and 4D-OBC extraction with integrated Kalman smoothing to the result of regression-based seed detection using either smoothed or unsmoothed input data. The comparison mainly regards the detection of surface activity in the time series of surface changes, which may improve for certain cases through the consideration of locally variable uncertainties. We further assess the 4D-OBCs that are extracted for each change detection method providing different seed candidates. No independent reference data about the occurrence of surface processes is available for quantitative validation of our analysis. We therefore evaluate our results by showcasing an example location which exhibits different occurrences of surface activity for our use case.
Based on this, we can further examine how the extraction of 4D-OBCs improves change information by identifying consecutive occurrences of surface activity within the same local area.

RESULTS AND DISCUSSION
Surface change analysis of the 3D time series yields different types of surface activity occurring at this study site: the formation of erosion rills on the slope, transport of debris through excavator works, and snow cover towards the end of the observation period (Fig. 2). Bitemporal changes for different timespans demonstrate how individual surface activities are aggregated or superimposed in the change information of specific epoch pairs (compare a-c in Fig. 2). In scenes where different processes act on the surface with spatial and/or temporal overlap, the detection of individual surface activities in their respective timing and duration can be The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France performed using the time series information. Surface activity detected as timespans of changes in the time series are used for the extraction of 4D-OBCs via spatiotemporal segmentation. As input to this, we subsequently assess the Kalman-based seed detection, which considers associated uncertainties, compared to the regression-based method for 4D-OBC extraction, which uses a globally defined minimum detectable change.

Assessment of Change Detection in Time
We examine the timespans of detected surface activities for the time series at a single example location, where multiple changes of different properties occur throughout the full observation period (Fig. 3). The uncertainty-based method with Kalman filtering detects six surface activities at the location. The regression-based change detection detects seven surface activities with temporal averaging and eight surface activities without any smoothing of input data. The detected surface activities and timespans differ, as the regression-based method detects an overall higher number of changes at the end of the observation period (starting Nov). Here, occurrences of snow fall and successive decrease of snow cover lead to the detection of multiple events of accumulation (snow fall) and erosion (snow cover decrease; Fig. 3b and 3c). The detected timespans mostly cover exactly three epochs from the start until the end of the change. In contrast, the Kalman-based method determines snow cover decrease as one surface process with a timespan of 65 h ( Fig. 3a; Nov-05 to Nov-07, marked with II). This oversegmentation in time of the regression-based method affects the performance of 4D-OBC extraction, because timespans which cover only part of the full surface activity may yield a change magnitude that does not exceed the globally defined minimum detectable change value (0.05 m for our dataset), even if their total change volume is much higher. Although changes are detected in time, the surface activity is disregarded as seed and will then be missing from the final result. In total, those changes can amount to a large share of change volume not considered in the analysis.
One seed is detected incorrectly by the regression-based method with temporal averaging, starting Aug-21 (Fig. 3b, marked with IV) until Nov-14. This is an effect of the grouping of epochs by gradient for linear regression (cf. Section 2.3.1), which disregards epochs in between. The seed candidate is later automatically discarded due to temporal overlap with other seed candidates of higher priority (based on a neighbourhood homogeneity criterion; cf. Anders et al., 2021). One timespan of erosion is only detected by the regression-based seed detection using unsmoothed change values as input data (Aug-08 to Aug-17; marked with V in Fig. 3c). The method with median-averaged change values (Fig. 3b) does not detect it as the smoothing presumably leads to a less pronounced linear representation of the erosion. After smoothing, the change does not exceed the globally fixed minimum detectable change value of 0.05 m anymore. It needs to be noted that the surface activity appears perfectly linear only due to direct interpolation of the data gap in this period. The Kalman-based method does not detect this erosion activity, as the uncertainties of change and change rate strongly increase during the time span of missing data (Fig. 3a).
The formation of the erosion rill in the beginning of the observation period (Jul-31) is only correctly detected by the Kalman-based method (Fig. 3a, marked with I). Here, significant change occurs with respect to the associated level of detection and leads to a relevant change on the slope surface. We assume that the regression-based method is more sensitive to the change time series not being perfectly linear. Therefore, the Kalmanbased method using the change rate can better detect surface activity with variable velocity during the full timespan of their occurrence, as opposed to abrupt events.
Surface activity can be adequately identified visually in our selected cases, and the behaviour of each method can thereby be explained and evaluated. An accuracy assessment of the seed detection methods will require further investigation with experimental data acquisition, so that the exact timing and properties of surface activity are known for validation.

Assessment of Spatiotemporal Extraction
Using uncertainty-based seed detection, four 4D-OBCs are extracted as final result at the example location depicted in Figure 3 by the Kalman-based method. Two seed candidates are discarded due to data gaps (no data values) at the start or end epoch of their timespan. The 4D-OBCs representing snow fall and snow cover decrease all exceed the defined maximum segment size during region growing. In these cases, a large part of the slope area is segmented even for the strictest threshold of time series similarity. Segments reach a large spatial extent for large-scale surface changes, i.e. overall surface increase or decrease in the scene. Such processes are usually appropriately quantified using standard methods of bitemporal change analysis. It is therefore suitable to limit the size of segments during region growing, as the computational cost increases with each grid location that needs to be additionally checked for its time series similarity.
Spatiotemporal segmentation with the regression-based seed detection with median-averaging of input data yields five 4D-OBCs as final result. All of these 4D-OBCs represent timespans of snow fall or snow cover decrease (cf. Section 3.1). They all reach the maximum segment size as large-scale surface change on the slope. The same result is yielded by the regression-based seed detection without any smoothing of changes in the input data. Here, an additional 4D-OBC is extracted which represents erosion as part of a rill forming on the slope (timespan marked with V in Fig. 3c).
The full formation of the erosion rill at the example location is only detected and subsequently extracted as correct 4D-OBC by the Kalman-based method (Fig. 4). This surface process is a distinct and relevant surface activity in the observed scene and is therefore important to identify in the time series of changes. Depending on the analysed timespan, the erosion process would also be missed in bitemporal change analysis (cf. Fig. 2). The erosion rill extracted as one of multiple surface activities at this single location throughout the full observation period demonstrates the key strength of 4D-OBC extraction. Methods considering the full time series information, for example for clustering (cf. Kuschnerus et al., 2021a;Winiwarter et al., 2022), also cannot resolve individual change occurrences temporally. It then depends on the magnitude of change in relation to the length of the full observation period if a single surface process is represented in the clustering results as distinct change pattern. In contrast, our method specifically targets the extraction of changes within their timespan as a temporal subset of the full observation. The detected changes and their timespans directly influence the performance of overall change extraction. The impact on change volumes quantified via 4D-OBCs in case the timespans are detected too conservatively was shown in Anders et al. (2022). Our comparison in this work demonstrates that the appropriate seed detection method may depend on the (different) types of surface activity which need to be identified in a scene. Besides the difference in design of the methods for seed detection, an important strength of the Kalman-based method is that changes can be confidently detected without global definitions of a minimum detectable change value. Considering spatially and temporally variable uncertainties holds potential to enable 4D-OBC extraction for very different types of surface activity that potentially occur in natural scenes without a-priori knowledge on their properties. The consideration of varying measurement uncertainties will further enable to incorporate data from different sources, i.e. platforms and sensors, in change analysis for comprehensive 4D monitoring (Pfeiffer et al., 2019;Winiwarter et al., 2021).
Results of our investigation in this paper further show that the extraction of 4D-OBCs can be rendered more flexible to the types of identified surface activity. The segmentation of 4D-OBCs does not rely on a specific model, but is free regarding the seed candidates, i.e. timespans of changes, which are provided for region growing. The presented change detection methods are bound to models of expected surface change behaviour (e.g., continuous vs. abrupt changes). Therein, the linear regressionbased method for seed detection is more sensitive to change rates which are not constant (i.e., if the change behaviour is not highly linear). Specific knowledge on target processes or external data may be used to extend the search for the occurrence of related surface activity in 3D time series data, for example in monitoring settings where meteorological variables are being recorded (e.g., Kromer et al., 2017;Kuschnerus et al., 2021b).

CONCLUSION
In this paper, we present a new approach of detecting surface activity in time series of surface changes for seed detection in the extraction of 4D objects-by-change (4D-OBCs). The method integrates Kalman filtering of change time series for consideration of spatially and temporally variable uncertainties to detect timespans of significant change. The method is showcased for different types of surface activity at a location which is affected by erosion of sediment and snow, where it detects relevant changes in their timing and duration. We compare the Kalman-based method to seed detection using linear regression of change time series. The regression-based method is more sensitive if change rates are not constant. This can lead to surface activity being detected with too short timespans or not at all. Depending on the types of observed surface activities, the derived change information is strongly affected, as surface activity is being missed or not fully quantified from its start to end. Where changes do not occur abruptly, the Kalman-based method using the change rate can hence better detect surface activity in its full timespan of occurrence.
For a location experiencing different surface processes throughout the full period of observation, we demonstrate that different occurrences of surface activity are individually extracted by the Kalman-based without a pre-defined minimum detectable change as magnitude threshold. Change is not detected by the method when long gaps occur in the data, even though the surface changes significantly before and after the gap, because the uncertainty of the Kalman filter strongly increases in such cases. Here, a combination of different change detection methods could provide a suitable approach to ensure complete detection of relevant surface activities in the future, using the strengths of each approach.