DEFORMATION MONITORING WITH OFF-THE-SHELF DIGITAL CAMERAS FOR CIVIL ENGINEERING FATIGUE TESTING

Deformation monitoring of civil infrastructure systems is important in terms of both their safety and serviceability. The former refers to estimating the maximum loading capacity during the design stages of a building project, and the latter means performing regularly scheduled maintenance of an already existing structure. Traditionally, large structures have been monitored using surveying techniques, while fine-scale monitoring of structural components such as beams and trusses has been done with strain gauge instrumentation. In the past decade, digital photogrammetric systems coupled with image processing techniques have also been used for deformation monitoring. The major advantage of this remote sensing method for performing deformation monitoring is that there is no need to access the object of interest while testing is in progress. The paper is a result of an experiment where concrete beams with polymer support sheets are subjected to dynamic loading conditions by a hydraulic actuator in a structures laboratory. This type of loading is also known as fatigue testing, and is used to simulate the typical use of concrete beams over a long period of time. From a photogrammetric point of view, the challenge for this type of experiment is to avoid motion artifacts by maximizing the sensor frame rate, and at the same time to have a good enough image quality in order to achieve satisfactory reconstruction precision. This research effort will investigate the optimal camera settings (e.g., aperture, shutter speed, sensor sensitivity, and file size resolution) in order to have a balance between high sensor frame rate and good image quality. The results will be first evaluated in terms of their repeatability, and then also in terms of their accuracy. The accuracy of the results will be checked against another set of results coming from high quality laser transducers. * Corresponding author.


INTRODUCTION
Structural health monitoring of civil infrastructure systems is important for ensuring both safety and serviceability over their lifespan.To verify the safety of a new structure, the maximum load-carrying capacity must be first estimated and then tested.In addition, regularly-scheduled inspections and maintenance operations must be done for already existing structures.One of the most common checks on the fine-scale level is the measurement of deflections in structural components such as beams.The measurement of deflections is necessary to verify that the occurring deformations are within the expected limits (Brownjohn, 2007).
Traditionally, fine-scale deformation monitoring has been done with mechanical, optical or electrical instruments designed for civil engineering purposes.Examples include wire strain gauges, optical fibre sensors, and laser transducers.Some of them achieve high precision, and can collect data samples at a very high frequency.However, they are either contact instruments, and require access to the area being monitored, or they have a limited range so they have to perform deformation measurements from a close distance.This imposes a risk of damage in the case of failure of the structural elements being tested (Gordon and Lichti, 2007).In addition, it is not economically feasible or logistically practical to use many instruments to cover an entire structure (Maas and Hampel, 2006), so highly trained and experienced structural experts must select only specific areas to be monitored.Another downside of these instruments is that they can perform measurements in one dimension only (Gordon and Lichti, 2007;Jiang et al., 2008;Maas and Hampel, 2006).Also, no permanent visual record is formed unless some basic notes are written with a marker directly on the specimen being tested.
In order to mitigate the limitations of the traditionally used instruments listed in the previous section, image-based remote sensing techniques can be implemented for deformation monitoring purposes.These techniques are capable of reconstructing entire 3D surfaces without the need to access the monitored object(s) (Lichti et al., 2000).This capability can be used for performing deflection measurements.In addition, a permanent visual record is established for each observed epoch (Jiang and Jáuregui, 2007).Several research initiatives based on laser scanning (González-Aguilera et al., 2008;Gordon and Lichti, 2007;Lee and Park, 2011;Monserrat and Crosetto, 2008;Park et al., 2007;Vezočnik et al., 2009), photogrammetric (Barazzetti and Scaioni, 2010;Fraser and Riedel, 2000;Jáuregui et al., 2003;Jiang and Jáuregui, 2010;Lin et al., 2008;Mills et al., 2001;Whiteman et al., 2002) and range camera (Lichti et al., 2012;Qi et al., 2014) methods for deflection estimation exist so far.
Both laser scanning and photogrammetric approaches to performing deflection measurements have advantages and disadvantages.A terrestrial laser scanner can acquire millions of points within seconds.In addition to being a non-contact and targetless reconstruction technology, laser scanning is also independent of the light conditions in the area of interest (Park et al., 2007).The downsides are that the single point positioning may be poor, the scanning process is not instantaneous, and due to the limited range of the instruments and the impact of occlusions, multiple point clouds must be collected in order for the entire area of interest to be covered (Mills and Barber, 2004).These multiple point clouds must be co-registered to a common reference frame.Also, the initial system cost can be unfeasibly expensive (Yoneyama et al., 2007).Photogrammetry, on the other hand, does traditionally require signalized targets, and it is a light-dependent technology.In addition, photogrammetric reconstruction involves image matching, and the scale definition must be brought in externally (González-Aguilera et al., 2008).However, image acquisition is nearly instantaneous, inexpensive photographic components are available off-the-shelf, and given a convergent geometry, the 3D reconstruction is precise (Mills and Barber, 2004).Finally, it should be noted that laser scanning and photogrammetry are not necessarily competing technologies.The type of optical imaging modality should be decided on per-application basis depending on the type of object monitored and the budget of the field campaign.In the case of kinematic objects and a tight budget, photogrammetry will be the preferred option.However, for static scenes, where a constrained budget is not an issue, laser scanning performs more than satisfactory as well (Rönnholm et al., 2009).A relatively new technology that combines both intensity and active depth imaging is the range camera (Lichti et al., 2012).Even though it can be seen as the perfect complement between photogrammetry and laser scanning, i.e., as an instantaneous laser camera, this sensor is still limited in terms of resolution, range and scene dependent errors.

LABORATORY SETUP, SYSTEM DESCRIPTION, AND CALIBRATION
A hydraulic actuator was installed in a structures laboratory in order to be used for a beam deformation experiment.The actuator had a capacity of 250 kN, and it was suspended from a heavy duty steel cross beam, which was supported by two massive steel columns bolted to a strong concrete floor.The concrete beam to be tested (dimensions of 3 m x 300 mm x 150 mm) was white-washed, and it had a steel reinforced polymer sheet glued to its bottom.It was placed under the actuator, and had two loading points through a steel spreader beam.Both static and dynamic loading conditions were to be tested.Static testing refers to progressively increasing the load applied by the actuator to the specimen at a very slow rate.As a result, the increase of the deflection in the tested specimen can be at the order of a couple of millimetres every minute.Dynamic testing, also known as fatigue testing, refers to repeatedly applying and releasing a load, or applying a range of loads, at a much faster rate simulating real-world operating conditions (e.g., bridge traffic).Thus, the deflection or the vertical motion occurring in the tested specimen can range in the order of several millimetres every second or a few times every second.Half of the full range of deflection is the amplitude of the vertical motion of the specimen.For a visual illustration of the experiment setup, and for a close-up of the hydraulic actuator and spreader beam, one can refer to Figure 1a and Figure 1b.
In addition to the actuator supporting frame, a lighter duty steel frame was designed and built around the actuator in order to hoist the components of the photogrammetric system in secure positions above the beam to be tested.The photogrammetric system was to be used to estimate the deformations of the top surface of the beam.However, since a large portion of the top surface of the beam was obstructed by the spreader beam, thirteen 150 mm x 50 mm white-washed thin aluminium plates were glued on the side of the beam (see Figure 1c).This way they could serve as offset witnesses to the bottom surface of the beam.Thus, the objects of interest for the photogrammetric system were primarily these offset/witness plates.In order to be able to evaluate the accuracy of the photogrammetric system, five laser transducers were used as control sensors: one Wenglor CP35MHT80, two Wenglor CP24MHT80, and two KEYENCE LK-G407 charge-coupled device (CCD) laser displacement sensors.One of the laser transducers can be seen in Figure 1c.The laser transducers were operating at a sampling frequency of 120 Hz, and their overall precision had been previously evaluated to be approximately 10-30 µm.The photogrammetric system used for this project consisted of eight cameras and two projectors placed on the lighter duty steel frame (see Figure 2).Both the cameras and the projectors were available off-the-shelf.The reason for using off-the-shelf components is that it is more cost effective and practical in case any of the units are malfunctioning and need replacement.The cameras used were the Canon EOS 1000D digital single-lens reflex (DSLR) ones.Each camera had a 22.2 mm x 14.8 mm complementary metal oxide semiconductor (CMOS) solid state sensor.The output images had a maximum resolution of 10.1 mega pixels (i.e., 3888 pixels in width and 2592 pixels in height), where the pixel size was 5.71 µm.According to the manufacturer's specifications, the cameras supported continuous shooting of up to three frames per second (Canon Inc., 2008).The cameras were to be used for photographing the concrete beam, while it was being deformed by the hydraulic actuator.They configured and synchronized (to 5 milliseconds) so that non-blurred images of both static and kinematic objects could be taken simultaneously.The synchronization was done through a hardware trigger (i.e., a wired remote control) connected to a hub, which could split the shutter release signal to all eight cameras.The acquired images were downloaded to a computer via a universal serial bus (USB) connection.The projectors used in this research project were the single-chip light processing (DLP) BenQ MP522 ST.They were shortthrow projectors with an extended graphics array (XGA) and a resolution of 1024 pixels x 768 pixels (BenQ Corp., 2008).The projectors were used to display a pattern in case artificial texture was necessary for the 3D reconstruction process.Figure 2.An example showing half the cameras and one of the projectors suspended from the support frame The photogrammetric system was calibrated in two phases: one for the geometrical camera calibration, and another for the position and orientation of the cameras with respect to an object space reference frame.The first one concerns the interior orientation parameters (IOPs), and the second one the exterior orientation parameters (EOPs).For this particular project, the camera lenses were calibrated with the zoom ring set to 18 mm.Also, before the camera calibration was done, the image stabilization, the automatic focus, and the sensor cleaning functions of the cameras were disabled to ensure the validity of the estimated parameters for subsequent data acquisition sets.In addition, the zoom and focus rings were taped so that the principal distance stayed fixed.Convergent camera station geometry (including rolled images) and a flat test field with checker board targets were used for the individual calibration of each of the eight cameras before setting up the experiment.Once the cameras were set up in the structures laboratory, their EOPs were estimated prior to the commencement of each test.
Physical signalized targets placed at different depth were used for that purpose.

CONDUCTED EXPERIMENTS
A multi-day beam deformation experiment was conducted in three phases: 1. Phase Istatic loading based on displacement control, which was done in order to settle the beam on its support; 3 mm displacements were applied at a rate of 1 mm per minute; 2. Phase IIstatic loading based on load control, which was performed in order to initiate cracks in the specimen; a maximum load of 60 kN was applied at a rate of 3 kN per minute; 3. Phase IIIdynamic loading based on load control, i.e., cycling between an expected low (e.g., 24 kN) and high (e.g., 72 kN) loads at a rate of either 1 Hz or 3Hz.
Phase III was run until the beam reached failure, i.e., the polymer sheet separated from the bottom of the beam, the reinforcing rebar inside the beam fractured, and the beam experienced significant cracking.Image data was collected at zero load/displacement before the experiment commenced, during the static loading in Phases I and II and the 1 Hz dynamic loading in Phase III, and after the end of the experiment for documenting the permanent damage.The 1 Hz dynamic loading was imaged by multiple two to eight second bursts of still frames, where the image resolution, shutter speed, and sensor sensitivity were varied.

EXPERIMENTAL RESULTS AND DISCUSSION
The goal of the conducted experiment was to attempt using offthe-shelf non-contact sensors in order to image a dynamically moving homogeneous object.The methods for performing 3D reconstruction and carrying out point cloud processing for deflection measurements are listed together with a discussion of the level of automation for the system, and the precision and accuracy of the final results.

3D reconstruction
Multiple light ray intersection was used for the 3D reconstruction.The matching of conjugate pixels was performed through a combination of area-and feature-based matching, where the corner detector of choice was the Harris operator (Harris and Stephens, 1988).Resampling according to epipolar geometry was also applied.A region of interest was also selected in every image.This was done manually, but the human operator interaction was partially mitigated.In order to avoid having to select the region of interest multiple times, the observed images in all epochs were first averaged.The averaged image was then used in the manual generation of a single binary mask, which was for the automatic selection of the region of interest in the images from all observed epochs.Overall, the current photogrammetric reconstruction process can be considered semi-automatic.The result of the reconstruction is the 3D object space coordinates of the offset/witness plates and the visible portions of the concrete beam.An example of the X, Y, Z point cloud resulting from the photogrammetric reconstruction can be seen in Figure 4a.

Point cloud processing
The Once the segmentation of the point clouds for all of the observed epochs was complete (see Figure 4b), the 3D centroid coordinates of the reconstructed offset/witness plates were computed by averaging the X, Y, Z coordinates for all the points belonging to the plate.The identification of the corresponding plates reconstructed at different epochs was established based on their 3D centroid coordinates.This was possible because the deformation in the beam between two consecutive epochs was much smaller than the spatial distance between the offset/witness plate locations.
Of interest in this study was estimating the vertical motion amplitude of each plate for a set of observed epochs.Since, during the dynamic loading experiment, the concrete beam was repeatedly subjected to the same load range, the vertical displacement of each plate exhibited a cyclic motion.This cyclic motion can be modelled as a sinusoidal curve, where the unknown parameters are the amplitude, the frequency, the phase, and the mean value of the signal.Even though the primary interest was the recovery of the amplitude, the other three parameters had to also be estimated as nuisance parameters.Since the sinusoidal curve model is non-linear, approximate values for the unknown parameters had to be provided, and the final values for these parameters were computed in an iterative manner.One way of deriving an approximate value for the frequency is through a Fourier transformation of the signal.The approximate values for the other parameters were derived using the linear fundamental harmonic equation.

Quality control discussion
Seven different bursts of still images were processed, and they were used to perform three tests.The tests were made in order to 1) identify the most appropriate image resolution, 2) choose the most suitable shutter speed and sensor sensitivity combination, and 3) assess the repeatability and accuracy of the estimated amplitudes.

Test on the image resolution
Three of the processed bursts of still images varied in their image resolution: large (see Table 5 in the Appendix), medium (see Table 6 in the Appendix), and small (see Table 7 in the Appendix).The image properties and the sinusoidal fitting results for the image bursts for the three different types of image resolutions are summarized in Table 1.The large resolution burst produced the most precise results, but had the largest file size and thus the lowest burst rate, while the small resolution burst had the least precise results, but the smallest file size and the highest burst rate.The burst rate listed here is the maximum number of consecutive JPEG files that each camera is able to output in continuous shooting mode without having the risk of the cameras going out of synchronization.Due to its low burst rate, the results from a large resolution burst cannot always produce a distinct peak at the correct frequency in the singlesided amplitude spectrum of the signal.Even though all three resolutions produce adequate results, the authors recommend using the medium resolution as its precision is only marginally lower than the one in the large resolution case and its burst rate is sufficiently high to identify the right frequency of the signal.9 in the Appendix).The idea here is that by increasing the shutter speed (or decreasing the integration time), the sensor frame rate would go up and motion artefacts such as image blurring could potentially be minimized.However, in order to maintain proper image exposure, the sensor sensitivity (or ISO value) must be increased, which could potentially increase the noise in the acquired images.Note that modifying the lens aperture was not an option due to depth of field restrictions.Thus the test here aimed at investigating the optimal balance between sensor frame rate and image quality.The approximate sensor frame rates (or sampling frequencies) and the sinusoidal fitting results for the three bursts are summarized in Table 2. Again, all three combinations of shutter speeds and sensor sensitivities produce similar results.Since the marginally higher frame rate in the case of the faster shutter speeds did not seem to improve the fitting results, the authors recommend the use of the slow shutter speed in order to guarantee the highest quality or the least noisy images possible.

Properties / Results
Shutter

Repeatability and accuracy assessment
Three of the processed bursts also had the same resolution (medium), shutter speed (1/15 sec) and sensor sensitivity (100 ISO).Detailed results can be seen in Table 6, Table 10, and Table 11 in the Appendix.These results yielded three different values for the amplitude estimation (A) for the vertical deflections at each plate.Thus, three combinations for the differences in the amplitude estimation (ΔA) were computed for each plate.As seen from Table 3, overall the ΔA values ranged from -0.10 mm to +0.10 mm, with no bias, and an RMSE value of 0.03 mm.
The derived amplitudes by the photogrammetric system were compared to amplitudes derived from the measurements made by the control laser transducers at five of the 13 offset witness plates.As seen from Table 4, the differences in the estimated amplitudes by the two systems ranged from -0.02 mm to +0.12 mm with a mean of 0.03 mm and an RMSE of 0.06 mm.Note that the laser transducer results shown here were processed from data whose collection was timed to match the image acquisition of the photogrammetric system (give or take a minute).The differences in the results show that the true motion amplitudes by the proposed system are valid at least at the 0.1 mm level accuracy.

Plate
No.
Min 1.20 2.17 2.53 2.17 1.30 Differences (LT -PS) [mm] 0.00 0.02 0.03 0.12 -0.02 Table 4. Differences in the estimated amplitudes by the laser transducers (LT in grey) and the photogrammetric system (PS in white) at five of the 13 offset witness plates

CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE WORK
The goal of this research work was to investigate the use of a photogrammetric system for targetless 3D reconstruction and precise vertical deflection measurements of a concrete beam loaded by a hydraulic actuator.The system was non-contact, and it was composed of off-the-shelf components such as digital cameras and projectors.The processing methodology was done at a relatively high level of automation with the only needed manual interaction being the selection of the region of interest in the images.The system was able to perform well while the object of interest was subjected to dynamic load cycling.The quality of the results was evaluated in terms of the image resolution, the combination of shutter speed and sensor sensitivity, and the repeatability of the estimated vertical motion amplitudes derived from images having the same acquisition settings.It was found that estimated vertical motion amplitudes had an RMSE of 0.03 mm, i.e., they were precise at a level below 0.1 mm.They were also checked against a more precise set of measurements coming from a set of laser transducers.The over RMSE there was 0.06 mm, which verified that the quantities estimated by the photogrammetric system were accurate at a level better than the 0.1 mm mark.
Future work will include performing more experiments with a higher frequency of loading, where newer cameras with a faster frame rate in continuous shooting mode will be used.In addition, an alternative method will be used for estimating the horizontal displacements of the offset witness plates and their amplitudes (if applicable).Finally, an algorithm for dense matching will be aimed at so that the use of the projectors will not be necessary anymore and continuous monitoring of deformation outside the laboratory can be achieved.
of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-5, 2014 ISPRS Technical Commission V Symposium, 23 -25 June 2014, Riva del Garda, Italy (c) Figure 1.Overview of the experiment setup (a), the hydraulic actuator and concrete beam (b), and close-up of one of the thirteen offset/witness plates with a control laser transducer installed underneath it (c)

Figure 3 .
Figure 3. Distribution of physical signalized targets used for the exterior orientation estimation first step of the point cloud processing is the segmentation.It is done, so that all points belonging to the top surface of the beam or to the same offset/witness plate are grouped together.The segmentation algorithm is performed by randomly selecting seed points and executing region growing based on a least squares plane fitting.Basically, a new point is added to a neighbouring plane only if its normal distance to the plane is within a set threshold(Al-Durgham and Habib, 2012).An example showing half the cameras and one of the projectors suspended from the support frame

Table 1 .
Summary of the image properties and the sinusoidal fitting results for the large, medium, and small resolution burstsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-5, 2014 ISPRS Technical Commission V Symposium, 23 -25 June 2014, Riva del Garda, Italy Test on the combination of shutter speed and sensor sensitivity Three of the processed medium resolution bursts varied in their combination of shutter speed (or integration time) and sensor sensitivity (or ISO value): 1/15 sec and ISO 100 (see Table6in the Appendix), 1/30 sec and ISO 200 (see Table8in the Appendix), and 1/60 sec and ISO 400 (see Table This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-5-195-20144.3.2