METROLOGY ENABLED REFLECTION TRANSFORMATION IMAGING TO RECONSTRUCT LOCAL DETAIL IN MANUFACTURED SURFACES

Understanding the performance of large high performance manufactured structures can require highly accurate dimensional measurement across large volumes with the often conflicting capability to record critical parts of the structure in fine detail. Examples include turbine blades, aircraft wings and off-site manufactured modular structures assembled on-site for city, energy and transport infrastructure. Established large-volume industrial metrology systems such as laser trackers and photogrammetry partially meet the need through the measurement of targets and reflectors, but are limited in capability to record high density local detail needed to capture the finest manufactured features. Whilst large-volume surface sensing is possible with laser radar, photogrammetric pattern projection and contact probing for example, the detail required at a local level typically demands local sensing which generally takes the form of a tracked sensor such as a triangulation laser scanner or hand held touch probe. Local sensing systems face challenges where surfaces have fine detail of similar magnitude to the local sensing system sampling capability and particularly for optical sensors where the light reflected back to the sensor by the surface includes specular reflections compounded by local geometry. This paper investigates how Reflection Transformation Imaging (RTI) with a dome camera and lighting system might be calibrated, characterised and tracked as an alternative technology that is more robust to material surface properties and capable of very fine surface detail capture. Laboratory results demonstrate the capability to characterise and locate the dome to sub-millimetric accuracy within a large-volume tracked space to achieve local surface sampling at the 30m x 30m level. A method utilising sparse touch probe points to seed conversion of low and high frequency normal maps into a common 3D surface is explored with local agreement with laser tracker surface probe check points to the order of 30m.


Context
Large high performance manufactured structures comprising assemblies of smaller structures utilise varying joining methods to connect individual components together. Tight tolerances in each sub-structure ensure that the surface geometry and connection interfaces of the assembled structure meets design objectives. Understanding the performance of such structures requires highly accurate dimensional measurement across large volumes with the often conflicting capability to record critical parts of the structure in fine detail. Examples include turbine blades, aircraft wings and off-site modular structures subsequently assembled on-site for city, energy and transport infrastructure.
Established large volume metrology systems (>50m 3 ) such as laser trackers and photogrammetry are optimised for the measurement of discrete targets such as corner cubes, spheres and retro-reflective markers, but are less well developed for local detail recording needed to capture the finest manufactured features. Whilst large volume surface sensing is possible, for example with: laser radar; photogrammetric pattern projection and; contact probing, the detail required at a local level often demands local sensing which generally takes the form of a tracked laser scanner or hand-held probe. These systems all face challenges, for example where surface detail is of the same order of magnitude as the probing capability, or where light received at the sensor includes specular components compounded by complex local geometry and varying surface finish.

Prior Work
Reflection Transformation Imaging (RTI) is widely used in the heritage recording sector where the emphasis is on qualitative visualisation to enhance legibility or better understand fine detail. The dome imaging technique in which a single camera views a surface from a single position whilst the angle of lighting is varied was initially developed by Malzbender et al (2001). The algorithm fits a biquadratic polynomial function to the set of intensities at each pixel location. Hemispherical harmonics were developed as basis functions by Gautron et al (2004). Interactive viewer software uses the cursor position, representing the coordinates of a 'virtual light source', to generate the intensity of each pixel as if it had been illuminated from that direction. The RTI method encodes surface detail at the pixel resolution of the camera, and hence with a suitable lens can provide excellent fine detail. Examples of applications are in the representation of coins (Mudge et al, 2005) and in archaeology (Earl et al, 2010). RTI can also provide information on physical changes in the surface geometry of the object itself, such as cockling and distortion of parchment (Manfredi et al., 2013). The photometric 'shape from shading' technique was introduced by Woodham (1980) in the context of computer-based image understanding. It estimates the surface orientation as the normal vector at each point of the object surface, using the principle that the intensity of the reflected light depends on the angle of incidence of the light onto the surface. With a perfect Lambertian surface and in the absence of noise, three intensity values from non-coplanar light sources are sufficient to solve for both normal direction and surface albedo. In practice, better results can be obtained for noisy image data by taking the median of results for many triplets of light sources (MacDonald & Robson, 2010). For a non-Lambertian surface the 'bounded regression' method selects a subset of the light sources, excluding those close to the specular angle or where the surface is in shadow (MacDonald, 2014). An alternative approach uses least median of squares for the regression and Gaussian radial basis functions with Tikhonov regularization (Zhang & Drew, 2014 (Coules et al 2019). Results demonstrated the surface detail that can be captured when sampling surfaces in the 1m to 1mm range and how these map into a range of engineering inspection tools.
Work described in MacDonald et al (2016) on alternative sensors for surface metrology compared the accuracy of 3D surface reconstruction between image sets of a small metric test object taken in an illumination dome by photometric stereo and improved structure-from-motion (SfM) against reference point cloud data from a 3D colour laser scanner. Connected work also showed how photogrammetry could be used to determine more accurate 3D coordinates of the dome lights, by using the lamps as targets (MacDonald, Ahmadabadian & Robson, 2015).
Integration of surface normals to reconstruct an accurate 3D surface remains an active research problem. Recent work has shown how photometric normals can be combined in the spatial frequency domain with a basic 3D 'support' to give a surface terrain (MacDonald 2015). A similar technique has been applied to the modelling of Roman coins, deriving the support surface from 3D laser scan data (MacDonald et al, 2017).

Work described in this paper
In this paper we build upon the qualities of the high frequency spatial detail attainable from illumination dome imagery to ascertain what is possible for local surface recording on a test panel located at UCL's large scale robotics laboratory at HereEast on London's Olympic Park (Figure 1). Figure 1. The 0.5m radius UCL illumination dome. Coded targets to characterise and track the dome are visible at the corners of the dome base plate and on the dome surface.

Overview
The experimental part of this paper is in two parts: 1) characterisation of the imaging dome to establish the internal and external geometry of the sensor and to support its tracking across a large volume and; 2) an investigation of the capability of the imaging dome to capture local surface form not just as normal vectors, which is well proven, but as a 3D point cloud. Laser tracker 6DoF surface probe data is used to provide a low density reference both to seed the 3D shape and also to provide independent check points to quantify the low frequency components of the reconstructed surfaces.

Surface dome image capture
The UCL dome consists of a Perspex hemisphere approximately 0.5m in radius, painted matt black both inside and outside. The flange is supported by an aluminium frame, attached to which a gantry made from steel tube provides a stable camera mount. The dome is configured with 64 electronic flash tubes which can be fired sequentially in synchronisation with images captured by a camera mounted at the nadir of the hemisphere, looking downwards onto the object under investigation. The dome and its calibration have been described in detail in MacDonald et al 2015. Whilst the exact camera and lens can be varied to suit the application, for this new work the sensor dimension, lens settings and taking distance to the surface have been updated and optimised to produce a design pixel footprint on the surface of 30m x 30m.
The dome imaging camera used throughout was a Nikon D600 DSLR fitted with a Nikkor 105mm f/2.8 G series macro lens. The sensor gives an image of 6016x4016 pixels, with a pixel size of 5.95 µm. With both autofocus and VR motor systems turned off, the camera was focussed on a detailed portion of the surface to be recorded and then focus locked to ensure consistent photogrammetric geometry within the camera. For each surface to be inspected, an image set comprising 64 exposures with sequential electronic flash illumination source directions was taken. An exposure of 1/50th sec for each image ensured that the complete sensor area was exposed to the light. An aperture setting of f/8 coupled with an effective ISO of 200 gave a wide enough latitude over the range of illumination angles to record the surface without loss of highlight or shadow detail.

Dome characterisation
Metrology grade reconstruction of surface normals by the imaging dome requires that the geometry of each of the components: light sources; camera pose, internal camera geometry, position and orientation of the dome at each captured surface location are known.

Macro-camera calibration
is challenging because of the long principal distance vs the dimensions of the sensor needed to achieve the design pixel footprint. This makes recording 3D target fields in the object space challenging because of narrow angle of view and a small depth of field. In this case we utilised a purpose-built "miniature Manhattan" artefact equipped with circular targets and target codes distributed across a 90x90x60mm working volume ( Figure 2). After surface recording, the locked-focus camera was removed from the dome and 19 convergent images taken of the Manhattan artefact and its 111 circular targets from a matching object distance (~620 mm) so that the Manhattan object base plane was in sharp focus. Nominal coded target 3D coordinates from the Manhattan object design were used to determine the orientation of each image. Since the lens needs to be moved away from the image plane to achieve sharp focus, the thin lens equation (1/f = 1/u + 1/v) is useful in estimating a starting value for principal distance. In this case a focal length (f ~105mm), with taking distance (u ~0.62m) gives a principal distance (v ~126.4mm). Output from a self-calibrating bundle adjustment with VMS software (Table 1)  Despite the convergent network geometry, the narrow viewing angle led to estimates of tangential, affinity and orthogonality that were highly correlated with principal point offset limiting the lens model parameters that can be reliably estimated. Notably for this particular lens, the effects of these parameters are below the pixel level. An image residual of the order of 0.5m is typical for our target based photogrammetric work with the D600. However relatively poor target image quality due to the extension of the Manhattan artefact beyond the lens depth of field resulted in a larger RMS image residual of 2m. Fortunately for this application the ~1/3 pixel image residual balances with the RTI method and its surface normal computation requirement at individual pixel level.

Determining dome geometry:
Photogrammetric application of the dome requires a knowledge of the camera pose with respect to the surface being observed and the constellation of lights in the imaging dome. Estimation of camera pose both within the dome coordinate system and to a wider laboratory coordinate system is complicated since the dome prevents direct line of sight surface observations from external cameras which might otherwise be used with a common object in a combined network with the dome camera imagery.
A solution is to place photogrammetric targets on both the inside and outside of the dome, to coordinate these along with the light sources and a scale bar with a network of convergent images taken in the round. In this case a self-calibrating bundle adjustment comprising 137 images of 135 retro-targets captured with a Nikon D500 camera and 18mm AF-D wide angle lens was computed to determine dome geometry (Figure 3). Outputs from the dome geometry bundle adjustment were the coordinates of retro-targets located on the dome and the centroid coordinates of the lamps within the dome to mean coordinate precisions of 50m and 250m respectively. These one-sigma precisions also include any flexing of the physical structure of the dome as it was moved between two imaging positions to capture both upper and lower surfaces. Lamp coordinates are less well defined than the circular retro-targets as the light output emanates from a cylindrical tube rather than from a small circle.

Dome camera orientation:
The final characterisation step establishes the orientation of the dome camera and its relationship to the targets on the exterior of the dome. An aluminium reference target plate (Sargeant et al 2016) equipped with a matrix of small photogrammetric targets was drilled to fit six 1.5" Hubbs split bearing photogrammetric targets (Figure 4). The unflat aluminium plate provides a multi-scale reference object which can be measured both by the dome camera using the small targets and by large volume metrology systems in the laboratory on the basis that each of the drilled holes can repeatedly support a 1.5" sphere. The plate was positioned on a stable base within a wider field of twenty 1.5" metrology nests. All nests were coordinated into our common laboratory coordinate system with a Hexagon AT960 laser tracker. The 1.5" laser tracker SMR was also used to create a reproducible offset target location for each of the six drilled holes on the aluminium plate. The nests and plate holes were then populated with 1.5" Hubbs split-bearing photogrammetric targets so that a photogrammetric network of both the plate and the nests could be used to establish the locations of the smaller photogrammetry-only targets located on the reference plate in the tracker defined coordinate system.
Without moving the plate, the dome was positioned so that the centre of the plate occupied the field of view of the D600 dome camera. An image of the plate was taken with the dome camera whilst a second hand-held DSLR camera was used to take a photogrammetric network of the targets on the outside of the dome in combination with the wider target field. A bundle adjustment combining all of the targets and image observations from both cameras was then computed. The adjustment output is a set of coordinate estimates on the common laboratory datum for: the dome; the position and orientation of the dome camera; and the location of each light within the dome. ( Figure 5). As a check, laser tracker nest coordinates for the seven common points on the reference plate were compared to those computed in the photogrammetric network in a least-squares best fit.
Photogrammetric data agreed with a maximum error of 80m and an RMS error of 50m.

Given successful characterisation, the characterised
dome can now be moved anywhere within the coordinated laboratory space and its location and pose determined with a photogrammetric network to coordinate the coded targets on the exterior of the dome. In the UCL laboratory this is achieved with a flexible set of ceiling and wall mounted cameras. Each camera references the datum defined by laser tracked target nests mounted on the laboratory floor to provide a flexible image network covering a 10 x 8 x 12m working volume. This solution makes it possible to track a characterised sensor, robot or object at the camera system frame rate of 2 to 4 fps with a sub-mm capability throughout the working volume (Robson et al 2019).

RTI SURFACE RECONSTRUCTION
For the second part of this paper, RTI outputs were produced by imaging the aluminium reference surface from a single location. Per-pixel normal maps (Section 1.2) were produced in each case to reveal local detail and to provide an indicator of the capability of the dome imaging sensor. Surface reference probe data could then be used to seed the output normal maps with the sparse coordinate data needed to reconstruct localised 3D surfaces.

Reference surfaces and direct referencing
Reference surfaces for this on-going work were captured using a Hexagon AT960 tracker with an optically tracked Hexagon Tprobe equipped with a 6mm diameter stylus. Surfaces include the aluminium reference plate which is the focus of this paper and outputs from a variety of manufacture and fabrication processes.
The use of physically common target centres across multiple metrology sensors allows direct point to point referencing avoiding the use of point cloud registration processes which cannot deliver accurate solutions for the geometrically smooth structures investigated in our work. For example, the photogrammetric targets on the characterised dome used in combination with a set of split bearing targets in laser tracked nest locations (determined in section 2.3.3) allow direct referencing of normal maps from the dome camera into the laboratory coordinate space. This relationship can be used to inter-compare and integrate dome-derived surface information with laser tracker surface probe data of the same area. The referencing method is appropriate for both single and multiple placements of the dome imaging system.

Reference plate surface reconstruction
3.2.1. Imaging and normal map generation: Dome image sets, each comprising 64 exposures with systematically changing illumination source direction, were captured with the Nikon D600 DSLR camera and 105mm macro lens (Section 2.2). The image area of the reference plate represents object dimensions of approximately 180 x 120 mm, at a resolution of 33.5 pixels/mm. Albedo and surface normals (Section 1.2) were computed using the bounded regression technique (Figure 7). Immediately apparent in the surface normal map are the edges of the square and circular patches corresponding to the edges of the 0.13mm thick retro-target material base (Figure 12). Secondly the high contrast random pattern on the reference plate, optimised for structural testing using digital image correlation techniques, is absent from the surface normal image. This is highly encouraging since the underlying printed surface is smooth to the touch.

Surface reconstruction:
Sparse seed points are needed to convert the normal map into a surface. Here registration between the laser tracker T-probe data and the dome camera pose can now be utilised to back project the 3D probe coordinates into the image plane of the camera (Figure 8). There isn't an exact pixel correspondence between seed probed points and image. In order to extract a set of sparse seed points suited to surface reconstruction, a sampling grid of 13x9 points was defined over the image area giving 12x8 square cells of 500 x 500 pixels. The probe point nearest to each (ideal) grid point was selected as a 'key point'. (Figure 9 top). The 533 remaining probe points provide check points independent of the reconstruction method. An exaggerated Z axis 3D plot of the surface with the dome image coverage is given in Figure 9 (bottom). Z coordinates in the image area range from -239 to -32 m. Figure 9. Top: Probe (red) and target (green) measurement coordinates within the image, and selected key points (blue). Bottom: Exaggerated Z axis plot of probe points on Reference Plate. Red points are within image area. Green points are targets. Blue points are the selected key points.
A surface was fitted (by the Matlab griddata function) to the selected key points, using the image pixel coordinates and the measured depth (Z) value. The 'v4' method was selected to fit the surface via biharmonic spline interpolation. This surface provided a 'support' for photometric mapping (Figure 10). The correspondence between measured and fitted support surface values is shown in figure 11, within a horizontal band at Y = 1500±50 (Figure 9).The correspondence is exact at the key points (blue circles) because the selected fitting method forces the surface to pass through the seed points. All other probe depths are within 20m of the estimated surface.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 11 Measured and fitted depth values within a horizontal band across the support surface.
The error distribution was calculated for differences between the Z values of the support surface and the probe data check points. Non-zero differences occur for all 533 check points. The mean absolute error between the support and measured probe points (excluding key points) was 13 m, with a st.dev of 19 m. The sampling density of the surface normal data enables noise mitigation by frequency analysis. In this case a Fourier Transform of the support surface was taken to obtain the low spatial frequencies in the surface reconstruction which correspond to the relatively smooth shape of the underlying surface. Partial derivatives (gradients) P and Q were calculated at every pixel from the photometric normal vectors derived from the dome image set (Section 1). A separate gradient value was computed from the normals for every pixel. These are illustrated in Figure 12 in a small detail of 9mm x 9mm cropped from the left side of Figure 7, showing one of the photogrammetric retro targets on the plate. The gradients for the whole image were transformed to enforce the integrability constraint, using the method of Frankot & Chellappa (1988) and projected into the frequency domain. Figure 13 Distorted surface arising from direct reconstruction from photometric normals.
A next logical step would be to reconstruct the surface directly from the photometric gradient spectrum, via an inverse Fourier transform. Unfortunately this process yields a surface that is grossly distorted, as if it were mapped onto an inflated rubber balloon ( Figure 13). The distortion is caused by inaccurate representation of the low spatial frequencies in the power spectrum. In this case the overall amplitude of the reconstructed surface within the image area, from lowest to highest points, is about 30 mm, when it should about 0.2 mm. A more sophisticated method is needed which can accommodate the discrepancy in the spatial frequencies between the true surface normals and the photometric gradients.
The chosen reconstruction method has to account for two populations of data, the low frequency shape of the 'support' surface from the probe points and the high frequency form of the local perturbations expressed in the photometric normals. Their different characteristics are highlighted in their respective frequency spectra (Figure 14). The local blending method used to combine the low spatial frequencies of the surface from with the higher spatial frequencies from the photometric gradients used in this paper draws on that used previously in the reconstruction of a small terracotta roundel of Chopin (MacDonald, 2015). In that case 16 probe points with measured depth were used for construction of the 3D support surface.
In the local blending method, the power of the high frequency components is adjusted to match that of the low frequency components within a circle of given radius around the centre of the shifted frequency plane. The two components are then blended by linear interpolation using a radial alpha function derived from a logistic function. The blending occurs in this case between a radius of 1.5 pixels and 4.5 pixels, corresponding to spatial frequencies of 1.5 and 4.5 cycles per image width. Figure  15 demonstrates its implementation over a 61 x 61 pixel window. The critical step is choosing the spatial frequencies for the overlap region of the blending function. The lower limit, defined by the radius of the inner circle in the frequency plane, needs to be sufficient to include all of the low frequency information in the support surface, which depends on the number and spacing of the key points and the interpolation method. The blending region, defined by the range of spatial frequencies between the inner and outer circles in the frequency plane, needs to be broad enough and smooth enough to prevent discontinuities in the frequency domain that could cause ringing or other artefacts in the spatial domain.
Applying an inverse Fourier transform to the blended spatial frequencies yields the reconstructed 3D surface with the underlying form of the support and superimposed fine detail from the photometric gradients (Figure 16 top).
The reconstruction from the combined datasets exhibits the expected depth range across the plate of 0.236mm established by the probe data and as can be seen in the extracted profile ( Figure  16 bottom). The method follows the support surface but adds fine detail derived from the photometric normals, which have been shown through heritage recording to provide excellent capability in recording the finest surface marks. The error distribution was calculated for differences between the Z values of the reconstructed surface and the measured probe data, as shown in Figure 17. The mean absolute error between reconstructed and measured points was 20 m, with stdev 29 m.
In this case the reconstructed surface no longer passes exactly through the key points used to seed the surface due to the influence of the photometric fine detail, but the errors are well distributed across all of the probe points. Figure 17. Histogram of differences between reconstructed surface and 533 measured probe points.

SUMMARY
This investigation has updated the imaging equipment in UCL's RTI imaging dome, calibrated it to take into account the internal geometry of its full frame DSLR camera and 105mm macro lens, determined the geometry of its 64 internal light sources to ~250m and the pose of its internal camera in a common network adjustment that connects internal and external features defined by coded targets to a precision of ~50m. A constellation of nine external coded photogrammetric targets on the RTI dome can now be employed within a low-cost large-volume photogrammetric tracking system to determine the position and pose of the dome as it is used to sample manufactured surfaces throughout a 10mx10mx8m factory-like space.
A best-fit analysis between common nest locations computed from laser tracker and photogrammetric observations to a target plate imaged within the dome produced agreement to better than 100m. This permits laser tracker probe measurements on the plate to be directly used without further registration both as reconstruction seed points and a check on 3D surfaces of the plate computed from RTI. This has removed the need to resort to best fitting between point clouds which is often the limiting factor in making accurate inter-comparison of surface measurement data without common reference points.
Whilst surface normal and fine detail from the RTI system can be registered with the manufactured surface, the integration of surface normals to reconstruct an accurate 3D surface without seed data remains a research problem. In this paper we have taken forward our earlier work in heritage recording to show how seed points and frequency-based filtering can produce a high quality 3D surface. In particular the advantages of blending lowfrequency normals from normal maps and high frequency normals from photometric stereo offer significant opportunities given the surface fitting that can be achieved. Note that the method employed in this study was a "one shot" merge of the probe data with the photometric normals. The quality of result could be further improved by application of an iterative technique using the errors at each step to adjust the gradients.
The next step, once our laboratories reopen following Covid-19, is to use our low-cost large-volume photogrammetric tracking system to compute the pose and location of the imaging dome as it is moved over larger manufactured surfaces. This will enable us to tile individual RTI outputs together in order to demonstrate the accurate capture and representation of some of the finest marks and features on 8m+ manufactured and natural objects.