REVIEW ON PHOTOGRAMMETRIC SURFACE INSPECTION IN AUTOMOTIVE PRODUCTION

When purchasing a premium car for a substantial sum, first impressions count. Key to that first impression is a flawless exterior appearance, something self-explanatory for the customer, but a far greater challenge for production than one might initially assume. Fortunately, photogrammetric technologies and evaluation methods are enabling an ever greater degree of oversight in the form of comprehensive quality data at different automotive production stages, namely stamping, welding, painting and finishing. A drawback lies in the challenging production environment, which complicates inline integratability of certain technologies. In recent years, machine vision and deep learning have been applied to photogrammetric surface inspection with ever increasing success. Given comprehensive surface quality information throughout the entire production chain, production parameters can be dialed in ever tighter in a data-driven fashion, leading to a sustainable increase in quality. This paper provides a review of current and potential contributions of photogrammetry to this end, discussing several recent advances in research along the way. Particular emphasis will be placed on early production stages, as well as the application of machine vision and deep learning to this challenging task. An outline for further research conducted by the authors will conclude this paper.


Overview: Automotive Production
As the task at hand concerns automotive production, a brief, highly simplified breakdown of the four main steps in this process chain will serve as a basis for context.

Stamping (press shop)
• Sheet metal is stamped into body components 2. Welding (body shop) • Body components are welded together 3. Painting (paint shop) • Raw car body is painted 4. Finishing (final assembly) • Exterior and interior are finished and outfitted Beginning with a metal coil, metal blanks are cut out and passed to the press shop (stamping plant). In the press shop these blanks are passed through a series of stamping tools in a press line, with finished, unpainted components emerging from the end. These components are then passed to the body shop where they are welded (or otherwise joined) into component groups and finally a finished, unpainted body (body-in-white). This body then passes to the paint shop, where several coats of paint are applied (usually a base coat, color coat and clear coat). Finally the painted chassis is paired with a powertrain and outfitted with all interior and exterior components in final assembly.
While the engineering particulars of each of these processes are not directly pertinent, two factors are crucial regarding photogrammetric inspection; material specularity and cycle times. * Corresponding author 1.2 Critical Factors for Inspection 1.2.1 Specularity: The first critical factor, specularity, describes the reflective properties of surfaces. The higher the degree of specularity of a surface, the more reflective it is. The lower this degree, the rougher the surface will be. For stages 1 and 2 the metal is raw and unpainted (diffuse surface), whereas for stages 3 and 4 it has been painted (specular surface). This will remain relevant since different photogrammetric technologies perform better at different degrees of specularity.

Cycle Times:
The second critical factor, cycle times, is simply the amount of time needed for a specific production step. In stage 1 this cycle time is very low, as modern press lines can produce components every three to five seconds. For all remaining stages the cycle time is significantly longer, as components are joined together and progress as a complete body through the remainder of production, rather than as individual parts. This cycle time is generally in the order of one to several minutes, depending on manufacturer throughput. These two cycle times will be henceforth be referred to as vehicle cycle time and component cycle time. These times are critical, as inspection must be performed without extending them.

Problem Formulation
Given a basic understanding of the process of automotive production, as well as the main limiting factors for photogrammetric inspection, a problem formulation can be postulated. This is stated as the acquisition, evaluation and interpretation of all necessary data pertaining to component surface quality throughout the entirety of production. This means an inline (or 100%) inspection of each component at each production step, with quality information being passed downstream. This is particularly important as manufacturers seek to transition to a more data-driven means of production as part of a larger transition to Industry 4.0.

SURFACE DEFECT LANDSCAPE
Before discussing the various photogrammetric technologies suited to the task of automotive surface inspection, it is important to outline the landscape of surface defects which are to be detected. As the presented research is being conducted in cooperation with BMW of Munich, Germany, the BMW -specific list of defects is shown here in non-exhaustive, redacted form.   Notches are tiny craters in the surface. Waviness denotes a series of dents and bulges in succession. Pimples are clusters of high spots (dirt spots) on the surface.

Slip Lines
Slip lines are visible lines on the surface caused by bending components around tight radii. They occur at the border between material which has been stretched over a radius and material which has not. No image available.

AUTOMOTIVE METROLOGY SOLUTIONS
With the production and inspection tasks laid out, as well as the target defect landscape, the next step is the selection of a surface inspection system. Whilst numerous and broad in appearance, these systems can generally be grouped into four categories: 1. Laserscanning

Camera setups
• "Raw" camera inputs, end-to-end processing These technologies will be discussed in further detail in the following, including a brief technological description and contemporary automotive applications.

Laserscanning
The oldest technology of the four categories is that of laserscanning. Well-established within the geodetic, photogrammetric and aerospace communities (Spring, 2020), laserscanning enables the 3D reconstruction of real-world objects. The specific type of laserscanning used to this end, and the one most commonly associated with the term in general, is polar laserscanning. For surface inspection, however, a different variant of laserscanning has become more prevalent; laser line triangulation. A schematic of the working principle of laser line triangulation is shown in figure 9.  (Dorsch et al., 1994) Polar laserscanning acquires discrete 3D points by deflecting a laser beam with a system of mirrors and measuring the timeof-flight and/or phase of the beam along the current, known longitudinal axis. Conversely, a laser line triangulation scanner projects a transverse laser line (or a pattern thereof) onto an object and observes the projection of said line or pattern on the object via an offset camera (or cameras). This is beneficial for surface inspection, as any longitudinal deviation (∆z in figure  9) of the projected line, such as that caused by a local surface disturbance, is transposed to a transversal deformation (∆x in figure 9) of the line's image on the sensor. This allows for faster acquisition and a more direct inference of surface quality information, with acquisition generally taking several seconds to a minute in the case of a car body component. Regarding specularity, a diffuse surface is generally needed in order to properly observe the projected laser line with the camera sensor.
Laser line triangulation is already in use in automotive production, accomplishing tasks such as gap and flush inspection and weld inspection. The application for surface inspection is a newer one, with an example being the Robiscan system from Automation W+R GmbH (Automation W+R GmbH, 2021).

Fringe Projection
First proposed in (Wust and Capson, 1991), fringe projection has become a mainstay in automotive production facilities worldwide. This technology works by projecting a structured pattern or series thereof (usually sinusoidal fringes) onto a component and observing said pattern(s). A schematic of the working principle of fringe projection is shown in figure 10. Figure 10. Fringe projection (Lin et al., 2016) In figure 10, a sinusoidal pattern is projected forwards onto a component. The camera and projector are hereby arranged in a fixed configuration, with their respective focal centers forming a triangulation baseline. For each visible point on the component, the corresponding phase φ(x, y) along the pattern, as encoded in the intensity I(x, y), is observed and triangulated to via the known projector-camera baseline. This can be done statically with a single acquisition, or dynamically with multiple acquisitions via phase shifting. This technology can be used both for geometric acquisition as well as surface inspection, depending on the configuration of the sensor. An excellent, if slightly dated, overview of fringe projection technologies is given in (Gorthi and Rastogi, 2010). With regard to cycle times, static fringe projection can make single acquisitions in a sub-second timeframe, while dynamic fringe projection generally necessitates several seconds per acquisition. For automotive components, multiple acquisitions (views) are generally required in order to capture the entire part. Regarding specularity, fringe projection systems usually require a diffuse surface in order to observe the projected pattern, although recent advances such as laserlight projectors have allowed for a modest increase in manageable specularity.
A contemporary example of a static fringe projection sensor is given with the ABIS II system from Zeiss Optotechnik GmbH, which is specifically tailored to automotive body inspection (Zeiss Optotechnik GmbH, 2021). An example of a dynamic fringe projection sensor is given with the ATOS family of sensors from GOM GmbH (recently acquired by Zeiss), which are tailored towards geometric measurements but also offer surface inspection functionality (GOM GmbH, 2021).

Deflectometry
A newer method, specifically suited to the task of specular surface inspection, is that of deflectometry (Knauer et al., 2004). The principle of deflectometry can be intuitively interpreted as a conceptual inversion of fringe projection. That is to say that, like with fringe projection, deflectometry makes use of a structured (usually sinusoidal) pattern and a camera or set thereof. The inversion lies in the fact that the pattern is no longer projected forwards onto a component, but rather reflected backwards off of it from a screen. A schematic illustrating the working principle of deflectometry is shown in figure 11. Figure 11. Deflectometry (Zhang et al., 2017) In figure 11, a reference fringe pattern (top-left) is displayed on a TV-style screen, with its reflection off of a component being observed by a camera. The acquired, deformed fringe pattern (top-right) then serves as a basis for evaluation. At any point on the component, any deviation in the surface normal Θ, such as the deviation caused by a local surface defect, will lead to a doubled deviation 2Θ between the nominal incidence ray wi from the surface to the screen and the actual incidence ray w i , leading to fringe deformation. Surface evaluation is then conducted in Fourier space of the deformed fringe pattern(s).
As with fringe projection, deflectometry can either be conducted with a single acquisition of with a series thereof. However, single-acquisition deflectometry requires strict boundary conditions, such as a flat surface and/or precise a-priori knowledge of component geometry. As is such, phase shifting deflectometry is usually standard practice. This also means that cycle times span the order of several seconds per component as phase shifting is conducted. Regarding specularity, the fact that this technology utilizes reflection rather than projection means that is specifically suited for specular surfaces, such as painted components. A contemporary example of a deflectometric inspection system is given with the reflectCONTROL system from Micro-Epsilon GmbH (Micro-Epsilon GmbH, 2021).

Camera Setups
The youngest method, and the most arbitrary in its definition, is that of camera setups. It is here that a disambiguation of the term is necessary. In the context of photogrammetric surface inspection, a camera setup denotes any system that makes use of digital cameras without the use of a structured pattern. In other words, there is no triangulation, phase analysis, or Fourier transformation taking place, but rather an analysis of raw 1 camera data in image space. A schematic showing the functional principle of a raw camera setup is shown in figure 12.
1 Image preprocessing notwithstanding In figure 12, a component passes under a camera (or a set thereof). The camera then acquires one or multiple images of the component, usually in conjunction with a strong light source. These images are then passed to a computer, where a wide variety of backend computational methods can be used to perform the inspection. After the images have passed through the inspection backend, a report is generated for each component. These reports can vary from a binary "OK" / "not OK" distinction to a detailed report including specific surface defect information, the latter of which being the preferable case. It is here that an important paradigm distinction should be noted, namely that between the more classic deterministic approaches and younger methods involving deep learning.
With regard to cycle times, camera setups outperform all prior technologies, as image acquisition time corresponds to the shutter speed of the camera or cameras. The aforementioned use of a strong light source also aids in reducing acquisition time. The main time constraint then becomes the speed of backend computation. Regarding specularity, camera setup based methods can be and have been implemented both for diffuse and specular components, as will be discussed later.

Overview
Given the previous overview of automotive production and metrological solutions, one is left with four main inspection tasks (one for each production step), as well as four categories of sensor to choose from. Taking the the issues of specularity and cycle times into consideration simplifies decision making, allowing a grouping into the following scheme (figure 13): This categorization of photogrammetric metrology solutions regarding their aptitude for each production step has been derived from prior and contemporary research by the authors in cooperation with BMW in Munich, Germany.
The first noteworthy observation to be made is that there exists at least one proven technology for each production step with the exception of the press shop. For this reason, particular emphasis will be placed on the state of surface inspection technology in this early production stage. It should also be noted that, in this context, deflectometry is restricted to the proven and commercially available optical variety.
For the first two production stages, where the target material exhibits diffuse reflective properties, both laserscanning and fringe projection come into consideration, as these forward triangulation methods generally necessitate a diffuse surface to function. For the latter two stages, deflectometry becomes the only proven viable option, as it is specifically designed to work for specular surfaces, as previously discussed.
Another observation is that the fourth sensor category, camera setups, has been deemed potentially feasible for each production stage. This is owed to the broad range of specularity that the method can deal with, as well as the fast acquisition time of digital cameras under strong lighting conditions. There are, however, limitations on the spectrum of observable defects with this method, as will be discussed later.
The final consideration before selecting an inspection paradigm is that of cycle times. For steps 2-4 one is bound by vehicle cycle time, ranging from the order of one to several minutes, as the components have been grouped together into a car body and progress synchronously. This allows a relatively long timeto-measure and with it the potential use of any eligible sensor category.
The stamping process is, conversely, bound by component cycle time, with intervals in the order of only a few seconds. Each interval, a new component is placed onto the outfeed belt of the press. It is within this brief interval that the inspection system must acquire, evaluate and interpret the data for the new component. Furthermore, harsh vibrations in the press and other disturbances cause issues for laser scanning and fringe projection, which acquire data over a longer time frame. Camera systems, conversely, can acquire data within a single shutter frame and are more convenient to install in press lines. This leaves the stamping process as the only production step without readily available surface quality control solutions. 2

STATE OF RESEARCH
So far the state of photogrammetric surface inspection in automotive production has been discussed from the perspective of an automotive manufacturer looking for proven, commercially available solutions. As already eluded to, however, there is ongoing research in the field of industrial surface inspection which pertains either directly or indirectly to the task at hand. Several examples from this contemporary research will be discussed in the following, sorted in accordance with the inspection task that they most pertain to. This review will serve as the basis for future surface inspection research, which will be presented and outlined in sections 5 and 6.

Press Shop Inspection
As mentioned prior, press shop production entails the creation of car body components from sheet metal blanks. This is the first stage where production-related surface defects, like those shown in section 2, can arise. A significant problem for photogrammetric inspection lies in the abundance of disturbances such as vibrations and conveyor belt movement. For this reason, press shop surface inspection is generally conducted visually and haptically by specialized workers, with sensor measurements being taken off-line at regular intervals, where these environmental and temporal constraints do not apply. This leads to a lack of comprehensive sensory quality information.
This deficiency is addressed in (Purr et al., 2015), who outline a general roadmap to a data-driven stamping process through continuous material parameter and attribute observation, albeit without an explicit discussion of necessary sensory advancements. Such advancements are discussed in (von Enzberg and Al-Hamadi, 2019), who utilize deformation simulations and model-based analyses to increase the accuracy of press shop inspection using contemporary sensors. A further example, this time incorporating the use of a novel sensor, is given with (Höfer and Beyerer, 2016), who utilize an infrared deflectometric approach, allowing for the deflectometric inspection of diffuse components. This alleviates the restriction of deflectometry to specular surface inspection as discussed in section 3.3, yet does not not explicitly address inline integration. Another novel sensor is proposed in (Strohmeier et al., 2019), utilizing an inverse fringe pattern in order to inspect for "necking" (a structurally relevant thinning-out of material and indicator of potential cracking) directly within the press line. This system, however, does not inspect for cosmetic surface defects. An inline inspection system for cosmetic surface defects is proposed in (Block et al., 2020), who use a raw camera setup in conjunction with deep learning for surface defect detection. While cycle times and vibrations are handled by this method, the spectrum of defects is restricted, focusing on imprint defects, analog to the "pimples" mentioned in section 2.
In summary, press shop surface inspection is very much in need of further research. As of writing, all relevant publications are forced to trade off between acquisition area, acquisition speed and the spectrum of detected defects. While such a trade-off is inherent to all metrological tasks, in the context of press shop inspection there is still no optimum which can satisfactorily meet the demands of a premium automotive manufacturer.

Body Shop Inspection
The constraints on body shop inspection are similar to press shop inspection in terms of material, as the body-in-white inherits its diffuse surface properties from the underlying stamped components. Unlike press shop inspection, however, the issues of vibration and cycle times are vastly reduced, as the car bodies progress steadily and uniformly through the body shop. This allows the successful use of techniques such as laser-line triangulation and fringe projection. As body shop inspection is considered a well-understood discipline, research in this area has stagnated, as researchers gravitate towards the younger, less-established disciplines of inline press shop and paint shop inspection. Advancements in body shop inspection now arise primarily through enterprise development of commercially available systems, such as the laserscanning and fringe projection systems mentioned in sections 3.1 and 3.2, respectively.

Paint Shop Inspection
The paint shop presents the first major junction for photogrammetric inspection. As mentioned in section 1.2.1, it is here that the components go from their initial, diffuse state to their final, specular one. It is also here that painting defects such as runs, inclusions and craters can occur. Furthermore, certain defects from prior processes can become significantly more noticeable after painting (e.g. dents and bulges) while others become minimized or even entirely covered (e.g. scratches). As discussed in section 3.3, deflectometry (of the optical variety) has emerged as the dominant technology for painted-body inspection, as it is tailored specifically towards specular surfaces. While industry-grade deflectometric solutions already exist to perform this task, research in this field is ongoing.
A recent example in photogrammetric painted-body inspection research is given with (Chang et al., 2019), who use a raw camera setup with a bright illumination source in conjunction with deep learning for surface defect detection. In this case the defect spectrum is simplified to concave and convex defects, such as dents and bulges. A further camera-based painted-body inspection system utilizing deep learning is proposed in (Park et al., 2020), who utilize LED mach bands and region-based convolutional neural networks (CNNs) in order to detect dents on painted body panels. A further example, this time reverting from the newer concept of raw camera setups to the established principle of optical deflectometry, is given with (Zhou et al., 2020), who couple a custom deflectometric setup with a deep learning backend for paint defect detection. This method returns a binary defect map, with no explicit defect classification taking place. Another deflectometry-based research example is given with (Arnqvist et al., 2021), who utilize a probabilistic weighted nearest-neighbor (k-NN) classifier to generate defect maps from fused deflectometric images, resulting in classified patches of three different defect types.
This review on painted-body research results in two preliminary conclusions. The first is that there is an ongoing parallelization of research between raw camera setups and advancements based on deflectometry. The second is that the limiting factor appears to be the spectrum of detectable defects, with most research focusing in on a subset thereof (generally two or three types). In fact, even when exploring surface defect detection outside the automotive realm, finding methods which can detect more than six distinct defect types becomes increasingly difficult. An example of this edge case is given with (He et al., 2019), who utilize semi-supervised learning in combinations with generative adversarial networks (GANs) to detect six different defect types on hot-rolled steel strips. If the entire defect landscape from section 2 is to be detected, further research is absolutely necessary.

Final Assembly Inspection
Final assembly inspection constitutes a final check of the finished automobile before delivering it to the customer. Final assembly is also the last stage where pre-delivery surface defects can occur, such as dents or scratches from unintentional contact with the vehicle. This inspection includes not only surface inspection, but also the inspection of gap and flush tolerances, exterior and interior fixture tolerances, and alignment checks. As the research presented pertains solely to surface inspection, and as the surface properties of vehicles in this stage do not change after painting, the same (primarily deflectometric) systems can be used as in the paint shop, requiring no further discussion.

WORKFLOW AND QUESTIONS
With the previous review of surface inspection technologies, solutions and research serving as a basis for understanding, the next step becomes the drafting of a potential surface inspection workflow which makes proper use of recent technological advancements. Such a workflow will be briefly outlined in the following, as well as some of the challenges and open questions which will be addressed by the authors in an ongoing research project. The workflow in question will pertain primarily to the press shop, the area most in need of further research and the core focus of the project.
At the end of the press line, where inspection has hitherto been conducted manually, a new sensor system is to be installed in the medium term. This system must be able to inspect diffuse metal components in this challenging production environment. It must therefore be able to handle vibrations and conveyor belt movement and be able to deliver comprehensive surface quality information for every 3 produced component. The main technologies which come into question are: static fringe projection, laserscanning, infrared deflectometry and camera setups with machine learning backends. On the basis of sensor system outputs, components can be flagged for rework, rejection, or direct progression to further production. Furthermore, this quality data can then be fed into an inter-process database, allowing an inference of surface behavior as components and vehicles move downstream. This will close a significant gap in the attainment of a truly data-driven production process and provide a valuable contribution to the ongoing transition to Industry 4.0.
This workflow is faced with multiple challenges and inherently raises several research questions. If a proof of concept and working prototype are to be developed, these must be clearly stated and systematically addressed. The most pertinent open research questions are postulated as follows: 1. How should defects be prioritized?
• As the landscape of surface defects is broad, it is prudent to rank them in accordance with their likelihood of appearance as well as the associated rework and rejection costs.
2. What are the exact environmental constraints?
• As the production environment is very challenging, a precise, quantifiable understanding of disturbing influences is required.
3. How does each technology perform regarding this task?
• This review demonstrates the potential feasibility of several different technologies to perform the task at hand. This needs to be more precisely quantified in order to select an ideal technology.
4. Which evaluation backend should be used?
• There exists a broad spectrum of evaluation backends for surface inspection. Experimentation must be conducted using several different varieties, with subsequent optimization leading to a deployable, optimal solution.

OUTLOOK AND CONCLUSION
In order to realize the workflow and answer the research questions outlined prior, a doctoral research project is currently underway by the authors of this work. In cooperation with BMW of Munich, Germany, a proof of concept is to be delivered by means of simulation, as well as a physical sensor prototype. The research will be subdivided into four stages, which are briefly outlined as follows:

Preliminary research
• Establish the state of research in photogrammetric surface inspection in automotive production, as presented here.

Modelling environmental constraints
• Determine and quantify environmental influences and disturbances as a basis for simulations.

Sensor evaluations
• Evaluate the performance of the aforementioned technologies with regard to their aptitude for press shop inspection.

Backend development
• Develop and optimize software backend, particularly though the use of neural networks, for surface defect detection and classification.
The doctoral research project at hand will serve to further the state of photogrammetric surface inspection and achieve new insights regarding the behavior of various technologies when applied to this challenging task. The authors also seek to solidify the place of photogrammetry in automotive production throughout and beyond the fourth industrial revolution.
In conclusion, the technology for photogrammetric surface inspection in automotive press shop production clearly exists. A turn-key solution to this task, however, does not. This can be explained intuitively, as sensor vendors have trimmed their enterprise systems more towards geometric measurements, which have a broader spectrum of application. Nonetheless, focus on surface inspection has gained significant traction in recent years, as this paper has demonstrated. New sensor technologies and advancements in machine learning are also enabling the detection of ever smaller defects with ever greater accuracy. It is therefore a logical and necessary course of action to pursue a symbiotic advancement of research perspectives in this field and industrial interests. The authors look forward to communicating results throughout the duration of the project.