A COMBINATION OF ADVANCED PHOTOGRAMMETRY AND LASER TRIANGULATION TECHNIQUES (VIDEO LASER SCAN TM) FOR THE FREE-FLYING 3D INSPECTION OF SUBSEA ASSETS

Over the last years, High-Definition (HD) video cameras have made visual inspections of subsea assets more efficient and a source of massive and critical information. But in many cases, this visual information is incomplete without dimensions and geometry and can be biased by the wrong optical perspective, the lack of scale, or bad water clarity. This paper will describe the development of a new Technology, called Video Laser ScanTM (or VLSTM), based on the combination of Photogrammetry and Laser Triangulation techniques, that provides additional Intelligence to such subsea inspections. In addition, we will highlight the innovation recently brought by Artificial Intelligence and 3D Holograms.


INTRODUCTION
There have been many phases in the development of Photogrammetry and applications to industrial 3D Inspection Projects. However, we can undoubtedly identify three significant steps: the invention of Photography (Niepce & Daguerre, 1830), the first measurements from images (Aimé Laussedat, 1849), and, more recently, the advent of the Digital Era (Computers, Algorithms, Digital Cameras, CAD software). Now is an exciting time for a significant new step around software development and technologies such as Image Processing, Artificial Intelligence (AI), Virtual Reality (VR), and 3D Holograms. Photogrammetry is also becoming a critical tool for creating virtual environments, such as the "Metaverse" (meta (beyond) + universe).

Photogrammetry & Laser Triangulation
Photogrammetry is a well-known 3D Measurement Technology allowing the computation of XYZ Coordinates of Points by processing images taken from different angles. But when it comes to measuring a perfectly homogeneous surface (like a white wall, for example), the lack of detail or texture makes this Measurement challenging, if not impossible. A contrario, Laser Triangulation allows the measurement of any surface by projecting a laser plane, generating a laser line on the surface, to be triangulated with a camera.
The idea of the Video Laser Scan is to combine both technologies, using Photogrammetry to run a first bundle adjustment, compute all camera locations and run the selfcalibration of the camera. Then, Laser Triangulation (with the same camera) can be used to triangulate points on the laser line projected on the surface (see Figure 1). This way, the Laser Triangulation becomes "mobile," all images extracted from the video file are resected by Photogrammetry, and the laser line is triangulated on every frame. Note that the density of the Point Cloud can be selected by processing more or fewer images.
With cameras shooting up to sixty images per second and vehicles moving slowly, generating a sub-millimetric density (or "resolution") becomes standard.
In Figure 1 below, we show the example of a pipe measurement. The Video Laser Scan System is flying along the pipe: Photogrammetry allows the measurement of points and geometrical features (camera only), and laser triangulation allows the measurement of the laser line in every frame (camera and laser). Advantages are multiple: all the advantages of Photogrammetry first, such as mobility and high accuracy, and all the advantages of Laser Triangulation, such as the generation of local highdensity Point Clouds on any surface, when needed. Therefore, with the same flight (or "run") and the same video file, the Video Laser Scan can achieve two different goals: generating a 3D As-Built CAD Model for engineering purposes (  This video file can be captured by any vehicle (no need for stability), including Remotely Operated Vehicles (ROV), allowing access to environments inaccessible to humans (nuclear areas, underwater, space). Furthermore, the video file can be interrupted by any problem, such as a sudden movement pushing the vehicle out of the area or the coming of obstacles such as fish or particles in subsea environments, as long as there is some overlap between all the files. This makes Photogrammetry the fastest and the most flexible Measurement Technique during on-site operations.

"Intelligent Point Cloud" Concept
The capability to measure photogrammetry points and generate local high-density point clouds simultaneously leads logically to the concept of "Intelligent Point Cloud." Instead of dealing with millions of 3D points and matching them with CAD objects, the idea is to measure only "characteristic" points and geometrical features such as lines or cylinders and integrate them into a weighted Bundle Adjustment (a priori sigmas depending on the type of feature are integrated into the bundle). Laser Triangulation (or "Laser Scan") is always active but is processed only on specific surfaces or components if needed. A typical example is shown in Figure 4: the Intelligent Point Cloud is the direct output of the 3D computation. It is made of points and geometrical features, ready for the 3D CAD Modeling phase. Note that such an Intelligent Point Cloud is generated by a single computation and has a lot of redundancy (many observations of the same item from many images, from different angles). Therefore, it provides a very homogeneous and accurate result, with reliable estimates of the final accuracy of each entity, thanks to redundancy and statistical tools. Whereas "non-topographic Photogrammetry" in the air with targets can achieve a typical accuracy of D/200,000 (D being the largest dimension of the measured object), such an advanced Photogrammetry technique, without targets, can still achieve about D/20,000 with the limit of +/-0,3 mm @2sigma, and in any environment. This makes a massive difference with 3D Imaging techniques that should be called "fast 3D rendering" techniques, which are not generating any 3D CAD Model as such. Their accuracy is poor and challenging to estimate, if not impossible. An image is a very rich source of information. It includes all visual and geometrical information (once the camera is calibrated). So, instead of getting millions of 3D points and spending time developing complex tools to transform them into 3D CAD objects (with the help of images!), the idea is to detect directly on the images the 2D features that will be part of the 3D objects. And that's what leads to the detection of 3D objects, as described further in paragraph 3. But at this stage, the Intelligent Point Cloud is only made of photogrammetry bundle adjustment points (enough for the 3D computation), construction or "characteristic" points (that will help build the 3D Model), and geometrical features such as lines, circles, and cylinders. The next step is to send this Intelligent Point Cloud directly to a CAD platform and work on the CAD model, potentially returning to the Photogrammetry process if additional points or features are needed.
At this stage, images corrected from distortions can also be superimposed on the CAD model, thus providing a powerful tool for verifying missing components (see Figure 5). In the case of an existing CAD Model, such a superimposition can be used to relocate components or delete those missing in the real world without rebuilding the entire CAD Model.

Hardware & Vehicles
Another benefit of the Video Laser Scan technology is using "off-the-shelf" and easy-to-use hardware: it needs only images with a laser line (see Figure 6). A laser line is just an option in cases 3D Analysis of local deformations is required. The accuracy and quality of the results will directly be correlated to the hardware quality and somewhat to the quality of Data Capture (smooth and slow flights will always be preferred). The classic subsea set-up includes an HD video camera shooting at least 1080p and 30 images per second and a 532 nm green laser line projector of about 100mW. Technology will always benefit from developing new and more powerful cameras and lasers. Some recent surveys used 4K footage and low light cameras, achieving one of the highest quality levels. Additionally, new and innovative software can improve video footage before processing, whether it lacks visibility or is somewhat blurry. As a result, we are close to the maximum achievable accuracy on the hardware side, and we've indeed reached almost 99% of classic Scopes of Work (SOW).
Besides the use of "off-the-shelf" cameras and lasers, the Video Laser Scan also has the flexibility of using any vehicle: if it can move around or along the object to be measured, any robot such as an ROV (Remotely Operated Vehicle), an UAV (Underwater Autonomous Vehicle) or AUV (Aerial Unmanned Vehicle) can be used.
Most vehicles already have HD cameras that can easily be used for such surveys. The Laser Line projector is also widely used in the subsea Industry for visual applications. If the vehicle doesn't have such equipment, or if this equipment is already used for other tasks, it's always possible to add an HD camera (and potentially a laser line projector) on the vehicle frame or the articulated arm if the vehicle has arms (see Figure 7).
The configuration with the Video Laser Scan System mounted on the articulated arm is undoubtedly the most efficient, allowing maximum accessibility thanks to the arm while keeping safety distances between the object and the vehicle (some components such as Mooring Chains can move and represent a potential danger of collision with the ROV).

Data Processing Automation
The Digital Era opened the way to increase the Automation of Data Processing levels.
There are mainly two phases in Data Processing Automation: Automation in the Photogrammetry Bundle and Automation in the 3D measurement. The first step in the Automation of the Photogrammetry Process is the extensive use of targets, coded targets, and automatic recognition. Therefore, the first step of the Automation of Photogrammetry into inaccessible environments has logically been eliminating targets and using "natural" points (putting targets in inaccessible environments is almost impossible). "Natural" points can be defined as; "any detectable and measurable detail on the image that can accurately be found on multiple images." This concept is extended to geometrical features such as lines, circles, cylinders, planes, and spheres. Today, the detection and measurement of natural points and geometrical features are automated via Image Processing Algorithms.
Once the bundle computation is finished, camera locations are known, and some 3D points and geometrical features are computed. The second step in Automation is triangulating additional geometrical features and points to build the CAD Model.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France When looking for an anomaly and a high-density Point Cloud, the Laser Line is also a feature that can be triangulated easily and automatically using Image Processing Algorithms again.

The Added Value of Artificial Intelligence
There is, and always will be, a difference between air (or in pool) images and subsea images: the real subsea world is full of biases such as particles in the water, fishes, high currents, and obstacles of all kinds. These obstacles will significantly decrease the level of Automation and make Image Processing Algorithms fail, at least partially. Human operation is then required to complete the process and measure what is missing: but this human operation can become very time-consuming in some cases. One solution could have been to improve Image Processing Algorithms by making them less sensitive to such changes, but finding a universal algorithm working in any condition seems a chimera. That's precisely where AI plays its whole part: teaching the software all kinds of situations (here, all kinds of images) and the appropriate solutions. The DataBase of actual images becomes fundamental for such a learning phase: the more images are processed, the better further processing will be until it reaches the highest accessible level of Automation. For example, without AI, the average level of Automation of the measurement of a laser line is around 80%. With AI, this level rises to around 98%. In Figure 8 below, relative to a subsea Mooring Chain measurement, the upper image shows the laser line measurement without AI, and the lower image shows the same Measurement with AI. Without AI, the ends of the curve's laser line are not measured but are quickly captured by AI in the lower image. AI compensates for any change in the laser line: break, shape, size, or light.

3D Object Recognition
At this stage, the global 3D CAD Modelling Process is about 80% automatic, with a fully automatic 3D computation process and fully automatic measurement of almost all required points and geometrical entities. But again, the human operator is needed to finalize the Model by building the objects from the measured points and all geometrical features. Again, this is time-consuming, and making it fully automatic can be considered the ultimate goal.
In the research we are conducting, we are investigating two main tools to reach such a goal: the first tool is being able to recognize a 3D object on a 2D image, again via Artificial Intelligence, and the second tool is being able to follow the object from one image to another. Having the object on multiple consecutive images will help calculate and model it immediately. However, only the final computation, including all the scenes, will provide the most accurate result (Figure 9). You'll notice that this is possible thanks to a video in which all images are consecutive, with minor field of view changes. In the past, it was impossible because photographs from different angles were significantly different. That's another benefit of video instead of still images. The idea is also to build a library of 3D Objects from nominal data (drawings) and objects previously measured. Again, the more objects are measured, the faster future projects will be.

Traditional Outputs
Once the Point Cloud or the CAD Model are generated, there are many ways to illustrate them and show the results of any measurements or 3D Analysis. One of the most common outputs is a 3D View of the CAD Model (see Figure 10). When the 3D Analysis of the shape is required, Cross-Sections are also a very interesting output (see Figure 11).

Augmented Outputs
As a measurement science, Photogrammetry has the unique benefit of providing the best visible information. Other measurement technologies will need to add a camera to their sensor to capture visible information! With such rich information, it's easy to create a more realistic 3D Model by adding the texture extracted from the images. The 3D Model can then be used with navigation software or virtual reality applications. Finally, the Photogrammetry output becomes the most relevant input for the Digital Twin: a genuine Digital Twin is not made of any fast 3D rendering Model but requires a true CAD Model made of CAD objects so that data, intelligence, and active functions can be added.
In the case study shown in Figure 12 below, a simple tablet (with an intrinsically safe case) was used to capture a video of a Refinery Unit and processed with the Video Laser Scan software and Microstation (Bentley).

Figure 12: Photogrammetry 3D As-Built CAD Model integrated into Virtual Reality Application
Such 3D As-Built CAD Models can be used for multiple applications, such as engineering, training, simulation, and communication.

3D Holograms
3D As-Built CAD Models are an excellent way to analyze a project, simulate operations, and design modifications if necessary. Plenty of dedicated software and information can easily be shared among multiple platforms. And when the site is accessible, nothing beats an on-site visit to better understand the configuration and its environment. But when the site is inaccessible, such as a subsea installation or a hot cell in a nuclear facility, the only option, until recently, was to build a physical mock-up, facing many technical and financial challenges and unrealistic approaches. The development of 3D Holograms is a revolution in managing 3D models of industrial assets: it gives access to the site by bringing a 3D artificial representation to the office on a meeting table. As a result, everyone can better comprehend the scene, and the team can exchange and collaborate more efficiently. Figure 13 below shows an example based on a 3D CAD Model built by Photogrammetry (Subsea Valve). The ultimate step will be to interact with the 3D Hologram as a Digital Twin and access functions such as the display or even the simulation of modifications. Let's consider a basic example, such as adding a pipe to an existing installation. We can easily imagine the operator touching one end and the other of the 3D Hologram where the pipe is supposed to be mounted. The software will automatically design such a pipe and register it in the CAD. All the necessary technologies exist; it's just a matter of developing the applications.

CONCLUSION
The world of 3D Measurement by Optical Techniques in the Industry and singularly the world of Photogrammetry is entering a new era: on the one hand, the quality of imaging is increasing exponentially (better resolution, better lightening, better sharpness), and on the other hand, the software is continuously offering new tools, boosted by new technologies such as Virtual Reality, Artificial Intelligence, and 3D Holograms, around the central concept of Digital Twins.
All industrial installations, starting with the inaccessible ones, will probably be entirely remotely controlled one day, making Virtual Reality the new reality. As probably the richest, most accurate, and more flexible source of information, Photogrammetry has a central role to play in this evolution. We can foresee a future where cameras are filming everywhere, automatically updating the Digital Twin Models, and helping operate and maintain all industrial assets. Human intervention might be restricted to meetings around 3D Holograms, Analysis, and the programming of interventions with robots.