3D MODELLING IN TEMPERATE WATERS: BUILDING RIGS AND DATA SCIENCE TO SUPPORT GLASS SPONGE MONITORING EFFORTS IN COASTAL BRITISH COLUMBIA

Structure-from-motion (SfM) has emerged as a popular method of characterizing marine benthos in tropical marine environments and could be of tremendous value to glass sponge monitoring and management efforts in the Northeast Pacific Ocean. However, temperate marine environments present a unique set of challenges to SfM workflows, and the combined impact that cold, dark, and turbid waters have on the veracity of SfM derived data must be critically evaluated in order for SfM to become a meaningful tool for ongoing glass sponge research. This paper discusses the design, development, testing, and deployment of an innovative underwater SfM workflow for generating high-resolution 3D models in temperate marine environments. This multi-phase research project (dry-lab, wet-lab, and field), while possibly seen as unconventional, was designed to innovate in two ways. First to build an operational data capture platform to support low-cost SfM-based seafloor surveys. And second, to enable systematic isolation and evaluation of SfM data capture parameters and their implications for representational veracity and data quality. This paper reports the challenges and outcomes from a series of field surveys conducted in Howe Sound, BC, one of which serves as the first of two data sets in a temporal analysis of 3D morphometric change. This research demonstrates that accurate, high-resolution morphometric characterization, of all benthic species and habitats, is dependent on a range of equipment, procedural, and environmental variables. It is also intended to share our applied problem-solving path to successful 3D capture, backed up by robust data science.


INTRODUCTION -THE SIGNIFICANCE OF GLASS SPONGES AND THE CHALLENGE TO MONITOR THEM IN THREE DIMENSIONS
The glass sponge reefs found throughout the western Canadian continental shelf are considered a key structural habitat due to their uniqueness, delicacy, and role in maintaining ecosystem dynamics (Fisheries And Oceans Canada, 2010). As filterfeeders, they remove waterborne bacteria with 95% efficiency, are a potentially important carbon sink (Kahn, Yahel, Chu, Tunnicliffe, & Leys, 2015), and their siliceous skeletons provide a significant silica sink (Chu, Maldonado, Yahel, & Leys, 2011). Glass sponge reefs support a variety of marine fauna, including bryozoans, gastropods, crustaceans, echinoderms, and fish (most notably, several species of rockfish) (Chu & Leys, 2010;Marliave, Conway, Gibbs, Lamb, & Gibbs, 2009). This ecological productivity attracts many commercial fisheries, resulting in an estimated 253 tons of coral and sponge bycatch between 1996 and 2002 (Leys, Mackie, & Reiswig, 2007) and widespread reports of damage (K. Conway, Barrie, Hill, Austin, & Picard, 2007;Kim W. Conway, Barrie, & Krautter, 2005;Cook, Conway, & Burd, 2008;Krautter et al., 2001). Consequently, Fisheries and Oceans Canada closed fisheries in coastal waters known to contain these sensitive reef habitats (2010), thereby protecting these ecosystems as researchers continue to assess reef health and their ability to adapt to environmental and anthropogenic influences. However, effective monitoring efforts are inhibited by insufficient pragmatic indicators of glass sponge reef functionstemming from their relative inaccessibility and a reliance on traditional 2D metrics that are subjective, time consuming, prone to error, and provide an inadequate characterization of the complex 3-D structure of glass sponges. Accurate characterization of this 3D complexity through time is critical for assessing and monitoring the health of glass sponge reef ecosystems, improving coastal monitoring efforts, and implementing informed marine management policies.
Habitat complexity is a critical attribute of underwater ecosystems, as kelp forests, coral reefs, and glass sponge reefs support a variety of marine fish and invertebrates, and higher levels of structural complexity are linked to increased biodiversity, productivity, and survivorship (Connell & Jones, 1991;Gratwicke & Speight, 2005). While traditional 2D metrics (e.g. percent cover or maximum relief) fail to sufficiently characterize habitat structural complexity, more recent methods (e.g. LiDAR), which can provide high-resolution, 3D characterizations, come with significant costs that place them out of reach for many researchers and monitoring organizations (J. H. R. Wedding, Friedlander, McGranaghan, Yost, & Monaco, 2008). However, Structure-from-Motion (SfM) has emerged as a popular alternative for documenting the 3D structure of marine benthos, particularly in clear, tropical waters.

The Rise of SfM in Marine Ecology
SfM, or multi-image photogrammetry, is a photogrammetric approach enabling 3D reconstruction and measurement from two or more 2D images. SfM emerged in the 1990s with the arrival of digital photogrammetry, and owes much of its existence to the computer vision community and the feature-matching algorithms introduced during the 1980s (McCarthy, 2014;Westoby, Brasington, Glasser, Hambrey, & Reynolds, 2012). While the photogrammetric community favours geometric quality over automation, those from computer vision stressed the importance of automation and auto-calibration, features characteristic of modern SfM software packages (Remondino et al., 2012) such as VisualSFM, Agisoft Metashape (formerly PhotoScan), and Pix4D. While these user friendly interfaces make 3D reconstruction from a collection of 2D photographs quite simple, they are often criticized as being black-boxes, due to their level of automation and lack of transparency (McCarthy & Benjamin, 2014;Remondino et al., 2012;Smith, Carrivick, & Quincey, 2016;Van Damme, 2015).

Developing a Rig for Operational SfM and Critical Benchmarking
The objective of this study was to develop an innovative SfM data capture workflow, supported by a preliminary sequence of data quality analyses conducted in a dry and wet-lab, that can be used to monitor changes to the complex, 3D morphology of glass sponges and glass sponge reefs in temperate marine environments. Given the annual growth rate of glass sponges is documented at 1-10 cm/year (Austin, Conway, Barrie, & Krautter, 2007;Kahn, Vehring, Brown, & Leys, 2016), 3D models derived from biannual SfM surveys needs to be sufficiently accurate to detect sub-centimetre scale changes.
Our approach focuses on a systematic analysis of survey parameters that are commonly reported or undertaken idiosyncratically to understand how camera type, camera settings, lighting, capture strategy, and underwater environments impact the veracity of SfM-derived 3D data. By implementing a systematic program of SfM benchmarking in controlled dry lab and wet lab conditions, we were able to establish a robust and informative context that provided a basis to understand the imaging and geomorphometric capacity of our workflow to resolve object morphology and features across a range of granularities, and to provide a basis for understanding the impact of combined equipment and environmental variability in the field.

EQUIPMENT AND BENCHMARKING
The glass sponges selected for this study are located off the west coast of Bowen Island, BC at a depth of 20-30 metres. These sites can experience strong currents, turbid waters, and low light levels; therefore, it was determined that providing camera and light stability would be critical to high-quality data capture. The following subsections describe the equipment and workflow benchmarking phases of this project, which provided critical data sets quantifying the performance of our SfM workflow in both a dry and wet-lab environment, and the first portion of the field work phase.

Data Capture Platform
The custom-built data capture rig (HEXYZ-1, named for the species, shape of rig, and acknowledging the objective of deriving 3D geometry) was designed to deliver camera stability, a controlled and repeatable data capture strategy, and a world stabilized light source ( Figure 1). An adjustable camera armature, extending from and rotating around the apex of HEXYZ-1's one-inch square tube aluminium frame, offered a stable, circular pathway for photograph and video capture. Seven extra-wide beam LED dive lights (BigBlue Model AL1200XWP), one at each of HEXYZ-1's six vertices (staggered high and low) and one at its apex, were attached to the frame within semi-opaque housings. This lighting configuration offered consistent, soft lighting that could be controlled (using 150, 300, 600, or 1200 lumen settings) to suite environmental conditions and prevented hard showing and over or under exposure. Figure 1. A custom designed data capture platform (HEXYZ-1) was constructed to provide camera stability, repeatability, and consistent, world-stabilized lighting in temperate marine environments.

Camera Equipment
Two different camera types were utilized in this research; a Sony Alpha α6500 mirrorless digital camera equipped with a Sony E PZ 16-50mm F3.5-5.6 OSS lens, and a GoPro HERO6 Black action camera. Both cameras were used in the dry-lab phase of this project, but only the Sony A6500 camera, mounted within an Aquatica A6500 underwater housing equipped with an 8-inch dome port, was used in the wet-lab and field phases.
The Sony A6500 was selected due to the availability of an underwater housing and as a highly rated digital camera suitable for SfM data capture. The GoPro HERO6 Black was chosen to evaluate the aptness of action cameras for SfM-based modelling, as action cameras, largely GoPro models, are often cited in the SfM research literature (e.g. Gutierrez-Heredia et al., 2016;Raoult et al., 2016;Thoeni, Giacomini, Murtagh, & Kniest, 2014;Van Damme, 2015). While the GoPro used here was significantly less expensive than the combined cost of the Sony A6500 and underwater housing, financial savings may come at the expense of data quality, which would limit the usefulness of action cameras in benthic surveys endeavoring to monitor smallscale (centimeter or less) morphometric changes.
Documented best practices indicate that photographs, as compared to frames extracted from video, provide superior SfM modelling results. Photographs and videos were collected in this study to calculate performance disparity.
Sony A6500 photographs were captured in the ARW format and converted to TIFFs for processing, whereas GoPro photographs, while captured in the RAW and JPEG format, were only processed as JPEGs due to problems with the quality of the RAW files. Individual frames from each 4K video were manually extracted as PNG files using VLC Media Player.

SfM Processing
Images from each of the SfM tests and surveys described throughout this paper were processed using Agisoft PhotoScan Professional (V1.4.0). In each instance, images were aligned using the high accuracy setting, markers were placed where visible, and a minimum of two scalebars were created using the reference objects (rulers) located by each subject, and image alignment was optimized using the optimize tool. Dense point clouds were then created using the high-quality setting and depth filtering was set to aggressive. Point clouds were then cleaned to remove erroneous points and high-polygon count meshes were created as necessary. Dense point clouds were exported for analysis in CloudCompare.

Phase One: Dry-lab Benchmarks
The objective of the first phase of this research project was to identify the optimal lighting configuration and which combination of camera type, camera settings, and data capture strategy resulted in the highest accuracy data (Lochhead and Hedley, 2020a). Five different lighting tests were conducted to assess the impact that a different number, configuration, and lumen output of lights had on model accuracy. Each test included 168 photographs of a glass sponge skeleton collected with the Sony A6500 (30mm focal length, f/5.0 aperture, ISO 640, and 1/60 shutter speed) mounted to HEXYZ-1. 3D model accuracy was determined by comparing linear measurements of five distinct features made within PhotoScan to those collected from the sponge skeleton using a digital calliper. The lighting configuration (shown in HEXYZ-1 Figures throughout) provided the smallest measurement error and was therefore applied to all subsequent tests.
Much of the research in phase one focused on identifying SfM data quality and data quality variability resultant from purposefully selected combinations of camera types, camera settings, data format, and data capture strategies utilizing HEXYZ-1 (Figure 2). These combinations yielded nine unique test configurations. Each test contained a total of 180 photographs (or 180 images extracted from videos), that provided either a unique perspective of the test subject in 10-degree increments along five encircling pathways (Figure 3), or a unique perspective collected from random positions within the confines of HEXYZ-1. An assemblage of wooden blocks of known dimension served as the test subject for these tests, rather than the glass sponge skeleton, in order to overcome imprecise manual measurements of a complex, irregular structure. SfM data quality was again determined by comparing linear SfM model measurements to real-world measurements. Overall, this battery of tests revealed that the Sony A6500 produced higher accuracy models than the GoPro, that having the Sony A6500 mounted to HEXYZ-1 (as opposed to being handheld) resulted in higher accuracy models, and that there was little difference between the mean measurement errors of tests containing photographs and those containing images extracted from videos. Figure 2. Dry-lab tests evaluated SfM data quality resulting from photographs and videos collected using different cameras, camera settings, light configurations, and with and without the stability of HEXYZ-1. . Figure 3. 180 images provided 360-degree coverage of the test subject. Images were captured from camera positions (black points) arranged in five halos around target objects (blue). This image shows a side perspective of the five halos of camera positions and target object.

Phase Two: Wet-lab Benchmarks
The objective of the second phase of this project was to evaluate the highest accuracy workflow from phase one in a controlled wet-lab, thereby quantifying the impact that a cold, dark marine environment has on SfM data quality (Lochhead and Hedley, 2020b). 180 photographs of an ornamental coral were collected underwater and on dryland with the Sony A6500 (16 mm, f/3.5, 400 ISO, and 1/60 shutter speed) mounted to HEXYZ-1 and using the data capture strategy depicted in Figure 3. Underwater photographs, captured from within a large saltwater holding tank, and dryland photographs, captured from beside that tank were both collected after sunset and from within the confines of semipermanent fabric building located at Fisheries and Oceans Canada's Centre for Aquaculture and Environmental Research in West Vancouver, BC (Figure 4).
Data quality was again determined by comparing the accuracy of a series of linear measurements made within PhotoScan to manual measurements made using a digital calliper. These tests revealed that, using the presented workflow, SfM models generated from underwater images can be as accurate as those generated from dryland images; in fact, the underwater photographs in this test produced a model of slightly higher overall accuracy (0.26% (wet) versus 0.28% (dry) differencecalculated as the mean of the absolute value of all measurement errors). However, that level of accuracy was only obtained when all 180 photographs were processed, as tests including 90, 60, and 45 photographs (processed in different configurations) resulted in greater measurement errors than those containing the complete set of photographs. Figure 4. The quality of SfM data sets, derived from photographs collected in a dryland (top) and wet-lab (bottom) environment, was evaluated to quantify the impact of a dark, cold water marine environment.

Phase 3: Glass Sponge Surveys
The workflow and data quality benchmarks delivered through phase one and two of this research project served to both guide the design and define the capacity of a SfM data capture strategy capable of monitoring the health and morphology of temperate marine ecosystems (specifically glass sponge reefs). Phase three of this research takes the next step towards the goal of informed reef-scale surveying in temperate waters by applying our workflow in the field. Photographs and videos of individual glass sponges were captured during a series of field surveys conducted off the coast of Bowen Island, BC (49°23'19.0" N 123°24'38.2" W) at a depth of 20-25 metres. All photographs and videos were collected using a Sony A6500 mounted within an Aquatica housing fitted with an 8" dome port (photographs: 16 mm, f/3.5, 400 ISO, and 1/60 shutter speed, video: 4K). All field surveys conducted with HEXYZ-1 utilized the lighting configuration presented in phases one and two (seven lights each producing 600 lumens). The specifics of each survey are presented in the following subsections. All SCUBA diving activities were conducted by members of the Howe Sound Research Group and their affiliates.
2.6.1 Survey 1: HEXYZ-1 was deployed from a small research vessel, lowered onto the seafloor, and then moved into position over a glass sponge by two divers; a small lift bag was utilized to support the weight of HEXYZ-1 as it was moved into place. Four marked stakes (for image alignment) and three scale markers (for model scaling) were driven into the seafloor surrounding the glass sponge. A total of 32 photographs (one circular pass) were collected with the camera attached to HEXYZ-1's armature ( Figure 5). All equipment, other than the marked stakes, was collected at the conclusion of this survey. 2.6.2 Survey 2: HEXYZ-1 was not deployed for this survey, in which two sets of photographs were collected manually by an underwater photographer. Scale bars were again placed around the glass sponge for reference. The first dive generated 67 photographs of the glass sponge, illuminated by a single camera mounted wide beam dive light (800 lumens), and the second dive yielded 73 photographs, illuminated this time by a single BigBlue extra-wide beam dive light (1200 lumens). The photographer was asked to collect sets of photographs encircling the glass sponge in a manner like that delivered by HEXYZ-1. All reference objects were collected at the end of this dive.

Survey 3:
HEXYZ-1 was again deployed from the research vessel by a team of divers, only this time the intended glass sponge could not be located. HEXYZ-1, the marked stakes, and the scale bars were instead placed around a different, smaller glass sponge. 36 photographs were collected in one circular path; however, none of those photographs contained enough of the glass sponge to be useful for 3D reconstruction. Only the camera and dive lights were collected at the conclusion of this survey.

Survey 4:
Returning the following day, and with HEXYZ-1 and the reference objects already on site, the dive lights were installed, and 75 photographs were captured from two paths encircling the glass sponge at different heights above the seafloor. Only the camera and dive lights were collected at the conclusion of this survey.
HEXYZ-1 remained on site for approximately three weeks, as divers were unable to safely return to the site due to poor weather conditions.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) 2.6.5 Survey 5: Having yet to successfully generate a complete data set, it was decided that videos, rather than photographs, would be collected to overcome the time limitations introduced by successive dives at depths of 20-25 meters. Three videos were collected for this survey, each video encircling the glass sponge at different heights and providing a different perspective. All equipment, other than the stakes, was collected at the end of this survey. A total of 189 frames were extracted from these videos.

DISCUSSION
A variety of SfM workflows have proven to be beneficial for marine surveying across a range of research disciplines. As a low-cost, non-intrusive, and relatively rapid method of 3D data capture, SfM has gained significant popularity. Yet temperate marine applications are not as commonly reported as those in tropical environments, and questions remained about SfM performance in challenging temperate environments. The objective of this research was to assess the impact of some of those challenges and develop a SfM workflow suitable for monitoring glass sponges in Howe Sound, BC.
Phase one and two of this research project provided those performance benchmarks, defining the capacity of our workflow under ideal conditions as well as in a cold, dark, and turbid marine environment. However, data collection in the field presented an additional set of logistical and methodological challenges which ultimately resulted in five disparate data sets.
The photographs and extracted images resulting from each of the five surveys were processed within PhotoScan using the parameters discussed above. Select results are presented in Table  1. No image enhancements or colour correction procedures were performed on any of the photographs. Table 1. SfM Field Survey Statistics The first deployment of HEXYZ-1 (Survey 1), while data was collected, was intended to evaluate the logistics of HEXYZ-1's deployment and data collection efforts. Survey 2 provided two valuable data sets containing manually collected photographs, the results of which could be compared to HEXYZ-1 collected data to identify differences resultant from lighting and camera control. Unfortunately, Survey 3 could not relocate the original glass sponge, and while a new sponge was identified, this survey did not produce a usable data set. Survey 4 did collect two circular sets of images using HEXYZ-1, but due to time constraints imposed by repeated dives at depths of 20-25 metres, did not produce a third set of images completing the intended data set. Survey 5 collected a set of three videos, having to settle for the lesser quality but more rapid video format, in order to generate the intended set of data providing three perspectives of the glass sponge.
For each survey, measurements of an additional scale marker, that was included in the survey but not utilized for model scaling, were made within PhotoScan to provide an approximation of model accuracy. The measurement made in Survey 1, despite the limited number of photographs processed, was only 1 mm less than the true distance value. The measurements made for Survey 2 was 1 cm over the true distance, and a measurement could not be made in Survey 2B due to excessive point cloud noise ( Figure  6). In both instances, the single dive light provided an insufficient and unstable light source that resulted in points which could not be removed using PhotoScan's aggressive depth filtering. Survey 3 did not produce a 3D model. The measurement made in Survey 4 was equivalent to the true distance, and in Survey 5 only differed by 1 mm. Surveys 1, 4, and 5 produced what visually appear to be highquality dense point clouds documenting the structure of the surveyed glass sponge. While Survey 2 produced two 3D point clouds, there are noticeable data quality issues with both data sets. These results suggest that, as was revealed during phase one tests, the use of HEXYZ-1 for data capture does improve the quality of the 3D data. However, those early tests also revealed significant differences between the quality of the SfM models derived from photographs and extracted images. This would imply that Survey 4 (photographs) should produce more accurate results than Survey 5 (extracted images). Figure 7 provides a side-by-side comparison of the two clouds, and other than the obvious colour differences, the dense point clouds appear similar. However, Survey 4 generated a dense point cloud with three times the number of points (2,410,338 versus 754,348) and twice the ground resolution (0.174 mm/pix versus 0.383 mm/pix) of Survey 5 (Figure 8). Figure 7. The dense point clouds from Survey 4 (left) and Survey 5 (right). Other than the obvious colour difference, photographs and extracted images seem to produce point clouds capable of documenting glass sponge morphology. Despite the difference in point density, a cloud-to-cloud distance comparison reveals that the dense point clouds from Survey 4 and 5 have a mean difference of only 2 mm (Figure 9). This minimal mean cloud-to-cloud distance is encouraging for future surveying efforts, as video capture required significantly less time than photographs. A team of two scuba divers were able to capture the three videos within the time that it would have taken them to capture one ring of photographs. More rapid data capture provides an opportunity to conduct successive surveys within the same diveto ensure survey precisionor more in-depth surveying, moving the camera to additional positions closer to the subject in order to increase pixel density. Overall, while the video format was not the intended data format, the achieved accuracy is sufficient for monitoring purposes and the more rapid survey logistics greatly assist with data capture. Figure 9. The mean cloud-to-cloud distance between the dense point clouds from Survey 4 and Survey 5 was 2 mm (top-down view of the point cloud).
This paper presents a series of field surveys that highlight some of the challenges associated with collecting photographs and videos of benthic species in temperate waters. Survey 5 represents the first of two data sets in a temporal series that will be used to assess morphometric change. While images extracted from video were not our ideal data format, the successive phases of this research project, and the preceding field surveys, provide valuable insight into the quality of this data set. Developing the data science behind our workflow was critical to future analyses of glass sponges, as the ability to generate 3D models of those sponges was never in question, but the accuracy of 3D models created by simply collecting and processing photographs or videos was. As illustrated here, subtle changes to lighting schemes, camera settings, and capture strategy can impact 3D model accuracy, and when monitoring millimetre or centimetre scale temporal changes, accuracy is critically important.

CONCLUSIONS
The glass sponge reefs found in the Northeastern Pacific Ocean are the only documented living glass sponge reef structures in the world. It is believed that these unique ecosystems are a key structural habitat, providing refuge to a wide variety of marine fauna. Yet the relative inaccessibility of glass sponge reefs has restricted research on reef function and limited the capacity to monitor reef health. Much of what is known about these reefs is the product of submersible and ROV based exploration projects, limiting research activities to those with significant capital and technological resources. The discovery of glass sponge bioherms in the Strait of Georgia and Howe Sound, B.C. at depths less than 60 m has provided a unique opportunity for SCUBA divers to record, study, and track changes in the 3-D structure and complexity of glass sponges over time. However, in situ manual measurements and traditional 2-D metrics are subjective, prone to error, and fail to accurately characterize the 3-D structure of these ecosystems. The SfM photogrammetric techniques employed by marine scientists in tropical climates, to both qualify and quantify coral reef ecosystems in 3-D, represents a workflow with the ability to transform what is known about the glass sponge ecosystem.
While SfM workflows can provide high-resolution 3-D models of the structural characteristics of benthic species and habitats that are critical to the analysis of these ecosystems (J. Pizarro et al., 2017), much of the published underwater SfM research is restricted to shallow water tropical climates that vary considerably from the coastal waters of the Pacific Northwest. While studies applying the SfM workflow in marine settings have achieved millimetre-scale accuracy, there is a lack of scientific literature documenting the impact of environmental variables such as light, turbidity, salinity and temperature on the 3-D SfM product, and consequently, the appropriateness of this technique in temperate waters.
The overarching objective of this paper is to share the challenges, strategies, methods and implications of an applied underwater SfM research project that, in the first instance, sought to develop a rig capable of delivering operational ecological surveying capability. However, rather than simply implementing yet another idiosyncratic marine SfM project, we thought deeply about the need to develop benchmarks for marine SfM more broadly. This led us to develop a rig which could also be used for careful, systematic testing of capture parameters, and their performance in dry-and wet-lab controlled conditions with comparable lighting to underwater field sites.
The systematic approach to our SfM research first defined the capacity of our workflow under ideal conditions, and the impact that variable data capture parameters have on SfM data quality in a dry-and wet-lab, before attempting to conduct underwater surveys of glass sponges in the field. These preliminary phases provided key data capture and data quality benchmarks that allowed us to adapt to challenging field conditions while understanding the implications that those adaptations would have on the quality of our data. While SfM can be used to generate 3D models of benthic species in temperate marine environments, this research highlights the impact that different capture strategies, different lighting conditions, and different data formats have on the quality of those 3D models. As the published annual growth rate of glass sponges is approximately 1-10 cm/year, those employing SfM to monitor glass sponge growth and health must acknowledge the subtle yet important role that these differences play if they are to detect real change.