LIGHT FIELD CAMERA AS TOOL FOR FORENSIC PHOTOGRAMMETRY

Light field cameras record both the light intensity received by the sensor and the direction in which the light rays are travelling through space. Recording the additional information of the direction of Light rays provides the opportunity to refocus an image after acquisition. Furthermore, a depth image can be created, providing 3D information for each image pixel. Both, focused images and 3D information are relevant for forensic investigations. Basic overview images are often acquired by photographic novices and under difficult conditions, which make refocusing of images a useful feature to enhance information for documentation purposes. Besides focused images, it can also be useful to have 3D data of an incident scene. Capital crime scenes such as homicide are usually documented in 3D using laser scanning. However, not every crime scene can be identified as capital crime scene straight away but only in the course of the investigation, making 3D data acquisition of the discovery situation impossible. If this is the case, light field images taken during the discovery of the scene can provide substantial 3D data. We will present how light field images are refocused and used to perform photogrammetric reconstruction of a scene and compare the generated 3D model to standard photogrammetry and laser scanning data. The results show that refocused light field images used for photogrammetry can improve the photogrammetry result and aid photogrammetric processing.


INTRODUCTION
Documentation of a discovery location of a deceased body is a standard procedure in legal medicine and forensics (Duncan, 2010;Grassberger and Schmid, 2009;Marsh, 2014).Therefore not only 2D imaging technologies but also 3D documentation techniques are used dependent on the cause of death of the discovered person (Naether et al., 2013).

Scene discovery
In Switzerland forensic pathologists or legal examiners are called whenever there is a deceased person with and the circumstances indicate an unnatural death or unclear situation (Schweizerische Gesellschaft für Rechtsmedizin, 2009).It is the forensic pathologists' responsibility to document and examine the corps and decide whether the person died of natural causes or if further examination on the body is required to establish the cause of death.When arriving at a scene one of the first tasks of a forensic pathologist is to document the discovery location and position of the corps to preserve potential evidence (Schweizerische Gesellschaft für Rechtsmedizin, 2009).The documentation is usually performed photographically with handheld consumer digital cameras.If an incident, such as a crime or an accident is suspected the scene is documented thoroughly using 2D and 3D documentation methods including photography, laser scanning and photogrammetry (Luhmann et al., 2006;Miller and Massery, 2016;Naether et al., 2013Naether et al., , 2008)).Ideally, every crime scene should be identified as such within the first stage of investigations.However, there may be homicides that become apparent only after a closer inspection of the corps and scene or even later during autopsy or case investigation.By this time, the crime scene was altered by moving and removing the body or other items.Thus hindering an accurate 3D documentation of the discovery situation and potentially having a negative impact on the subsequent investigation.

Scene documentation
The usual documentation technology is photography.Whenever a scene is supposed to be photographically documented, the performance of a standard technique is essential.Thereby the knowledge about technical aspects of cameras including the manual mode and profound training is more important than expensive equipment (Redsicker et al., 2001;Robinson, 2016).The first images should usually provide an overview of the scene while subsequent photos should provide more detailed information including injuries or forensically relevant items (Duncan, 2010;Grassberger and Schmid, 2009).A common problem is that non-professional photographers capture these photographs.At the same time, the surrounding conditions might be difficult, with poor lighting, confined space conditions or light reflecting surfaces.As a consequence it is likely that the images are not usable due to incorrect exposure, (optical and motion) blur, incorrect white balance, noise, or other factors.Some parameters can be adjusted in post processing but especially optical blur presents a challenge for these photographs.
In cases where it is suspected that the person has died of an abnormal cause the scene is photographed by experts to document injuries and items with sufficient details.However, their aim is not to perform a photogrammetric 3D documentation.The scene is not only photographed but documented in 3D when a crime is suspected (Buck et al., 2007b;Naether et al., 2013).The most common method for these 3D scene documentations is 3D laser scanning.For larger scale areas aerial photogrammetry and aerial laser scanning is used.Scene scans should also document the position of relevant traces, evidence and objects.These objects are then also scanned in more detail using pattern light scanning, handheld scanners or photogrammetry (Buck et al., 2007a).Besides the objects also the corps found at the scene should be documented in the scene and afterwards with more detail independently of the scene with post-mortem computer tomography and, additionally with surface scanning methods when there are pattern injuries visible (Ebert et al., 2016;National Academy of Science, 2009;Thali et al., 2003).
The acquired data can be used for 3D case reconstruction to reconstruct the incident and answer specific forensic questions which can include visibility circumstances, witness statement plausibility, accident dynamics or even virtual incident scene visits (Buck et al., 2013(Buck et al., , 2007b;;Ebert et al., 2014).

Light Field Cameras
Light-field cameras, also called Plenoptic cameras, record the light intensity received by the sensor and the direction in which light rays travel in space.The vectors describing the amount of light flowing in each direction is called light field (Adelson and Bergen, 1991;Georgiev, 2008).This additional information of the direction of light rays allows to refocus an image, change the observers position and to estimate a depth map of the scene (Ng et al., 2005;Perwass and Wietzke, 2012;Wang et al., 2015;Wanner and Goldluecke, 2014;Yu et al., 2013) (Figure 1).
To record the direction of a light ray a light field camera contains a main lens, a microlens array and an image sensor (Knight et al., 2012;Ng et al., 2005).This main lens focuses the light onto the microlens array while the micro lens separates the received light rays on a number of pixels on the image sensor behind each microlens (Ng et al., 2005).This allows to record the direction from which a light ray was received by the camera (Adelson and Wang, 1992).However, the recoding of light rays direction and subsequent refocusing can contain errors (Jeon et al., 2015) (Figure 2).The aim of this work is to test whether refocused light field camera images can be used to perform photogrammetric calculations to create an accurate 3D surface model.

METHOD
To test the use of the light field camera images, a discovery scene of a corps was simulated.This scene was used to acquire 3D data with different methods and compare the results.

Scene Setup
The scene was setup in the corner of an office room at the Institute of Forensic Medicine, Zurich.Therefore, a desk and chair were placed in the room and a mannequin was arranged on the floor with the feet under a desk, while the upper body was besides the desk.This creates a steep gradient between lit area located beside the table and a shadowed area identified under the table, which presents a challenge for the photographic task.Furthermore, the scene was enhanced with a paper bin, papers on the floor and a broken suitcase.To provide a scale for the photogrammetric measurements a folding ruler as used for forensic measurements was placed in the scene.For a more accurate check, three measurement crosses were place in the scene, in both lit and shadowed area (Figure 3).

Scene Scan
At first the scene was scanned using laser scanning with a Z+F Imager 5010C (Zoller + Fröhlich GmbH, 2017).Two scans from different positions were performed.The positions were chosen in a way to document as much as possible of the scene and minimize occlusions.However, as the scene was located in a corner of a room the scan had some significant holes, which cannot be prevented using solely laser scanning.This presents a usual problem at scenes but as long as there is no relevant information missing these holes do not present a major problem for subsequent forensic reconstructions (Figure 4).
After the laser scan, the scene was photographed using a Nikon D700 (Nikon Corporation, 2018) digital single lens reflex (DSLR) camera fitted with a Nikon 24-70mm (Nikon Corporation, 2018) zoom lens.The first image was acquired to document an overview of the scene, as it would normally be performed at a discovery scene.Then the lens was changed to a fixed Nikon 50mm (Nikon Corporation, 2018) focal length lens, which was focused once and then used to document the scene using photogrammetry.Due to the different illumination, the camera was set up in a way that illuminated areas were not over exposed, resulting in slightly underexposed areas in the unlit area under the desk.This was adjusted in post processing by increasing the brightness of dark areas to provide sufficient detail in these areas.58 overlapping images were acquired.
Finally, the Lytro Illum (Lytro.Inc., 2018) light field camera was used to perform a first overview image and subsequent photogrammetric images.In difference to the DSLR camera, the light field camera has a zoom lens permanently connected to the camera body so that for the photogrammetry the camera was once set to a focal length of 50mm and then not altered after.
The focusing was also performed just once and then not altered after.The dynamic range of the camera appears to be smaller than the one of the DSLR camera noticeable by the dark areas, which appeared completely black.Due to this issue, the shutter was adjusted to allow bright and dark areas to be exposed properly.In total 58 overlapping images were acquired.

Data processing
The laser scan data was post processed with Z+F LaserControl (Zoller + Fröhlich GmbH, 2017) where the point clouds were aligned using dense cloud matching.Afterwards the matched point clouds were cropped to the area of interest and processed in Geomagic Studio (Geomagic, 2016).The processing included meshing the point cloud, texturing it and then exporting the final model as .objfile (Reddy, 2017).
The DSLR images were first processed in Darktable, where the brightness was adjusted (David and Houz, 2017).Furthermore, a copy of the images with a reduced pixel count of 2450x1634 pixel was created, which is equivalent to the light field images pixel count.Then both image sets, the full and the reduced resolution images were post processed in Agisoft Photoscan (Agisoft, 2018).To align the cameras, Agisoft automatically calculates a camera calibration so that no separate camera calibration was required.The settings for the processing steps were set to high, while the mesh was limited to 1.5 million faces and the texture to 4096 x 4096 pixels.Before exporting the final 3D model, the scene was scaled using the scale bar placed in the view.Furthermore, the measurement crosses were detected automatically and compared between real distance and distance measured in the 3D model.The final 3D model was exported in .objfile format.
For subsequent comparison the created 3D models from laser scan, DSLR and light field camera were imported in Cloud Compare (Cloud Compare, 2018).Within Cloud Compare both the DSLR and light field model where aligned to the laser scan model using the "Align by equivalent point pairs" feature using a total of eight feature points.Then the aligned photogrammetric models were sampled with 3'000'000 points changing the meshed object back to a point cloud.This was necessary to calculate the distance between laser scan and each of the photogrammetric models using the "Compute cloud/mesh distance" feature.Even if the feature allows to compare two meshes it was found that the returned distances between the two meshes were disproportionally large and inaccurate especially at the boundaries of the models (Daniel, 2015).With sampled point clouds, this problem was solved and a colour coded point cloud showing distance between laser scan model and photogrammetric models were created (Figure 5).
Due to holes in the models, the calculated distances contain huge discrepancies so that for a detailed comparison not the complete model was compared but ten sampling areas were cropped from the models.The ten areas were chosen in areas where no holes occurred in either of the models and with five areas flat and another five in rougher parts of the scene (Figure 6).

RESULTS
The comparison of photogrammetric model based on DSLR images to models based on refocused light field images allows to establish whether refocusing light field images negatively influences the photogrammetric procedure.Comparing the measurement crosses distributed in the scene shows that the scaling for both, the full resolution DSLR and the light field images is within 0.2 mm accuracy.However, for the reduced resolution DSLR images the discrepancy was twice as large (Table 1).The subsequent alignment of the photogrammetric models to the laser scan returned similar root mean squares (RMS) for each model within sub millimetre range (Table 2).
The calculated distances between laser scan and each of the photogrammetric models exposed that the laser scan had significant holes, while the photogrammetric models covered these holes.In the areas of these holes the calculated distances are the largest with up to 0.33 m (Figure 6).For the sample areas the calculated distances are between 4 mm to 58 mm.For the flat areas the average distances are between -2 mm and +2 mm while for the rougher areas the distances are between -31 mm and +33 mm (Table 3 and 4).
Looking at each model separately shows that the model create with the light field camera has similar distances compared to both DSLR models, the high resolution and reduced resolution images (Table 5).For flat areas the distances between laser scan model and photogrammetric models was within a few millimetre, for the rougher areas it was up to several centimetre.

DISCUSSION
Using the light field camera is similar to a normal DSLR.The only difference is that the aperture is fixed and the focusing can be performed faster.This is possible, as the focusing does not need to be performed as accurate due to the refocus ability of the light field images.Furthermore does this ability allow the operator to change the distance between camera and object during image acquisition.Usually this should be avoided with standard DSLRs as changing the distance between camera and object requires refocusing the lens and therefore changes the interior orientation.
Furthermore, we were able to show that refocused images acquired with light field camera can be used to create photogrammetric models.The refocused images could be processed without problems and even the automatic detection of targets or scaling of the model was possible.The model was of similar accuracy as the high-resolution DSLR model despite the low pixel count of the images (Table 1).The texture of the models was best for the reduced resolution DSLR and light field images while for the high-resolution DSLR images some of the texture was fragmented (Figure7).However, it was not possible to evaluate in Agisoft Photoscan why this fragmentation happened.
Comparing the photogrammetric models to the laser scan showed that the light field model is of similar geometric quality as the DSLR models (Table 3, 4 and 5).It was expected to notice some differences due to the lower pixel count of the images, however, there were no significant differences visible.

CONCLUSION AND FUTURE WORK
In conclusion, it can be said that light field cameras can be useful to perform photogrammetric documentations even after refocusing them and offer a variation of future research opportunities.
Subsequent experiments will examine if the light field information recorded during image acquisition can be used to create accurate 3D models (Murgia et al., 2016a;Wang et al., 2015;Zeller et al., 2015).Recent research shows that the creation of 3D models is possible with light field camera images, however the authors do not present how they created the 3D point cloud or compared their result to a ground truth model (Murgia et al., 2016b;Perra et al., 2016).In other trails it will be tested whether the use of light field cameras might improve forensic scene documentation and subsequent use of these documents as evidence in court.

Figure 1 .
Figure 1.Light field image acquired with Lytro Illum and refocused after image acquisition for close, middle, far and complete image.

Figure 2 .
Figure 2. Error while refocusing in image and depth map.The cone is not represented correctly and the transition to the cone in the back appears incorrect.

Figure 3 .
Figure 3. Scene overview before alteration (placing a forensic scale) of the scene.

Figure 4 .
Figure 4. Colourised laser scan model of the scene.The holes are visible especially in the top right corner behind the mannequin.

Figure 5 .
Figure 5. Distance point could light field model to laser scan.Green small distances red, larger distances.The red at the top right represents the hole in the laser scan.The colour bar on the right represents the colour coding of the discrepancies between -0.26 m to 0.33 m.

Figure 6 .
Figure 6.The sampling areas (cropped out holes) are distributed over the flat areas on the floor and the rougher areas along the mannequin the suitcase and the desk chair.Furthermore, the top left corner in red shows a large hole in the laser scan which could be documented using photogrammetry.

Figure 7 .
Figure 7. On the right, the fragmented texture of the high resolution images.In the middle the texture on the reduced resolution images.On the left the texture of the light field images.

Table 1 .
Discrepancies of the measurement cross measured in 3D model to real value.

Table 2 .
Root mean square for the photogrammetric models matched to the laser scan model using eight tie points.

Table 3 .
Flat sample areas with positive and negative distance to laser scan model.

Table 4 .
Rough sample areas with positive and negative distance to laser scan model.

Table 5 .
Average absolute distance for flat and rough areas.