IMMERSIVE VISUALISATION OF CONSTRUCTION SITE POINT CLOUD DATA, MESHES AND BIM MODELS IN A VR ENVIRONMENT USING A GAMING ENGINE

: By adopting Building Information Modelling (BIM) software, the architecture, engineering and construction (AEC) industry shifted from a two-dimensional approach to a three-dimensional one in the design phase of a building. However, a similar three-dimensional approach for the visualisation of the current state of the construction works is lacking. Currently, progress reports typically include numerous pictures of the construction site or elements, alongside the appropriate parts of the 3D as-design BIM model. If a proper transition to a 3D design versus 3D current state were achieved, the evolved type of reports would become more comprehensible, resulting in more well-informed decision-making. This requires a single, unique software platform that is able to import, process, analyse and visualise both the as-design BIM model as well as the recorded data of the current construction state. At present however, the visualisation and interpretation of the different datasets alone requires already multiple software packages. As a partial solution this work presents a platform to easily visualise and interpret various data sources such as point clouds, meshes and BIM models and analysis results. Recent advances of gaming engines focus on and allow for an excellent visualisation of mesh data. Therefore all of the aforementioned data sources are converted into mesh objects upon importing. Moreover, gaming engines provide the necessary tools to traverse the scene intuitively allowing construction site managers and other stakeholders to gain a more complete and better oversight of the construction project. Furthermore, these engines also provide the possibility to take the immersion to the next level: incorporating the 3D entities into a Virtual Reality (VR) environment makes the visualised data and the executed analyses even more comprehensible. By means of a case study, the potential of the presented approach is showcased. The real-world construction site recordings, models and analyses are visualised and implemented in VR using the Unity gaming engine.


INTRODUCTION
Recent remote sensing techniques serve as an excellent basis for progress monitoring.The resulting products typically are point clouds or meshes.A proper visualisation platform for the captured data on construction sites can be a game changer for the AEC industry.Often, the vast amounts of captured and processed data not only pose a challenge for analysing software but also simply displaying the data can be troublesome.Therefore there is large demand for platforms that are easy to work with and, more importantly, are able to import and visualise the various data formats encountered in the construction industry.In contrast with often expensive and field-specific software, gaming engines present a viable solution.These pieces of software are heavily optimised to display complex meshes.When converting the different data formats into mesh objects upon importing, fast and workable visualisations become possible (figure 1).Furthermore, gaming engines allow for various kinds of user input and interaction, allowing for a transition from a mere visualisation project to one where alterations to and interaction with the scene as well as analyses on deviations and achieved accuracy become reality.These kind of projects are also called serious games, since they allow professional users to visualise and interact with the data as if playing a game.
Visualisations are one of the most important and comprehensible inputs for decision makers in the AEC industry.Currently, re-ports written by construction site managers and field workers are the most widely used instrument to inform decision makers on the achieved construction progress (Zollmann et al., 2014).Frequently, pictures of construction elements are accompanied by the according 2D plan sections or BIM model visualisations.Pictures efficiently describe the current situation, more than words or numbers can.They inform construction site managers, project managers and other stakeholders that errors have been made in a specific section by effectively showing the discrepancies between the reality and the plans.Subsequently, these people base their decision on what they deduct from what they read and foremost what they see on the pictures and visualisations in the report (Golparvar-Fard et al., 2009).
When taking the visualisation a step further, by advancing from a two-to a three-dimensional level, decisions can be taken more well-informed.Furthermore, implementing data of the design and the executed works into a VR environment allows decision makers, much more than currently is the case, to work completely remotely.They can perfectly estimate the element dimensions and the severity of the errors and assess whether or not the planning should be altered or if the built element should be deconstructed.
The emphasis of this work is on the visualisation of data captured on construction sites and the derived analyses.The importing procedure of various data formats encountered in the construction industry into the gaming engine Unity is discussed by means of a practical case study on a real-world construction site in Ghent, The remainder of this work is structured as follows.In section 2. related works on visualising different construction-related datasets are discussed.Subsequently, importing the data and the associated conversion necessities are presented in section 3. The implementation in VR is the subject of section 4. Section 5. provides the discussion of the achieved results and an outlook on future further developments and finally, the conclusions are presented in section 6.

RELATED WORK
Visualisations in the AEC industry form an important input for decisions about the further process of the construction works.Currently, the most common instrument as a basis for these decisions are progress reports.Although these allow for a decent understanding of the delays, difficulties and errors by means of on-site pictures, a transition to 3D and by extension VR is far advantageous and will result in more well-informed decisions.Project managers and construction workers can use the threedimensional visualisations to check what the different construction elements should look like.Furthermore, if multiple epochs of the 4D as-design BIM model are loaded, they can assess the foreseen progress.Moreover, users can view the imported datasets from every possible angle, augmenting the level of immersion, especially when the visualisation is implemented in VR.Also later, after building completion, the visualisations can serve well for facility management, where possibly additional information can be added to the as-built BIM model such as described by (Wang, 2013, Shi et al., 2016, Zou et al., 2018).
The importance of using VR in the AEC industry is reflected by the extensive literature on this subject.Asgari et al. (Asgari and Pour, 2017) stress the unique opportunities of this approach as it yields multiple advantages over current processes.First of all the scene is represented much more comprehensibly, resulting in a more well-informed decision-making process.Moreover, the persons involved in the decision-making are no longer limited to those that have visited the construction site recently or have read the progress reports.Remotely visiting the site becomes a possibility.Furthermore, the models and VR environment can also be used for training staff for a multitude of tasks among which safety precaution is an important one (Li et al., 2012, Sacks et al., 2013).
During the design phase of the building, employing VR offers the advantage that the model can be viewed more immersively and comprehensibly.By integrating the as-design model in a VR environment and doing so in real-time, the designers and stakeholders are able to immediately see changes to the design.Johansson and Du et al. (Johansson, 2016, Du et al., 2018b) both developed such applications that visualise a BIM model in realtime while designing.Their applications can handle large BIM models and are both implemented as an Autodesk Revit plug-in.
The 2D plans and more recently the 3D as-design BIM models are one of the most important products for the construction pipeline.Therefore extensive literature is available on aiding construction workers with correctly interpreting these.Protchenko et al. (Protchenko et al., 2018) present their work on an Augmented Reality (AR) application that superimposes the 3D asdesign BIM model over the 2D plans to avoid misconceptions on constructing elements.Wang et al. (Wang et al., 2014) present similar work on an AR application, although more focussed on the construction of MEP installations.By extent, their application also allows the user to visualise the different stages of the construction, further decreasing the possibility for misconceptions and errors in the construction phase.In the work of Du et al. (Du et al., 2018a) a more elaborated application is presented where multiple stakeholders can use a cloud-based VR system simultaneously and remotely for easier decision making.Also similar commercial applications exist that offer these opportunities, such as IrisVR Prospect (IrisVR, 2019) and Enscape (Enscape, 2019).
In all aforementioned work, one of the crucial necessities is the implementation of the as-design BIM model into a VR or AR environment.This can be achieved through gaming engines that form the bridge between the BIM and the VR environment.However, current BIM software packages such as the commonly used Autodesk Revit, typically use native formats to save the created models.These are often not readable for gaming engines and thus a conversion is required.Different solutions exist.One of them is exporting to a *.3ds file (Protchenko et al., 2018) which is a type of format that is readable by gaming engines such as Unity.This method however only allows for importing the elements' geometry into the gaming engine platform, while texture information is lost.A more common approach is to import or link the Autodesk Revit model into another Autodesk software package: Autodesk 3ds Max and then saving the model as a *.max file so it can be imported into a gaming engine environment (Johansson, 2016, Motamedi et al., 2017, Wang et al., 2018).Alongside these conversion options, our work presents several others.
As far as point cloud data are concerned, typically, field-or brandspecific software or open source solutions such as CloudCompare are used to visualise and interpret these.Although these software packages are very well suitable for these tasks, the transition from a visualisation of the data to an immersive VR experience within their software environment is not possible.However, for a better comprehension of the data, this would be beneficial, especially when the opportunity exists to simultaneously visualise the point cloud data with other data types such as the as-design BIM model.Despite the promising possibilities this offers, only limited literature on visualising point cloud data in VR through gaming engines exists.Gong et al. (Gong et al., 2017) present their work on a tool to update the design of systems in the manufacturing industry.To immersively visualise the current state of the installation, the laser scanning point cloud data is imported in a gaming engine environment.However, the exact process is not discussed in their work.Although this is the case in (Bruder et al., 2014), the data is not directly implemented in a gaming engine environment but is visualised in VR using a combination of the 3DTK point cloud library and the Viargo VR interaction library.Stets et al. (Stets et al., 2017) managed to import point cloud data into the Unity gaming engine, but do not describe the process either.Bergé et al. (Bergé et al., 2016) use the Point Cloud Library (PCL) in Unity to visualise point clouds captured by an unmanned combat aerial vehicle equipped with a laser radar sensor.De Lima Hernandez et al. (De Lima Hernandez et al., 2019) embedded uncoloured point cloud data of stone fragments of ancient Egyptian artefacts into the Unity gaming engine by using PCL as well.This allows them to digitally puzzle the pieces together, preserving their delicate state.The most extensive works on importing point cloud data and especially the visualisation of it are presented by Discher et al. (Discher et al., 2018) and Thiel et al. (Thiel et al., 2018).They present their work on effectively rendering point cloud data in VR using a multipass rendering approach, hence solving frequently occurring visual artefacts.
The implementation of meshes, a derivative product of point clouds, that represent the current state of the construction into the gaming engine environment is a possibility.This is frequently done in the heritage sector (Anderson et al., 2010, Carrozzino and Bergamasco, 2010, Rua and Alvito, 2011, Bassier et al., 2018).However, in the construction industry this occurs much less frequently.The reason for this is that the time for decision-making, and thus the time for recording the data and embedding these into a VR environment, is rather limited.One of the works that does implement construction site meshes, created by photogrammetric processing of aerial images, into a VR environment is that of Zollmann et al. (Zollmann et al., 2014).This allows them to monitor and document the construction progress on a regular basis.
The AEC industry is highly interested in construction site and progress monitoring.Therefore analyses of the current state of the construction performances in comparison with the designs are required.A method presented by Kim et al. (Kim and Kano, 2008) virtually creates a picture of the as-design model, from the same point of view as the fixed camera pictures taken on the actual construction site.In a similar work, a virtually created image of the 3D model is superimposed on a recorded panorama of the construction site (Gheisari et al., 2016).Both these methods allow for intuitively checking the progress and possible discrepancies between reality and design.By developping this approach further into a real-time application, faster decision-making becomes possible (Kim et al., 2018).In another work a mobile AR application that can be used on the construction site is presented.One function of the application allows users to visualise the designed 3D model in-place on the construction site.However, the focus in this work is more on the assembly installations rather than the construction of buildings (Wang et al., 2014).While these applications offer the advantage of visualising both the reality and the design, the actual determination of progress still has to be carried out by the application's user.Various research has been published to overcome this by developing algorithms that automatically determine the achieved progress (Golparvar-Fard et al., 2009, Turkan et al., 2012, Kim et al., 2013).
Another interest of the AEC industry is a regular qualitative assessment of the constructed elements.Although applications such as Cloudcompare enable for quality assessments of the achieved performances, only limited related work on implementing this type of analyses in an immersive VR environment has been published.Kwon et al. (Kwon et al., 2014) present their work on an AR application that can be used in real-time during the construction to verify whether the dimensions of the constructed elements are correct.However, the location of the elements cannot be checked with their application.The work presented by Zhou et al. (Zhou et al., 2017) includes this feature and checks for the correct placement of tunneling elements.
Although the aforementioned applications also offer various possibilities for qualitative evaluations, one of their drawbacks is that the virtual positioning of elements on construction sites with little to no earlier constructed elements or on highly cluttered construction sites is cumbersome and thus error-prone and less accurate.The lack of necessary qualitative features hampers an easy and accurate determination of position and orientation on the construction site.Therefore our work takes an opposite approach by bringing the reality, accurately captured by remote sensing techniques, into a virtual platform instead of taking the virtual model to reality.Although real-time solutions are precluded this way, the obtained accuracy is far better.This advantage is crucial, especially for quality assessments that check for wrongly placed elements.Furthermore, the AR methods, overlaying the model on the reality, result in a partial loss of important information since the 3D model is downgraded to a two-dimensional picture that does not contain the same level of information.Therefore our method proposes an opposite approach: instead of downgrading the 3D design model, we upgrade the representation of the current state to a three-dimensional one by capturing the site with remote sensing techniques and implementing the resulting analyses into a three-dimensional fully immersive VR environment.This way the maximum amount of information and the high accuracy are preserved.

GAMING ENGINE VISUALISATION
For a proper visualisation it is paramount that all datasets share a common reference system.If this is not the case, several anomalies can occur.First of all a positional shift between two models is possible.Furthermore, it is also possible that the scale is incorrect.This certainly is the case for photogrammetric reconstructions of the construction site if no ground control points, that mostly form part of the reference system, are used.Finally, also the orientation can be wrong (Vincke and Vergauwen, 2019).These anomalies misinform decision-makers, certainly if multiple datasets are visualised simultaneously, resulting in erroneous decisions.For instance if discrepancies between the designed and the actual, built walls are noticed in the VR application, this could lead to tearing down the structure and rebuilding these elements at the correct place.
Gaming engines are typically heavily optimised to visualise mesh data.However, other data types and formats are present in the AEC industry.To display these correctly, a conversion of the data into mesh objects is required, since the used version of the Unity gaming engine (version 2018.2.14)only accepts this type of input.In the following paragraphs, the embedding procedure and data conversion to mesh objects are explained.Moreover, the benefits and possibilities of the visualisation for each data type are discussed.

Meshes
Meshes of construction sites hold valuable colour and texture information.For visualisation purposes this is crucial.The geometry of meshes is derived from point clouds.The colour information is either vertex-based in case of laser scanning or trianglebased with textures created from imagery in case of photogrammetric approaches.
As mentioned before, gaming engines form an excellent platform for the visualisation of meshes as they are originally designed for this.As a result, textured mesh models are swiftly imported into the Unity gaming engine without any problems.However, depending on the software the mesh was created in, it is possible that a correction is needed for the flipping of Y-and Z-axes, since Unity per default considers the Y-axis as the vertical axis.
The advantage of mesh visualisations is that a scene can be displayed with a large degree of textural detail while not occupying a lot of storage space or requiring high amounts of computational resources.However, the drawback is that these data tend to be less geometrically accurate, certainly at element edges.Scenes wit a lot of clutter and sharp-edged elements, of which construction sites are excellent examples, pose severe challenges to create high quality geometrical meshes.

BIM models
The use of BIM in the AEC industry is heavily increasing.The models serve as 3D plans for the construction works.Furthermore, it is also possible to extend the model with a time schedule, hence transforming the 3D model to a 4D BIM model.Similar to 3D BIM models, these serve as 3D plans for the construction works but they form also an important basis for the monitoring of progress.Moreover, both 3D and 4D models also allow for construction site monitoring and quality assessment.
The visualisation of as-design BIM models, but also of as-built BIM models, is more complex.First of all it is essential to convert the data to a format, readable by the Unity gaming engine.Therefore the data are converted into meshes.To this end, several possibilities exist (figure 2).A first approach is to save the BIM model as an *.ifc-file.Subsequently, this file can be imported in Blender via an open source plug-in, called Ifc OpenShell, that converts IFC-elements into mesh objects.Following, the Blender project is saved as *.blend-file and imported into Unity.However, only the model's geometry is exported without any texture information.As described in section 2., a second and more commonly used method converts the BIM model into a mesh using Autodesk 3DS max.The third method partly makes use of another visualisation software package called Lumion.The Lumion LiveSync add-in for Autodesk Revit allows users to import their model into Lumion.The free plug-in exports a Collada (*.dae) file that describes the mesh surfaces.Moreover, it also exports the material textures, present in the project.Subsequently, instead of importing these files into Lumion, the Collada files can be imported into Unity.Opposed to Lumion, Unity allows scripting and other user inputs, making it much more extensible for our application.The presence of texture is paramount, especially for visualisation purposes such as in the VR environment.Therefore the last 2 methods are the most successful ones, with the latter giving slightly better visual results.
The visualisation of BIM models is invaluable.As a result the BIM model can be intuitively traversed in the gaming engine and by extent in VR, such as described further.Furthermore, apart from project managers, who have access to specialised software packages to view these models, also other stakeholders such as clients easily can use this solution.Since the visualisation is build in Unity as a standalone serious game, the need for expensive field-specific software, only to visualise the designs and captured stage of the construction works, is eliminated.

Point cloud data
The visualisation of point cloud data is very important to determine the current state of the construction works.These point clouds are either captured by laser scanning or are created via photogrammetry.
Similar to BIM models, the visualisation of point cloud data is troublesome.Opposed to BIM models, point cloud data do not consist of planar surfaces, hence preventing a similar simple conversion to a mesh model.The visualisation of points in Unity is infeasible because points are infinitely small and therefore not visible.Hence, using an open source plug-in developed by Keijiro (https://github.com/keijiro/Pcx),each point is converted into a tiny mesh face upon importing the data into Unity.Depending on the size and type of project, the size of the mesh faces can be adjusted to increase their visibility.The small faces are coloured based on the colour information of the vertices.
Despite requiring much more resources, the visualisation of point cloud data offers a big advantage.Opposed to meshes, often derived from these point clouds, the captured data is shown in its raw, unprocessed form in case of laser scanning and in a less processed form in case of photogrammetry.Consequently, the geometrical quality of these data is higher, certainly at pro-or intruding elements and edges, often encountered at construction sites.

Analyses
The monitoring of progress at construction sites and certainly the assessment of possible deviations, require an analysis of the captured data and plans.Both phenomenons can be detected by comparing multiple datasets.By colouring one dataset according to the discrepancies between the two, deviations or progress can clearly be visualised, such as in figure 3.
Depending on the data type, importing the analysis results is done according to one of the aforementioned procedures.In our work a deviation analysis between the laser scanning point cloud and the as-design BIM model was executed.CloudCompare was used for this assessment, after which the coloured point cloud was imported into the Unity environment.
The ability to immersively visualise an analysis result in VR is advantageous insofar that it becomes much more comprehensible.This certainly is the case if various entities can be shown simultaneously.This is extremely useful for showing the results of the quality assessment such as is the case in figure 3, but also construction progress can be visualised in a similar way.

IMPLEMENTATION IN VR
The serious game is further developed into a VR application using the Oculus API.By implementing this, it is possible to traverse the scene and visualise the imported data and analysis in high detail in VR.Furthermore, the experience becomes more immersive, augmenting the level of comprehension of the project and the analyses.
The ability to toggle on or off datasets and display various instances simultaneously is paramount (figure 4).Therefore we implemented this feature as it adds to the comprehension of the displayed data.If both the as-planned and as-built state of the construction are at hand, rudimentary visual progress monitoring is possible.The comparison between "what is" and "what should be", intuitively happens when traversing the scene.Furthermore, it allows for a raw deviation analysis when toggling on and off elements to see whether or not these are located according to the as-design BIM model.For both analyses the use of a common reference system for the various datasets, i.e. a correct localisation, orientation and scale, is of large importance.
The number of points needed for a proper visualisation of a construction site exceeds by far the number of mesh faces for this.As a result, smoothly displaying such large amount of data in VR becomes cumbersome.However, gaming engines typically deal well with visualising vast amounts of meshes by only loading and displaying the meshes in the field of view of the user.Therefore, before importing, it was decided to split the point cloud into dozens of parts such as in figure 5.This way the necessary resources are divided by a factor 3 to 4, improving the graphical results substantially.

DISCUSSION
The conversion and importing procedure of the various datasets still require several software packages.However, in our work embedding the multitude of data types in the gaming engine environment is only showcased as a proof of concept.Implementing all of the different datasets, such as both the point cloud data from the photogrammetry pipeline as well as the laser scanning point cloud, is rather trivial.Previously, it was not possible to simultaneously visualise point cloud data of a certain stage of the construction works, together with the as-design BIM model.Recently, several commercial software applications made this possible.However, these solutions are mostly rather expensive.This way frequent visualisation was only available to designers, construction site managers and project managers who have access to these software packages, but not for other stakeholders such as clients.Also if the different parties use different software packages this can be cumbersome.By embedding the datasets in a open API platform such as provided by gaming engines, the simultaneous visualisation of various data types becomes possible and can be consulted by all the project's stakeholders.
Although the current importing procedure of point cloud data works perfectly, there still are some artefacts in their visualisation due to the high density of the recorded points.This was already partially solved by splitting the point cloud data into several parts.Because the circular or triangular representations of the points sometimes almost coincide, depending on the position of the user, the software has difficulties to properly display them since they all want to protrude to the foreground, hence resulting in a frequently occurring flickering phenomenon.

Future work
The presented framework can be developed much further.Several future developments and implementations will be carried out.First of all the conversion methods into Unity can be optimised such that the data is automatically converted into mesh formats upon importing can be optimised.As described, it is currently necessary to use external software packages for this.Research will be done if and how these conversions can be implemented into Unity such that the data is internally converted into a readable format upon importing.For the analysis solutions for instance this can be achieved by linking parts of the open source API of CloudCompare into the application.
Another challenge lies in the optimisation of the current visualisation performance.As mentioned previously, flickering problems frequently occur.Furthermore, when viewing large point clouds or a large number of point cloud parts, the frame rate in the VR environment drops below a satisfactory threshold.Also the import and visualisation procedure can be worked out further to speed up the processes, since the time available for the whole procedure should fall within the limited desired time span of the construction industry.If a construction element is located incorrectly, this should be noticed as soon as.
Further possibilities to enhance the application are possible as well.For instance measurement tools can be included to intuitively assess the dimensions or even the discrepancies when traversing through the scene.The game can also be developed further to suit facility management requirements by adding extra necessary information.

CONCLUSION
Although the implementation of products of the design phase, i.e. the as-design BIM model, in a gaming engine environment is researched profoundly, the work on gaming engine implementations of datasets that visualise the current situation and the derived qualitative analyses is rather limited.Our work presents a solution for this, allowing an implementation and visualisation of many different datasets encountered on construction sites, such as as-design and as-built BIM models, point cloud data, meshes and quality analysis results.Moreover, our system is also capable of showing these datasets simultaneously in a VR environment, augmenting the level of immersion.As a consequence, the targeted users, such as project and construction site managers, management people but also other stakeholders such as clients, have an increased understanding of the project and its challenges and problems.Using the open platform will result in a less errorprone building process closer to the proposed schedule and will lead to better informed decisions.Furthermore, future optimisations are still possible to enhance the application to an even more useful instrument in the processes of construction, progress monitoring, decision-making and quality assessments.

ACKNOWLEDGEMENTS
This project has received funding from the FWO research foundation (under FWO PhD SB fellowship 1S11218N) and the Geomatics research group of the Department of Civil Engineering, TC Construction at the KU Leuven in Belgium.We would also like to thank Willemen Group for giving access to their construction site.

Figure 1 :
Figure 1: Three datasets (as-design BIM model, photogrammetry point cloud and deviation analysis on laser scanning point cloud) are successfully imported into the Unity gaming engine.The models overlay precisely due to a common reference frame as is proven by the non-altered point of view.(Remark: this figure is a photoshopped combination of the three original images, from an unaltered point of view.)Belgium.These results are then further extended towards a more immersive VR visualisation.

Figure 2 :
Figure 2: Diagram of the different methods to import a BIM model into the Unity gaming engine environment.( 1 We used Autodesk Revit as BIM software.)

Figure 3 :
Figure 3: The ability to show datasets simultaneously in the user's VR view, increases level of comprehension substantially.The discrepancy between the left grey part of the as-design BIM column and the point cloud is smaller compared to the right side, hence the typical CloudCompare blue/green and yellow/red colourisation.

Figure 4 :
Figure 4: The ability to toggle on or off certain models benefits the workability of the game.The depicted script allows to couple the Oculus Rift controller's buttons to a certain model.

Figure 5 :
Figure 5: By splitting the point cloud data into multiple parts, the game's frame rate can be increased substantially because only the point cloud parts in the field of view of the user should be rendered.