Qualitative testing of an advanced terrestrial laser scanner simulator: users experience and feedback

The current situation, namely the Corona crisis in 2020, has demonstrated that digitalization of teaching is not only an option, but also the indispensable necessity of present reality. The pandemic has particularly affected those academic programs where classroom teaching is unavoidable, for example, in practical courses with surveying equipment. Within the project VirScan3D, a terrestrial laser scanner simulator has been developed as educational tool for teaching and learning practical laser scanning processes. This article describes the functionality of the newly developed software and simulator, the evaluation of the simulator by qualitative testing as well as usability testing results conducted by independent users recruited in university courses.


INTRODUCTION
The current situation, namely the ongoing Corona crisis of 2020, has demonstrated that digitalization of teaching is not only an option, but also the indispensable necessity of present reality. The pandemic has particularly affected those academic programs where classroom teaching is unavoidable, for example, in practical courses with surveying equipment. Figure 1: Example of a heritage building as virtual object to be scanned in the 3D terrestrial laser scanner simulator VirScan3D. Visible are the avatars hand holding a checkerboard target and the BLK 3D laser scanner, which will be placed in the selected virtual environment.
This project, with the acronym VirScan3D, is funded 2019 to 2021 by DAAD within a program for 'Supporting the internationalisation of Ukrainian universities: German-Ukrainian higher education institution collaboration'. The development was originally initiated by the lack of highlevel technical equipment in many countries, for example in Ukraine.
Previous work was published in Chizhova et al. (2020) and Popovas et al. (2021). We have presented a concept and first results of a terrestrial laser scanner simulator -a software tool, which could be a valuable educational tool for geomatics and engineering students as well as for heritage conservation students (s. Fig. 1 for a view of the interface of the simulator).
The developed software tool of VirScan3D (in the following also called: simulator software) allows users to reproduce the sequence of practical actions at the stage of fieldwork, and create realistic data in the absence of a real measuring device (Fig. 2). During the Corona crisis, it became obvious that a simulator-based virtual teaching tool is of even higher interest for remote online teaching and learning.
Users can plan their fieldwork in the virtual environment, i.e. specify 3D scanner type and target positions or change scanner settings. In addition, users can observe the process of virtual scanning and analyse the results. The VirScan3D simulator software can be used to create virtual 3D point clouds from different stations; in a further step, these point clouds have to be registered (and geo-referenced) and processed to higher-level information such as 3D models, architectural drawings or maps. The virtual 3D scanner emulates commercially available instruments with realistic outer appearance of the device and matching userinterfaces; it creates 3D point clouds in a virtual simulation according to user-selected individual specifications and settings.
In this described project, currently, the development of a laser scanning simulator is in the testing phase of the software's' first version. This article describes the development results and qualitative user interface testing, an evaluation conducted by independent users from university courses at the Centre for heritage conservation studies and technologies at the University of Bamberg.

STATE OF THE ART
In the educational field, the term of ‚educational simulation' has been formed, which refers to different environments representing the behaviour of any real objects, systems, phenomena for study and investigation (Aldrich, 2005;Shannon, Johannes, 1976). The ability to reflect and interact with a simulated object ensures a highly effective educational process in an adequate framework (Fallon, 2019;Widiyatmoko, 2018).
Simulators can be considered to be a product. High-quality development of a simulator definitely requires a user perception/ feedback analysis. Social sciences (anthropology, sociology, psychology) have created a wide pool of methods to study consumer needs (UXresearch). These methods have to be adapted for current UX tasks without losing data quality and keeping within financial limits.
The main challenge is to understand the target users and helps to place these at the centre of design process. UX-research is introduced by quantitative and qualitative testing. Quantitative methods investigate mass events, phenomena and processes in order to find patterns and the ability to predict events. Qualitative research focuses individuals' life experiences in a particular context using observation or interaction with test participants. As an example, evaluation of a metric test object informed by 'user requirements for better 3D recording of cultural heritage artefacts' has been presented in Hess 2016). Generally, qualitative testing draws from the context in which events occur and attempts to 'describe occurrences, as a means of determining the process in which events are embedded and the perspectives of those participating in the events, using induction to derive possible explanations based on observed phenomena' (Gorman, 2005).
The field of UX-research has a wide range of research methods available depending on the stage of product development. At this stage of product design, the methods most often used are for example card sorting, usability studies, participatory design etc. (Schuler et Namioka 1993), that belongs to qualitative methods of UX-research. Usability studies (or usability tests) are the most effective approach to determine ways to improve the software in the next development cycle, based on user experience and their feedback.
In general, the process of technical software testing consists of 'finding bugs'. In detail, the methodology of software testing has been described in (Lewis, 2009) and is based on ISO 9001 norm (ISO 2018).
Qualitative data is usually collected by audio recording, video recording, transcribed texts from interviews with single participants, or focus groups. Interpretation is achieved by analysis of recordings or notes through the formulation of categories against which a text is analysed. An introduction to qualitative research methods, data collection and interpretation methods, alongside case studies, is given by (Gorman, 2005).
Usability inspection method have been described by Mack and Nielsen (Mack and Nielsen, 1993) and use heuristic data (i.e. collection of clear feedback on command methods, often still pen and paper forms). Participants with domain knowledge improve the usability testing. The inspection aims at identifying usability problems in an interface and aim for the highest interface efficacy (in a design iteration). The success of a formative testing will be the absence of usability problems. Additionally, information can be teased out of the participants by Think-Aloud-Protocol, i.e. by asking participants to speak along as they complete tasks. Makri reflects on the advantages and disadvantages of Think-Aloud protocol and applies the 'pret a rapporter' framework, (Blandford et al. 2008) and (Makri et al., 2010). Interviews can subsequently be transcribed and analysed, with software such as for example Nvivo (QRS International, 2021).

QUALITATIVE USER TESTING METHOD / TEST DESIGN
We will now report on the results of the 3D laser scanner simulator testing using one of qualitative methodsusability studies -by engaging a focus group of users as testers and analysing their feedback. In this case, the qualitative simulator testing includes the evaluation of technical software aspects and user perception of user interface and general user-friendliness and usability.
The VirScan3D simulator evaluation has been designed in the form of a qualitative user test with 1) re-useable rated questionnaires, 2) free text and 3) free, live discussion.
The test execution has been presented in five parts: a. Introduction to the VirScan3D project and brief description of the simulator software aims (10 minutes). b. Use of VirScan3D simulator software: opening/launch, in-scene navigation and functions for orientation by the users themselves/ unaided (10 minutes). c. VirScan3D simulator: task of active survey preparation and scanning process of the building in the simulator by the users themselves/ unaided (50 minutes). d. User questionnaire: evaluation and rating of the user interface and the simulator (20 minutes, details of questions see section 3.2) e. Joint discussion of the impressions of using the simulator with the VirScan3D project staff (20 minutes).
The evaluation has been designed with a reproducible technical set-up, including video and audio recording. The qualitative user testing took place in the computer cluster of the Centre for Heritage Conservation Studies and Technologies (KDWT) at University of Bamberg (s. Fig.  3).

Report on user testing execution
The user testing was conducted in December 2020, just before the universities' complete shutdown of in-presence teaching during the Corona Pandemic. Prescribed hygiene rules and social distancing were followed. Two tutors, also VirScan3D project staff, were present during the testing.
User testing part a): On purpose, to foster the expression of self-exploration and unbiased impressions, no instructions were given on how to use the simulator. During software testing, participants were asked to explore the basic functionality, such as simulator software start, -navigation/ movement in the virtual 3D environment, -selection and placement of targets, -3D laser scanner device placing and initialisation of 3D scanning, -functional calls in the simulator software (by the users themselves), -difficulties and simulator software errors, -additional options/ functionalities to be implemented.
User testing part b): Then, the participants had to select an architectural object inside the 3D virtual environment on their own and to 3D record/ scan it, including activities such as survey planning and necessary preparations (target choice placing, laser scanner positioning etc.).
It was observed that -even if not encouraged by the tutors and not necessarily desired by the tutors -conversations and discussions and mutual helping began in all three groups of these engaged and interested students during part a) and b) of the user testing. All students were keen to contribute to a discussion and to solve problems amongst themselves, and the tutors were asked very few questions.
User testing part c): An additional questionnaire, filled in during the user testing, had been prepared using an online form. In the questionnaire not only the technical testing of the software functionality was queried, but this evaluation also included a survey of participant perception and preferences regarding the simulator software, if they imagined themselves as possible future user. The questions were specifically directed at: intuitive understanding of the software interface of the simulator software and its commands, -difficulties in finding the necessary functions in the simulator software, -necessity of additional explanation to the new user, -difference in using a virtual scanner in comparison to a real device in life, -simulator software usability for planning.
In addition to the multiple-choice function in the questionnaire, the participants had the opportunity to contribute with a free text, which was subsequently analysed. The duration of filling the questionnaire was ca. 20 minutes.
User testing part e): This was followed by a discussion with the tutors and impressions about the simulator software the users had just tested. Additionally, the discussion has been audio recorded for the purpose of further improvements to the simulator in the next iterative programming step.

Report on survey participants and participation mode
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B5-2021 XXIV ISPRS Congress (2021 edition) The qualitative user testing took place in three consecutive groups in the universities' computer cluster and online. The current version V05.7 of the simulator has been tested by a group of students of the master course 'Digital Technologies in Heritage Conservation' at the University of Bamberg. Their educational backgrounds were quite heterogeneous, ranging from engineering to humanities. All of them were able-bodied and none of them reported colour-blindness.
Altogether, 22 people participated in the user testing. Two participated online by hybrid participation (video and screen sharing). 20 questionnaires were filled in German and 2 in English. The participants were aged between 21 and 35 (the majority of them between 21 and 28). Gender was optionally registered (Fig. 4). Participating students from first and second year of the masters' program had different levels of experience with laser scanning. In particular, they were asked if they knew how 3D laser scanning technology is applied and how the data is processed (s. Fig. 5).

Figure 5:
Knowledge about 3D scanning technology and 3D data processing.
The first-year students already were two months into their studies. Most of them had used laser scanning in reality at least once (s. Fig. 6). The simulator is based on the Unreal gaming engine, and therefore the controls are similar to common gaming controls. Therefore, it was important to know, which experiences the participants already have in computer games. The skills should be rated from 0 (none) to 10 (advanced users) by the participants (s. Fig. 7). Figure 7: Distribution of computer gaming skills by participants.
As can be seen from diagram above, there was a quite equal distribution between inexperienced users and advanced users in terms of computer gaming experience.

Report on user testing of the simulator software
The intuitive interaction within the simulator depends on many factors, firstly, the previous knowledge about laser scanning and gaming skills of each user. A minor percentage of participants (especially without gaming skills) were not familiar with the WASD keys for moving (s. Fig. 8).

Figure 8: Movement/ navigation evaluation within the simulator
Some participants found the movement/ in-simulator navigation confusing and not sufficiently intuitive due to individual movement perception. "The room 'moves' with you when you move. Towards the end of the experiment, I also had the feeling that it was getting slower and slower", -feedback from one of the participants. Overall, majority of test participants did not experienced difficulties while moving within virtual environment.
Placing, movement (i.e. free navigation in the virtual environment) and elimination (i.e. deleting) of equipment or already placed scanners or targets were not intuitive for a small group (s. Fig. 9). Some of them had difficulties to combine the correct keys with a mouse. Figure 9: Evaluation of target and scanner placement.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B5-2021 XXIV ISPRS Congress (2021 edition) Many participants found the need to implement additional introductions and to optimise existing help menus. By finding the necessary functions in the simulator software, the group divided quite evenly (s. Fig. 10). Description of movement controls, jumping, and mouse control description either in beginning or under the "Help" should be implemented in the later versions of the simulator software.

Summary of observations on the user testing
Obviously, the simulator represents an idealised virtual process, resulting in some differences to real conditions. The simulator software simplifies the process in many ways. On-site, on the one hand, there are always difficulties with disturbing factors, especially with regard to a scanned object. Furthermore, in reality there are often technical and logistical difficulties.
Some things can be easily achieved by one-click in a virtual environment and difficult to imagine in reality, like: walking through walls, -placing targets on any surface or in the air, -jumping or teleporting with equipment in hands, -the absence of obstacles such as pedestrians or cars arbitrary setting of the scanner height in the air etc.
In reality, it is not easy to reach a scanner height of 10 metres and place a target marker on a tower. In addition, it is necessary to learn how to balance the tripods. Apart from that, as already mentioned above, the possibility of compatibility control is necessary.
During the measurement, the laser beam hitting the surfaces are shown and the coverage scanned area can be shown. Although this does not correspond to reality, it was visualised for didactic reasons. The opinion was also frequently expressed that an overview plan with scanning stations and targets would be helpful for a measurement planning.
At the current stage of development, it is not possible to reproduce real scanning time, because it depends on computer performance as well as individual scanning settings. Sometimes the simulator software could even crash, if too many stations or recordings with the highest resolution have been chosen. Many participants agreed that the remaining scanning time as well as produced data volume could be displayed.
Generally, the software was understandable for most of the users, but additional explanation and development of further functionality is required. On the question of whether the software is intuitively understandable and the participant feels like an advanced user, the simulator had middle scores (s. Fig. 11).
In addition to the technical aspects, general questions about the simulator's usability for teaching, planning, research as well as overall perception were also discussed.
As expected, many differences were evident comparing a real and a virtual device. The answers about this difference were mostly distributed from middle to 'too big, have nothing in common' (s. Fig. 12). Asking about the simulator need for measurement planning, most of the participants approved the software capabilities in the simplification of on-site processes (s. Fig. 13).
Replacement of the real scanner with the simulator for learning 3D laser scanning methods had no clear tendency  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B5-2021XXIV ISPRS Congress (2021 to be observed. The answers were relatively evenly distributed (s. Fig. 14).

DISCUSSION
The objective of this research was to analyse acceptance and usability of a complex virtual simulator software for terrestrial laser scanning. Adapting and programming of a real scanner is not trivial and requires a deep understanding of internal functions as well as practical experiences. Therefore, after development of the first prototype version (Chizhova et al., 2020;Popovas et al., 2021), it has to be evaluated how users, especially students, interact with the system in order to improve functionality and usability.
From a technical point of view, several errors and bugs (e.g. walking through the walls) were first identified and already fixed in the newest version of the software. Generally, the testing allowed defining the tendencies for further development of the simulator according to the participants' suggestions.
Some user insights are worth to mention specifically: introduction of movement controls and more informative help menu, -scanning status bar and remaining scan time indicator, -possibility to use several different scanner models at the same time, -a 'minimap' (small overview map in the bottom right corner, for example) of the spatial environment for better orientation and more evenly distributed positioning of targets and positioning of scan stations. -Some suggested functionalities during testing are now already implemented in the current version of VirScan3D simulator, like a scanning status bar and remaining scan time indicator and visual aid of movement control.
Since not all material properties were implemented in the simulator, materials show incorrect intensity values in the resulting simulated point cloud. For noise simulation, random noise distribution was implemented initially, which differs from reality. Further investigation and implementation will be undertaken in the future.
Another important point is the simulator application and dissemination as a teaching tool. The pandemic situation   with travel restrictions, ongoing online or hybrid teaching as well as future reduction of flights and travel from an ecological and economical point of view, is a good opportunity to develop the simulator as an applicable pedagogical alternative.
Besides the elimination of technical errors, the development of the simulator will be based on the expansion of its functionality inspired by the user evaluations. The simulated 3D-scanning process, as well as data processing steps, should be brought closer to reality.
The intention is that the quality of the captured 3D data will be more applicable for teaching and research. Furthermore, a closer didactical observation option of the geospatial perception during the recording simulation should be improved to guarantee the desired teaching impact.
In this context, the following steps will be taken on the simulator software VirScan3D: technical improvements, -functionality extensions, -further tests involving wider participant groups, -project extension and new funding acquisition.
It is expected that implementation of above-mentioned measures will allow to improve the simulator to the extent, where it can be efficiently used as a valuable educational tool. With such a virtual system, users can learn and prepare to operate 3D terrestrial laser scanners, plan how to properly distribute targets and select optimal scanner locations, generate close-to-reality datasets. Moreover, extended version of the VirScan3D simulator might be used as a research tool, which allows analyse simulated data from different virtual sensors in different virtual environments.

ACKNOWLEDGEMENTS
The project is funded by DAAD (German Academic Exchange Service) from 2019 to 2021 within a program for 'Supporting the internationalisation of Ukrainian universities -shaping the digital future together: German-Ukrainian higher education institution collaborations'.
The financial support and opportunity for this collaboration between three universities, two in Germany and one in Ukraine, is gratefully acknowledged.