A USABILITY EVALUATION OF A 3D MAP DISPLAY FOR PEDESTRIAN NAVIGATION

: This paper is focused to address the map display usability for finding given POI addresses in a popular urban city area. LOD 1 of 3D representations of city buildings are presented into a 2.5D map for pedestrian navigation test. This 3D map display is evaluated against familiar 2D map system on the test participants’ smartphones. 16 participants were involved in the field test. The typical walking model of a searching task that is focused only to look for a certain address of building is chosen as the way finding model during the field test. Three kinds of navigation processes i.e. self-orientation, spatial knowledge acquisition and navigation decision for searching task were evaluated for each test participant. Usability measures of 3D map-based display over 2D-map based display for pedestrian navigation were collected from test participants’ mobile devices. In addition to that, activities of test participants in terms of acceleration and orientation information are used to support analysis of pattern and trends of test participants. As the testing app is also intended to support smart city application, its ability to provide user report on complaints was also assessed. Most participants agreed with the statements in the questionnaire that were organized into three sections, i.e. addressing participants’ interaction, participants’ responses in navigation processes and crowdsensing. The results suggest that 3D map-based pedestrian navigation is more usable to be used to look for a certain address of building in central tourist area of urban city.


INTRODUCTION
Pedestrian navigation is a research domain that is a multidisciplinary research related to navigation research (Montello 2005), location-based services (Chen et al. 2010) and human cognitive (Seer, Brändle, and Ratti 2014).In more particular, pedestrian navigation is the interaction field of study related to geospatial information, transportation intelligence, and built environment (Ma and Yarlagadda 2014).Developing mobile navigation systems that provide sufficient support to users to orient the reality of urban environment against the maps and information has been done in many researches.Effective and efficient route finding has been tested with the use of landmark visibility information (Delikostidis et al. 2013), sensor extension with GPS/GNSS module (Shi et al. 2015;Hsu, Yanlei, and Kamijo 2016) or RFID (Saeed et al. 2010), improvised filtering, route planning and maps (Khider, Kaiser, and Robertson 2012).In condition where GPS is obstructed and denied, dead reckoning method and inertial localization have been widely adopted (Shin et al. 2016;H. Li 2015;Ruotsalainen et al. 2013).In regard to map display technology, the problem with current 2D pedestrian navigation systems is about their adaptability for helping pedestrian travellers, especially when they are looking for specific target in the middle of similar urban canyon.Meanwhile, 3D pedestrian navigation systems may offer more realistic perspectives about urban environments, but it may provide too much complexity for the users.Differences in terms of visual attention produced by users in using 2D and 3D map browsers have been identified in other work (Liao et al. 2017).
Landmarks and 3D information is assumed to have crucial factors in assisting users to structure spatial knowledge and to recognize environments around the buildings shown on the map.
In a typical crowded urban environment, pedestrian navigation can be useful to support different types of navigation.The first type would be that users use their navigation system to find a specific but unfamiliar address (typical to searching task on search engine using fix keywords).The other type would be that users use their navigation system to look for interesting targets relate to a particular theme (typical to browsing activities through web pages), a typical activity done by tourists.Effective and efficient navigation can be measured based upon users' movement and accuracy toward the target.This paper focuses in testing a 3D visualization of urban environment to support pedestrian navigation.The case of searching for specific address and the case of looking for particular targets are tested to real users.For this purpose, digital map of 3D models of buildings and their corresponding POI (Point of Interest) are prepared as an android application.The app has an ability to locate the current position and to provide route navigation to find specific target of buildings.The study area is located in central tourist area of Yogyakarta City, which known as Malioboro pedestrian street.Users' movement are recorded and evaluated to assess the users' preferences and feedback.We would like to investigate which display will be effective for finding given POI address and for looking around objects in central tourist area of urban city.3D map-based routing is prepared as an app.Velocity, direction progress and orientation information support are also tested whether they are important to contribute the navigation task assigned to test users.As the app is also intended to support smart city application, its potential to provide crowd report will also be assessed.

Pedestrian navigation using 2D and 3D maps
Users preferences and modelling for pedestrian navigation using Location Based services have been a topic of research in Cartography and Informatics (C. Li 2006;Liao et al. 2017;Shin et al. 2016;Shi et al. 2015;Ma and Yarlagadda 2014).Technological framework that is applied for navigation systems depend heavily on positioning framework.The positioning framework can be delivered as network assisted solution or mobile device-based positioning services.Rapid developments of smartphones and apps development have created more opportunities to improve positioning techniques at the mobile device site with GPS, inertial sensors, RFID readers embedded on the device.The general framework of a client mobile device, hereafter will be called as apps, is integrating 2D and 3D terrain data, Server, Cellular Network with GPS data and apps at the user side (Noguera et al. 2013).
Pedestrian navigation model can be classified into at least two main models: continuous model (e.g.social task force) and discrete model (e.g.cellular automata) (Ma and Yarlagadda 2014).Changes of speed and space occupied can be used as parameters to identify motivation yet hindrances found during the path.People walking model than can be verified using position and orientation generated by users on the field during their task completion.Using web interaction as metaphor, the walking model can be of a typical of a searching task, where a specific place and target has been in traveller's mind.Meanwhile it can be the case that walking model can be of a browsing task, where a specific task with broader criteria rather than finding just one address is formulated.In pedestrian navigation situation, such browsing metaphor is applied when users only focus to visit historical buildings along the path or to do window shopping for unique city souvenirs.According to three kinds of navigation processes (Liao et al. 2017), self-localization, spatial knowledge acquisition and navigation for searching task will be narrowed only to a certain address of building.Meanwhile, selflocalization, spatial knowledge acquisition, and navigation for browsing task will be broader and involved more planning thinking in the beginning of the task execution.

LOD 1 and 3D representations
Level of details have been used to rank degree of completeness of building representations for graphic visualizations.LOD can be formalized as LOD 1 as simplest primitive representations, LOD 2 as building with roof representations, LOD 3 as refinement of LOD 3, and LOD 4 represents exposed building and interiors (Biljecki, Ledoux, and Stoter 2016;Biljecki et al. 2014).LOD for web processing and mobile applications are LOD 2 and LOD 1 (Nouvel et al. 2017).For developing an app in support of pedestrian navigation, LOD 1 is seen sufficient (Over et al. 2015).LOD 1 can be generated from digitalization of 2D scanned map (Ledoux and Meijers 2011;Gimenez et al. 2015), OSM (Goetz 2013), aerial photographs (Aljumaily, Laefer, and Cuadra 2016), or terrestrial or aerial laser scanning (Over et al. 2015).LOD 1 and other 3D representations of real world have been applied for improving better pedestrian way finding and navigation in urban environments (Liao and Dong 2017;Hsu, Yanlei, and Kamijo 2016).Before the emerging of app and GNSS uses, local landmark has been promoted to improve wayfinding experiences (Raubal and Winter 2002).It has been tested that the 3D building map in GNSS navigation system can be used to improve positioning accuracy (Hsu, Yanlei, and Kamijo 2016) and the 3D map is suggested to enrich user experiences in personal navigation (Chen et al. 2010).While the term 3D in navigation has been applied to the context of navigation of floors (Shin et al. 2016) or 3D (X,Y,Z) movements (Khider, Kaiser, and Robertson 2012), the term 3D here is limited to the LOD 1 building representation (Biljecki, Ledoux, and Stoter 2016).

Usability evaluation of 3D navigation
The term usability is used to describe how the product influences user acceptance in using the product (Folmer and Bosch 2004).There have been many variations in defining usability attributes.Usability attributes as defined in ISO 9241 refer to effectiveness, efficiency, and user satisfaction with which targeted users achieve specified goals in particular environments.Quesenbery (2004) defines the usability as the 5Es: effective, efficient, engaging, error tolerant, and easy to learn.In order to check whether the intended product or a service meet the required usability attributes against the expected users, usability evaluation is done.Usability refers to the technical aspects of a visualization (Bleisch 2012).Usability of pedestrian navigation has been evaluated in (Delikostidis et al. 2013) by investigating the influence of landmark visibility to improve navigation usability and in (Liao et al. 2017) by exploring differences in the use of 2D and 3D maps in pedestrian navigation.Usability evaluation on the effectiveness in using different map displays has been applied to reveal usability attributes of the 5Es effective, efficient, engaging, error tolerant, and easy to learn (Aditya 2010).Similar evaluation is conducted in this research to understand user preferences and requirements for better LOD 1 of 2.5D map-based navigation over 2D map-based navigation.This LOD 1 of 2.5D map-based navigation is simply called 3D navigation in this paper.

Designing the 3D Pedestrian Navigation App
In this research work, a prototype which is prepared as a testing app to understand the usefulness of LOD1-3D models for city navigation was developed.Inputs that were required by the apps include: (1) 3D building models originally created from OpenStreetMap's building features, (2) location information is derived from the GPS sensor in the android unit, (3) route services accessed from MapBox API, and (4) a simple basic map accessed from MapBox API.
The app is intended to support navigation and routing in urban environments.The urban buildings are generated from OSM data and visualized as LOD 1 models.The app has ability to self-localization, show buildings and the search target on the map displayed through the apps.The map was developed using MapBox API showing streets, simple 3D building units which are clickable to show their building attributes (see Figure 1 and Figure 2).
Technically, additional libraries are required to build the app.This includes a core component to display the map on the app (com.mapbox.mapboxsdk:mapbox-android-sdk:5.1.3@aar).

User Interaction and User Activity
This paper will focus to differentiate the usability measures of LOD 1 3D map against 2D map of the city buildings for pedestrian navigation purposes.Here the user interaction and user activity will be evaluated based upon users' experiences and feedback.For the user interaction, the usability measures were focused on the user search and the user's use of menus, buttons, and displays.In case of the user activity, the usability measures were applied into three parameters of navigation processes: self-localization, spatial knowledge development, and navigation decision.In addition to that, spatial reporting in regard to urban disturbances and violations were also evaluated.It is a case scenario when travellers can create onthe-spot reports or complaints related to specific disturbances attributed to urban environments found along their navigation path, such as smell and sound pollution.This mobility function is intended to support crowd sensing implementation for city planning.The potential use of the pedestrian navigation app to provide crowd sensing for urban city planning is also assessed in this study.It is expected that the crowd can participate to deliver spatially references complaints to noise and smell in the study area.Users are expected to make measurements and to give report through their smartphones in order to assess disturbances of smell and sound along the pedestrian path and its surrounding 3D objects.
The utility and usability of the 3D Pedestrian Navigation App for supporting users' pedestrian navigation and for supporting crowdsensing application are tested through the field test.The typical implementation of searching task in pedestrian navigation is used as the walking model.The typical implementation of browsing task like window shopping for unique souvenirs along the street is not tested in this research.
In order to derive the usability attributes as suggested by Quesenbery (2004), a questionnaire survey to test participants and elaboration of comments and test participant's sensor recordings were used as indicators to develop attribute values in regard to 5Es: how the app is considered to be effective, efficient, easy to learn, error tolerant, and engaging to be used by target users (Table 1).In this paper, self-orientation stage is started when test participants start to search the given destination targets, gain the results, orient his/her current/next position, and follow the navigation instructions.Spatial knowledge development includes test participants ability to differentiate the building target to others and to finalize the task.Navigation decision include users' preferences toward the display and their decision to follow the route to reach the target.

Usability Evaluation Test
Usability evaluation of pedestrian navigation and pedestrian crowdsensing were tested on the field.The field test was executed to reveal possible usability problems of 3D Pedestrian Navigation.The evaluation was done to gather information about the interaction and activities of each user when using this app.The location of this research is in the central tourist area of Yogyakarta City, which is known as Malioboro pedestrian street.

Test Participant (TP):
16 Test Participants are involved in the field test.The participants consist of 10 males and 6 females.They were undergraduate students.The participants were divided into 8 groups with two peoples in each group.Each participant walks to three different destinations.Each destination is a specific place that has been designed to be visited by test participants.Each participant must install apps on their smartphones.The apps installed include the testing 3D Pedestrian Navigation app, Google Mobile Maps, MyTracks and AndroSensor on their smartphones.

Procedure:
The applied usability evaluation was a combination of questionnaire survey and observation that include recordings of users' movement, direction and orientation.TPs navigation experiences were observed from TPs' origin points to given destination points.Their experiences in using the app for supporting their navigation task are the basis for assessing the app's utility and usability.The destination points have been designed based on a scenario movement.Users used of 3D map-based Pedestrian Navigation and Google Maps in this activity.3D map-based navigation routing can be accessed through participants' mobile devices.
The evaluation was prepared to investigate which display will be effective for finding given POI address and for looking around objects in central tourist area of urban city.Users' movement are recorded and evaluated to assess the users' preferences and feedback.Users were asked to run MyTracks app during the test.Acceleration, direction progress and orientation information were collected to identify whether the task assigned can be completed correctly.The sensor data belonging to participants were recorded by using AndroSensor app.The Androsensor app has ability to collect readings of accelerometer, gyroscope, light, orientation, proximity, sound and magnetic sensors.
Each participant has three destination points.The destination points of one participant were not the same with other participants.All participants find the destination point by typing the assigned POI through search textbox available on the app.Then, the participants walk to the target based on the direction of navigation display and instructions seen on the app.The participants are allowed to stop or change the direction.The scenarios reflected a real use of an electronic navigation tool to help users look for a specific POI.The test was done to assess the user's experiences using 3D Pedestrian Navigation app and Google Map in smartphone (see Figure 3).
During their navigation test, participants' movements were recorded by using the MyTracks app.The result of each participant's walking track is saved into a polyline geometry as a file in *.kml format.Data are plotted by using QGIS to view the TP's spatial footprints during the test.This data is used to verify tracks and patterns of participants' movements toward a destination target.Data on the acceleration, orientation, and direction progress on their navigation experiences were recorded by using AndroSensor app.These sensor recordings in regard to each participant's activity was saved into *.csv format.Subsequently, sensor recordings of test participants are processed using FusionTable in order to produce graphs.
Figure 3. Routing information with 2D map display (left) and with 3D map display in the pedestrian navigation app (right) Once completing the test activity, all participants complete the online questionnaire through http://ugm.id/3DMalioborosurveylink.The questions on the questionnaire were mainly used to assess the users' interaction and activities when using the app.
The questionnaire is divided into three sections consisting of 25 questions.The sections are as follows: 1. Section 1-Questions related to test participants experiences on their interactions with the app.2. Section 2 -Questions related to their activity in using the app (2D/3D) on the field to assist their self-orientation, spatial knowledge development, and navigation decision.3. Section 3 -Questions about their experiences in crowd sensing and in using the app.
The research data acquired from the usability testing comprised of five information: 1. users' movements as tracks represented in *kml format.2. users' screen captures of their navigation activities from 3D map display dan 2D Google map display.3. users' acceleration, direction and orientation derived from their mobile sensors.4. users' responses to the test questionnaire.5. users' feedback.
Steps undertaken during the test are as follows:  Test Participants (TP) turn on AndroSensor measurement to record inner sensor and MyTracks apps to record TP's walking path.Before starting to walk, Google Maps and 3D Pedestrian Navigation app should be opened. Each TP got three random Places of Interest to visit.TP starts the test by typing the names of POI onto search textbox both on Google Maps Navigation and the 3D Pedestrian Navigation App. Each TP look at the 3D map based display and follow the instructions given by the screen in order to achieve the target chosen navigation.
 Each TP can switch to the 2D map based display when considered the 3D map display is not helpful for his/her navigation purposes. For each target destination, each TP should stop AndroSensor recording.They were asked to restart their movement recording using AndroSensor for their subsequent destination.Meanwhile their walking path was recorded from the beginning till the end of the test for all three destinations using MyTracks. During the test, when they have complaints in regard to urban environments along the path, they were asked to take photo and to make comment. On the completion of the test, all TPs were requested to fill in an electronic questionnaire.
Table 2. Statements and responses related to Section 1 Using the Likert scale values from 1 to 5, the responses of Section 1 is presented in Table 3.

Average Responses
3.9 3.7 3.7 3.4 3.6 3.5 3.1 Table 3.The summary for TP responses to Section 1 From 7 questions that were asked to TPs, it can be summarised that the TP responses were all more to agree that to disagree.However, this is very clear that the representations of individual building in test areas are found to be not optimal (1Q7).6 TPs found difficulties in differentiating individual buildings on 3D map.This is very reasonable as the 3D display is limited to LOD 1 where 2.5D data structure individual buildings are presented as 3D boxes with different heights.4. Statements and responses related to Section 2 Using the Likert scale values from 1 to 5, the responses from TP for Section 2 can be summarised that the respondents gave the lowest average responses at the 10th question that is 2.8.While the highest average responses at the 7th question that is 4.4.7 The representation of 3D buildings ease me to distinguish buildings 0 6 5 3 2 Total 2 13 34 48 15

Section 1-Statements related to test participants experiences on their interactions with the app (1Q)
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W10, 2018 13th 3D GeoInfo Conference, 1-2 October 2018, Delft, The Netherlands From 13 questions that were asked to TPs, it can be summarised that the TP responses were all more to agree that to disagree.However, in case of 2.Q10 it can be seen that the participants prefer looking at the map display then choosing to their own path than following directions given by the app.In total 7 TPs responded that they did not follow the directions shown in the map display (2.Q10).Three of them also argued that the visual lines to 3D target is not clear (2.Q11).This can be understood as these participants have been familiar to the location.Thus, further field-test for TPs in more unfamiliar location could be suggested.The statements related to navigation processes correspond to self-orientation, spatial knowledge development, and navigation decision activities.
The processes started from TPs search for targeted POIs until their wayfinding activity to arrive at the targeted building and stop their interaction.
The test was intended to collect users' experiences in using the LO1 of 3D map for pedestrian navigation.This is important to improve the app utility based on TPs feedback.The test was also focused to enable participants to collect any complaints and disturbances related to sound and smell disturbances on the area.This was done to enrich design requirements on the potential use of the app for crowdsensing uses.In Section 3, statements number 1 to 3 were intended to evaluate the applicability of the app to be used for crowdsensing uses.
Statements number 4 and 5 can be used to improve the app's utility (Table 5).
Table 5. Statements and responses related to Section 3 Using the Likert scale values from 1 to 5, the responses for TP for Section 3 is presented in Table 6.

Statements of Section 3
3Q1 3Q2 3Q3 3Q4 3Q5 Average Responses 3.1 3.1 3.4 3.3 3.5 Table 6.The summary for TP responses to Section 3 statements As seen in Table 6, it can be summarised that the TP responses were all more to agree than to disagree.In regard to potential use of the app to make a complain report while using the app for pedestrian navigation, the app got positive support from TPs.The statement 1 (on sound disturbances or violations) was made in negative sentence to challenge the TP critical observations.The response is positive in terms that some TPS, although minor, still found there were some sound disturbances.The lack of sound and smell disturbances complained can be the case since the area is considered as the city's prime tourist area that is well preserved.In addition to that, the test was done during the opening hours of shops and places of attractions.The summary of evaluation from TPs on the app (Section 1/1Q, Section 2/2Q, Section 3/3Q) is given in Figure 4.It also provide overview on responses' composition between male and female participants.In the test, 6 test participants were female and 10 were male.
Figure 4.The summary of responses to the app's usability on interactions, navigation processes, and crowdsensing utility.

The Visualization of Users' Movements
The goal to save users' track using MyTracks and to collect TP movements using AndroSensor app was aimed at checking the pattern of TP navigation paths and the correctness of the visited target.The readings from accelerometer, gyroscope, orientation, and sound sensors for each TP were processed and plotted into the graphs format.The sensor data processing of all TP was done using Fusion Table .Tracks of test participants show that the navigation app can be used by the TPs in urban environements.All targeted POIs can be visited by the TPs.The length of navigation activity ranges from 15 minutes to 59 minutes in order to find 3 targeted POIs. Figure 5 shows the overall field test activity done by 16 participants.The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W10, 2018 13th 3D GeoInfo Conference, 1-2 October 2018, Delft, The Netherlands (Figure 6).In addition to such spatial patter as seen in Figure 6, the pattern can also be checked from the sensor data, especially on linear acceleration (towards X, Y, and Z movements) and device orientation.The black textbox over the sample plots indicates the position when the TPs found their navigation targets (Figure 7).

Usability Attributes
The responses to questionnaire's questions or statements on user interaction, user navigation activity and app's utility and crowd sensing were used assess the values of usability attributes offered by the app.From responses to 25 statements in this questionnaire, the assessments were made.The correlation between questionnaire questions to usability attributes was given in Table 1.All responses tend to agree to the statements.In addition to questionnaire, comments and feedback from test participants (TP) were also collected.The most critical feedback was on the availability and quality of pedestrian navigation lines available in 2.5D map.It is very clear that the accuracy and correctness of 2D based map (Google Navigation) are more appreciated than the one used for 2.5D (using MapBox API).A summary of the responses of participants to the comparison displays based on the 5E variable is shown in the Table 7.
From the questionnaire responses and the feedback collected, the comparison between 2D and 3D map based pedestrian navigation can be drawn.In regard to users-interaction, as also emphasized in the TP comments, the benefit for the use of 3D against 2D map based was not so emerging.Here, the function in regard to search for POIs will apply the same between 2D and 3D options, but the results selection and display using 3D map is highly appreciated.In regard to use of 3D map to support self-orientation, the responses are positive.Even more for the case of spatial knowledge development and navigation decision, the 3D map provide effective and efficient means to accelerate test participants to go approaching the destination.TPs also expressed their concerns that amidst their convenience to use 3D map to for supporting their navigation, they have difficulties to differentiate individual buildings.As discussed in Section 4.1, 6 TPs found difficulties to differentiate individual buildings as the LOD 1 buildings used in the system are only differentiated by the size and height of boxes.
Table 7. Usability attributes of 3D map for pedestrian navigation

CONCLUSIONS
The usability attributes of uses of 3D map display in form of LOD 1 of buildings for pedestrian navigation in urban area are all positive.The typical walking model of a searching task that is focused only to look for a certain address of building is chosen as the way finding model during the field test.It can be confirmed that effectiveness and efficiency of 3D map display to support self-orientation, spatial knowledge development, and navigation decision are well delivered by the app.The results were also confirmed by the tracks and records of test participants' activity.Although the results show users preferences for 3D over 2D map to help self-orientation, to recognize the surrounding, and to make navigation decisions are quite obvious, some unclear answers are still gained in terms of navigation clarity.7 TPs said that they did not follow the directions shown in the map display and three of them argued that the visual lines to 3D target is not clear and correct.Here the app needs to use better pedestrian navigation wayfinding APIs.The results also suggest that the development of LOD 1 for pedestrian navigation is acceptable but in case that the navigation require faster building comparison, the LOD 1 is not sufficient.Crowdsensing application using the navigation app is possible especially for its ability to provide an app, seen to be easy to learn, error tolerant, and engaging.

Figure 1 .
Figure 1.The application development results

Figure 5 .
Figure 5.All tracks during the test (left); their enlarged view (right).Each individual track can be linked to the corresponding sensor recordings belonging to each test participant.From sensor data collected, the behaviour on the navigation activity can also be assessed.The trend of 3 selected TPs to have similar acceleration pattern before finding the POIs indicates that the map display could help their spatial orientation and navigation decision.Here, those selected 3 TPs (TP 10, TP 14, and TP 15) have closed targeted POIs along the same pedestrian lane

Table 1 .
Observing user interactions and user activities for assessing usability attributes of a pedestrian navigation app.