WHICH WAY IS ‘YILDIZ AMFİ’? AUGMENTED REALITY VS. PAPER MAP ON PEDESTRIAN WAYFINDING

This paper investigates the use of Augmented Reality (AR) in pedestrian wayfinding in two aspects. First, an experiment was conducted to understand whether an AR-based mobile platform improves finding the direction of a query destination compared to a paper map. A total of 54 participants were enrolled to represent each group, in which the task was to show the direction of a query point-of-interest (POI). The experiments were carried out at the Beytepe Campus of Hacettepe University. The results suggest that AR-based platform significantly improves the task completion time compared to a paper map. Second, an online questionnaire was conducted to understand the preference of participants in terms of visualising the distances of POIs on an AR-based platform. Four different methods were utilised which vary the colour and size of a POI depending on its distance to the user. The results suggest that the majority of the participants preferred visualising POIs with the same colour but with different sizes depending on their distance to the user. This finding adds further support to the default visualisation adopted in Mapbox, the technology that was used to develop the AR-based platform.


INTRODUCTION
One of the enduring human activities is wayfinding, since most of the daily activities involve people moving between different, occasionally unknown, locations. Although wayfinding depends on external issues such as the effective delivery of maps and signs (Calori, 2007), it is also an ability depending on sociodemographic factors such as education, gender and age (Ulrich et al., 2019). The prevalent use of smart phones and the technological progress on Augmented Reality (AR) has much to offer in terms of easing the process of wayfinding. Researchers have already investigated the use of AR in this context, and the results are encouraging for both indoor (Kim et al., 2015) and outdoor environments (Ramos et al., 2018).
Advancement of technology and its ubiquitous use is a requirement to rely on AR for wayfinding. Yet, it is also evident that the 'users' are and should be the ultimate reference to evaluate the effectiveness of a wayfinding system. Only having understood the perception of users in this emerging technology, it would be possible to design and develop effective user interfaces. There are various issues to consider regarding the design of an effective AR platform including, but not limited to, the variation between individual users, representation of occlusion, depth visualization and text legibility (Kruijff et al., 2010). Consequently, it is an open research agenda to understand user experience, and develop effective visualisation techniques (Gabbard and Swan II, 2008;Zhou et al., 2008).
Most of the research evidence on evaluating user experience of an AR platform relies on qualitative measures. For instance, Olsson and Salo (2011) conducted an online survey which aimed to understand the overall use of AR platforms including Wikitude, Layar and Juanio. Most of the respondents consider that AR 'enables various things to be done better than earlier, especially gaining new perspectives on places and objects'. Previous research evidence favours AR in terms of user experience; however, there have been limited research on providing quantitative evidence. * Corresponding author Quantitative evidence on wayfinding evaluated participants' performance in terms of number of errors they made during wayfinding or how much time they spent to complete the route (Ishikawa et al., 2008). In other words, previous studies have required participants to walk a path, which is a time-consuming experiment that limits the number of enrolled participants. On the other hand, the initial step of pedestrian wayfinding is direction estimation, which is often an overlooked task. When a person is navigating towards, for example a lecture hall in a campus environment or towards a tourist destination in an urban environment, the first step would be estimating the direction of the destination from the initial position.
The aim of this paper is to investigate the effectiveness of the developed AR-based platform in direction estimation in a campus environment. Consequently, there are two main contributions of this research. First, the effectiveness of the developed AR platform is investigated in terms of direction estimation in a campus environment. The participants were asked to show the direction of a lecture/seminar hall by relying on two mediums: paper map and AR-map. The time it took for a participant to show the direction of the query point was recorded. Three different test sites were employed to have a better understanding on whether the complexity of the environment affects the performance of the participants. The results suggest that the AR-based platform leads to a faster direction determination. In addition, the AR-based platform, given that it is correctly calibrated, would provide the correct location of a query point. On the other hand, if a participant has low map literacy, then the estimated direction might be wrong.
The second contribution of this paper is to report the outcomes of the preference of users in terms of visualizing Point-of-Interest (POI) data in an AR platform. An online survey is created that collected anonymous information on participants and asked them to choose one of the four different ways to visualize POI data. These methods differ in how they represent POI data depending on their distance to the user's location by varying the colour and size of the points. The explored possibilities assume i) same size and colour, ii) same colour and different size, iii) discrete colour and iv) gradient colour. Once a participant fills the online survey, the overall results of the previous participants were also displayed, which is a contribution towards open data in AR research.

LITERATURE REVIEW
Traditional AR-based platforms were built to operate on indoor environments for various purposes ranging from medical visualization to manufacturing. Two main difficulties prevented the adoption of the AR technology in outdoor environments. First, traditional AR systems were based on head-mounted displays. These expensive devices are difficult to use in outdoor environments as they restrict the free movement of the users. Second, outdoor environments are much more dynamic in which lighting and environmental conditions vary substantially compared to a carefully designed indoor environment, often including markers to facilitate image registration and tracking (Azuma et al., 1999;Nagymáté and Kiss, 2019). Nevertheless, with the decreasing costs of new generation smartphones possessing advanced software development kits such as ARCore or ARKit for Android and iOS devices respectively, it is now possible to abstract most of the aforementioned challenges from the developers (Nowacki and Woda, 2020). Several AR apps are already available in mobile application stores for displaying the direction of a point-of-interest (POI) and they are briefly compared in Table 1.
Two different errors may occur when visualizing POIs in an AR platform for direction estimation: alignment error and registration error. Alignment error is defined as the wrong geolocation of the POIs. It might be due to wrong calibration of the sensors of the AR platform. On the other hand, registration error is defined as the 'dynamic errors caused by lag in the system and distortion in the sensors (Azuma et al., 1999). A registration error would cause the POIs to fluctuate dynamically on an AR platform. By compensating a registration error, it would be possible to stabilize the display even if the AR platform is under motion. In order to avoid the occurrence of these errors, a calibration process is often required (Rojtberg and Gorschlüter, 2019).
Investigating ways in which to visualise POIs is one of the withstanding issues in AR research due to its common use in various domains, most of which include a spatial dimension, including tourism (Loureiro et al., 2020;Yovcheva et al., 2013), entertainment (Sari, 2020) and archaeology (Pierdicca et al., 2015). There are different ways to visualise POIs in an AR environment, and it is difficult to select the 'best' one. According to Kruijff et al. (2010), environment on which the augmentations would occur, hardware to capture reality, display device, differences between users, and the way in which augmentation is designed affect how AR is perceived.
One of the recent studies that compared a map view (i.e. Google Maps) with an AR view facilitated both a qualitative and quantitative survey in a campus environment (Ramos et al., 2018). Participants had to carry out two tasks: i) finding the name of the building and its opening hours that is in front of the participant, and ii) finding the closest coffee shop to participant. The results suggest that even though the participants primarily relied on the map view to search for information, the completion time of both tasks were shorter in AR view. Specifically, participants on average required 9.1 seconds and 14.21 seconds to complete the tasks respectively when relying on the paper map. On the other hand, these times decrease to 6.75 seconds and 12.05 seconds in the AR view. However, they have relied on 20 participants, which might limit the statistical justification of these results. Nevertheless, the results encourage the use of AR in a campus environment for wayfinding.
User Experience (UX) is an important research area within AR. For instance, Kim et al. (2015) relied on five indicators (identifiability, accessibility, comprehensivity, interactivity and visual clarity) for assessing the UX for their AR based indoor (i.e. a hospital) navigation system. However, the AR based platform they developed require barcodes to be placed on walls. In addition, the study had a single destination, whereas in a real-life context there could be many possible destinations. Displaying all of the available information to users might lead to clutter and reduce the effectiveness of the display (Julier et al., 2002). Therefore, researchers try to understand the perception of users by conducting surveys. Gabbard and Swan II (2008) noted that environmental conditions (e.g. fluctuations in natural light, heterogenous background etc.) make it difficult to present legible augmented information. Therefore, it is important to achieve greater contrast between textual augmented information and the real-world background, and it is necessary to understand the user response times to read the provided textual information. However, their research was limited to utilising four static background images, and it is equally important to investigate their methodology in different environments and through a video.

METHODOLOGY
This research investigated the effectiveness of the developed AR platform in two different aspects. First, statistical analysis was carried out to understand whether the developed AR platform improves direction estimation of a query destination. Second, user preferences in terms of visualising POI distances was investigated through an online qualitative survey. This task involves different POI visualisation methods including changing the size and colour of the POI depending on its distance to the user. All of the developed AR solutions were saved as an Android application file (i.e. in 'apk' format) in Unity. The Mapbox Software Development Kit (SDK) was used on top of Unity game development engine.

Direction Estimation Task
This task requested participants to estimate the direction of a given conference/seminar room using a paper map and an ARbased platform. Specifically, participants were asked to show the direction of the query point on these two mediums. This task was important as it is the first-step in pedestrian navigation and wayfinding. In this way, whether the AR platform improves direction estimation for pedestrian navigation can be investigated. The experiments were carried out at three different test sites to have a better understanding of whether the surrounding environment effects the success of direction estimation. The design of the overall process to achieve the first task is illustrated in Figure 1. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-4/W3-2020, 2020 5th International Conference on Smart City Applications, 7-8 October 2020, Virtual Safranbolu, Turkey (online)

Name
• Which way is meeting room X?
•  Figure 1. AR vs paper map for direction estimationthe questionnaire design Two students were enrolled to conduct the experiment with participants. While one of the students filled the answers provided by the participants, the other student guided the participant to complete the direction estimation task and determined the completion time. The estimated direction is assumed to be correct for the AR platform by definition. On the other hand, a participant might fail to show the direction of the query point by using the paper map due to low map literacy. Two students next to a participant testing the AR platform in front of Hacettepe University's iconic lecture hall, Yıldız Amfi, is illustrated in Figure 2a. A screenshot from the AR-based platform is illustrated in Figure 2b.
(a) (b) Figure 2. Two students and a participant performing the direction task (a) and an exemplar screenshot from the mobile device (b)

Visualising Distance
In a complex environment possessing different type of POIs, it is important to investigate how to visualise them on an AR-based platform. This task investigates the preferences of participants in this context. Specifically, POIs can be classified into different categories depending on their spatial and thematic features. Distance between the object and user is the spatial feature, whereas thematic features include the type of the venue (e.g. sports/business/meeting room, café, dormitory etc.), its popularity, opening-closing times etc.
This task investigated four different methods based on the spatial aspect of the POIs. Specifically, different size and colour could be augmented based on the distance between the query point and the user. Consequently, effectiveness of four different scenarios were evaluated, where POIs have i) same size and same colour (URL 1), ii) different size and same colour (URL 2), iii) discrete colour (URL 3) and iv) gradient colour (URL 4). The videos were uploaded to Youtube and exemplar screenshots could be seen in Figure 3. Figure 3. POI visualisation based on the distance: same size and colour (a), same colour different size (b), discrete colour (c), gradient colour (d) The default setting to visualise POIs in Mapbox is the 'same colour different size' option, where the size of the POIs get smaller with the distance from the user. Specifically, Figure 3b is closest to the default way of visualising POIs in Mapbox. In order to assure that all of the POIs are visualised in same size, the more distant POIs to the user must be displayed larger. In order to achieve this, equation 1 was used to adjust the point size depending on the user.
, where s(poi) denotes the size of the POI, d(poi, user) denotes the distance between the POI and the user, and finally c is a constant that is identified by trial and error.

CASE STUDY: BEYTEPE CAMPUS
The AR platforms required to achieve the two tasks were developed for the Beytepe Campus of Hacettepe University, which is one of the renowned universities of Turkey. As of July 2018, approximately 3600 academics serve to 50 thousand students on a range of disciplines with a strong focus on medical studies and engineering. The Beytepe Campus is located outside of city centre, but can be reached through a metro line and with public buses. The campus sits on a land of approximately 5.8 km 2 .
The map illustrating the 12 POIs are illustrated in Figure 4. Most of the POIs correspond to conference rooms, while few others correspond to a dormitory or a refectory. Some of the seminar rooms such as 'Canan Dağdeviren' or 'Hayrettin Gürbüz' do not exist in reality, and those names correspond to a well-known academics associated with those departments. In order to increase the legibility of the majority of the POIs located in the centre of the campus, the POI 'Tunçalp Özgen Kültür ve Kongre Merkezi' was not shown at its correct location. While transferring the POIs to the AR platform, their exact locations were used.

Figure 4. Data collection sites and POIs of the direction estimation task
While the data collection sites A and B were surrounded with buildings, site C was an open area with a main road next to it. Site B is close to the iconic Yıldız Amfi, where the majority of students attend lectures. The participants were randomly selected from the passers-by. A total of 108 people was surveyed. The overall results of the first task are illustrated in Table 2.  Table 2. Overall results of the first task Participants were randomly divided into two groups: 'map' group members received the paper map, and the 'AR' group received a calibrated smart phone. Consequently, each medium was tested by 54 participants. Participants of both groups were asked to show the direction of a random query point. The boxplot of completion times using these two mediums are illustrated in Figure 5.

Figure 5. Completion times of the first task
The median completion time of the task was 29 seconds in the map-based approach. On the other hand, participants of the AR based platform determined the POI, hence, its direction in approximately 10 seconds. This substantial difference between two groups can be regarded to the nature of the task. Specifically, the map-based approach requires participants to first determine the location of the POI on the map, and then estimate its direction. The latter objective might require more mental effort since it requires participants orienting themselves with respect to the map. This process is also error prone, since participants may fail to show the correct direction of the query point.
The distribution of completion times of map based and AR based approaches are illustrated in Figure 6a and Figure 6b respectively. The wide range of duration it took to estimate the direction of a POI in map-based approach might be attributed to the substantial variations on the map literacy of the participants. In addition, the variation might also be attributed to participants' knowing the correct location of a query point, hence, pointing out its direction in a fast manner.
(a) (b) Figure 6. Distribution of completion times of map based approach (a) and AR based approach (b) The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-4/W3-2020, 2020 5th International Conference on Smart City Applications, 7-8 October 2020, Virtual Safranbolu, Turkey (online)

Method Effectiveness
Statistical analyses have been conducted to investigate whether the difference between the completion time of paper-based approach is significantly higher than AR-based approach. These statistical analyses were conducted in SPSS. First, data normality have been checked by using the Shapiro-Wilk test by using a significance level of 0.05. The results suggest that even though paper map task completion times follow a normal distribution, AR map task completion times do not. Therefore, Mann-Whitney U test was used to determine whether there is a significant difference between the two mediums.
Mann-Whitney U test is a non-parametric method used to compare the mean of two independent groups. The null hypothesis, H0, suggests that two groups come from the same population. On the other hand, the alternative hypothesis, H1, suggest that there is a significant difference between the two groups. The assumptions of the Mann-Whitney U test are: • The dependent variable should be measured on an ordinal scale or a continuous scale.

•
The independent variable should be two independent, categorical groups.

•
Observations should be independent.

•
Observations are not normally distributed.
The dependent variable (task completion time) is continuous variable, which satisfies the first assumption. Data contains two independent categorical groups, paper map and AR-based platform, which satisfies the second assumption. Observations were independent, since randomly chosen different participants were enrolled in the experiment. Finally, the aforementioned analysis suggest that AR completion times do not follow a normal distribution.
The p-value of the Mann-Whitney U test was zero, which suggests the rejection of the null hypothesis. Consequently, it can be concluded that there is a statistically significant difference between the completion times of the task for map-based approach and AR-based approach. The latter approach decreases the time it takes to find the direction of a query POI. This conclusion might be attributed to the data collection sites.

Data Collection Site Effectiveness
It is important to investigate whether different data collection sites effect the completion time of pointing out the direction of a query point. The variation of completion times at different sites for map-based approach and AR-based approach are illustrated in Figure 7.
(a) (b) Figure 7. Variation of task completion times for map-based approach (a) and AR-based approach (b) at different data collection sites It was observed the densely built data collection site A led to the highest task completion time. Strangely, site B which is also in a densely built environment but in a more central location led to the lowest task completion time. This suggests that when people are in a familiar environment, they can locate themselves easier; hence, estimate the direction of query destinations faster. This advantage; however, appears to have lost when using the ARbased platform. Nevertheless, the AR platform improved the task completion times at all data collection sites.
In order to understand whether there is a significant difference between data collection sites, first Shapiro-Wilk test is conducted to check the normality of the task completion times. The results suggest that the task completion times for all the data collection sites in map-based approach follow a normal distribution. On the other hand, task completion times for the AR-based approach at sites A and C do not follow a normal distribution.
One-way ANOVA analysis was carried out to investigate whether there is a significant difference between different data collection sites for the map-based approach. The assumptions of ANOVA were satisfied. Specifically, the participants were randomly selected, task completion times follow a normal distribution and homogeneity of variances, which was satisfied by applying the Levene test. The ANOVA analysis revealed an F value of 7.936 which corresponds to a p-value of 0.001. Since p < 0,05. the null hypothesis is rejected and there is significant difference between task completion times of different data collection sites. Post-hoc analysis was carried out using the Fisher's LSD test. The pair-wise comparison of data collection sites revealed that the site B was significantly different from sites A and C.
The non-parametric Kruskal-Wallis test was used to investigate whether there is a significant difference between different data collection sites for the AR-based approach. This is due to the fact that the task completion times of different data collection sites do not follow a normal distribution for the AR-based approach. The null hypothesis H0 suggest that the average task completion times at different data collection sites were equal. The statistical value H was found to be 8.133 which corresponds to a p-value of 0.017. Since, p<0,05 the null hypothesis was rejected. Consequently, there was a statistically significant difference between data collection sites. Dunn's test was carried out to understand which data collection site was different. The data collection site C was identified to be substantially different from the sites A and B. Eight of the 12 POIs were located on the northern side of site C as can be observed from Figure 4, which might have made it difficult to identify a given query point in the AR platform due to the clutter of labels.
Finally, user preferences regarding the use of AR for wayfinding was investigated. Almost 95% (i.e. 102 participants out of 108) of the participants would consider relying on the use AR for pedestrian navigation compared to their traditional method. This outcome is staggering as it demonstrates that a vast majority of participants were welcoming this new technology. Amongst these 102 participants, a 91 of them suggest that AR eases pedestrian navigation. Compared to the other criteria of relying on AR such as it being fun (4 participants), or participants preferring using advanced technology (3 participants), it is important that a large number of participants actually find the AR platform intuitive and easy to use. Three improvements were suggested: i) including direction signs (19 participants), ii) including search box (18 participants) and iii) improving legibility of the POI names (1 participant). The 6 participants who would not choose the AR-based platform for wayfinding have different rationale such as maps being simpler or such apps would reduce the battery life. One of the participants suggest that the use of the AR platform caused dizziness, while another one suggests that it might disturb other people.

POI Visualisation Method
The second objective of this research was to understand user preferences in terms of visualising distances in an AR platform. An online survey was created and shared with the online community (https://forms.gle/Pd5Nd41z51Ay4enX8).
A total of 33 participants were enrolled in this experiment. Each age group has similar number of participants. Specifically, there were 7, 13 and 13 participants that were in the age groups 19-25, 26-35 and 35+ respectively. None of the participants were colour blind. A majority of participants were from Turkey (22/33) and there were participants from Ukraine (2), United Kingdom (5), Macedonia (1), Montenegro (1), Netherlands (1) and South Africa (1).
The main question of this task was 'which POI visualisation method would you prefer to estimate the distances?'. There were four options to choose from as aforementioned. The following results were observed: • Same colour different size -13 participants • Same colour same size -11 participants • Discrete colour -5 participants • Gradient colour -4 participants The results indicate that relying on different colours to display colours are less preferred by participants. It should be noted that the same colour options include the distance information to each POI, which may decrease the mental burden on the participants. Therefore, while displaying POIs on an AR platform it is better to also include a textual information that states the distance between the POI and the user. This finding adds further support to the default way of displaying POIs on an AR platform that Mapbox relies, which is 'same colour different size'. In this way, users could have the opportunity to have a quick understanding of nearby facilities. In addition, users would also prevent reading the distances to distant POIs. Consequently, the findings of this task suggest that users of an AR platform tend to choose the least mentally demanding approach to visualise POIs.
The last question of this task was to complete the sentence: 'I would use AR for direction and distance estimation in a real-life context if…'. The participants could select more than one the following options, and the following answers were given: • it will not over-consume my battery (19 participants). The results reveal an important outcome. The majority of the participants would like to see that an AR platform would indeed not overconsume their battery. Consequently, developers should also consider how to reduce the computational complexity of an AR platform. This is a challenging task as participants were also interested in having more advanced user interfaces that include a search box or filtering by the type of the POI.

DISCUSSION
The developed AR platform requires calibration before each use in order to reduce alignment errors. The calibration of the mobile device used in this research was achieved by opening the application while the head of the smart phone is facing towards the north. This could be automatically achieved by developing image based matching methods or by geometric analysis in which users were asked to walk few steps in a given direction. However, there might also be dynamic errors, which are known as registration errors. This was actually witnessed in URL 1, and the registration error is illustrated in Figure 8. Such errors would cause frustration especially in dense urban environments. 0:54 (a) 0:55 (b) Figure 8. The POI labelled as 'A blok' moves substantially in less than a second in two consecutive frames (0:54 -0.55) In order to have loyal users that rely on an AR platform for pedestrian navigation, it is therefore critically important to provide a reliable system that keeps registration errors at minimum.

CONCLUSIONS
The recent advances in AR technology encourage researchers to investigate its use in pedestrian navigation. This research was an attempt to understand how well AR might improve direction estimation of a query destination. Considering that direction estimation is the first step in pedestrian wayfinding, its fast and correct estimation is important. This research attempted to answer two questions: i) how does the direction estimation time varies on a printout map and an AR platform, and ii) which POI visualisation method in an AR platform is preferred. The outcomes of the first task suggest that AR platform significantly reduces the task completion time. The outcomes of the second question indicate that participants to the online survey prefer the least mentally demanding method to visualise POIs, which is same colour different size. This adds further support to the default setting of some of the existing AR-based apps and Mapbox. Future research would focus on quantifying registration errors and investigate the effect of having different number of POIs on the AR platform. Furthermore, the experimentation needs to be repeated on different test sites, and with more participants. Last, the experiment could be extended such that participants were asked to walk to the query POI. While the participants were moving, their cognitive work load (e.g. how many times did they need to look at the map/AR, did they walk while looking at these platforms etc.) could be measured and analysed.