MODELLING OF COLLECTIVE MOVEMENT IN IMMERSIVE ENVIRONMENTS

Immersive technologies allow us to map physical reality by means of 4D virtual systems in ever higher spatial and temporal detail, up to a scale level of 1:1. This level of detail enables the representation of phenomena that have been widely ignored by the geovisualization research agenda as yet. An example for such a large scale phenomenon is the collective movement of animals, which can be modelled and visualized only at a fine grained spatio-temporal resolution. This paper focuses on how collective movement can be modelled in an immersive virtual reality (VR) geovisualization. In a brief introduction on immersion and spatial presence we will argue, that high fidelity and realistic VR can strengthen the users’ involvement with the issues visualized. We will then discuss basic characteristics of swarming in nature and review the principal models that have been presented to formalize this collective behavior. Based on the rules of (1) collision avoidance, (2) polarization, (3) aggregation and (4) self-organized criticality we will formulate a viable solution of modelling collective movement within a geovisualization immersive virtual environment. An example of use and results will be presented. 1. IMMERSIVE ENVIRONMENTS IN GEOVISUALIZATION


Introduction
The increasing use of immersive virtual reality (VR) technologies for geovisualization matters is challenging GIScience in several ways: Since VR capabilities of current geographic information systems (GIS) are limited, middleware (e.g.game engines like Unreal Engine or Unity) is required to visualize GIS data on VR output devices (Boulos et al., 2017).
Also the transfer of fundamental GIS concepts and data models to immersive VR applications is complex: Generalization on different scale levels, for instance, has always been a defining criteria in both analogue and digital cartography.Immersive VR environments, however, provide vivid illusions of (nongeneralized) reality at spatial scales of up to 1:1, giving the user a sense of being virtually located in a real place.Such vivid illusion is not only a matter of spatial scale but of time as well.In most GIS applications, temporal change is either neglected or represented in a classified way and with a rather coarse resolution.By contrast, successful immersion also depends on a continuous and realistic visualization at temporal scales of up to 1:1 (Cummings & Bailenson, 2016), especially when animate nature is mapped.
The present paper focuses on a particular implication of the aforementioned 1:1 ratio between spatial and temporal aspects of immersive VR and physical reality, namely the modelling of collective movement of animals.We can observe such collective movement almost every day, be it in a flock of birds, a swarm of insects, a cattle herd or a school of fish.Consequently, whenever * Corresponding author we want to successfully immerse a user in a virtual environment, we have to model these animate aspects of nature not only at the individual but also at an aggregate level at a large (spatial and temporal) scale level.
A very brief discussion of concepts that are fundamental to the understanding of virtual reality within the scope of this text shall precede the subsequent analysis of algorithms and designs on modelling swarming behaviour.

Immersion & Presence
Immersion can be understood as a technological characteristic of a VR system describing "[…] the extent to which the computer displays are capable of delivering an inclusive, extensive, surrounding and vivid illusion of reality to the senses of a human participant" (Slater & Wilbur, 1997, p. 3).In accordance with this definition, a highly immersive environment will rely on a 3D stereo display with a 6 degrees of freedom (DoF) tracking system (Schulze et al., 2011).
Immersion is decisive for the formation of presence (Cummings & Bailenson, 2016), roughly defined here as the sense of "being there" in a virtual environment (Skarbez et al., 2017).This experience of being situated in a VR space can, in turn, increase the impact of a geovisualization product (Bailey et al., 2016).In a nutshell we can, thus, generalize that immersive VR systems may facilitate presence while presence may strengthen the users' involvement with the issues visualized in VR space.

Ratio in Immersive Environments
Realism and fidelity of the VR environment usually are not considered as defining criteria for the formation of presence (Cummings & Bailenson, 2016).However, for geovisualization matters it can be important "that the scenario corresponds to reality to the maximum extent possible" (Skarbez et al., 2017, p. 96:5), i.e. to virtually emulate the real world (Alexander et al., 2005).
In practical terms, this claim for correspondence between virtual and physical reality, together with the aforementioned definition of presence as the feeling of being there lead us to geovisualization immersive virtual environments (GeoIVE; Hruby et al., 2018), which allow the user to experience in first person a physical reality that is distant in time and/or space.
Figure 1.Ratio of 1:1 between physical reality (above) and virtual reality (below).Example from a GeoIVE of a Caribbean coral reef (cf.Hruby et al., 2018) We propose to call such relationship between virtual and real space a 1:1 scale, where the user perceives a VR representation of a real place at a level of detail as he would do being physically there.This will allow her to unambiguously (i.e.: 1:1) assign a virtually experienced model to the corresponding real object.
Regarding the representation of swarming behavior, a 1:1 scale not only depends on the 3D modelling of animal groups at a level of detail that allows recognizing the species.Also the collective movement of the swarm members needs to be simulated in a highly detailed manner.

Understanding Collective Behaviour
A considerable amount of literature is documenting the benefits of animal group formation (cf.Krause & Ruxton, 2002, for overview and further readings).However, the set of rules followed by the individual animal in order to achieve a specific collective behavior is unexplored as yet.
Self-organization has been one of the main concepts used so far to model and explain collective animal behavior: "The central tenet of self-organization is that simple repeated interactions between individuals can produce complex adaptive patterns at the level of the group."(Sumpter, 2006, p. 5).
Within the framework of self-organization, several principles of collective animal behavior have been proposed (Bouffanais, 2016;Sumpter, 2006): Production of patterns is a decisive characteristic of swarming (Bouffanais, 2016).Patterns can emerge from positive feedback (in space), e.g.reinforcement or "recruitment to a food source" (Bonabeau et al., 1997, p. 183) or synchronization (in time), while negative feedback mechanisms help stabilizing patterns.However, collective structures can arise out of randomness at the individual level as well (Theraulaz et al, 2003).
The emergence of order can be also analyzed from a swarm intelligence perspective (Engelbrecht, 2006), especially where modelling a virtual swarm system is the primary research interest.
Finally, and with regard to the collective processing of information, also the concept of leadership is worth considering.
In accordance with Sumpter (2006), there is only an apparent contradiction between self-organization and leadership: "Rather than being opposites, self-organized positive feedback and leadership are instead powerfully combined to produce coordinated collective migration" (p.17).A classification and formal description of leadership structure within groups of animals has been presented by Garland et al. (2018).
Basic SPP models are designed upon a set of three rules guiding the movement of each agent (i.A zonal model has been presented to spatially organize these rules in a metric framework (Brace et al., 2016;Garland et al., 2018), where spherical regions are drawn concentrically around a focus agent, so that zones of repulsion (rule A), orientation (rule B) and aggregation (rule C) can be differentiated.
This means in terms of figure 3, that the focus agent will steer: (A) to avoid collision with flockmates in the repulsion zone (red), (B) to align with flockmates in the orientation zone (blue) and (C) to stay close to flockmates in the aggregation zone (green).
Flockmates outside the aggregation zone are supposed to be imperceptible and, hence, of no relevance for the focus agent.The three basic rules shown in figure 2 have been extended by numerous authors during the last three decades: Considered have been, for instance, perturbation (Vicsek et al., 1995), social interaction (Strömbom, 2011;Garland et al., 2018), adhesion (Grégoire et al., 2003) or particle shape (Peruani et al., 2006).

MODELLING OF COLLECTIVE BEHAVIOUR
The basic SPP rules may lead, in the medium term, to a rather homogeneous and isotropic movement of each agent.Hence, in order to simulate a more natural behavior of virtual animal groups in an immersive environment, we propose to consider also theories of self-organized criticality (Bak, 1996).These concepts allow us to better represent cohesion and unity of real animal groups, where not just aggregation but also disintegration and reorganization can be observed under certain conditions.
To achieve a behavior that maximizes the individual's autonomy within a swarm we the following rules to model collective movement: 1. We will discuss these variable in further detail subsequently and show how polarized non-colliding aggregate motion can be modelled for critically self-organized agents.

Collision Avoidance
Starting point for each agent's (A) movement is the definition of cuboid regions in which the agent can move on the (positive and negative) X-, Y-and Z-axis.Regions, defined through regional variables (tab. 1) are a particular characteristic of GeoIVE as they represent ecosystems where a given species typically can be observed.
In the case of a virtual coral reef, for instance, regions may represent the main reef zones and, thus, assign species to corresponding environments.For example, agents of a fish species that is typically found in the reef crest will only move in the virtual region representing the reef crest.
Belonging to a predefined 3D region, an initial position is assigned to each agent Ak accordingly.Considering then the aforementioned instability of Ak, the agent's linear (v_max, v_min) and angular (v_ang) velocity and turning rate (tab. 1) can vary randomly within a capsule-like surrounding volume V (fig.4).Separation distance D of a given individual Ai is then defined in a way that no intersection with the separation distance of the closest agent Aj can occur: where Ai = [pos_xi, pos_yi, pos_zi] Aj = [pos_xj, pos_yj, pos_zj] for n ∈ ∩ ℕ

Polarization
Organizing agents in terms of swarming behavior assumes that the individual direction vectors share a general alignment.To model such a collective directional movement, we introduce the concept of leadership.
First we consider, for n ∈ ℕ, a swarm in terms of a set A: where each agent Ak can be a possible leader AL.This configuration allows us to achieve a collective movement of a group following a single leader.In order to extend this approach towards multiple and different concepts of leadership (Garland et al., 2018) that may generate different, but still polarized behavior we can modify equation (2) to: where AL are different leaders pertaining to set/swarm A, and where AL has smaller cardinality than A.
It should be pointed out here that we can, in consequence of equation ( 3), even simulate swarms that are no longer polarized but still demonstrate collective movement: Whenever A's cardinality equals the cardinality of AL, each agent becomes a leader AL.In such constellation all agents will follow each other, thus forming groups without following any leader in particular, this is: #(AL) = #(A).

Aggregation
Equation (1) establishes a minimum radius within which the variables defined in table 1 can be parametrized as a function of the natural behavior of the species represented.Turning rate and velocity parameters of fish, for instance, have to be specified (also in relation to other species of the same ecosystem) in order to provide a realistic behavior of the agents in the corresponding GeoIVE.Moreover, real-world flockmates aggregate on the swarm-level but, on the individual-level, usually do not move in a completely identical and synchronized manner.
To simulate both aggregation on the collective (macro) level and randomness on the individual (micro) level, i.e. in the behavior of the virtual agents, the variables defined in table 1 can be modified independently for each agent Ai without affecting the behavior of a neighboring agent Aj by ≤   } and where Q and P are two neighboring capsules V of agents Ai and Aj; to ensure that Ai cannot affect the behavioral variables of Aj (tab.1), we set Q ∩ P ∅ for all x and y.

Self-organized Criticality
Self-organization is probably the most important characteristics of our approach to ensure a realistic simulation of real-world swarming behaviour, as it allows us to introduce a certain amount of chaos into the swarm system.This kind of noise is typical for animal swarms, whose individuals will usually differ slightly regarding size, shape and behaviour, even within the general patterns of the swarm they belong to.Hence, in a GeoIVE selforganized criticality helps to avoid the unnatural impression of cloned agents with a cloned behaviour and thus facilitates the formation of spatial presence (cf.section 1.2).
To include self-organization into our model, a critical point between instability and order is defined for all behavioral variableson condition that the number of leaders AL shall be neither 0 nor 1: where A is a set of agents whose cardinality #(A) must be greater than the cardinality #(AL) of a set of leaders.

EXAMPLE OF USE
So far, we have applied the model outlined above on virtual schools of fish.Moreover, we simulated the movement of solitary animals (e.g.sea turtles, sharks) in a predefined region by considering only the rules of collision avoidance and selforganized criticality.Since these tests have been realized within a GeoIVE of a submarine ecosystem, we will briefly discuss this particular setting subsequently.

Area of Study
The model outlined above has been applied to a GeoIVE of a 3.5 x 3.5 km coral reef in the Mexican Caribbean, near the city of Cancun.This GeoIVE has been built upon information on bathymetry and benthic habitats derived from WorldView-2 imagery and in-situ data.Species of reef fauna and flora were defined in accordance with the National Information System on Biodiversity (SNIB).The study area, methods of data acquisition and 3D modelling are presented in further detail in Hruby et al. (2018).

Software Setting & Implementation
As stated in section 1, game engines can serve as middleware to visualize GIS data within an immersive environment.In this project, Unreal Engine was used to build a GeoIVE and is therefore the software our model of collective movement has been tested on.
In general terms, a simulation of collective movement of animal groups in a GeoIVE has to support accurately rigged and animated 3D species models.These 3D models have to be organized under the four rules defined earlier in this paper.
Unreal Engine and comparable software (e.g.Unity) provide (to our best knowledge) no solution out-of-the-box (e.g. by particle systems, path animation or hierarchical instanced static meshes) for the modelling of collective but self-organized behaviour of highly detailed animated objects.Hence, we translated our model of a critically self-organized, polarized, non-colliding and aggregate motion into C++.The latest code is maintained on GitHub (Magallanes, 2018).However, the model has been discussed here in terms that should be general enough to allow for a transfer to other software environments as well.Figure 4 shows an example of implementation:

Results
The model presented in this paper, yet work in progress, allowed us to populate a GeoIVE of a submarine ecosystem (cf.section 4.1) with currently approx.5000 agents representing 21 fish species.Following a set of four basic rules, these agents move autonomously across predefined regions (corresponding georeferenced real-world reef zones) in species-specific swarms of 2 to 25 members.As shown below, collective movement resulting from our model can produce regular and well-aligned groups (fig.5: above), but also rather irregular, yet aggregated, formations (fig.5: below) of critically self-organized agents.

Outlook
Compared with path-animated solutions (cf.Hruby et al., 2018), time-savings and performance improvements of the present approach are considerable.However, in the face of swarming behavior of real animals several limitations are apparent: For instance, our current model neither considers (1) ecosystemic roles and relations of predator vs. prey (2) nor do the agents follow any natural stimulus (e.g.foraging).More complex algorithms will be necessary to amplify what we (deliberately) have called movement in the title to real behavior in terms of an artificial intelligence (AI); (cf.Asgari et al., 2016;Witkowski & Ikegami, 2016).

Conclusion
We started paper by arguing that the formation of spatial presence within immersive VR can strengthen the users' involvement with the issues visualized, which turns the feeling of being there into an important cognitive aspect for geovisualization matters.As we tried to show, presence also depends on high fidelity and realism, so that geovisualization immersive virtual environments (GeoIVE) will be typically display representations at a 1:1 scale.However, such a realism is not just a matter of highly detailed 3D models, but also of visualizing temporal change, e.g.movement, in a natural manner.
Collective movement of animals, a basic manifestation of collective behaviour, was the aspect of a spatio-temporal 1:1 scale we dedicated the main part of the paper to.
Since Reynold's boid concept (1987) a continuously growing number of models has been published to better formalize collective animal behaviour (Vicsek & Zafeiris, 2012).However, many of these models are rather designed to deeper understand general or particular aspects of behaviour, while (geo)visualization issues do not figure prominently as yet.
Against this background we presented a viable implementation of a SPP model (extended by self-organized criticality) into a GeoIVE of a submarine ecosystem.The approach discussed in this text is still in process.However, even the current stage of development meets the basic demands of an immersive and realistic visualization of collective movement of several thousand agents.The model presented is currently applied in a GeoIVE of coral reef in the Mexican Caribbean.
It is worth mentioning that our visualization of collective movement is built upon geospatial data (e.g.regarding benthic habitats and bathymetry), but without GIS software, which is due to the currently rather limited VR-capabilities of GIS.For example, modern GIS neither support VR output devices nor the rendering of atmospheric conditions and lighting.Considering, finally, that both GIS and game engine software lack the AI features necessary to model and visualize behaviour beyond simple rule based movement, turns the interrelationship between GIS, VR and AI into a promising field of future geovisualization research.

Figure 3 .
Figure 3. Zonal model of collective movements based on the basic SPP model; modified from Brace et al. (2016) and Garland et al. (2018)

Figure 4 .
Figure 4. Implementation of collective movement in Unreal Engine under the rules of collision avoidance, polarization, aggregation and self-organized criticality: Circles visualize V for each agent Ai from different points of view.

Figure 5 .
Figure 5. Collective movement of agents in a GeoIVE of a submarine ecosystem.Regular (above) and irregular (below) motion patterns produced by the rules of (1) collision avoidance, (2) polarization, (3) aggregation and (4) self-organized criticality

Table 1 .
Behavioural and regional variables used to model virtual agents' movement in swarms