FEASIBILITY STUDY OF INEXPENSIVE THERMAL SENSORS AND SMALL UAS DEPLOYMENT FOR LIVING HUMAN DETECTION IN RESCUE MISSIONS APPLICATION SCENARIOS

Significant efforts are invested by rescue agencies worldwide to save human lives during natural and man-made emergency situations including those that happen in wilderness locations. These emergency situations include but not limited to: accidents with alpinists, mountainous skiers, people hiking and lost in remote areas. Sometimes in a rescue operation hundreds of first responders are involved to save a single human life. There are two critical issues where geospatial imaging can be a very useful asset in rescue operations support: 1) human detection and 2) confirming a fact that detected a human being is alive. International group of researchers from the Unites States and Poland collaborated on a pilot research project devoted to identify a feasibility of use for the human detection and alive-human state confirmation small unmanned aerial vehicles (SUAVs) and inexpensive forward looking infrared (FLIR) sensors. Equipment price for both research teams was below $8,000 including 3DR quadrotor UAV and Lepton longwave infrared (LWIR) imager which costs around $250 (for the US team); DJI Inspire 1 UAS with commercial Tamarisc-320 thermal camera (for the Polish team). Specifically both collaborating groups performed independent experiments in the USA and Poland and shared imaging data of on the ground and airborne electro-optical and FLIR sensor imaging collected. In these experiments dead bodies were emulated by use of medical training dummies. Real humans were placed nearby as live human subjects. Electro-optical imagery was used for the research in optimal human detection algorithms. Furthermore, given the fact that a dead human body after several hours has a temperature of the surrounding environment our experiments were challenged by the SUAS data optimization, i.e., distance from SUAV to object so that the FLIR sensor is still capable to distinguish temperature differences between a dummy and a real human. Our experiments indicated feasibility of use SUAVs and small thermal sensors for the human detection scenarios described above. Differences in temperatures were collected by deployed imaging acquisition platform are interpretable on FLIR images visually. Moreover, we applied ENVI image processing functions for calibration and numerical estimations of such a temperature differences. There are more potential system functionalities such as voice messages from rescue teams and even distant medication delivery for the victims of described emergencies. This paper describes experiments, processing results, and future research in more details.


INTRODUCTION 1.1 State-of-the-art geospatial data acquisition and processing for rescue operations
Rescue operations for natural and man-made disasters response take significant amount of various organizations and humans effort and financial expenses.For example, theUS National Parks System rescue operations have a budget of 12+ million dollars USD annually (Heggie et al, 2009).McKie & Richardson (2003) described the human-factor analysis of a rescue and indicated the conspicuous feature of a human tendency to disregard opportunity costs when the life of identifiable individuals are visibly threatened.Due to this fact, we may observe operations when hundreds of people and multiple sets of equipment are deployed to save only one human life.There are multiple projects supporting rescue operations.UN-affiliated International Search and Rescue Advisory group (INSARAG, 2016) has developed multiple standards for human life saving operations.Use of geospatial data is crucial for the rescue operations.One of the most reliable systems delivering geospatial data supporting rescue operations is EU-ICARUS (DeCubber et al, 2013) which deploys multiple aerial (solar aeroplane by ETH; quadrotor by ASCAMM and indoor multicopter by SkyBotix), unmanned ground platforms (large UGV by Metalliance and small UGV by Vanguard), and marine-based platforms (U-Ranger by Calzoni and ROAZ II by INESC).Sensors-suite combining video and still visible spectrum cameras is described (Ahmed et al, 2008), which deploys RPH2 (RPH2, 2016) helicopter UAV that allows to care up to 100 kilograms of payload.This system functioned efficiently during disaster response and provided efficient data fusion from its sensor suite.Andriluka et al (2010) showed that vision-based human detection algorithm from UAV imagery produced feasible workflows combining various computer vision detectors such as pictorial structures (Andriluka et al, 2009) and discriminately trained models (Felzenszwalb et al, 2009), similar to components of Support Vector Machine (SVM).Rudol et al, (2008) introduced human body detection via positioning algorithms using visible and infrared imagery.Cascaded Haar classifiers (Viola et al,2001) were successfully applied in Gaszczak et al(2011) for real-time detection on UAV imagery humans and cars.Donelli (2011) detected living humans trapped under rubble (usinf light-weight microwave life detector based on compact X-band radar that was installed on a small quadcopter.This work introduced the independent component analysis algorithm successfully detects breathing and heartbeat rates through 15 cm rubble.The research described in this paper is a proof of concept for using small UAVs equipped with infrared and visible diapason sensors for detection of living humans in outdoor settings.

Why SUAV?
The demand is quite high for various services needing geospatial data acqusition in rescue situations.Therefore, the pace of information collection and processing should be at an appropriate rate.Typically, these processes are local in nature, affecting small areas, scattered in space and are quite numerous.The use of manned aircraft often results in unnecessary costs.From an economic stand point it becomes clearly unjustified.Even if there is a justification of this expense related to the urgency and importance of the task (such as emergency management situations), these projects are seldom absent of favorable weather conditions, leading to the disruption of the project schedule.In these described circumstances, the use of easy to transport, remotely-controlled aircrafts equipped with the necessary imaging equipment allows very quick performance of all the necessary project stages and allows for flexible adjustments to the ongoing technology, even in the absence of stable weather conditions needed for the traditional aerial survey.
During emergency response application scenarios, manned aerial vehicles are often used in support of the operation.A manned aerial vehicle can help rescue workers reach hard-to reach areas quickly, provide visual support from the air, and transport injured personnel to medical facilities quickly.However, these manned aerial vehicles are very expensive to purchase, operate, and maintain and also require pilots with extensive training.In addition, very strict requirements are needed for areas of take-off and landing, and these areas are often far from the search and rescue area (Calabrese et al, 2014).Furthermore, the safety of the aerial vehicle and the pilot is of great concern.Because of these difficulties, the use of small unmanned aerial vehicles (SUAVs) in rescue operations has gained great interest for researchers as it is demonstrated in previous section.
Multiple UAV classifications exist and there is not a single standard.The US Department of Defence (UAV Roadmap, 2016) uses a classification based on weight, altitude, and speed, which is commonly accepted among UAV hobbyists and professionals (Table 1).Thus our research hypothesis is that we need to deploy a sensor which will be capable to distinguish a temperature difference between a living human and a dead human body.In average, we assume this difference would be in a range of 5 degrees Celsius.Therefore, temperature resolution of selected sensor should not exceed this range and potential thermal sensors should be able to capture the difference in temperatures.

Experimental Scenarios
International team of researchers in the USA and Poland selected experimental areas which for both study areas can be characterized as a semi-forest areas in Poland and grassland in Michigan US on small to medium hilly terrain.Both teams used human dummy objects (see Figure 1) and real people during SUAV imaging acquisition experimental sessions.Images were taken in the late fall of each region, with snow on the ground in the US (the Keweenaw Peninsula, Michigan near Lake Superior) and no snow on the ground in Poland.4. Assessment of meteorological, terrain, and lighting conditions, setup of sensor data acquisition parameters of shooting; 5. Flight execution; 6. Assessment of radiometric and photogrammetric characteristics of imaging and visual analysis, 7. Data processing of the results, 8. Rescue decision support outputs and recommendations.
For developing the rescue flight specification it is necessary to determine the desired accuracy of image processing and respective sensor parameters along with navigation accuracy requirements.It is empirically defined that the minimum extent of spatial object which is distinguishable on an aerial image can be obtained via: (1) where δl = non-contrast zone width on border of an image of object and a background (for aerial images δl = 0.05 ÷ 0.15 mm) and ΔD = contrast or difference of optical density of object and background (ΔD =0.8 ÷1.6).Taking into account that rescue search may be performed in adverse illumination conditions we will accept the lowest indicators on width of a noncontrast and the average size of contrast δl =0.05mm, ΔD=0.8mm.Then the extent of detectable object has to be ≥ 78 microns on the image scale.For stable object detection we will accept 3-fold coefficient of reliability in object recognition.In this case the object will be steadily identified if its size is not less, than 0.234 mm on the image scale.
To identify a possible flying heights we assumed the following sizes of human objects on the ground: -5-7 year old child, sizes lying on the ground is 1 meter, seating is 0.5 meter; -older child, teen and adult, sizes lying on the ground is 1.6 meter, seating is 0.8 meter.
Specific flying heights and equipment specification for the US and Polish teams are listed in Table 2.  To use the inexpensive Lepton LWIR ($259) development kit required some hardware installation and software development.
Toolkit has drivers which enabled to integrate it with 3DR IRIS SUAV. Figure 3 shows SUAVs with sensor systems ready for the field experiments on geospatial imaging acquisition.

Experimental Data
As it was mentioned in previous section we performed flights using different flying heights and speeds while collecting data over dummy and human subjects on both SUAV platforms.Samples of collected raw imagery for Polish team is shown in Figure 4 and for the US team in Figure 5, respectively.It is clear that the image quality of the Tamarisc sensor is much higher than the Lepton LWIR.Objects of interest (humans) are distinguishable on both sensors imagery.Moreover experiments indicated that animals (dogs) also can be identified.

EXPERIMENT RESULTS
The next step in our experiments was to attempt automated extraction of the humans, that would then be clearly interpretable by analysts using visual interpretation.This effort can be useful in two ways: a) accelerate human-analyst work during visual interpretation; b) increase speed of rescue decision support systems by obtaining automatically generated alerts on appearance of the objects of interest in geospatial imagery.The latter is exceptionally important due to a vast amount of multisensor geospatial data needed to be processed in a timely manner during rescue operations..The majority of natural landscape and aerial images are usually formed by pattern combinations -various types of boundaries created by changes in feature signs such as color and texture, bringing a lot of difficulties in automated image processing.Thus, a potentially reliable algorithm needs to consider all combination of different types of image attributes together in order to provide correct segmentation of real natural images.Furthermore, image segmentation itself is an ill-posed problem.It often requires user assistance to select a proper scale for detecting edges and boundaries, in order to segment the proper regions of interest.
Spatial gray level co-occurrence matrices estimate image properties related to second-order statistics.Haralick (1979) suggested the use of gray level co-occurrence matrices (GLCM), which have become one of the most well-known and widely used texture features.The GxG gray level co-occurrence matrix Pd for a displacement vector d=(dx,dy) is defined as the entry (i,j) of Pd is the number of occurrences of the pair of gray levels i and j which are a distance d apart.Formally, it is given as: . is the cardinality of a set.The software ENVI (Harris, 2016;   http://www.exelisvis.com/IntelliEarthSolutions/GeospatialProducts/ENVI.aspx)has realization of occurrence and co-occurrence texture segmentation algorithms mentioned above.Figure 5 depicts grey level and pseudo-colour results of the Tamarisc thermal imagery processing via ENVI software.

CONCLUSIONS AND FUTURE RESEARCH
Preliminary experimental results indicates that living humans are detected in a very reliable way in both positive (around 13°C) in Poland and negative (-5°C in the US) temperature intervals.For statistical definition of the proposed method reliability more experiments are required.Our future research will be devoted to increasing accuracy and minimizing processing and analysis time for obtaining results, including further software and algorithm optimization along with deploying new SUAV sensors, such as portable X-Ray systems.

Figure 1 .
Figure 1.Dummy deployed in the US (CPR training mannequin); note the snow on the ground.The purpose of the described research is optimization on timing and cost of the following two tasks: -human search, -define quickly if identified human is alive.Technological processes for realization of these tasks can be divided into following consecutive operations: 1. Development of the rescue flight specification; 2. Choice of the UAV platform and sensors suite; 3. Development of the flight plan;

Figure 2
Figure 2 depicts thermal sensors for the US and Polish teams deployed in research experiments.

Figure 3
Figure 3. a) LWIR sensor mounted on 3DR IRIS drone b) Tamarisc check before flights on DJI Inspire drone.

Figure 6 .
Figure 6.Lepton LWIR imagery segmentation results; analogously Figure 6 depicts grey-level and pseudo-colour results of the Lepton LWIR thermal imagery segmentation.

Table 1 :
UAVs Classification; AGL is above ground level and MSL is mean sea level.

Table 2 :
Equipment and sensors specification.