MULTI SENSOR AND SMART GIMBAL FOR ENHANCED POWERLINE MONITORING ON HELICOPTER AND UAVS

: The inspection of overhead high voltage power lines is important because power supply has to be seen as a critical infrastructure. Monitoring power lines using helicopters and UAVs requires a setup of various sensors. The combination of high-resolution aerial cameras for capturing better than 2 mm GSD, LiDAR collecting 200 point/m², thermal cameras detecting hot spots in cm accuracy of anomalies and UV-cameras checking for corona discharges are important elements in the newest monitoring setups. The combination of online analytics with the Corona Camera (defined as a combination of UV detector and RGB monitoring camera) and the Smart Gimbal technology enhances the quality and speed of the inspection. Besides that, it reduces the need of too many sensors. We show different integrations and their pros and cons and as an outlook where the technology with respect of entering AI technologies will go. A basic task is the handling of the enormous data volume, at this point online analytics may help to reduce the data-volume beside a rapid response.


BACKGROUND
High and ultra-high voltage overhead power transmission lines are part of a sensitive and very important infrastructure. The international power line network helps to bridge gaps but especially due to new technologies like windfarms and solar-power plants, the distribution and storage becomes more important than in the past. Thus, the need of maintenance and detection of issues/findings became an obligation and remote sensing technologies help to automate or at least document this process. Still today observers visually inspect infrastructure of cables and poles during the helicopter missions. Insulators, wireclamps, bumpers and many other features must be monitored and maintenance frequently. During the mission observers typically use handheld cameras for the documentation but are meanwhile supported by more complex technologies e.g., LiDAR to detect vegetation that grow too close to the power line. In the last 7 years the integration of multi sensor systems with higher resolution entered the inspection workflow. Thermal, Corona and high-resolution visual cameras became the dominant part but also LiDAR and other electronic devices help to detect anomalies on the overhead lines. While some of the sensors provide visual data or measurements (RGB and LiDAR) others, e.g., thermal and Corona Cameras, directly show anomalies that may indicate a finding. The combination of both enables the automated detection if issues.

SENSOR SETUP IN HELICOPTERS
The most complex sensor setup is used by Siemens in the SIEAERO program. Flying with a side and height-offset * Corresponding author of 40 and 15 m from the power line this distance typically fulfills the core of security regulations. During the flight the data are captured with forward and sideward oblique angles that give the best view into the pole structure. Using 4 high resolution cameras take oblique images in order to detect the findings. The cameras are rotated in forward and backward direction and tilted in a certain angle downward to capture the pole with 2 cameras entirely. 2x100MP forward and 2x100 MP backward cameras, one nadir orthophoto camera, an array of 4 thermal sensors, LiDAR and Corona discharge detection system make this setup unique. Nevertheless, each flight hour generates about 2.5 Terabyte of RAW data that has to be processed.
All data need to be preprocessed using GNSS/INS. Processing of LiDAR data need a perfect GNSS/INS trajectory but also the small Thermal camera data need to be directly referenced. The RGB images may be processed by tie point matching, but due to the huge number of captured images the direct referencing is a much more useful approach. Procedures for the internal, Sensor to IMU, bore sight and sensor to sensor calibration are an intensive process to get the entire sensor output prepared for a fast processing and analytic pipeline. More difficult is the integration of the video stream generated by the Corona camera. This camera uses a narrow band daylight blind and UV amplified Sensor that only detects discharges and in parallel to a coregistered RGB Video camera that is used to overlay the discharges to the video stream. However, this setup with a specific focus needs a dedicated mission plan that takes all poles and infrastructure into account. Missing information or imprecise mission navigation can cause gaps in the images or lower data quality. This indeed blocks the defined processing workflow and may result in a re-flight of the power line. For planning the mission, a GIS based tool was developed that takes the pole structure into account as well as the footprint of the images related to a certain trigger point. It is important to have the exact coordinate of the pole, the height and length of the traverses and the layout of the structure e.g., the places of the Insulators and cables. In order to be effective during the mission, a flight path as a curved line was calculated that entered a Flight Management System with specific instruments to show the curves to the pilot in advance. All sensors were triggered and controlled automatic without the need to interfere by an operator.
The basic idea of such a complex sensor setup is to combine sensors that directly detect anomalies like the corona discharges or thermal effects. To reduce the number of sensors as well as the number of data to be captured, 2 strategies help to get the system smaller and smarter.

ONLINE ANALYTICS ON THE CORONA-CAMERA
A Corona Camera is defined by the combination of an ultraviolet sensitive camera capable to capture images of Corona-Discharges and a RGB Camera of the same view. The developed Corona Camera System is a combination of a daylight blind UV amplified camera and a parallel mounted RGB camera. Both cameras have the same sensor installed and can make use of the same interfaces with control libraries. The image capture rate should be close to but not exactly the same as the power grid A/C frequency of 50 Hz. With 48 Hz both cameras are capturing images simultaneously. Using PTP and GNSS interface, both cameras are perfectly synchronized and thus enable real-time image pre-processing. Both cameras are pre-calibrated to each other, that way the images can be overlaid to each other, displayed, analyzed and saved in real-time. The output data of the UV Camera contains 8bit grey scale pixel values and can be analyzed online by detecting UV radiation as pixel areas exceeding a certain threshold in size and intensity. Detected UV radiation patterns above the thresholds are then highlighted in red on the RGB image. As a part of the online -analytics, during detection of corona discharges the data are stored with the full frame rate of 48 Hz. For time spans without UV events only for documentation purposes the storage frame rate is reduced to 1 fps. Thus, the data volume for the corona discharge images is reduced dramatically, depending on the number of findings, to 0.1-5%. Analyzing the intensity of the discharges over a certain period is a task that has to be managed in further developments. Also online scanning the detection area on the RGB image was planned, so the discharge area can be marked with specific tags (e.g., detected discharge near isolator). For this task an information data interface and fast image data access was provided during software development. But with regard to the high utilization of the computing unit in case of computer vision applications not only strong hardware, but also continues code optimizations of the scanning and detecting process have to be considered.

SMART GIMBAL
To provide the same capturing quality as with the before mentioned setup, a smart gimbal development was initiated. A standard gimbal typically has two functions, on one side to stabilize the flight conditions (roll, pitch and heading), it can on the other side can direct the camera to a specific target, for example to inspect different parts of a transmission pole. For optimal response time, high angular accelerations, high precision and low power consumption (enhanced battery life on UAVs) there are two options for the motor decision: Stepper motors or brushless DC (BLDC) motors with a decent amount of pole pairs. Stepper motors will result in a higher holding torque while BLDC motors achieve slightly better accelerations and higher angular rates. In both cases encoders need to be installed to track the position of the rotor and to enable closed loop operation. Most common setups use BLDC motor controllers with field orientated control capability (foc). This ensures low power consumption (<100mA per motor) when the gimbal operates in a good balanced state. On the other hand, the overall low motor temperature creates a high temperature buffer enabling high current peaks for the motors, so it handles fast acceleration and high torque without overheating. This is especially important to ensure fast turning rates from pole to pole and to resist gusts of wind during operation.
In contrast to the standard setup with stabilization only, the smart gimbal is a fine-tuned all-in-one system with an onboard high-end single-board computer with several interfaces to manage more specific functionalities and interactions between external devices and the gimbal controller unit. Thus, the smart gimbal is able to calculate and execute new operations on the fly and also recalculate them with high refresh rates while moving. While the compensation of the flight parameters typically is automated using GNSS/INS information and the planned track, the controller can make use of pre-planned POIs to follow specific 3D points along the flight path. The approach of using a database for managing point of interests increases the efficiency of the inspection process while decreasing the workload of the operator. The gimbal controlling unit is able to access the nearest PoI with its own coordinates with just one simple database query. Also, it has the ability to log important information to the database entries. Those could be if a corona discharge was detected or the angles/angular rates of the gimbal at the time images were captured. Besides that, it also is able to interface with the cameras, e.g., to refocus the lens by the distance between the actual and the target position in real-time.
The main advantage is that a smart gimbal enables the forward and backward view of the sensor-head, compensates flight conditions, acts to compensate inaccurate flight paths and also compensates uneven terrain in the perspective view. This is shown in the Figure 4 as a nadir view and in figure 5 to demonstrate the influence of the terrain in the side view. A smart gimbal solution needs beside an IMU and GPS a dedicated mission plan that accounts for the optimal flight path and the 3D information of the Poles as POIs. The 3D terrain information is important as well. A definition of rotation speed from POI 1 to POI 2, correcting the focus and the capability to get information of the wires even in-between the poles, are challenging.
Our designed smart gimbal carries three 100 MPix RGB cameras and two thermal or alternatively one corona detection system. With this combination, movements of the aircraft are compensated, motion blurs minimized, and the transmission tower is always kept in the field of view. Furthermore, the online evaluation of the gimbal parameters (angles, current offset angles to POI, current angular rates etc.) can be used to enhance the image quality. This is possible because of the computing unit is monitoring those values while communicating with the camera. The focal distance can be calculated and set in advance before the trigger is released at the point of minimum motion of the gimbal. This feature is especially important for longer focal lengths where small angular rates have a huge impact on motion blur.
Backward, forward and sideward views prevent objects of interest from being hidden behind other parts of the installation. Besides that, contour flights in hilly terrain are also possible. An additional challenged is to modify the system to a single camera with a longer focal length that captures data with a higher resolution of the infrastructure but cannot capture the entire pole in a single image. To solve this issue several POIs on the same pole may be defined and an intelligent algorithm applied that makes use of a sequential operation of the gimbal and the camera. There are several aspects to consider here. The most critical part is the time the gimbal needs to move from POI to POI. The closer the pole the greater the angle to travel and so the drive time increases. If this process of moving the gimbal, refocus the lens and capture one image e.g. takes about one second, the vehicle moves about 10 meters at a speed of 36km/h. That would be 30-40 meters covered distance until the entire pole is completely captured from one side. Minimizing this range with higher acceleration torque as well as fast image data processing and refocusing during gimbal motion period. Another task consists of getting the best order of captures, so that the gimbal's travel distance is minimized. This way in a continuous sweeping motion all parts of a transmission tower can be captured in sequence. Figure 6: Adjustment to the Terrain following while bypassing

ADJUSTMENT TO UAVS
Besides the regulations (BVLOS Missions) for using UAVs, a reduced sensor setup is needed in order to reduce payload weight. We testes integrations on 3 different models for power line inspections.
The AeroSpector Quadcopter from GGS, specially designed for such monitoring work, carries 6.5 kg payload and provides 45' flight-time. This is sufficient for a power line observation length of 10 km (+-5 km both sides) but needs a reduced sensor setup due to the payload compared with a helicopter. The before mentioned gimbal was adjusted and then used to capture thermal anomalies in forward and backward directions well as high resolution images with 2 mm GSD of the entire power line. In a second project the combination of aerial and thermal cameras in a SwissDrones UAV was used to perform a mission in France. The variable setup to exchange cameras and focal length was used to adapt to the regulations of the power-line providers. In that case the camera layout was designed in that way, that 2 cameras with different focal length capture the entire pole with similar resolution while a thirds camera captures in a single shot the entire pole with its environment. The Mission was performed in the Département Creuse at an overhead power line of the network provider RTE.
In a third project a multi sensor system was adjusted to a one axis smart gimbal using the CAMCOPTER S100 r UAV from Schiebel. This test was part of the Innspektor project with Siemens, Schiebel, GGS and Lufthansa. The CAMCOPTER S100 is a huge UAV that is able to carry up to 50 kg for several hours. A LiDAR was fix mounted at the UAVs nose while the other sensors with RGB, NIR and UV were adjusted to the main payload bay in a smart gimbal below the CAMCOPTER S100 Figure 9: Installation at the CAMCOPTER S100 before the mission in Austria Figure 10: Report generated by AI analytics out of a Mission with Siemens on a CAMCOPTER S100 The gimbal was specifically redesigned for this Innspektor application and has a high freedom to rotate around the z-axis. This was done according to the left-or righthand side mission who was needed to adapt the flight of the CAMCOPTER S100 to the wind conditions. A testmission in Austria was performed in summer 2021 successfully.

CONCLUSION
While UAVs are suitable for limited areas due to VLOS mission regulations, the technology of a smart gimbal and the online analytics can be part of the helicopter surveys as well. The interest in this technology is increasing and the possibilities, to run more online analytics opens new fields for the inspection. To do real-time analytics on the UV and the Thermal band (these sensors are used to find anomalies anyway) can setup the other sensors to capture more or less data. Using UAVs, the small corona images can be linked e.g., via 5G to the ground control for immediate validation and needed repairs. AI can assist to find decisions based on multi data analytics -in real time online.
The demand for such inspection work is growing and beside the overhead lines also the inspection of windpower plants becomes very important. Using thermal cameras, the use in the inspection of solar power plants also is in focus of the electric power supplier. The technologies demonstrated are just one step in the more and more automated inspection workflow.