SETUP OF A CORONA CAMERA AND IMAGE CO-REGISTRATION / CALIBRATION

Power lines are an important infrastructure and need special attention. Their functionality is of high importance for our life. Their monitoring is important to guarantee their sustainable operation. It is done mostly by aerial survey using different technologies. In addition to standard setups, e.g. LiDAR and high resolution RGB cameras, often UV cameras, so called Corona Cameras, are used to detect unwanted discharges. In many cases this UV cameras are combined along with RGB cameras with a similar field of view in order to superimpose the detected UV signals onto a visual image of the inspected objects. Discharges at power lines indicate findings that should be monitored with such additional sensors like thermal or high resolution mapping cameras then. The use of Corona-Cameras for such inspection work needs several steps of calibration, co registering and direct referencing methods in order to get them in a good georeference to the other sensors.


BACKGROUND
Power lines belong to the most important infrastructure especially in the industrialized urban world. Our daily life highly depends on a sustainable electric power supply. Monitoring them to guarantee their reliable operation is a legal obligation in many countries and/or part of homeland security. Big efforts are put into overhead line inspections to prevent issues on the infrastructure itself or of vegetation that grows to close. Every year thousands of km are monitored mainly with helicopters either by manual inspections with observers and/or combined with sensors that capture data for documentation or also analytics on the captured datasets. A typical setup is the use of LiDAR that nicely maps the power line and detects vegetation issues due to first and last pulse analytics. Aerial cameras are used for detailed imaging with resolutions of better than 1 cm in combination. More and more complex systems are used for automated issue detection like for example in the "SIEAERO" Project of Siemens in Germany. Many sensors deliver huge datasets which enter automated feature and issue detection algorithms using AI Techniques. Beside LiDAR and high resolution imaging specific sensors are used for detecting anomalies. Beside thermal cameras that capture images of hotspots, the use of Corona-Detectors became important. Discharges in the UV band indicate issues on the infrastructure. Their detection guides the automated algorithms to analyze on the other data what the problem finally is.

The cameras and their features
Detecting discharges needs a UV sensible camera with related lenses that consists of UV-permeable optical components in combination with a daylight blocking filter with a narrow bandwidth of 265nm +-15 nm. Daylight blocking is important so that natural UV radiation does not interfere with the detection of unwanted discharges. The transmitted radiation must be amplified by a multi-stage UV sensitive amplifier and projected on a phosphorescent material which is connected to a CMOS Sensor via fiber optic components.
The advantage using such a construction is, that behind the phosphorescent plane you can use any kind of industrial camera to capture the emitted photons. Because of the small spectrum a monochrome sensor is sufficient. As mentioned before, it is not necessary to put a sensor with bayern-pattern into the UV sensing system. So the optimal way is to use an achromatic sensor to capture the intensity of the multi-stage amplification. Corona discharges appear as blobs of white pixels on the otherwise black images. To identify the location of detected discharges, a second camera, working in the visual spectrum is used in parallel with the UV sensitive camera. To get the most of information out of the VIS camera, a RGB-bayern-pattern Camera with a relatively high resolution and high frame rates is preferred.
To achieve the best result, two almost similar industrial cameras were chosen for the task. This does not only provide the best PTP-synchronization compatibility, but also ensures that both Sensors have the same sensor size. Using similar camera lenses, theoretically the fields of view of both cameras are almost identical, i.e. the visual cameras covers the same region of interest as the UV sensitive camera. One major difficulty is to compensate the intensity losses caused by the daylight filter of the UV Camera. Because of its structure the remote areas at the frame hit the filter by a wider angle that causes higher absorption. Also, diffraction artifacts, caused by the grid structure of the filter, can cause problems as mentioned at the border of the image.

Camera synchronization
Discharges are detected as single dots in an 8Bit monochrome video stream with up to 50 frames per second in order to capture discharges that can be initiated by the AC frequency of the power lines. A basic issue was to synchronize both cameras, UV and visual, to an exact GPS time tag. This was managed by synchronizing both cameras with the precision time protocol (PTP) and a GNSS based PTP timeserver (Eidson, J. C, 2006). The time of all devices will be synchronized to the GPS-time provided by the timeserver. Logging the data with the exact timestamp is the basis for a direct referencing to a GNSS-INS trajectory, post processed after the mission.

Figure 1. PTP Synchronization scheme
Each frame capture gets triggered by a pre-calculated timestamp of the internal clock, which is periodically synchronized to the GPS Time in order to correct the drifts of the internal clocks. Because the cameras get triggered at a specific GPS-timestamp and the flight data also is referenced to this time, both data streams can be interpolated afterwards in order to enable photogrammetric processing.

Figure 2. Precision of the cameras PTP-synchronization
The precision of the synchronal image capture can be measured via the sync outputs of the cameras. If the frame acquisition begins the IO Pin is set to high and can be measured by an oscilloscope. The result is shown in 2. The delay between the exposures of the cameras is 500-560ns. The handling of the image data, capture and storage, is a further big challenge.

Capturing Images
The basic idea of the UV and RGB Camera combination is to overlay detected UV radiation with the RGB image data. While the RGB Camera shows frames of the real scenery, the discharges create only single photon point-clouds visible as BW signal. The overlay of these points on the RGB image enables a real-time monitoring and to detect and inspect the place of discharge visually e.g. insulators, cable clamps etc. To get the best result, a tradeoff between the highest possible camera resolution and a suitable frame rate must be done. Limiting factors are the operating system where the image processing takes place beside the maximum available network bandwidth on the other hand. The result is considered to be a lightweight system to be operated by UAVs, but both, the computing unit and the network solution, will grow with increasing the processing performance. To reach the target of 50 frames per second, a reduction of the dataflow on the network is needed. Because the multistage amplifier limits the optical resolution of the UV-camera, the sensors pixel resolution can be reduced by a procedure called binning or decimation. "Decimation is used primarily for the reduction in the number of pixels and the amount of data while retaining the original image area angle and image brightness" (Allied Vision, 2021) . This process can be applied to both the horizontal and vertical resolution. In this case a vertical and horizontal decimation of 2 (skipping every second pixel) gives an improvement of 37.5% compared to both cameras running at full resolution: (1) (2) data saved = data nobinning −data binning 2•data nobinning = 37.5% As shown in Figure 3, at a sensor resolution of 1936x1216, frame rates up to 40fps are possible with 1GigE bandwidth. To further improve this performance up to 50 fps either a 2.5 or 10GigE capable system can be installed or the height and width can be reduced by the ROI (Region of Interest) feature. Another solution might be applying decimation on the RGB-Camera too, but because this leads to information losses in the real visual picture it is not recommended.

Image overlay
After capturing images from both cameras, this data needs to be combined in case of a detected discharge. For this purpose, an RGB-to UV-image calibration or co-registration has been performed. For this step, 10 special electronic targets have been developed that emit UV light in the needed narrow UV band width. Placing them on a facade, determining the relative coordinates with a total station and capturing images at different perspectives with both cameras enable a relative orientation of the UV data using a transformation into the RGB image.
For this task, several options are available. Not only linear geometric transformations like the affine-or perspective transformation but also polynomial transformations must be taken in consideration. This is due to several source of errors like misalignment parallaxes error or lens distortion of the camera optics. Depending on the impact of the errors mentioned before a linear transformation may be insufficient. But also, the increasing processing power, needed for image transformation of higher orders must be kept in mind.
x and t y discribes translation.
a 20 and a 21 yields two more degrees of freedom another method needs to be applied to project the UV-image on the color image with a non-linear transformation.
Considering that the transformation takes place up to 50 times a second, the calculation needs to be done quite fast. To do so, instead of calculating the new position of the coordinate with polynomials for every frame, a map can be created within the calibration process. This results in a large lookup table, storing shifting data for every pixel in the image. This saves a lot of computation load. After calculating the UV-pixel positions in the RGB-image a second step has to be done. In order to calculate the 8Bit-value in the new picture, an interpolation method has to be applied. This is shown by the schematic in Figure 4. The UV-image does not contain the most precise and sharp information about the corona discharge. Therefore the choice of the interpolation method can be reduced to two methods while polynomial and higher ordered interpolation methods will increase the arithmetic operations without a noticeable effect. Both interpolation principles, nearest-neighbor and bilinear, are represented in Figure 5.

Calibration Process and Resulting Image
There are several steps to be performed to successfully get the UV data overlaid on the RGB image. First is to analyze the distortion of both lenses (K. Kraus, 1994). To calibrate the RGB camera a line pattern with equal intervals can be captured. In terms of getting the most precise information, the captured image was evaluated with an automatic detection programmed with OpenCV. Thus, not only time is saved by the automated processing, also the calibration points are detected with subpixel precision by getting its center of mass (shown in Figure 6). To receive the same data from the UV-camera was more complex. Reproducing a pattern with equal distances is achieved by moving a UV LED target with a linear motion through the diagonal of the field of view. For this task a setup with a stepper motor can deliver high precision linear movement.
Based on the achieved data, an evaluation process was done to analyze which transformation will result in a good error-to-performance ratio.

ONLINE DATA EVALUATION
Finally, to process all these steps and ensure that live video output as well as saving the relevant data is possible, the operating software has to deal with several tasks. PTP-synchronized capturing, image processing, displaying as well as saving the result to an appropriate drive needs to be done almost simultaneously to reach the requested frame rates.
To fulfill this requirement a multithreading environment is essential. Splitting up the tasks in different sections and connecting them with generous sized data buffers leads to establish a dynamic system. In case of a powerful multicore system or even one with a graphic processing unit, the latency of the live output will be negligible. Also, less powerful systems like modern and lightweight single board computers can handle this operation even though with a noticeable output delay. The concept is represented by the flow diagram in figure 10. For further improvements of the performance on slower hardware the concept can be slightly modified.

Figure 9.
Flow diagram a software structure to achieve live output Instead of displaying and saving all image data, the relevant data is processed only. In this case processing will only applied if the UV-camera detects photons above a defined threshold. This can simply be done searching for indicators in the histogram of the image or furthermore an efficient outline detection to analyze the dimension of a discharge cloud. This will make the system even more dynamic, since the processing load is higher when detecting discharges and that again will fill up the buffer. On the other hand, if nothing is detected, the system has time to process queued frames and clean up the buffer. This not only reduces the overall system load, it also prevents unnecessary data to be stored into the HDD. In addition and depending on the requirements of the user, less data output might end up in a faster post processing and therefore a better workflow.

Evaluation of the accuracy
Another task to be performed is the evaluation of the accuracy. As already mentioned, the designed UV-System it is rather complex to evaluate the difference between the overlaid UV-and the RGB-image. To measure the difference at certain points the calibration LED-targets can be placed to different positions and their pixel positions be measured in the images before and after the calibration process. But to evaluate the entire overlaid image another method can be applied.
For calculating the error depending on the position all sections of the image, it has to be represented by the evaluation process. Therefor a UV-and bright white LED were placed on a circuit board as close as possible. This construction can be passed through great sections of the entire image area. While doing that, the PTP-synchronized cameras capture the image with a decent framerate. For this, the exposure time needs to be that short that the motion blur does not disturb the measurement. In our software, two related incoming images which are captured at the same time, can be processed. Not just the UV-image represents the position of the LEDs as a white filled circle, also the RGB-image shows the same white dot due to the high intense light. By auto detecting the midpoints of both white group of pixels, the position of the LEDs in both pictures can be stored. NOTE: Bright reflected sunlight can be a major issue during this process. After doing this, for each taken image pair, the positions were used to evaluate the error at the corresponding x-and y-position: The error in relation to the position can be plotted for further inspection. Figure 11 shows the uncalibrated overlay results and those of the perspective transformation. The calibration images were taken at a distance of 4 m that reduces but still not erases the parallax effect. Because of multiple factors, like lens distortion in combination with the parallaxes error, also small misalignments of the system can lead to a nonlinear relationship between the images, although those cameras are almost similar.
Further research needs to be done to get a lower average error. The difficulty is maintaining efficiency although nonlinear transformations like polynomial approximation is applied. One option could be a remapping process (OpenCV,accessed 2021), where all approximation results are saved to a lookup table (LUT) to improve performance.

ENTERING IN THE MULTI SENSOR SYSTEM AND THE DATA WORKFLOW
The new synthetic image still is not calibrated like images received by a metric camera in order to enter a photogrammetric workflow. Nevertheless due to the dominant RGB data, a classic tie point matching can be applied in contrast to the pure UV image. That way, the synthetic images can enter the entire workflow in calibration the multi sensor head. Usually using ground control points, the images pass the typical workflow by using tie-point matching, first alignment, and entering Ground control points and calculate a final adjustment and calibration that results with parameters like radial distortion, calibrated focal length and PPS. Using GNSS-INS data will in addition provide the boresight angles and offset parameters in order to direct reference the camera data. Usually all sensor data are related to the GNSS-INS deice, that way the PTP on a PTP GPS serer gives the best synchronization that is available for that technology.
We do not expect the most perfect calibration since we still have to deal with rolling shutter technologies, but at least we are able to adjust the synthetic images with a few pixels offset to the sensor data out of very high resolution data e.g. LiDAR and RGB images. Not the synthetic image will be used for reconstructing a digital twin of a power-line, this data will be super positioned to the other data in order to guide automated AI algorithms to the exact points of the findings.

OUTLOOK
Monitoring Powerlines became a growing task within the last 10 years. Multiple sensor installations support data analytics using AI algorithms. To assist the software in managing the huge amount of data, sensors that show anomalies assist in searching at the right data position for the objects. In both directions, using high resolution data and sensors that directly guide to findings, AI technology will improve.
The new PTP technology enable to use non photogrammetric sensors now precisely georeferenced in multi sensor systems. The idea, to use smaller and different sensor technologies in a combined way as synthetic images will assist in smart sensor integrations. Other technologies e.g. radar and HF receivers will surely enter the monitoring of powerlines in the future