EP3811286A1

EP3811286A1 - Method and assembly for detecting objects on systems

Info

Publication number: EP3811286A1
Application number: EP19766186.1A
Authority: EP
Inventors: Josef Alois Birchbauer; Vlad Comanelea-Serban; Olaf KÄHLER
Original assignee: Siemens Energy Global GmbH and Co KG
Current assignee: Siemens Energy Global GmbH and Co KG
Priority date: 2018-08-21
Filing date: 2019-08-20
Publication date: 2021-04-28
Also published as: US20210174061A1; CN112639803B; CN112639803A; US11989870B2; WO2020038944A1; EP3614299A1; BR112021002143A2

Abstract

The invention relates to a method for detecting objects on systems, having the steps of: - providing a three-dimensional representation of the system, wherein the position and orientation of the representation and the system are known, and - capturing a first image and a second image of the system, the two images being captured from different positions above the system. The invention is characterized in that for a plurality of sections of the system, a respective comparison of the first and the second image is carried out using a parallax effect. If the images in a region surrounding the system match, an object is detected on the system. The invention additionally relates to a corresponding assembly.

Description

description

Method and arrangement for recognizing objects in systems

The invention relates to a method according to the preamble of claim 1 and an arrangement according to the preamble of claim 10.

The previously unpublished European patent application 17161027.2 dated March 15, 2017 with the title "Method and arrangement for condition monitoring of a system with operating means" discloses a method for condition monitoring of a system with operating means, in which by means of a first vehicle with an overview sensor arrangement for opti system data, overview data are recorded, and

by means of an evaluation device in the overview data, the operating resources are recognized and the positions of the operating resources are determined taking into account the position of the first vehicle, whereby detailed images of the operating facilities are determined by means of a second vehicle with a detail camera that is aligned with the respective positions of the operating means means are generated. For example, only a single aircraft such as a drone or a helicopter is used to detect masts and isolators when flying over an overhead line using the overview camera, determine the position of the isolators and then use the detail camera to obtain high-resolution images of the isolators. In this way, defective insulators can be identified easily and reliably.

Numerous detection methods in images are known from the prior art. In particular, object detection in two-dimensional (2D) images has recently received increasing attention. But classic detections, for example in thermographic images or UV images, can also be found in constant industrial use. There are also numerous other 2D detection methods, e.g. based on anomalies lien or also on color differences. While the majority of the previous work has been limited to 2D detection, there are only a few works that attempt to transfer detections into a three-dimensional (3D) space.

For example, the publication "Probabilistic Integration of Cues From Multiple Cameras" by

J. Denzlerl et al. , which deals with the integration of image recordings from different cameras. From the publications "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks" by Shaoqing Ren et al., "SSD: Single Shot MultiBox Detector" by Wei Liu et al. and "Using Histograms to Detect and Track Objects in Color Video" by Michael Mason et al. , 2001, methods for object detection in image data are known. The publication "Anomaly Detection: A Survey" by Varun Chandola et al., ACM Computing Surveys, 2009, deals with the detection of deviations from expected patterns in data.

The doctoral thesis "Improving Object Detection using 3D Spatial Relationships" by Tristram Southey, MSc., University of British Columbia, 2013, is also known. A three-dimensional image analysis is described.

When inspecting systems such as Overhead lines have been used overflights with helicopters and image recordings to detect damage or objects on the overhead line. The decision whether an object such as A bird's nest, a balloon or a kite (children's toys) lying on the conductor ropes or below on the ground is difficult to hit with a pure aerial view and prone to errors. So far, this has usually been done by manually evaluating the image recordings. If objects on the line are mistakenly recognized, this results in useless costs and efforts for triggered maintenance.

Detections in one or more image recordings cannot always be clearly assigned to a specific 3D object. Since the individual 2D images do not contain any depth information, the distance to an object along the line of sight cannot be determined. Detections can therefore not be restricted to the relevant areas of 3D space, which can lead to irrelevant false detections

leads. If, for example, objects on an overhead line are to be differentiated from objects under the overhead line, automated image evaluation on aerial photographs (bird's eye view) does not make this trivial.

The object of the invention is to provide a method with which objects in systems can be recognized automatically and reliably.

The invention solves this problem with a method according to claim 1.

According to the invention, the problem of object recognition is solved by using the 3D information associated with the respective 2D points. Due to the parallax effect, objects under the system such as an overhead line is shown in the pictures at different points in relation to the line. A parallax effect arises when an observer shifts his own position and an apparent change in the position of an object occurs. The effect of parallax is described in detail on Wikipedia, for example (permanent link:

https: // de. wikipedia. org / w / index. php? title = parallax & oldid = 17 8305744).

A system can be, for example, an electrical system such as an overhead line or an overhead line. However, it can also be a pipeline. For example, an object can be a bird's nest, a car, or a kite.

The object of the invention is to provide 3D information in conjunction with 2D detections in 3D space in order to Reduce alarm rate compared to simple 2D detections.

It is an advantage of the invention that the frequency of false positively recognized objects on a system, i.e. an alarm rate is reduced by the combined use of several views in a comparatively compute and memory-efficient method. This results in a comparatively more reliable automatic detection of objects from aerial photos and a significant reduction in the effort for manual post-processing. Even false detections that only appear in a single image from a camera can be reliably detected with the method.

According to the invention, it is possible to differentiate safely, quickly and automatically between objects on a system - ie above ground level - and objects below the system - ie close to the ground level. This is an advantage because of dangers or damage to a system such as an overhead line must be removed immediately by maintenance technicians. If an object such as If a bird's nest or a kite is incorrectly recognized as being on the line, the line is switched off and / or maintenance is unnecessarily triggered, which causes costs and reduces the availability of the system.

Furthermore, it is an advantage of the invention that the method described is comparatively little computational. Since scalar values at the known 3D points - the sections of the system - only have to be read out in the two two-dimensional images, the memory and computing requirements are comparatively significantly smaller than in alternative methods.

In a preferred embodiment of the method according to the invention, the three-dimensional representation of the system provided is used to restrict a search space for the system or to include the recognized object as a component assign the system in the three-dimensional representation. Surprisingly, it was found that the use of the three-dimensional representation already available makes it possible, in particular, to reduce the computing power required for image evaluation by restricting the search space. Furthermore, a recognized object can be assigned to a component contained in the three-dimensional representation, so that relevant and irrelevant objects can be separated easily and reliably for further evaluation. With the approaches allow the required computing power to be significantly reduced. Both options have proven to be very beneficial. For example, the first option allows a false negative rate to be feared in the case in question, at the expense of a slightly higher computing power. The second option allows, for example, by means of a less computation-intensive pre-classification, to identify potential problem areas while the images are being taken and, if necessary, to carry out a more precise inspection of these objects automatically. A combination of these two options has also proven to be advantageous.

In a further preferred embodiment of the method according to the invention, the three-dimensional representation is recorded as a three-dimensional point cloud (PCD), the three-dimensional point cloud (PCD) being semantically segmented in order to restrict a search space for the system in the three-dimensional point cloud (PCD). A restriction of the search space in this way has proven to be particularly advantageous for typical applications, such as those that occur in the inspection of overhead lines.

In a further preferred embodiment of the method according to the invention, the three-dimensional representation is obtained by means of a “light detection and ranging (LIDAR)” sensor and recorded as a three-dimensional point cloud (PCD). This is an advantage because LIDAR provides highly precise information about the position can be obtained from objects In connection with the position of the aircraft, the objects can be located in three-dimensional space.

In a further preferred embodiment of the method according to the invention, the three-dimensional point cloud (PCD) is semantically segmented in order to restrict a search space for the system in the three-dimensional point cloud (PCD). This is an advantage because the object recognition is restricted to the relevant area, which considerably reduces the calculation requirements and / or increases the speed of the calculations. The complexity is reduced because the search space is restricted to relevant scene content. If the evaluation is carried out on board the aircraft, weight can be saved in this embodiment because a less powerful computer device is required. A typical example is LIDAR data of a high-voltage line, in which (automatically) those points are determined that belong to the overhead lines or are approximated by a parametric model of a chain line. An example of a method for segmenting image data is known from the publication "Mask R-CNN" by Kaiming He et al.

A classic 2D detector is used in this restricted search area, which is pre-trained for certain accident classes. Alternatively, an anomaly detection automatically becomes a model of the norm

the conductor region is determined (e.g. using auto encoders) and outliers are detected. Both approaches determine in

The result is a probability of potential detection-relevant states of individual pixels or image areas. In one variant, the image space can not only be limited to visible light, but can also extend to adjacent spectral ranges such as (thermal) infrared and ultraviolet light. In a further development of the aforementioned embodiment, it is provided that the detection answers or pixel color values in the individual images for each 3D point of the system

to query and consolidate these individual detections. One of the following linear or nonlinear mathematical approaches can be used as a consolidation function: extreme value determination (minimum / maximum), median determination, mean value determination, other robust statistics such as using quantiles.

The following process steps can be carried out:

- As INPUT, 3D point clouds and, in addition, highly precisely recorded images are obtained;

- The 3D point cloud is optionally semantically segmented (optional, since the entire PCD can also be semantically relevant for the inspection task);

- The 3D points remaining after segmentation are projected into the 2D images;

- At the resulting points in the 2D image, detection results are either generated selectively or the existing detection results are read out (the latter if the detection in the image space was carried out across the board);

- For each 3D point, the associated individual detections in the image space are checked for consistency using a consolidation function;

- Optionally, the remaining 3D points can again be projected back into the image space and thus result in the final detection result in the image space.

In a further preferred embodiment of the method according to the invention, an overhead line is used as the system, and the semantic segmentation is carried out by using a model of a chain line for detecting conductor lines of the overhead line. This is an advantage because the search space can easily be restricted. In a further preferred embodiment of the method according to the invention, the position and orientation of the representation are determined by means of a position determining device. This can be done, for example, by means of a receiver for "Global Positioning System (GPS)" signals, the orientation depending on the viewing direction of the sensor arrangement (LIDAR or camera). The viewing direction can be determined, for example, by means of a tilt sensor in conjunction with a compass, which are provided in the aircraft.

In a further preferred embodiment of the method according to the invention, the images are recorded by means of a camera for visible light. The light visible to humans is usually specified with wavelengths between 380 nm to 780nm (permanent link:

https: // de. wikipedia. org / w / index. php? title = electromagnetic s_spectrum & oldid = 178702023).

In a development of the aforementioned embodiment, who set the other cameras to improve reliability. For example, several cameras can be used redundantly or with different magnifications or detailed resolutions. This is an advantage because it increases the likelihood of being able to obtain all the required image data in just one flight over the system.

In a further preferred embodiment of the method according to the invention, the camera is guided along the system with an aircraft in order to record the first and the second image at the two different positions.

In a further preferred embodiment of the method according to the invention, the three-dimensional representation of the system is projected into the two images in order to determine the sections from each.

In a further preferred embodiment of the method according to the invention, the evaluation device is advises provided. This is an advantage because an evaluation and object detection can take place directly during an overflight. The images and coordinates of the recognized objects can be saved and transmitted to the operator of the system after the flight. Alternatively, the detected objects can be transmitted by radio data communication during the flight.

In a further preferred embodiment of the method according to the invention, the evaluation device is provided as a central server. This is an advantage because it saves weight and space in the aircraft. For example, all the data recorded by the camera and the LIDAR can be temporarily stored on a data memory and, after the end of the flight, transmitted to the evaluation device for evaluation. Alternatively, the data can be transmitted to the evaluation device by means of radio data communication even during the flight.

Furthermore, it is an object of the invention to provide an arrangement with which objects in systems can be recognized automatically and reliably.

The invention solves this problem with an arrangement according to claim 10. Preferred embodiments result from claims 11 to 15. For the arrangement according to the invention and its embodiments, the same advantages as described at the outset for the method according to the invention result.

For a better explanation of the invention show the schematic representation

Figure 1 shows an example of a semantic segmentation of

LIDAR image data, and

Figure 2 shows an example of images of an overhead line in different frequency ranges, and Figure 3 shows an example of anomaly detection of objects on an overhead line, and

Figure 4 shows an example of a detection of the position of

Objects on an overhead line through the parallax effect.

Figure 1 shows an example of a semantic segmentation of lidar image data. The viewing angle cp of the LIDAR with respect to the location coordinate x is shown. A color scale 3 shows how strongly the LIDAR signals were received. It can be seen that after a successful segmentation of the overhead line cable, line 1 is highlighted using a model of a chain function. The other lines 2 remain in the background.

Figure 2 shows an example of images of an overhead line in different frequency ranges. An image in the visible frequency range (VIS), in the infra-red frequency range (IF) and in the ultraviolet frequency range (UV) is shown from left to right. In the visible frequency range (VIS) there are 1 bird nests on the lines. In the infrared area (IF), a particularly heated area 99 of an isolator is shown on a mast. In the UV range (UV), corona discharges can be clearly seen on the lines 1.

FIG. 3 shows an example of an anomaly detection of artificially inserted objects on an overhead line. The picture is taken from above during an overflight. In this case, cable 1 run over wooded areas and a road 4, which bifurcates in the upper part of the picture. There is a car 5 on street 4. A kite 6 is arranged on one of the conductor ropes. In the picture, the evaluation algorithm correctly marks the objects as deviating from the expected course of the ropes. However, the algorithm cannot easily obtain the depth information, ie it cannot decide whether the car and in particular whether the kite is on the line or below on the ground.

In order to distinguish objects on the ground from objects on the line, the invention proposes an exploitation of the parallax effect. Figure 4 shows two scenes side by side. Two masts 9 each carry an overhead line. Trees 10 can be seen below the overhead line. A first and a second image are recorded at two different positions 7, 8 during an overflight of the line.

In the left image it can be seen that when both viewing directions are aligned with the two images at a point 11 on the overhead line previously recognized by segmentation in 3D space, both images target a section 11 of the line in the line of sight. If an object is arranged directly on or on the line, the object appears on the line at the same location from both perspectives. It is different in the right picture for tree 10. In the two pictures, tree 10 does not appear at the same location on the line, but because of the parallax effect from viewing direction 7 on section 11 and from viewing direction 8 on section 12 of the Management. This means that the tree 10 does not have to be arranged at the same height as the line, but rather below it. This principle enables a simple automated distinction between objects arranged on or on a system and objects arranged on the ground.

Claims

claims

1. Method for recognizing objects (5, 6) in systems 1), with the steps:

- Providing a three-dimensional representation of the plant ge, the position and orientation of the representation and the plant are known, and

- Capture a first image and a second image of the system, the two images being taken from different positions (7, 8) above the system (1), characterized in that

A comparison of the first and the second image is carried out for a multiplicity of sections of the system (1) using a parallax effect, with an object (5) on the system (5) if the images match in an area surrounding the system ( 1) is recognized.

2. The method according to claim 1, characterized in that the three-dimensional representation of the system provided is used to restrict a search space for the system or to assign the recognized object to a component of the system in the three-dimensional representation.

3. The method according to any one of the preceding claims, characterized in that

the three-dimensional representation is recorded as a three-dimensional point cloud (PCD), the three-dimensional point cloud (PCD) being semantically segmented in order to restrict a search space for the system in the three-dimensional point cloud (PCD).

4. The method according to claim 3, characterized in that an overhead line is used as a system, and that

the semantic segmentation is carried out by using a model of a chain line for the detection of conductor cables (1) of the overhead line.

5. The method according to any one of the preceding claims, characterized in that the three-dimensional representation is obtained by means of a “light detection and ranging (LIDAR)” sensor and is detected as a three-dimensional point cloud (PCD) and / or that the position and orientation of the representation is by means of a Position determining device can be determined.

6. The method according to any one of the preceding claims, characterized in that the images are recorded by means of a camera for visible light.

7. The method according to any one of the preceding claims, characterized in that the camera is guided along the system with an aircraft in order to record the first and the second image at the two different positions (7, 8).

8. The method according to any one of the preceding claims, characterized in that the three-dimensional representation of the system (1) is projected into the two images in order to determine the sections.

9. Arrangement for recognizing objects (5, 6) on systems (1), comprising:

- An aircraft with a "light detection and ranging

(LIDAR) "Sensor for recording a three-dimensional representation of the system, and

- With a position determining device for detecting the position and orientation of the three-dimensional representation of the system, and

- With a camera for visible light, which is formed for capturing a first image and a second image of the system, the two images being taken from different positions (7,8) above the system (1),

characterized in that

an evaluation device is designed for a large number of sections of the system using a pa rallaxen effect each carry out a comparison of the first and the second image, wherein in the event of a match of the images in an area surrounding the system (1) Be an object on the system (1) is detected.

10. The arrangement according to claim 9, characterized in that the "light detection and ranging (LIDAR)" sensor is designed to detect the three-dimensional representation as a three-dimensional point cloud (PCD), and that

the evaluation device is designed to semantically segment the three-dimensional point cloud (PCD) in order to restrict a search space for the system (1) in the three-dimensional point cloud (PCD).

11. The arrangement according to claim 9 or 10, characterized in that the system has an overhead line, and that the evaluation device is designed to perform the semantic segmentation by using a model of a chain line for the detection of conductor cables (1) of the overhead line.

12. Arrangement according to one of claims 9 to 11, characterized in that the evaluation device is designed to project the three-dimensional representation of the system (1) in each case in the two images in order to determine the sections each Weil.

13. Arrangement according to one of claims 9 to 11, characterized in that the aircraft is an aircraft, a helicopter or a drone.

14. Arrangement according to one of claims 9 to 13, characterized in that the evaluation device is provided in the aircraft.

15. Arrangement according to one of claims 9 to 14, characterized in that the evaluation device is provided as a central server.