JP2023128381A - Flight device, flight control method and program - Google Patents

Abstract

To perform an inspection by autonomously flying a flying object along an inspection target.SOLUTION: A flight device that photographs a predetermined inspection target in the air is provided with: one or more cameras that photograph a surrounding of the flight device itself; a first image processing unit that analyzes an image from the camera to detect the inspection target; a second image processing unit that analyzes the image from the camera so as to detect an obstacle around the own flight device, which may impede flight; and a control unit that dynamically determines a flight path according to specified flight path information, a detection result of the inspection target, and a detection result of the obstacle, and that controls the flight of the flight device along the determined flight path.SELECTED DRAWING: Figure 1

Description

The present invention relates to a flight device, a flight control method, and a program.

As background technology in this technical field, there is Japanese Patent Application Publication No. 2016-111414 (Patent Document 1). The bulletin states, ``[Problem] To provide an aircraft position detection system that can accurately determine the position of an unmanned aircraft without relying on GPS signals, and an aircraft using the same. [Solution] Distance measurement. By grasping its own position using data and two-dimensional image data obtained from imaging, the unmanned aerial vehicle 100 can be flown and inspected while maintaining a constant distance from the structure OBJ without relying on GPS signals. can be done.''

Japanese Patent Application Publication No. 2016-111414

With the recent decline in the working population, there is a need to improve work efficiency and reduce or replace dangerous work. Against this background, unmanned aerial vehicles (UAVs), such as drones, are increasingly being used for pesticide spraying, aerial photography, inspections, transportation, etc. Inspection work, especially inspections at high places and in areas that are difficult for humans to access, such as mountainous areas, has a high potential for danger, and there is a need for an inspection method that can replace direct visual inspection, and UAVs are ideal for such purposes.

Patent Document 1 describes that the UAV can estimate its own position and fly a UAV even in an environment where it is difficult to receive GPS signals. However, in order to photograph the desired inspection target without going out of frame, precise routing is required to determine where the UAV will fly, and the accuracy of its own position and three-dimensional map is required, as well as the flight route. There is room for improvement in that there is no way to avoid obstacles if they are present above.

Therefore, an object of the present invention is to autonomously fly a flying object along an object to be inspected and perform an inspection. According to the present invention, for example, by recognizing the inspection target and surrounding obstacles, calculating the relative position with the inspection target, and controlling to maintain a predetermined positional relationship, accurate self-position estimation and route No settings are required, just rough waypoint instructions, and you can avoid obstacles on the route while continuing to photograph the inspection target without going out of frame.

In order to solve the above object, one of the representative flight devices, flight control methods, and programs of the present invention photographs the surrounding area with one or more cameras equipped on the flight device, and analyzes the images from the cameras. Detects objects and obstacles to be inspected, dynamically determines the flight path according to the specified flight path information, the detection results of the objects to be inspected, and the detection results of the obstacles, and then flies along the determined flight path. It controls the

According to the present invention, it is possible to continue photographing the inspection target without going out of the frame while avoiding obstacles on the route by simply giving rough waypoint instructions without requiring precise self-position estimation or route setting. can. Problems, configurations, and effects other than those described above will be made clear by the following description of the embodiments.

1 is a configuration diagram showing the configuration of a UAV of Example 1. FIG. 5 is a flowchart of control in Example 1. FIG. FIG. 3 is a diagram showing the operation of the UAV in tracking mode during an inspection operation. FIG. 3 is a diagram showing the operation of the UAV when avoiding obstacles during an inspection operation. FIG. 3 is a diagram showing a route taken by a UAV during an inspection operation. FIG. 2 is a diagram showing the operation of the UAV during an inspection operation (when important inspection points and obstacles coexist on the route). FIG. 2 is a configuration diagram showing the configuration of a UAV according to a second embodiment. 7 is a flowchart of control in Example 2. FIG.

Embodiments of the present invention will be described below with reference to the drawings.

The configuration and operation of a flight device, a flight management method, and a program that are one embodiment of the present invention will be explained using FIGS. 1, 2, 3, 4, 5, and 6.

FIG. 1 is a configuration diagram showing the configuration of a UAV according to a first embodiment. The UAV 10 shown in FIG. 1 is an aircraft used for inspection, and includes an inspection camera 101, an obstacle detection camera 102, a signal processing section 103, a GPS reception section 104, an IMU 105, a control section 106, and a recording section 107. It will be done.
In this embodiment, electric wires, utility poles, and power distribution equipment attached to utility poles are exemplified as objects to be inspected. The inspection object includes an extending object that is an extending object to be inspected, and a relay object that is an inspection object that has a relay function for the extending object. An electric wire is an example of an extending object, and an electric wire and power distribution equipment are examples of a relay object. The extended object can be used to set a flight path along the extended object. The relay target physically or functionally relays the extended target. For example, a utility pole is a physical relay object that supports electric wires, and power distribution equipment is a relay object that maintains the electrical characteristics of electric wires.

The inspection camera 101 is attached to the UAV 10 and is a camera that acquires an external image of the inspection target. The acquired external appearance image is sent to the subsequent signal processing unit 103.

The obstacle detection camera 102 is attached to the UAV 10, and is a camera that acquires external images of surrounding obstacles and distance information to the obstacles, and is, for example, a stereo camera. The acquired appearance image and distance information are sent to the subsequent signal processing unit 103.

The signal processing unit 103 processes the images and distance information transmitted from the inspection camera 101 and the obstacle detection camera 102, and sends the processing results to the subsequent control unit 106. The signal processing unit 103 includes an inspection target detection unit 1031 and an obstacle detection unit 1032.

The inspection target detection unit 1031 performs recognition processing on the external appearance image of the inspection target transmitted from the inspection camera 101 and the obstacle detection camera 102, and determines where the inspection target is located in the image and how large it is. It is determined whether the image is captured and the determination result is sent to the control unit 106.

The obstacle detection unit 1032 performs recognition processing on the external appearance image and distance information of surrounding obstacles transmitted from the obstacle detection camera 102, and determines where the obstacle is located in the image and at what distance. It is determined whether or not it exists, and the determination result is sent to the control unit 106.

The GPS receiving unit 104 receives signals from GPS satellites, calculates the position information of the UAV 10, specifically, the latitude, longitude, and altitude, and sends it to the control unit 106.

The IMU 105 is an inertial measurement unit (IMU) attached to the UAV 10 and sends signals from a gyro, an accelerometer, and a compass to the control unit 106.

The control unit 106 dynamically determines a flight path according to the specified flight path information, the inspection target object detection result, and the obstacle detection result, and controls the flight of the UAV 10 along the determined flight path. Dynamic determination of a flight route is, for example, a process of detecting electric wires that are objects to be inspected and modifying the route as needed to fly along them. Similarly, it also includes a process of determining a route that allows photographing objects to be inspected (electrical wires, utility poles, accompanying power distribution equipment, etc.) while avoiding detected obstacles. Specifically, the control unit 106 performs a reading operation on the recording unit 107, reads out predetermined route information, collates the current position information input from the GPS reception unit 104, and determines the position of the UAV 10 in the next time step. Decide on your career path. Additionally, the external image and distance information inputted from the signal processing unit 103, the latitude, longitude, and altitude information inputted from the GPS reception unit 104, the signal information of the gyro, accelerometer, and compass inputted from the IMU 105, and the recording unit described later. Based on the route information input from 107, the UAV 10 moves in a desired direction by controlling the attitude of the UAV 10 to stabilize it, and at the same time controlling the balance of voltages applied to multiple thrust generation motors (not shown). control so that

The recording unit 107 is a recording medium that stores route information that the UAV 10 should follow, and is, for example, a nonvolatile semiconductor memory.

FIG. 2 is a diagram showing a flowchart of control in the first embodiment.
FIG. 3 is a diagram showing the operation of the UAV 10 in the tracking mode during the inspection operation.
FIG. 4 is a diagram illustrating the operation of the UAV 10 during obstacle avoidance during an inspection operation.
FIG. 5 is a diagram showing a route taken by the UAV 10 during an inspection operation.
FIG. 6 is a diagram showing the operation of the UAV 10 during an inspection operation (when important inspection points and obstacles coexist on the route).

The operation of this embodiment will be explained below. This embodiment shows a method of inspecting electric wires, utility poles, and power distribution equipment attached to these using a UAV.

The flight device, the flight management method, and the program of this embodiment are designed to avoid obstacles detected by the obstacle detection camera 102 while being inspected by the inspection camera 101 under the control of the control unit 106 included in the UAV 10. The purpose is to photograph certain electric lines, utility poles, and associated power distribution equipment, and obtain images that can be used for inspection purposes. During flight, it is characterized by detecting electric wires and flying along them.

In this example, there are two operation modes: an operation mode (tracking mode) in which the vehicle flies along the electric wire until a utility pole or distribution equipment is detected, and an operation mode (inspection mode) in which the utility pole or distribution equipment is inspected with priority. exists. In the inspection mode, the aircraft is controlled to, for example, fly around the inspection target along a orbit created separately for inspection, and obtain an image of the entire surroundings of the inspection target.

The method of equipment inspection in this embodiment will be explained using the flowchart of FIG. The procedure of FIG. 2 starts when the UAV 10 takes off and first detects a power line.

First, the control unit 106 receives signals from the gyro, accelerometer, and compass from the IMU 105 (step S101), and calculates voltages to be applied to the thrust generation motors necessary for the UAV 10 to hover through attitude control processing ( Step S102). Next, the control unit 106 receives latitude, longitude, and altitude information from the GPS receiving unit 104 (step S103), and calculates the current position of the aircraft through self-positioning processing (step S104). Here, self-positioning refers to processing that removes error factors and uncertainties inherent in GPS signals by appropriately filtering the obtained raw latitude, longitude, and altitude information using a Kalman filter, etc. Point.

Next, the control unit 106 performs a branching process to determine what the current operating mode is (step S105), thereby causing a branching depending on the operating mode. When the operation mode is the tracking mode, the control unit 106 reads out the route information recorded in the recording unit 107 and compares it with the self-position calculated in step S104 to determine the next direction to proceed. (Step S106).

When the operation mode is the inspection mode, the control unit 106 compares this orbit with the self-position calculated in step S104, since the orbit around the inspection object has been set as described above. By doing so, the next direction to proceed is determined (step S107).

Next, obstacles in the course direction determined in step S106 or step S107 described above are detected (step S108). The presence or absence of an obstacle is determined by the obstacle detection unit 1032 analyzing an external image of the obstacle and distance information acquired by the obstacle detection camera 102. Here, the optical axis direction of the obstacle detection camera 102 is attached facing the front direction of the UAV 10 (controlled so as to match the traveling direction), and does not have an angle adjustment function. Therefore, for example, by setting the angle of view of the obstacle detection camera 102 to a wide angle, it is possible to detect all obstacles in the traveling direction and in a range deviated from the traveling direction. Alternatively, the configuration may be such that the angle of the obstacle detection camera 102 is adjusted by the control unit 106. The control unit 106 calculates the position of the obstacle with respect to the traveling direction by understanding the relative relationship between the traveling direction of the UAV 10 and the optical axis direction of the obstacle detection camera 102. be able to.

Next, the control unit 106 corrects the course depending on the presence or absence of an obstacle and the distance to the obstacle (step S109). Specifically, first, if the distance to the obstacle is longer than a predetermined threshold, no special processing is performed in this step. If the distance to the obstacle is less than a predetermined threshold, the route is corrected to avoid the obstacle. The correction method may be, for example, a method of correcting the course to raise the UAV 10 so as to straddle the obstacle, or a method of detecting the shape and type of the obstacle in the obstacle detection unit 1032 and detecting the shape and type. It is also possible to modify the course according to the situation (for example, if the obstacle is a tree, the vehicle should straddle it, or if it is on a pole such as a street lamp, it should be dodged horizontally).

Next, a branch process is performed to determine what the current operation mode is (step S110), and a branch is generated depending on the operation mode. If the operation mode is tracking mode, electric wire detection processing is executed (step S111). The electric wire detection process is realized by processing either or both of the image data input from the inspection camera 101 and the obstacle detection camera 102 in the inspection target detection unit 1031. The detection means may be, for example, edge detection, a straight line detection algorithm such as Hough transform, or a method using machine learning such as deep learning. By detecting the electric wire, it is determined at which position and at what angle the electric wire appears in the image.

Based on this information, the control unit 106 further modifies the course of the UAV 10 (step S112). For example, when it is determined that the electric wire is visible in the upper half of the image, the course of the UAV 10 is corrected to raise the UAV 10. In another example, if an electric wire is detected as a diagonal line in an image, the UAV's traveling direction is adjusted (for example, if it is rising to the right, it is adjusted to turn to the right), so that the electric wire and the inspection camera are It becomes possible to set the relative position so that 101 faces directly, and flight along the electric wire can be realized.

Next, inspection target object detection processing is executed (step S113). The electric wire detection process is realized by processing either or both of the image data input from the inspection camera 101 and the obstacle detection camera 102 in the inspection target detection unit 1031. The detection means may be, for example, template matching, a cascade classifier in which weak classifiers are arranged in series, or a method using machine learning such as deep learning.

Next, branch processing (step S114) is executed depending on the presence or absence of the inspection target. If it is determined that the object to be inspected does not exist, no special processing is performed and the process returns to step S101. If it is determined that the object to be inspected exists, its position and size in the image are calculated. Based on its position and size, it is possible to estimate in which direction and at what distance the object to be inspected is located relative to the UAV 10. If it is determined that the target object is closer than a predetermined threshold, a orbit (inspection route) around the target object is set based on this information. Next, the current position is stored as the starting point of the inspection route (step S116). The storage destination may be the recording unit 107 or a temporary storage medium such as a RAM (not shown). Next, the operation mode is changed from the tracking mode to the inspection mode (step S117), and the process returns to step S101.

On the other hand, in the branch processing in step S110 described above, if the operation mode is the inspection mode, it is determined that the current position matches or is sufficiently close to the starting point of the inspection route stored in step S116 (step S118; YES). ), the operation mode is changed from the tracking mode to the inspection mode (step S119), and the process returns to step S101. If it is not the starting point of the inspection route (step S118; NO), the process directly returns to step S101.

Through the series of operations described above, it is possible to realize an operation in which the UAV 10 flies along the electric wire and autonomously switches between an inspection route that goes around the object to be inspected and a normal route that flies along the electric wire. . In particular, even if the route information stored in the recording unit 107 does not necessarily follow the electric wire or the object to be inspected, flight along the object to be inspected can be achieved by analyzing images obtained by the inspection camera 101. is possible.

Next, a method for setting a route for flying along electric wires will be explained using FIGS. 3 to 6. FIG. 3 is a diagram illustrating a method of setting a route along an electric wire using the route information recorded in the recording unit 107 and the direction of the electric wire and the distance to the electric wire detected by the inspection target detection unit 1031. The power lines and the path of the UAV 10 are depicted looking down from directly above. In the figure, WP1 and WP2 are passing point information (waypoints) recorded in the recording unit 107. It is necessary to set this waypoint information in advance, but since the location information of the inspection target is often unclear, it is necessary to set rough location information so that the UAV 10 can perform a flight directed by waypoints. The aircraft flies while adjusting the route appropriately so that it follows the electric wires. The method will be explained below.

The UAV 10 is originally controlled to fly along a route L1 (dotted line in the figure) connecting WP1 and WP2. However, the route L1 is not necessarily parallel to the electric wire P that is the object to be extended, and for example, in the example shown in this figure, the UAV 10 flies so as to gradually move away from the electric wire P. In such a case, the electric wires and objects to be inspected appear smaller than expected in the image acquired by the inspection camera 101, and therefore may not be useful for the intended inspection. Therefore, it is possible to control the UAV 10 to fly parallel to the electric wire by, for example, feedforward control as described below.

First, it is assumed that the UAV 10 is present in the WP 1 as an initial state. Next, when the UAV 10 flies along the initially set route L1, the assumed position after the minute time Δt has elapsed is defined as U1. In this case, the UAV 10 attempts to fly along the route L11 from WP1 to U1. Here, it is assumed that the inspection camera 101 is attached at right angles to the traveling direction of the UAV 10. Then, when photographing the electric wire with the inspection camera 101, the electric wire is photographed diagonally by the angle formed by the route L1 and the electric wire P. The angle at which the image is photographed can be estimated, for example, in the process of recognition processing in the inspection target detection unit 1031. Therefore, it is possible to correct the original flight path L11 to the flight path L21 so that the flight path of the UAV 10 becomes parallel to the electric wire P. As a result, the destination of the UAV 10 is corrected from U1 to V1.

A similar correction is made in the next time step, and when the user turns from the current position V1 to the assumed destination U2 after a minute time Δt has elapsed, the path L21 and the electric wire P are A new destination V2 is calculated so as to correct the angle. By repeating this, it is possible to set routes L22, L23, . . . along the electric wire P.

Next, a route setting method when an obstacle is found on the route will be described using FIG. 4. First, it is assumed that the UAV 10 is present in the WP 1 as an initial state. Next, when the UAV 10 flies along the initially set route L1, the assumed position after the minute time Δt has elapsed is defined as U1. In this case, the UAV 10 attempts to fly along the route L11 from WP1 to U1. Here, it is assumed that the inspection camera 101 is attached at right angles to the traveling direction of the UAV 10. Then, when photographing the electric wire with the inspection camera 101, the electric wire is photographed diagonally by the angle formed by the route L1 and the electric wire P. The angle at which the image is photographed can be estimated, for example, in the process of recognition processing in the inspection target detection unit 1031. Therefore, it is possible to correct the original flight path L11 to the flight path L21 so that the flight path of the UAV 10 becomes parallel to the electric wire P. As a result, the destination of the UAV 10 is corrected from U1 to V1.

A similar correction should be made in the next time step, but if an obstacle T (tree in this case) exists at the position shown in the figure, the obstacle detection camera 102 detects that the obstacle T exists. is detected. Here, consider a situation in which the distance D from V1 to the obstacle T is less than a predetermined threshold (it cannot get any closer). In this case, priority is given to maintaining a certain distance from the obstacle T rather than setting a route along the electric wire. Therefore, consider a virtual spherical surface Q with a radius D centered on the obstacle T, and consider setting the following route on this spherical surface Q. There are several ways to set the next route. For example, as mentioned above, you can set the route so that it passes directly over the obstacle T, or you can set the route so that it goes around the obstacle T while maintaining altitude. Alternatively, these may be determined according to the detection result of the type of obstacle T. The figure shows a route setting that goes around an obstacle T. As described above, the next route W2 can be determined. By repeating this, it is possible to sequentially set the routes after W3.

When avoiding an obstacle by going around it, the UAV 10 moves around the obstacle by sliding while maintaining the front direction. Therefore, the obstacle detection camera 102 continues to take pictures in the same direction. Then, when the obstacle moves out of the photographing range of the obstacle detection camera 102, that is, when the obstacle goes out of the frame, the UAV 10 finishes detouring around the obstacle and returns to the route for tracking the electric wire.

Next, the operation when an object to be inspected is detected along the route will be described using FIG. 5. First, it is assumed that the UAV 10 is present in the WP 1 as an initial state. Next, when the UAV 10 flies along the initially set route L1, the assumed position after the minute time Δt has elapsed is defined as U1. In this case, the UAV 10 attempts to fly along the route L11 from WP1 to U1. Here, it is assumed that the inspection camera 101 is attached at right angles to the traveling direction of the UAV 10. Then, when photographing the electric wire with the inspection camera 101, the electric wire is photographed diagonally by the angle formed by the route L1 and the electric wire P. The angle at which the image is photographed can be estimated, for example, in the process of recognition processing in the inspection target detection unit 1031. Therefore, it is possible to correct the original flight path L11 to the flight path L21 so that the flight path of the UAV 10 becomes parallel to the electric wire P. As a result, the destination of the UAV 10 is corrected from U1 to V1.

Similar corrections should be made in the next time step, but if there is an inspection target O (in this case, a utility pole that is a relay target and the power distribution equipment attached to the utility pole) at the position in the diagram, the inspection The presence of the inspection object O is detected by the camera 101. Here, consider a situation in which the distance D from V1 to the object to be inspected O is less than a predetermined threshold. In this case, a circumferential trajectory X with a radius D that passes through the current position V1 and is centered on the object O to be inspected is defined, and the UAV 10 is controlled to fly along this circumferential trajectory X in the operation mode. is switched (inspection mode). The current position V1 is stored as the starting point of the circular orbit X. Thereafter, points X2, X3, X4, X5, etc. on the circumferential orbit X are sequentially determined based on the position information received from the GPS receiving unit 104, and the UAV 10 is controlled to pass through these points. fly Eventually, the UAV 10 goes around the circumferential orbit By estimating the direction of the wire from the image obtained by the inspection camera 101, the robot is controlled to fly along the wire.

When orbiting the inspection target, the UAV 10 performs attitude control so that the inspection target enters the photographing range of the inspection camera 101. Therefore, the obstacle detection camera 102 photographs the direction in which the UAV 10 is traveling at that time (circumferential direction of the orbit).

Further, as a combination of these, when an obstacle T is discovered while flying on the orbit X in the inspection mode, priority is given to avoiding the obstacle T. A specific example is shown in FIG. FIG. 6 shows the operation when an obstacle T (in this case, a tree) is discovered while flying in orbit X to inspect the object to be inspected O (in this case, a utility pole). In this case, the orbit X is inside the spherical surface Q representing the closest distance to a predetermined obstacle, and there is a possibility of collision with the obstacle T. Therefore, in such a case, a new orbit X' is defined along the spherical surface Q, and the route of the UAV 10 is changed so that it flies along X'. When it is determined that the distance from the obstacle T is sufficient (when it is determined that the UAV 10 will not enter the inside of the spherical surface Q), the UAV 10 is controlled to fly on the orbit X again. By performing such control, it becomes possible to acquire image data for use in inspection while circling around the inspection target while avoiding obstacles.

Although the present embodiment has been described above using FIGS. 1 to 6, the present embodiment is not limited to the above configuration. For example, although the inspection camera 101 and the obstacle detection camera 102 have been described as separate cameras, they may be configured by a single camera that has both functions. This provides benefits such as reducing the total weight of the UAV 10 and extending flight time.

Furthermore, the inspection camera 101 may be a camera (for example, a stereo camera, a TOF camera, etc.) that itself has a distance measurement function. As a result, information on the distance to the wire and the relative angle to the wire can be determined more accurately from the wire image obtained by the inspection camera 101, and the advantage is that the accuracy of the wire tracking operation can be improved. There is.

The inspection camera 101 may also include a distance sensor (for example, an ultrasonic sensor, a radar, etc.). As a result, information on the distance to the wire and the relative angle to the wire can be determined more accurately from the wire image obtained by the inspection camera 101, and the advantage is that the accuracy of the wire tracking operation can be improved. There is.

Further, although the obstacle detection camera 102 has been described as a stereo camera in the embodiment, distance information is obtained by other methods, such as a TOF camera, or a combination of a camera and another distance sensor (such as an ultrasonic sensor or radar). It may be a configuration, or it may be a configuration including a plurality of these. This has the advantage that it is possible to detect obstacles not only in the direction in which the UAV 10 is traveling, but also above, below, and behind the UAV 10, thereby further reducing the possibility of a collision of the UAV 10.

The obstacle detection camera 102 also uses a method of obtaining distance information through video analysis, such as a distance estimation method using deep learning, or a method of estimating the distance to surrounding objects from changes in multiple images taken at different times. A method such as Visual SLAM or SfM (Structure from Motion) may be used. As a result, the stereo camera can be replaced with a monocular camera, and there are advantages such as, for example, it becomes possible to employ an ultra-wide-angle camera and detect surrounding obstacles over a wide range.

Moreover, although the obstacle detection camera 102 has been described as a single stereo camera, the obstacle detection camera 102 may be a combination of multiple cameras or sensors. For example, a configuration may be adopted in which a plurality of stereo cameras are arranged and the entire periphery of the UAV 10 is photographed. Alternatively, a method may be used in which images are taken in a specific direction with a camera and sensors are used to explore in other directions. This has the advantage that it is possible to detect obstacles not only in the direction in which the UAV 10 is traveling, but also above, below, and behind the UAV 10, thereby further reducing the possibility of a collision of the UAV 10.

Furthermore, although the recording unit 107 has been described as being attached to the UAV 10, part or all of the recording unit 107 exists in a different location (for example, on the ground) from the UAV 10, and they are connected by, for example, wireless communication. It may be configured as follows. As a result, restrictions regarding the capacity of the recording unit 107 are relaxed, or by making it possible to rewrite from another device (for example, a PC) not shown, it is possible to dynamically change or add routes. This has the advantage of improving the flexibility of system operation.

Further, although the UAV 10 has been described as having a configuration including a GPS receiving unit 104 and an IMU 105, some or all of these functions may be replaced by information obtained by image analysis of the inspection camera 101 and the obstacle detection camera 102. It may be a configuration. For example, part or all of the self-position estimation using GPS may be replaced or combined with another self-position estimation method such as Visual SLAM. This makes it possible to reduce positional errors in tracking and orbiting operations even when the radio wave conditions are poor, such as when conducting inspections in areas with poor visibility, thereby reducing the potential for dangerous situations such as collisions or crashes of the UAV10. This has the advantage of being able to reduce

Furthermore, although the inspection target detection section 1031 and the obstacle detection section 1032 in the signal processing section 103 have been described as separate components, they may be a single component. In this case, for example, a single arithmetic unit may be used in a time-sharing manner to perform both the inspection target detection process and the obstacle detection process. This allows the number of component points to be reduced, which has the advantage of reducing the weight of the UAV 10 and extending the flight time.

Further, in the above embodiment, an example was shown in which there is an inspection mode in which the aircraft flies around the inspection target, but the inspection mode is not essential, and a configuration having only a tracking mode may be used. This makes it possible to reduce the number of combination patterns with obstacle avoidance operations, speed up calculations, stabilize flight by reducing the time constant of feedback control, and further simplify the calculation device to improve UAV10 performance. It has the advantage of being lightweight.

This embodiment requires precise self-position estimation and route setting by recognizing the inspection target and surrounding obstacles, calculating the relative position with the inspection target, and controlling to maintain a predetermined positional relationship. By simply giving a rough waypoint instruction, you can avoid obstacles on the route and continue photographing the inspection target without going out of frame.

In the first embodiment, a configuration in which the photographing range of the inspection camera is fixed with respect to the UAV is illustrated. In the second embodiment, a configuration and operation that enable changing the photographing range of the inspection camera will be described using FIGS. 7 and 8.

FIG. 7 is a configuration diagram showing the configuration of a UAV according to the second embodiment. The UAV 20 shown in FIG. 7 is a flying object used for inspection, and includes an inspection camera 201, an obstacle detection camera 102, a signal processing unit 103, a GPS receiving unit 104, an IMU 105, a control unit 106, a recording unit 107, and a cloud It has a stand 208 and a zoom control section 209.

The inspection camera 201 is attached to the pan head 208 and is a camera that obtains an external image of the inspection target. The acquired external appearance image is sent to the subsequent signal processing unit 103.

The control unit 206 performs a reading operation on the recording unit 107, reads out predetermined route information, collates the current position information input from the GPS reception unit 104, and determines the course of the UAV 20 in the next time step. Additionally, the external image and distance information inputted from the signal processing unit 103, the latitude, longitude, and altitude information inputted from the GPS reception unit 104, the signal information of the gyro, accelerometer, and compass inputted from the IMU 105, and the recording unit described later. Based on the route information input from 107, the UAV 20 moves in a desired direction by performing attitude control to stabilize the UAV 20 and at the same time controlling the balance of voltages applied to multiple thrust generation motors (not shown). control so that Further, the operation of the pan head 208 and the zoom control unit 209 is controlled based on the external image and distance information input from the signal processing unit 103.

The pan head 208 is a pedestal that is attached to the UAV 20 and on which the inspection camera 201 is mounted. The direction (pan angle, tilt angle) of the inspection camera 201 is set under control from the control unit 106.

The zoom control unit 209 is attached to the lens portion of the inspection camera 201, and sets the zoom magnification of the inspection camera 201 under control from the control unit 206.

The operation of this embodiment will be explained below.
This embodiment shows a method of inspecting electric wires, utility poles, and power distribution equipment attached to these using a UAV. However, the inspection camera 201 is not attached to the UAV 20 at a fixed angle, but the inspection camera 201 is attached on the pan head 208, and a zoom control unit 209 is attached to the lens part of the inspection camera. This embodiment differs from the first embodiment in that the pan head 208 and the zoom control section 209 are connected so that they can be controlled by the control section 206.

The flight device, the flight management method, and the program of this embodiment are designed to avoid obstacles detected by the obstacle detection camera 102 while being inspected by the inspection camera 201 under the control of the control unit 206 included in the UAV 20. The purpose is to photograph certain electric lines, utility poles, and associated power distribution equipment, and obtain images that can be used for inspection purposes. During flight, it is characterized by detecting electric wires and flying along them.

In this example, there are two operation modes: an operation mode (tracking mode) in which the vehicle flies along the electric wire until a utility pole or distribution equipment is detected, and an operation mode (inspection mode) in which the utility pole or distribution equipment is inspected with priority. exists. In the inspection mode, the aircraft is controlled to, for example, fly around the inspection target along a orbit created separately for inspection, and obtain an image of the entire surroundings of the inspection target.

The method of equipment inspection in this embodiment will be explained using the flowchart of FIG. 8. The processing procedure of FIG. 8 starts when the UAV 20 takes off and first detects a power line.

First, the control unit 206 receives signals from the gyro, accelerometer, and compass from the IMU 105 (step S201), and calculates voltages to be applied to the thrust generation motors necessary for the UAV 20 to hover through attitude control processing ( Step S202). Next, the control unit 206 receives latitude, longitude, and altitude information from the GPS receiving unit 104 (step S203), and calculates the current position of the aircraft through self-positioning processing (step S204). Here, self-positioning refers to processing that removes error factors and uncertainties inherent in GPS signals by appropriately filtering the obtained raw latitude, longitude, and altitude information using a Kalman filter, etc. Point.

Next, the control unit 206 performs a branching process to determine what the current operation mode is (step S205), thereby causing a branch depending on the operation mode. When the operation mode is the tracking mode, the control unit 206 reads out the route information recorded in the recording unit 107 and compares it with the self-position calculated in step S204 to determine the next direction to proceed. (step S206).

When the operation mode is the inspection mode, the control unit 206 compares this orbit with the self-position calculated in step S204, since the orbit around the object to be inspected has been set as described above. By doing so, the next direction to proceed is determined (step S207).

Next, obstacles in the course direction determined in step S206 or step S207 described above are detected (step S208). The presence or absence of an obstacle is determined by the obstacle detection unit 1032 analyzing an external image of the obstacle and distance information acquired by the obstacle detection camera 102. Here, the optical axis direction of the obstacle detection camera 102 is attached to face the front direction of the UAV 20 (controlled to match the direction of travel), and does not have an angle adjustment function. Therefore, for example, by setting the angle of view of the obstacle detection camera 102 to a wide angle, it is possible to detect all obstacles in the traveling direction and in a range deviated from the traveling direction. Alternatively, the configuration may be such that the angle of the obstacle detection camera 102 is adjusted by the control unit 206. The control unit 206 calculates the position of the obstacle with respect to the traveling direction by understanding the relative relationship between the traveling direction of the UAV 20 and the optical axis direction of the obstacle detection camera 102. be able to.

Next, the control unit 206 corrects the course depending on the presence or absence of an obstacle and the distance to the obstacle (step S209). Specifically, first, if the distance to the obstacle is longer than a predetermined threshold, no special processing is performed in this step. If the distance to the obstacle is less than a predetermined threshold, the route is corrected to avoid the obstacle. The correction method may be, for example, a method of correcting the course so as to raise the UAV 20 so as to straddle the obstacle, or a method of detecting the shape and type of the obstacle in the obstacle detection unit 1032 and detecting the shape and type of the obstacle. It is also possible to modify the course according to the situation (for example, if the obstacle is a tree, the vehicle should straddle it, or if it is on a pole such as a street lamp, it should be dodged horizontally).

Next, a branch process is performed to determine what the current operation mode is (step S210), and a branch is generated depending on the operation mode. If the operation mode is tracking mode, electric wire detection processing is executed (step S211). The electric wire detection process is realized by processing either or both of the image data input from the inspection camera 201 and the obstacle detection camera 102 in the inspection target detection unit 1031. The detection means may be, for example, edge detection, a straight line detection algorithm such as Hough transform, or a method using machine learning such as deep learning. By detecting the electric wire, it is determined at which position and at what angle the electric wire appears in the image.

Based on this information, the control unit 206 adjusts the setting values of the pan head 208 and the zoom control unit 209 (step S212). For example, when it is determined that the electric wire is visible in the upper half of the image, the tilt angle of the pan head 208 is adjusted so that the angle of the camera is directed upward. In addition, if the electric wire is viewed obliquely, the camera and the electric wire are set to face each other by adjusting the panning angle of the pan head 208.

Next, inspection target object detection processing is executed (step S213). The inspection target object detection process is a process of detecting inspection targets other than electric wires, and the inspection target detection unit 1031 uses either or both of the image data input from the inspection camera 201 and the obstacle detection camera 102. This is realized by processing at. The detection means may be, for example, template matching, a cascade classifier in which weak classifiers are arranged in series, or a method using machine learning such as deep learning.

Next, using the result of the inspection object detection process, branch processing (step S214) is executed depending on the presence or absence of the inspection object. If it is determined in the inspection object detection processing that there is no inspection object, no special processing is performed and the process returns to step S201. If it is determined in the inspection object detection process that an inspection object exists, its position and size in the image are calculated. Based on its position and size, it can be estimated in which direction and at what distance the inspection object detected in the inspection object detection process is located relative to the UAV 20. If it is determined that the target object is closer than a predetermined threshold, a orbit (inspection route) around the target object is set based on this information. Next, the current position is stored as the starting point of the inspection route (step S216). The storage destination may be the recording unit 107 or a temporary storage medium such as a RAM (not shown). Next, the inspection target detection unit 1031 determines the size and position of the inspection target, and the pan head 208 adjusts the pan and tilt angles of the camera so that the inspection target is large in the image and appears near the center of the image. At the same time, the zoom magnification is controlled by the zoom control unit 209 (step S220). Thereafter, the operation mode is changed from the tracking mode to the inspection mode (step S217), and the process returns to step S201.

On the other hand, in the branching process in step S210 described above, if the operation mode is the inspection mode, it is determined that the current position matches or is sufficiently close to the starting point of the inspection route stored in step S216 (step S218; YES). ), the operation mode is changed from the tracking mode to the inspection mode (step S219), and the process returns to step S201. If it is not the starting point of the inspection route (step S218; NO), the inspection target detection unit 1031 determines the size and position of the inspection target, and makes sure that the inspection target is large in the image and appears near the center of the image. The pan head 208 controls the pan angle and tilt angle of the camera, and the zoom control unit 209 simultaneously controls the zoom magnification (step S221). After that, the process returns to step S201.

Through the series of operations described above, it is possible to realize the operation in which the UAV 20 flies along the electric wire and autonomously switches between an inspection route that goes around inspected objects other than the electric wire and a normal route that flies along the electric wire. It is possible. In particular, even if the route information stored in the recording unit 107 does not necessarily follow electric wires or other objects to be inspected, it can be traced along the object by analyzing the images obtained by the inspection camera 201. The major advantage of this embodiment is that it is possible to fly the robot in a controlled manner, and that an enlarged image of the object to be inspected can be stably obtained through the interlocking operation of the pan head 208 and the zoom control unit 209.

Although this embodiment has been described above using FIGS. 7 to 8, this embodiment is not limited to the above configuration. For example, although the inspection camera 201 and the obstacle detection camera 102 have been described as separate cameras, they may be configured by a single camera that has both functions. This provides benefits such as reducing the total weight of the UAV 20 and extending the flight time.

Furthermore, the inspection camera 201 may be a camera (for example, a stereo camera, a TOF camera, etc.) that itself has a distance measurement function. This has the advantage that information on the distance to the wire and the relative angle to the wire can be determined more accurately from the wire image obtained by the inspection camera 201, and the accuracy of the wire tracking operation can be improved. There is.

In addition, the inspection camera 201 may also include a distance sensor (for example, an ultrasonic sensor, a radar, etc.). This has the advantage that information on the distance to the wire and the relative angle to the wire can be determined more accurately from the wire image obtained by the inspection camera 201, and the accuracy of the wire tracking operation can be improved. There is.

Further, although the obstacle detection camera 102 has been described as a stereo camera in the embodiment, distance information is obtained by other methods, such as a TOF camera, or a combination of a camera and another distance sensor (such as an ultrasonic sensor or radar). It may be a configuration, or it may be a configuration including a plurality of these. This has the advantage that it is possible to detect obstacles not only in the direction in which the UAV 20 is traveling, but also above, below, and behind the UAV 20, thereby further reducing the possibility of a collision between the UAV 20.

The obstacle detection camera 102 also uses a method of obtaining distance information through video analysis, such as a distance estimation method using deep learning, or a method of estimating the distance to surrounding objects from changes in multiple images taken at different times. A method such as Visual SLAM or SfM (Structure from Motion) may be used. As a result, the stereo camera can be replaced with a monocular camera, and there are advantages such as, for example, it becomes possible to employ an ultra-wide-angle camera and detect surrounding obstacles over a wide range.

Moreover, although the obstacle detection camera 102 has been described as a single stereo camera, the obstacle detection camera 102 may be a combination of multiple cameras or sensors. For example, a configuration may be adopted in which a plurality of stereo cameras are arranged and the entire periphery of the UAV 20 is photographed. Alternatively, a method may be used in which images are taken in a specific direction with a camera and sensors are used to explore in other directions. This has the advantage that it is possible to detect obstacles not only in the direction in which the UAV 20 is traveling, but also above, below, and behind the UAV 20, thereby further reducing the possibility of a collision between the UAV 20.

Furthermore, although the recording unit 107 has been described as being attached to the UAV 20, part or all of the recording unit 107 exists in a different location (for example, on the ground) from the UAV 20, and they are connected by, for example, wireless communication. It may be configured as follows. As a result, restrictions regarding the capacity of the recording unit 107 are relaxed, or by making it possible to rewrite from another device (for example, a PC) not shown, it is possible to dynamically change or add routes. This has the advantage of improving the flexibility of system operation.

Further, although the UAV 20 has been described as having a configuration including a GPS receiving unit 104 and an IMU 105, some or all of these functions may be replaced with information obtained by image analysis of the inspection camera 201 and the obstacle detection camera 102. It may be a configuration. For example, part or all of the self-position estimation using GPS may be replaced or combined with another self-position estimation method such as Visual SLAM. This makes it possible to reduce positional errors in tracking and orbiting operations even when the radio wave conditions are poor, such as when conducting inspections in areas with poor visibility, thereby reducing the potential for dangerous situations such as collisions or crashes of the UAV20. This has the advantage of being able to reduce

Furthermore, although the inspection target detection section 1031 and the obstacle detection section 1032 in the signal processing section 103 have been described as separate components, they may be a single component. In this case, for example, a single arithmetic unit may be used in a time-sharing manner to perform both the inspection target detection process and the obstacle detection process. This allows the number of component points to be reduced, which has the advantage of reducing the weight of the UAV 20 and extending the flight time.

Further, in the above embodiment, an example was shown in which there is an inspection mode in which the aircraft flies around the inspection target, but the inspection mode is not essential, and a configuration having only a tracking mode may be used. As a result, it is possible to reduce the combination patterns with obstacle avoidance operations, speed up calculations, stabilize flight by reducing the time constant of feedback control, and further simplify the calculation device to improve UAV 20. It has the advantage of being lightweight.

Further, in this embodiment, when detecting and photographing electric wires and objects to be inspected, the pan head 208 and the zoom control unit 209 are controlled, and the route of the UAV 20 is not corrected based on the detection results of the electric wires and objects to be inspected. However, the present invention is not limited to this example. For example, the idea of route correction shown in the first embodiment and the idea of adjusting pan/tilt/zoom shown in the second embodiment are not exclusive, and a system that realizes both may be used. As a result, it is possible to configure a system that combines the advantages of this embodiment, in that an enlarged image of the object to be inspected can be obtained, and the ability to set the attitude of the UAV 20 at an angle suitable for photographing by changing the route. can.

This embodiment requires precise self-position estimation and route setting by recognizing the inspection target and surrounding obstacles, calculating the relative position with the inspection target, and controlling to maintain a predetermined positional relationship. By simply giving a rough waypoint instruction, you can avoid obstacles on the route and continue photographing the inspection target without going out of frame.

As explained above, it is possible to avoid obstacles on the route and continue photographing the inspection target without going out of frame by simply giving rough waypoint instructions without requiring precise self-position estimation or route setting. I can do it.

Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations.

For example, in each of the above embodiments, the configuration in which route information as flight route information is stored in the recording unit 107 was explained as an example, but the flight route information is stored from a control device such as a server that is separate from the flight device. It may also be transmitted by communication means.

Further, in each of the above embodiments, an electric wire is exemplified as an extending object, and a utility pole is exemplified as an inspection object (relay object) having a relay function for the extending object. It is not limited to this. For example, it is applicable to train overhead lines, monorails, cable cars, cableways, etc. Here, the cableway includes a ropeway, a gondola lift, a ski resort lift, and the like.

In addition, in the above embodiment, the case where the extended object is used for tracking and orbits around the relay target is illustrated, but for example, if an abnormality is detected in the extended object, the location where the abnormality was detected is It may revolve around the extended object as the center.

In addition to inspecting extended objects and relay objects, it can also be used to inspect waypoints. That is, by recording the difference between the direction of the waypoint and the flight direction while flying along the extended object, it is determined whether the position of the waypoint is appropriate.

In addition, each of the above configurations may be partially or entirely configured by hardware, or may be configured to be realized by executing a program on a processor. Further, the control lines and information lines are shown to be necessary for explanation purposes, and not all control lines and information lines are necessarily shown in the product. In reality, almost all components may be considered to be interconnected.

10...UAV
101...Inspection camera 102...Obstacle detection camera 103...Signal processing section 1031...Inspection target detection section 1032...Obstacle detection section 104 ...GPS receiving section 105 ...IMU
106... Control unit 107... Recording unit 20... UAV
201... Inspection camera 206... Control unit 208... Pan head 209... Zoom control unit

Claims (14)

A flight device that photographs a predetermined inspection target in the air,
one or more cameras that photograph the surroundings of the flight device itself;
a first image processing unit that analyzes an image from the camera to detect the inspection target;
a second image processing unit that analyzes images from the camera to detect obstacles that impede flight;
a control unit that dynamically determines a flight path according to specified flight path information, the detection result of the inspection target object, and the detection result of the obstacle, and controls the flight of the flight device along the determined flight path; A flight device comprising: The flight device according to claim 1, wherein the control unit sets a flight path along an extended object that is an extended inspection object. The flight device according to claim 2, wherein the control unit uses an electric wire as the extending object. 2. The control unit, when detecting a relay target that is an inspection target having a relay function for the extended target, sets a flight path that goes around the relay target. The flight device according to 2. 3. The flight device according to claim 2, wherein the control unit sets a flight route that approaches a point designated by the flight route information while maintaining a distance from the extended object. The flight device according to claim 1, wherein the control unit sets the flight path so as to maintain a predetermined distance from the obstacle. The flight device according to claim 1, wherein the control unit gives priority to setting a flight path related to the obstacle over setting a flight path related to the inspection object. a first camera capable of photographing a direction perpendicular to the traveling direction of the flight device;
a second camera that captures at least an image of the traveling direction of the flight device,
The first image processing unit analyzes the image from the first camera to detect the inspection target,
The flight device according to claim 1, wherein the second image processing unit detects the obstacle by analyzing an image from the second camera. 9. The flight device according to claim 8, wherein at least one of the first camera and the second camera has a function of measuring a distance to an object. 2. The flight device according to claim 1, wherein the flight route information is transmitted by a communication means from a control device remote from the flight device. The flight device according to claim 1, wherein the control unit acquires position information used for flight control using a GPS (Global Positioning System). The flight device according to claim 1, wherein the control unit calculates the position information used for flight control by analyzing images from the one or more cameras. A flight control method for controlling a flight device that photographs a predetermined inspection target in the air, the method comprising:
a first image processing step of detecting the object to be inspected by analyzing images from one or more cameras included in the flight device;
a second image processing step of analyzing images from the one or more cameras included in the flight device to detect obstacles that impede flight;
Flight control that dynamically determines a flight path according to specified flight path information, the detection result of the inspection object, and the detection result of the obstacle, and controls the flight of the flight device along the determined flight path. A flight control method characterized by comprising steps. A program that causes a computer to control a flight device that photographs a predetermined inspection target in the air,
a first image processing process that analyzes images from one or more cameras included in the flight device to detect the inspection target;
a second image processing process that analyzes images from the one or more cameras included in the flight device to detect obstacles that impede flight;
Flight control that dynamically determines a flight path according to specified flight path information, the detection result of the inspection object, and the detection result of the obstacle, and controls the flight of the flight device along the determined flight path. A program that causes a computer to execute a process.

Images