JP7184381B2 - Unmanned aerial vehicle, flight control device for unmanned aerial vehicle, flight control method for unmanned aerial vehicle, and program - Google Patents

Description

Application of Article 30(2) of the Patent Act July 6, 2017 Publication with explanation [Publication, etc.] July 28, 2017 Publication with demonstration test [Publication, etc.] August 10, 2017 Publication with explanation [Publications, etc.] August 17, 2017 Disclosure by explanation [Publications, etc.] August 21, 2017 Disclosure by demonstration test [Publications, etc.] September 28, 2017 Disclosure by sales [Publications, etc.] ] September 28, 2017 Disclosure through sales [Publications, etc.] October 25, 2017 Disclosure through sales [Publications, etc.] October 27, 2017 Disclosure through demonstration tests

The present invention relates to an unmanned aerial vehicle, a flight control device for an unmanned aerial vehicle, a flight control method for an unmanned aerial vehicle, and a program. More particularly, the present invention relates to flight control devices, flight control methods, etc. for controlling the distance between an unmanned aerial vehicle and a target element.

In recent years, unmanned aerial vehicles that control flight by controlling the rotational speed of multiple rotor blades have been distributed in the market, and are widely used in industrial applications such as photographing surveys, pesticide spraying, material transportation, and hobby applications. .

In one example, an unmanned aerial vehicle flies according to an external input signal input from an external input device such as a proportional controller (propo). Even if there is a risk of collision due to approaching the vehicle, it may not be possible to recognize this and avoid the collision. Even when an unmanned aerial vehicle flies along a preset flight plan route by executing an autonomous control program with a flight controller, there are obstacles that were not taken into consideration when creating the flight plan route. In some cases, it may not be possible to avoid collision of the aircraft against this.

JP 2012-198077 A

Toshiba Corporation, "Development of imaging technology that can simultaneously acquire a color image and a depth image from a single image taken with a monocular camera", [online], Toshiba Corporation Research and Development Center, [searched on October 16, 2017 ], Internet <URL: https://www.toshiba.co.jp/rdc/detail/1606_01.htm> Vinay R., “What is OpenCV?”, [online], Intel, [Searched on October 23, 2017], Internet <URL: https://software.intel.com/en-us/articles/what- is-opencv> Andrew J.Davison, et al., “MonoSLAM: Real-Time Single Camera SLAM”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.29, NO.6, JUNE 2007, [online], [October 2017 Searched on the 20th], Internet <URL: https://www.doc.ic.ac.uk/~ajd/Publications/davison_etal_pami2007.pdf> Georg Klein, “Parallel Tracking and Mapping for Small AR Workspaces - Source Code”, [online], Georg Klein Home Page, [searched on October 20, 2017], Internet <URL: http://www.robots.ox .ac.uk/~gk/PTAM/> Georg Klein, et al., “Parallel Tracking and Mapping for Small AR Workspaces”, Proc. International Symposium on Mixed and Augmented Reality (ISMAR'07, Nara), [online], [searched October 20, 2017], Internet <URL: http://www.robots.ox.ac.uk/~gk/publications/KleinMurray2007ISMAR.pdf> Shuichi Maki, 3 others, "Development of mobile 3D measurement system using laser distance sensor", [online], 2014 Institute of Image Information and Television Engineers Winter Conference, [searched on October 20, 2017], Internet< URL: https://www.jstage.jst.go.jp/article/itewac/2014/0/2014_4-13-1_/_pdf>

In view of this, it is an object of the present invention to provide a flight control device, a flight control method, etc. for measuring the distance between an airframe and a target element during flight and controlling the distance according to the measured value. .

In order to solve the above problems, the present invention provides a distance sensor that measures the distance between an unmanned aerial vehicle that flies under control using an external input signal and/or flight plan information generated in advance and a target element. and a distance sensor equipped with a camera for photographing and a measurement value determination circuit for determining a distance measurement value using the photographed image information, and a distance measurement value measured by the distance sensor, during flight and a control signal generating circuit for generating a control signal for controlling the distance between the unmanned aerial vehicle and a target element. Here, “according to” the measured value of the distance and “generating a control signal for controlling the distance” means that the control signal for controlling the distance is generated regardless of the measured value of the distance. does not mean that must be generated, in one example, a control signal for controlling the distance is generated only when the measured distance is out of a predetermined range.

The unmanned aerial vehicle may be an unmanned aerial vehicle that flies under control using at least an external input signal, and the external input signal may be a signal input in real time from an external input device during flight of the unmanned aerial vehicle. may be a signal obtained by changing an external input signal according to the distance measurement.

The unmanned aerial vehicle may be an unmanned aerial vehicle that flies under control using at least flight plan information, and the flight plan information may be flight plan information generated in advance before flight by a computer executing a program.

The measured value determination circuit may be integrated with the control signal generation circuit.

The control signal generation circuitry may be configured to generate a control signal to move the unmanned aerial vehicle away from the target element when the measured value is less than the first reference value. However, some additional condition may be imposed in addition to the condition "when the measured value is smaller than the first reference value" as the condition for "generating a control signal for moving away from the target element". Also, even if the condition "when the measured value is smaller than the first reference value" is not satisfied, it is not prohibited to "generate a control signal for moving away from the target element".

The control signal generation circuitry may be configured to generate a control signal to bring the unmanned aerial vehicle closer to the target element when the measured value is greater than a second reference value greater than or equal to the first reference value. However, in addition to the condition "when the measured value is greater than the second reference value equal to or greater than the first reference value" as a condition for "generating a control signal for approaching the target element", some additional condition may be imposed, and even if the condition "when the measured value is greater than the second reference value equal to or greater than the first reference value" is not satisfied, "the control signal for approaching the target element It is not prohibited to "generate

The first reference value and the second reference value may be equal.

The control signal generation circuit generates a control signal for moving the unmanned aerial vehicle away from the target element when the measured value is smaller than the first reference value and the measured value decreases over time, and the measured value is the first reference value. 2 reference value and the measured value increases over time, it may be configured to generate a control signal to bring the unmanned aerial vehicle closer to the target element. However, the condition for "generating a control signal to move the unmanned aerial vehicle away from the target element" and the condition for "generating a control signal for moving the unmanned aerial vehicle closer to the target element" are, respectively, that "the measured value is the first In addition to the condition "when the measured value is smaller than the reference value of 1 and the measured value decreases over time", and the condition "when the measured value is greater than the second reference value and the measured value increases over time" may impose some additional conditions, and the condition "when the measured value is smaller than the first reference value and the measured value decreases over time", "the measured value is less than the second reference value and the measurement value increases over time”, even if the conditions are not met, “generate a control signal to move the unmanned aerial vehicle away from the target element”, “unmanned aerial vehicle Generating a control signal for approaching the target element is not prohibited.

The flight control device may further include an external environment imaging camera that captures images in a direction different from that of the imaging camera.

The flight controller may further comprise a relative position measurement sensor for measuring the relative position of the unmanned aerial vehicle with respect to elements present in the unmanned aerial vehicle's environment.

The target element may be an inspected structure.

The present invention also provides an unmanned aerial vehicle comprising the above flight control device.

In addition, the present invention captures an image of a target element, and uses the captured image information to determine the distance between an unmanned aircraft that flies under control using an external input signal and/or pre-generated flight plan information, and the target element. measuring a distance by determining a metric; and generating a control signal for controlling the distance between the unmanned aerial vehicle and a target element in flight in response to the distance metric. A flight control method for an aircraft is provided.

In addition, the present invention provides a method for determining the distance between an unmanned aerial vehicle that flies under control using an external input signal and/or flight plan information generated in advance using image information captured by a camera that captures a target element, and the target element. A program for causing the measured value determination circuit to determine the measured value and causing the control signal generation circuit to generate a control command value for controlling the distance between the unmanned aerial vehicle and the target element during flight according to the measured value of the distance. offer. The program is recorded in a computer-readable non-volatile (non-temporary) recording medium such as a hard disk, CD-ROM, arbitrary semiconductor memory (even if it is recorded in one recording medium, two or more It may be distributed and recorded on a recording medium.) It is also possible to provide as a program product.

By flying the unmanned aerial vehicle while controlling the distance between the unmanned aerial vehicle and the target element according to the present invention, it is possible to at least reduce the risk of the unmanned aerial vehicle colliding with a structure to be inspected, an obstacle, or the like during flight.

1 is a perspective view of an unmanned aerial vehicle that is an embodiment of the present invention; FIG. FIG. 2 is a diagram of the unmanned aerial vehicle of FIG. 1 viewed in the negative z direction; FIG. 2 is a block diagram showing the configuration of the unmanned aerial vehicle in FIG. 1; 4 is a flowchart for explaining distance measurement between an unmanned aerial vehicle and a structure to be inspected, and distance control according to the measured value. A diagram for explaining the principle of distance measurement by a stereo camera (cited from FIG. 1 of Japanese Patent Application Laid-Open No. 2012-198077, with only the definition of coordinate axes changed). FIG. 5 is a block diagram showing the configuration of a stereo camera and a measurement value determination circuit (cited from FIG. 5 of Japanese Patent Application Laid-Open No. 2012-198077, only the reference numerals are changed); FIG. 4 shows an unmanned aerial vehicle flying a distance d from a structure to be inspected; FIG. 4 is a diagram for explaining that control is performed to move the unmanned aerial vehicle away from the structure to be inspected when the distance d between the unmanned aerial vehicle and the structure to be inspected is smaller than a first reference value _D1 ; FIG. 5 is a diagram for explaining that control is performed to bring the unmanned aerial vehicle closer to the structure to be inspected when the distance d between the unmanned aerial vehicle and the structure to be inspected is greater than a second reference value _D2 ; To explain that when the first reference value _D1 and the second reference value _D2 are equal, control is performed to set the distance between the unmanned aerial vehicle and the structure under inspection to _D1 = _D2 . illustration. FIG. 4 illustrates that the flight of the unmanned aerial vehicle is substantially restricted within a two-dimensional plane when the distance between the unmanned aerial vehicle and the structure under inspection is controlled to D ₁ =D ₂ ; FIG. 4 illustrates that the flight of the unmanned aerial vehicle is substantially constrained to a one-dimensional line when the distance between the unmanned aerial vehicle and the inspected element is controlled to D ₁ =D ₂ ; FIG. 5 is a modified example of the flowchart of FIG. 4; The figure which looked at the prototype of the unmanned aerial vehicle which is one Embodiment of this invention from the positive direction of z. A photograph of a prototype of an unmanned aerial vehicle that is an embodiment of the present invention, viewed from a direction slightly oblique to the positive direction of z. FIG. 10A is a block diagram showing the configuration of the prototype shown in FIGS. 10A and 10B; FIG. 10B shows a distance setting knob on an external input device for steering the prototype of FIGS. 10A and 10B; 10A and 10B show take-off pads used for initial setup of the prototype of FIGS. 10A and 10B; FIG.

An unmanned aerial vehicle, a flight control device for an unmanned aerial vehicle, a flight control method for an unmanned aerial vehicle, and a program according to an embodiment of the present invention will be described below with reference to the drawings. However, the unmanned aerial vehicle, the flight control device for the unmanned aerial vehicle, the flight control method for the unmanned aerial vehicle, and the program according to the present invention are not limited to the specific embodiments described below, and can be appropriately modified within the scope of the present invention. Note that For example, the unmanned aerial vehicle according to the present invention may be a manual type or an autonomous flight type, or a semi-manual type unmanned aerial vehicle combining these. Any operation is possible as long as similar operations are possible. Alternatively, the operation to be performed by a single illustrated component may be performed by a plurality of components, such as distributing the function of a main processing circuit to a plurality of processing circuits. As an example, in FIG. 3, the measured value determination circuit is hardware separate from the control signal generation circuit (for example, a circuit consisting of a processor, a memory, etc., as a digital signal processing unit attached in advance to a commercially available stereo camera). , the main processing circuit performs such digital signal processing (a photographing camera is composed of two monocular cameras, and the image data photographed by each is output to the main processing circuit). The measurement determination circuit may be integrated with the control signal generation circuit by taking The autonomous control program of the unmanned aerial vehicle may be recorded in a recording device such as a hard disk drive, read out by the main computing circuit, and executed (the autonomous control program shown in the figure may be stored in a distance control module, etc. It may be decomposed into a plurality of program modules, or any other program may be executed by a main arithmetic circuit or the like.) Similar operations may be executed by an embedded system using a microcomputer or the like. . It is not necessary for an unmanned aerial vehicle or a flight control device for an unmanned aerial vehicle according to the present invention to include all of the components shown in the following embodiments, and all of the shown method steps or instructions for causing a processing device to execute them. need not be included in the unmanned aerial vehicle control method or program according to the present invention. The rotors for flying the unmanned aerial vehicle are not limited to six rotors R1 to R6 as shown in FIGS. any rotating blade, such as a propeller). The unmanned aerial vehicle may be any unmanned aerial vehicle, such as a single rotor helicopter or fixed wing aircraft. The body size of the unmanned aerial vehicle is also arbitrary.

Overview of configuration and flight control of unmanned aerial vehicle FIG. 1 shows a perspective view of an unmanned aerial vehicle that is an embodiment of the present invention, and FIG. omitted). The unmanned aerial vehicle 1 has a main body 2, six motors M1 to M6 (FIG. 2) driven by control signals from the main body 2, and the motors M1 to M6 are driven to rotate and fly the unmanned aerial vehicle 1. 6 rotors (rotary blades) R1 to R6, arms A1 to A6 (FIG. 2) connecting the main body 2 and the motors M1 to M6, respectively, a stereo camera 3 for photographing the front during flight, and a flight An external environment photographing camera 4 for photographing areas other than the front and a landing leg 5 contributing to prevention of overturning during takeoff and landing are provided. As shown in FIG. 1, a roll angle, a pitch angle, and a yaw angle are defined as rotation angles about the x-axis, the y-axis, and the z-axis. A throttle amount is defined as an amount corresponding to the ascent and descent of the airframe (the number of rotations of the rotors R1 to R6 as a whole). As shown in FIG. 2, rotors R1, R3 and R5 rotate clockwise when viewed in the negative z direction, and rotors R2, R4 and R6 rotate counterclockwise when viewed in the negative z direction. That is, adjacent rotors rotate in opposite directions. The six arms A1 to A6 have the same length and are arranged at intervals of 60° as shown in FIG. The unmanned aerial vehicle 1 may also be equipped with additional cameras, payloads, etc. (not shown) depending on the application.

3 is a block diagram showing the configuration of the unmanned aerial vehicle of FIG. 1. FIG. The main body 2 of the unmanned aerial vehicle 1 includes a main arithmetic circuit 7a, which is composed of a processor, a temporary memory, etc., and performs various arithmetic operations. A signal conversion circuit 7b composed of a processor, a temporary memory, etc., which performs processing such as conversion to signals (PWM, Pulse Width Modulation signals), and an arithmetic circuit including a main arithmetic circuit 7a and a signal conversion circuit 7b. circuit 8), speed controllers (ESC: Electric Speed Controllers) ESC1 to ESC6 that convert the pulse signals generated by the control signal generation circuit 8 into driving currents for the motors M1 to M6, and various data with the outside. A communication antenna 12 and a communication circuit 13 responsible for transmitting and receiving signals, a sensor unit 14 including various sensors such as a GPS (Global Positioning System) sensor, an attitude sensor, an altitude sensor, an azimuth sensor, and an autonomous control program 9a (distance control module 9b ), a recording device 10 composed of a recording device such as a hard disk drive for recording various databases 9c, etc., a battery device such as a lithium polymer battery or a lithium ion battery, and a power supply system 11 including a power distribution system to each element It has Also, the unmanned aerial vehicle 1 includes a stereo camera 3 and a measured value determination circuit 6 configured by a processor, a temporary memory, etc., which determines a measured value of distance by performing digital signal processing of image information captured by the stereo camera 3. and an external environment photographing camera 4 that photographs a direction different from that of the stereo camera 3 during flight and records it in its own memory (image information photographed by the external environment photographing camera 4 is recorded at any time by a recording device). 10).

In addition, the unmanned aerial vehicle 1 may be provided with arbitrary functional units, information, etc. according to the functional application.
As an example, when the unmanned aerial vehicle 1 autonomously flies according to the flight plan (autonomous flight mode), the flight start position, the target position, and the checkpoint positions ( latitude, longitude, altitude) Flight plan information (external computer generates flight plan information) By executing the program, information generated in advance using conditions, routes, etc. input by the user of the unmanned aerial vehicle 1 from an external interface) is recorded in the recording device 10, and a flight plan of the unmanned aerial vehicle 1 is recorded. The two-dimensional map or three-dimensional map information around the flight plan route included in is also recorded in the recording device 10, and the main arithmetic circuit 7a reads the flight plan information and executes the autonomous control program 9a to execute the unmanned aerial vehicle according to the flight plan 1 flies. Specifically, the current position, speed, etc. of the unmanned aerial vehicle 1 are determined from information obtained from various sensors of the sensor unit 14, and compared with target values such as the flight plan route, speed limit, altitude limit, etc. determined in the flight plan. As a result, the main computation circuit 7a computes control command values relating to the throttle amount, roll angle, pitch angle, and yaw angle, and the main computation circuit 7a converts these into control command values relating to the rotational speeds of the rotors R1 to R6, and outputs signals The signal conversion circuit 7b converts the data indicating the control command value related to the rotation speed into a pulse signal and transmits it to the speed controllers ESC1 to ESC6. The flight of the unmanned aerial vehicle 1 is controlled by controlling the rotation speeds of the rotors R1 to R6 by controlling the driving of the motors M1 to M6. As an example, in response to a control command to raise the altitude of the unmanned aerial vehicle 1, the number of revolutions of the rotors R1 to R6 is increased (reduced when the altitude is lowered), and the unmanned aerial vehicle 1 is moved forward (the positive direction of x in FIG. 1). ), control such as decreasing the rotation speed of the rotors R1 and R2 and increasing the rotation speed of the rotors R4 and R5 (reverse control for deceleration) is performed. Flight record information such as the flight path actually flown by the unmanned aerial vehicle 1 (body position of the unmanned aerial vehicle 1 at each time, etc.) and various sensor data are recorded in various databases 9c at any time during the flight.

The unmanned aerial vehicle 1 receives external input command values (throttle amount, roll angle , pitch angle, and yaw angle) (manual mode), the main arithmetic circuit 7a uses the external input command value to control the autonomous control program 9a (the unmanned aerial vehicle 1 as an aircraft dedicated to manual control by an external input device is configured, a separate control program recorded in the recording device 10) is executed to calculate the control command value for the rotational speed of the rotors R1 to R6, and this data is converted into a pulse signal by the signal conversion circuit 7b. After conversion, the speed controllers ESC1 to ESC6 and the motors M1 to M6 are used in the same way to control the rotational speeds of the rotors R1 to R6 to perform flight control.

Alternatively, when flying the unmanned aerial vehicle 1 in an attitude control mode (an example of a semi-manual mode) in which only the attitude of the aircraft is autonomously controlled, attitude information obtained by measuring the attitude sensor (gyro sensor, magnetic sensor, etc.) of the sensor unit 14 The main arithmetic circuit 7a executes the autonomous control program 9a using the data indicating the attitude control command values (roll angle, pitch angle, yaw command value for attitude control) and the external input command value (command value for throttle amount, roll angle, pitch angle, yaw angle) indicated by the external input signal received from the external input device. are combined to calculate (synthesized) control command values relating to the throttle amount, roll angle, pitch angle, and yaw angle, which are converted into control command values relating to the rotational speeds of the rotors R1 to R6 (the main computation circuit 7a autonomously Calculations and conversions are performed by executing the control program 9a), and the flight is controlled in the same way.

Examples of autonomous flying unmanned aerial vehicles include Mini Surveyor ACSL-PF1 (Autonomous Control Systems Laboratory Co., Ltd.), Snap (Vantage Robotics), AR. Drone 2.0 (Parrot), Bebop Drone (Parrot), etc. are commercially available. In the flight control of the unmanned aerial vehicle 1 described below, the unmanned aerial vehicle 1 basically flies according to external input signals from an external input device or the like, and only the attitude and the distance to the target element are autonomously controlled. Similarly, flight control including distance control is possible in the unmanned aerial vehicle 1 that performs fully autonomous control flight or fully external control flight.

Distance Measurement and Distance Control Based on Measured Values The unmanned aerial vehicle 1 in this embodiment measures the distance between the unmanned aerial vehicle 1 and a target element such as a structure to be inspected using a stereo camera 3 and a measured value determination circuit 6. The control signal generation circuit 8, which measures during flight and receives in real time a signal indicating the measured value of the distance from the measured value determination circuit 6, controls the distance in real time according to the measured value of the distance during flight. (the main arithmetic circuit 7a generates a control command value, and the signal conversion circuit 7b converts the control command value data into a control signal as a pulse signal) to perform distance control. FIG. 4 shows the distance measurement using the stereo camera 3 and the measured value determination circuit 6, followed by the control instruction value obtained by the main arithmetic circuit 7a executing the autonomous control program 9a including the distance control module 9b. 4 shows a flowchart of processing including generation processing of .

First, during the flight of the unmanned aerial vehicle 1, the stereo camera 3 photographs a target element (a structure to be inspected 15a shown in FIG. Using the image information captured simultaneously by C0 and C1 (see FIGS. 5 and 6 described later), the measured value d of the distance between the unmanned aerial vehicle 1 and the structure to be inspected 15a is determined (step S402). The measured value determination circuit 6 outputs a signal indicating the measured value d of the distance to the main arithmetic circuit 7a (step S403). Below, Patent Document 1 (Japanese Patent Application Laid-Open No. 2012-198077. Inventor: Shin Aoki, Title of Invention: "Stereo Camera Device, Parallax Image Generation Method", Applicant: Ricoh Co., Ltd., Application Number: Japanese Patent Application No. 2011-61729, (filed on Mar. 18, 2011), the principle of distance measurement and the configuration of the stereo camera 3 will be described.

Distance measurement by stereo camera FIG. 5 is a quote from FIG. 1 of Patent Document 1 (only the definition of coordinate axes is changed). The principle of distance measurement by the stereo camera 3 will be explained with reference to FIG.

"[0003]
FIG. 1 is a diagram for explaining the principle of distance measurement by stereo cameras arranged in parallel. Cameras C ₀ and C ₁ are placed a distance B apart. The focal lengths, optical centers, and imaging planes of the cameras C0 and C1 are as follows.
focal length: f,
Optical center: O ₀ , O ₁
Imaging plane: s ₀ , s ₁
An image of an object A located at a distance d in the optical axis direction from the optical center O ₀ of the camera C0 forms an image at P ₀ which is the intersection of the straight line AO ₀ and the imaging plane s ₀ . On the other hand, in the camera C1, the same subject A forms an image at the position _P1 on the imaging surface _s1 . Here, let P 0 ' be the intersection of a straight line passing through the optical center O ₁ of the camera C1 and parallel to the straight line AO ₀ and the imaging surface s ₁ , and let p be the distance between the points _{P 0 '} and P ₁ . do. "(citing paragraph [0003] of Patent Document 1)

"[0004]
P ₀ ' is the same position as the image P ₀ on the camera C0, and the distance p represents the amount of positional deviation between the images of the same subject on the images taken by the two cameras, which is called parallax. .
Triangle: AO ₀ -O ₁ and triangle O ₁ -P ₀ '-P ₁ are similar, so
d = Bf/p
is obtained. If the distance B (baseline length) between the cameras C0 and C1 and the focal length f are known, the distance d can be obtained from the parallax p. "(citing paragraph [0004] of Patent Document 1)

The principle of distance measurement by a stereo camera has been described above with reference to FIG. 1 and paragraphs [0003] to [0004] of Patent Document 1. In the following, as shown in FIG. 5, the distance d in the optical axis direction between the subject (target element) A and the optical centers O ₀ and O ₁ will be referred to as "the distance between the unmanned aerial vehicle 1 and the target element A". For example, the distance between the object (target element) A and the imaging planes s ₀ and s ₁ in the optical axis direction = d + f, such as defining the "distance between the unmanned aerial vehicle 1 and the target element A". is. The “distance d” in the following description is not limited to d defined by the above formula d=Bf/p, but may be any “distance” defined as such.

FIG. 6 is a block diagram showing the configuration of the stereo camera 3 and the measurement value determination circuit 6, which is quoted from FIG. 5 of Patent Document 1 with the reference numerals changed. Hereinafter, referring to paragraphs [0030] to [0036] of Patent Document 1 (reference numbers will be changed), their configurations will be described.

"[0030]
〔Constitution〕
FIG. 5 shows an example of a schematic configuration diagram of the stereo camera 3. As shown in FIG. A right camera C1 and a left camera C0 are arranged in the camera section 300 . The right camera C1 and the left camera C0 have the same lens and the same CMOS image sensor, and the right camera C1 and the left camera C0 are arranged such that their optical axes are parallel to each other and their imaging surfaces are on the same plane. It is The left camera C0 and the right camera C1 have the same lens 301, diaphragm 302, and CMOS image sensor 303. (Paragraph [0030] of Patent Document 1 is quoted. However, reference numerals have been changed.)

"[0031]
The CMOS image sensor 303 operates with a control signal output by the camera control unit 308 as an input. The CMOS image sensor 303 is assumed to be a monochrome image sensor of 1000×1000 pixels, and the lens 301 focuses an image of a field of view of 80 degrees on one side and 160 degrees on both sides in the imaging area of the CMOS image sensor 303 by equidistant projection method. It has the characteristic to image. (Paragraph [0031] of Patent Document 1 is quoted. However, reference numerals have been changed.)

"[0032]
Note that the lens characteristics are not limited to equidistant projection characteristics, and may be lenses that are used as fisheye lenses, such as equisolid angle projection or orthogonal projection, or lenses that have central projection characteristics with strong barrel distortion. As with the equidistant projection, both lenses have a smaller enlargement ratio in the periphery of the image than in the central projection, so that the same effects as in the present embodiment can be obtained. "(citing paragraph [0032] of Patent Document 1)

"[0033]
Furthermore, even when a lens having central projection characteristics with small distortion is used, the effect of the same tendency can be obtained by reducing the number of pixels of the deformed image. "(citing paragraph [0033] of Patent Document 1)

"[0034]
The image signal output from the CMOS image sensor 303 is output to the CDS 304 and subjected to noise removal by correlated double sampling. The image signal is stored in a frame memory 307 capable of storing the entire CMOS image sensor 303 . (Paragraph [0034] of Patent Document 1 is quoted. However, the reference numerals have been changed.)

"[0035]
The image signal stored in the frame memory 307 is subjected to distance calculation and the like by the digital signal processing unit 6, and depending on the specifications, the format is converted and displayed on display means such as a liquid crystal. The digital signal processing unit 6 is an LSI including a DSP, CPU, ROM, RAM, and the like. Functional blocks to be described later are provided, for example, by this digital signal processing unit 6 in terms of hardware or software. Note that the camera control unit 308 may be arranged in the digital signal processing unit 6, and the illustrated configuration is an example. (Paragraph [0035] of Patent Document 1 is quoted. However, the reference numerals have been changed.)

"[0036]
The digital signal processing unit 6 outputs each pulse of the horizontal synchronizing signal HD, the vertical synchronizing signal VD and the clock signal to the camera control unit 308 . Alternatively, the camera control section 308 can generate the horizontal synchronization signal HD and the vertical synchronization signal VD. A camera control unit 308 has a timing generator and a clock driver, and generates control signals for driving the CMOS image sensor 303 from HD, VD and clock signals. (Paragraph [0036] of Patent Document 1 is quoted. However, reference numerals have been changed.)

In the description of Patent Document 1 cited above, the camera control unit 308 may be referred to as a camera control circuit 308 hereinafter. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. CDS is an abbreviation for Correlated Double Sampling, and the CDS 304 is hereinafter referred to as the CDS circuit 304 . AGC is an abbreviation for Automatic Gain Control, and the AGC 305 is hereinafter referred to as the AGC circuit 305 . A/D is an abbreviation for Analog/Digital, and the A/D 306 is hereinafter referred to as the A/D converter 306 . DSP is an abbreviation for Digital Signal Processor. CPU is an abbreviation for Central Processing Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. LSI is an abbreviation for Large-Scale Integrated Circuit. In this embodiment, the digital signal processing unit (measurement value determination circuit) 6 calculates the distance measurement value by executing a program for determining the distance measurement value stored in the ROM by the CPU. . In one example, the measurement value determination circuit 6 determines a distance measurement value such as d=Bf/p for each pixel included in both images captured by the cameras C0 and C1, and calculates the distance measurement value for each pixel. A distance image having a color corresponding to the distance measurement value of the pixel is generated, and a measurement value corresponding to the distance between the target element and the unmanned aerial vehicle 1 obtained from the distance image data is output to the main arithmetic circuit 7a.

1, 5, and paragraphs [0003]-[0004], [0030]-[0036] of Patent Document 1, the principle of distance measurement, the configuration of the stereo camera 3 and the measurement value determination circuit 6 However, it is also possible to determine the distance between the subject (target element) and the unmanned aerial vehicle 1 using a photographing camera and a measurement value determination circuit other than the stereo camera 3 and the measurement value determination circuit 6. For example, using a monocular camera instead of the stereo camera 3, the target element is photographed twice at short time intervals, and the position of the monocular camera at each time of photographing (detected by the sensor unit 14 and measured via the main arithmetic circuit 7a input to the value determination circuit 6) is used in the same manner as the positions of the cameras C0 and C1 in FIG. In addition, Toshiba Corporation has developed an imaging technology capable of simultaneously acquiring a color image and a distance image from a single image taken with a monocular camera (Non-Patent Document 1), and this technology is used to measure distance. may Also, if the camera is equipped with a zoom lens, the measurement accuracy can be improved.

In one example, the signal indicating the measured value of the distance output from the measured value determination circuit 6 to the main arithmetic circuit 7a is It may be a signal indicating the shortest distance (in this case, the element with the shortest distance from the unmanned aerial vehicle 1 among the elements included in the photographed image is the "target element"), or the measurement value is determined. By the operation of the circuit 6, a specific element is detected by an arbitrary image processing algorithm, the distance between the element and the unmanned aerial vehicle 1 is determined by the measurement value determination circuit 6 based on the principle described above, and a signal indicating the distance is used as the main calculation. You may output to the circuit 7a. For example, the image processing function of Open CV (Open Source Computer Vision Library), which is an open source library published by Intel Corporation, can detect a specific object from a photographed image based on its contour (non-patent Reference 2). In this case, such an image processing program is previously installed in the memory of the measured value determination circuit 6, and the image information recorded in the frame memory 307 is executed by the processor of the measured value determination circuit 6. , and the distance between this element and the unmanned aerial vehicle 1 can be determined.

Distance Control According to Distance Measured Value When the distance measurement value d is determined as shown in steps S401 to S403 described above and a signal indicating this is output to the main computation circuit 7a, the main computation circuit 7a performs distance control. The processing after step S404 is performed by executing the autonomous control program 9a including the module 9b. Note that steps S401 to S403 are repeated at predetermined time intervals, and therefore the entire processing according to the processing flow of FIG. 4 and the subsequent control processing are also repeated at predetermined time intervals. It is not essential to determine the measured value d of the distance for each frame (for each frame) and output a signal indicating it to the main arithmetic circuit 7a. The processing of the entire flowchart may be performed. The same applies to the modification shown in FIG.

In this embodiment, the unmanned aerial vehicle 1 receives external input command values (throttle amount, roll angle, pitch angle, and yaw angle command values) input in real time during flight by an external input signal from a proportional controller, A (synthetic) control command in which posture control command values (command values relating to roll angle, pitch angle, and yaw angle) generated using data from the posture sensor by the arithmetic circuit 7a executing the autonomous control program 9a are combined. value (in one example, the throttle amount of the external input command value is used as the command value for the throttle amount, and the roll angle in the external input command value and the command value for attitude control is used as the command value for each of the roll angle, pitch angle, and yaw angle The command value obtained by adding the command values of , the command value obtained by adding the command values of the pitch angle, and the command value obtained by adding the command values of the yaw angle are used respectively. (Fig. 7A).

When receiving the input of the signal indicating the measured value d of the distance, the main arithmetic circuit 7a converts the measured value d to the first reference value D ₁ (preliminarily input from the outside, etc.) by executing the distance control module 9b. is recorded in the recording device 10, and is read out by the main arithmetic circuit 7a executing the distance control module 9b.The same applies to the second reference value _D2 .) (step S404). If the measured value d is smaller than the first reference value D ₁ (Yes), the unmanned aerial vehicle 1 is too close to the structure to be inspected 15a as shown in FIG. A control command value for moving away from is generated (step S405). In one example, in order to move the unmanned aerial vehicle 1 in the backward direction (in the direction opposite to the x direction in FIG. 1), there are (Synthetic) Update the amount related to the pitch angle in the control command value with the amount corresponding to rotating the fuselage in the direction of the arrow indicating the pitch angle in Fig. 1 (the front part of the fuselage rises and the rear part of the fuselage descends). By doing so, a control command value for moving the unmanned aerial vehicle 1 away from the structure to be inspected 15a is generated.

If the measured value d is not smaller than the first reference value _D1 in step S404 (No), the unmanned aerial vehicle 1 is not too close to the structure to be inspected 15a. The process proceeds to step S406. The main arithmetic circuit 7a compares the measured value d with the second reference value _D2 by executing the distance control module 9b (step S406). Here, the second reference value _D2 is a reference value greater than or equal to the first reference value _D1 . If the measured value d is greater than the second reference value D ₂ (Yes), the unmanned aerial vehicle 1 is too far from the structure to be inspected 15a as shown in FIG. 7C. is generated (step S407). In one example, in order to move the unmanned aerial vehicle 1 forward (in the x direction in FIG. 1), the throttle amount, roll angle, pitch angle, and yaw angle obtained by combining the external input command value and the attitude control command value (synthesis) Among the control command values, the amount related to the pitch angle is updated with an amount corresponding to rotating the aircraft in the opposite direction of the arrow indicating the pitch angle in FIG. 1 (the front portion of the aircraft descends and the rear portion rises). Thus, a control command value for bringing the unmanned aerial vehicle 1 closer to the inspected structure 15a is generated.

If the measured value d is not greater than the second reference value _D2 in step S406 (No), the unmanned aerial vehicle 1 is not too far from the structure to be inspected 15a, so the process of step S407 is not performed, and the process ends. The process proceeds to step S408. The main arithmetic circuit 7a generates control command values relating to the throttle amount, roll angle, pitch angle, and yaw angle as control command values obtained by combining the externally input command values and attitude control command values (step S408).

According to the processing flow of FIG. 4, control command values relating to the throttle amount, roll angle, pitch angle, and yaw angle are generated in any of steps S405, S407, and S408. Subsequently, the main arithmetic circuit 7a executes the autonomous control program 9a to convert these control command values into control command values relating to the rotational speeds of the rotors R1 to R6, and the signal conversion circuit 7b converts them into pulse signals. A control signal is generated, and each of the speed controllers ESC1 to ESC6 converts the pulse signal into a drive current and outputs it to each of the motors M1 to M6, thereby controlling the driving of the motors M1 to M6 and controlling the rotational speeds of the rotors R1 to R6. The flight of the unmanned aerial vehicle 1 is controlled by controlling the . As a result, the distance d between the unmanned aerial vehicle 1 and the structure to be inspected 15a is controlled within the range from the first reference value _D1 to the second reference value _D2 . Since the processing according to the flow of FIG. 4 and the subsequent control processing are repeated at predetermined time intervals, unless the unmanned aerial vehicle 1 falls within the range from the first reference value _D1 to the second reference value _D2 , The unmanned aerial vehicle 1 continues to be controlled so as to head within the range.

Here, when the first reference value _D1 and the second reference value _D2 are equal, the distance d between the unmanned aerial vehicle 1 and the structure to be inspected 15a is controlled toward a constant distance equal to the reference value. (Fig. 7D). In this case, the flight of the unmanned aerial vehicle 1 is controlled toward an equidistant plane 16a from the structure to be inspected 15a (FIG. 8A). is possible. When the target element is an inspected element 15b such as an electric wire instead of the inspected structure 15a, the flight of the unmanned aerial vehicle 1 is controlled by controlling the distance d between the unmanned aerial vehicle 1 and the inspected element 15b on the same principle. It is also possible to control toward the equidistant line 16b from the inspected element 15b to make it substantially one-dimensional (FIG. 8B).

Whether or not to perform the distance control as shown in FIG. 4 is determined by, for example, receiving a mode switching signal transmitted from the proportional controller by the communication antenna 12 and the communication circuit 13, and the main arithmetic circuit 7a receiving the mode switching signal. can be switched by executing the autonomous control program in response to the input of As a result, after the unmanned aerial vehicle 1 is brought close to the structure to be inspected 15a and controlled so that the stereo camera 3 faces the direction of the structure to be inspected 15a, the mode switching signal for turning on the distance control mode is proportionally・Transmit from the controller to the communication antenna 12, shift to the distance control mode, and perform inspection work (image information captured by the stereo camera 3 is transferred from the measurement value determination circuit 6 to the main arithmetic circuit 7a as a still image or moving image). , and transmitted to the operator's external computer via the communication circuit 13 and communication antenna 12 at any time, real-time inspection is possible. A camera separate from the stereo camera 3 may be provided in the unmanned aerial vehicle 1 as an inspection camera, and image information captured by the separate camera may be used in the same way. is completed, a mode switching signal for turning off the distance control mode is transmitted from the proportional controller to the communication antenna 12 to end the distance control mode, and control such as returning the unmanned aerial vehicle 1 becomes possible.

As already described, the unmanned aerial vehicle 1 flies under control based on a (combined) control command value obtained by combining an external input command value input in real time and an attitude control command value generated by executing the autonomous control program 9a. However, similar distance measurement and distance control are possible even when the unmanned aerial vehicle 1 flies according to the control using the flight plan information described above. The control flow is basically the same as the flow shown in FIG. Using the main arithmetic circuit 7a to execute the autonomous control program 9a, the unmanned aerial vehicle 1 is controlled to fly away from the structure to be inspected 15a, and is included in the flight plan information recorded in the recording device 10. The flight plan route is changed so as to detour away from the structure to be inspected 15a without passing through the position of the unmanned aerial vehicle 1 when step S405 was executed. Alternatively, for example, when a control command value for bringing the unmanned aerial vehicle 1 closer to the structure under inspection 15a is generated in step S407, the main arithmetic circuit 7a executes the autonomous control program 9a using this control command value. , the unmanned aerial vehicle 1 is controlled to fly in a direction approaching the structure to be inspected 15a, and the flight plan route included in the flight plan information recorded in the recording device 10 is recorded as the unmanned aerial vehicle 1 when step S407 is executed. Change to detour in the direction approaching the structure to be inspected without passing through the position of Since the processing according to the flow of FIG. 4 and the subsequent control processing are repeated at predetermined time intervals, unless the unmanned aerial vehicle 1 falls within the range from the first reference value _D1 to the second reference value _D2 , The unmanned aerial vehicle 1 will continue to be controlled to come within range, and the flight plan path will continue to change. When the unmanned aerial vehicle 1 does not receive an external input signal, it flies under control using flight plan information, and when it receives an external input signal, it temporarily performs manual control by prioritizing control based on the external input command value. Similar distance measurement and distance control are possible even when flying in a mode such that Alternatively, when the control command value for approaching the structure to be inspected 15a is generated, the flight plan route is detoured in the direction away from the structure to be inspected 15a or in the direction approaching the structure to be inspected 15a. Be changed. Similarly, the process according to the flow chart of FIG. 9, described below, is also operable for various flight modes of the unmanned aerial vehicle.

FIG. 9 shows a modification of the flowchart of FIG. 4 described above. The processing of steps S901 to S908 is the same as the processing of steps S401 to S408 in FIG. 4, and the description thereof will be omitted as appropriate. In the flowchart of FIG. 9, comparison processing in steps S909 and S910 is newly added.

In steps S901-S903, similarly to steps S401-S403 in FIG. The value determining circuit 6 determines the measured value d of the distance between the unmanned aerial vehicle 1 and the structure to be inspected 15a using the image information captured simultaneously by the left and right cameras C0 and C1 (see FIGS. 5 and 6) ( step S902). The measured value determination circuit 6 outputs a signal indicating the measured value d of the distance to the main arithmetic circuit 7a (step S903). 9 and subsequent control processing are repeated at predetermined time intervals in the same manner as the flow chart of FIG. Here, the main arithmetic circuit 7a associates the measured value d of the distance indicated by the input signal with the measurement time corresponding to the measured value d (the time at which the signal was input), and the measured value d and It continues to record in the recording device 10 as data of a set of corresponding measurement times.

When the main arithmetic circuit 7a receives the input of the signal indicating the measured value d of the distance, it compares the measured value d with the first reference value D ₁ by executing the distance control module 9b (step S904). If the measured value d is smaller than the first reference value D ₁ (Yes), the main arithmetic circuit 7a further executes the distance control module 9b so that the measured value d (latest measured value) and the previous The previous distance measurement value d ₀ indicated by the signal received from the measurement value determination circuit 6 is compared (S909). If the latest measured value d is smaller than the previous measured value d ₀ (Yes), the unmanned aerial vehicle 1 is too close to the inspected structure 15a and the measured distance value is decreasing over time. , a control command value for moving the unmanned aerial vehicle 1 away from the structure to be inspected 15a is generated (step S905). When the main arithmetic circuit 7a receives the input of the signal indicating the measured value of the distance in the first measurement and the "previous" measured value does not exist, the comparison in step S909 is omitted (Yes and Considered), the process of step S905 is performed.

If the measured value d is not smaller than the first reference value _D1 in step S904 (No), or if the measured value d is smaller than the first reference value _D1 in step S909, the latest measured value d is the previous is not smaller than the measured value d ₀ (No), the unmanned aerial vehicle 1 is not approaching the inspected structure 15a too much, or is moving away from the inspected structure 15a or is maintaining an equal distance. The process of step S905 is not performed, and the process proceeds to step S906. The main arithmetic circuit 7a compares the measured value d with the second reference value D2 by executing the distance control module _9b (step S906). As already described, the second reference value _D2 is a reference value greater than or equal to the first reference value _D1 . If the measured value d is greater than the second reference value D ₂ (Yes), the main arithmetic circuit 7a further executes the distance control module 9b so that the measured value d (latest measured value) and the previous A comparison is made with the previous distance measurement value _d0 indicated by the signal received from the measurement value determination circuit 6 (S910). If the latest measured value d is greater than the previous measured value _d0 (Yes), the unmanned aerial vehicle 1 is too far from the structure to be inspected 15a, and the measured distance has increased over time. , a control command value for bringing the unmanned aerial vehicle 1 closer to the structure to be inspected 15a is generated (step S907). When the main arithmetic circuit 7a receives the input of the signal indicating the measured value of the distance in the first measurement and there is no "previous" measured value, the comparison in step S910 is omitted (Yes and Considered), the process of step S907 is performed.

If the measured value d is not greater than the second reference value _D2 in step S906 (No), or if the measured value d is greater than the second reference value _D2 in step S910, the latest measured value d is the previous is not greater than the measured value d ₀ (No), the unmanned aerial vehicle 1 is not too far from the structure to be inspected 15a, or is approaching the structure to be inspected 15a or is maintaining an equal distance. The process of step S907 is not performed, and the process proceeds to step S908. The main arithmetic circuit 7a generates control command values relating to the throttle amount, roll angle, pitch angle, and yaw angle as control command values obtained by combining the externally input command values and attitude control command values (step S908).

According to the processing flow of FIG. 9, control command values relating to the throttle amount, roll angle, pitch angle, and yaw angle are generated in any of steps S905, S907, and S908. Subsequent conversion of the control command value, generation of the control signal, etc. are as already described with reference to FIG.

Prototype The inventor of the present invention designed a prototype of the unmanned aerial vehicle 1 of the present invention that performs distance measurement and distance control according to the distance measurement. However, in the configuration of FIG. 3, in addition to the various sensors of the sensor unit 14, this prototype machine has views and photographs viewed from below (the z direction in FIG. 1) and from a slightly oblique direction in FIGS. 11, it has a lower camera 17 and a SLAM (Simultaneous Localization and Mapping) processing circuit 18 . However, the landing leg 5 is omitted in FIG. 10A. The lower camera 17 is a monocular camera that takes pictures of the lower direction (the z direction in FIG. 1) during flight. are used for signal processing and recording. The SLAM processing circuit 18 is a commercially available circuit board equipped with a CPU, a GPU (Graphics Processing Unit), memory, etc., and uses the memory by recording programs and data for executing Visual SLAM. . Visual SLAM is a technique for estimating a self-position and a map in parallel by tracking a plurality of feature points between a plurality of frames of continuously captured images. Various algorithms have been developed, such as PTAM (Parallel Tracking and Mapping) (Non-Patent Documents 4 and 5). By executing a program in which the SLAM processing circuit 18 implements such an algorithm, the image signal recorded in the frame memory of the lower camera 17 is used to perform self-position estimation and mapping by Visual SLAM. Self-position (the relative position of the unmanned aerial vehicle 1 with respect to elements existing around the unmanned aerial vehicle 1), velocity (determined by time differentiation of the position), attitude (by geometric calculation from the arrangement of a plurality of feature points in the photographed image determined using the sensor data from the sensor unit 14 in the configuration of FIG. Signals indicating these quantities are output to the main arithmetic circuit 7a, and the main arithmetic circuit 7a uses the information input from the sensor section 14 in the configuration of FIG. Use the information entered. The map information estimated by the SLAM processing circuit 18 is also output to the main arithmetic circuit 7a and recorded in the recording device 10. FIG. Except for the configuration related to SLAM, it is basically the same as the configuration described with reference to FIGS. 1 to 9. FIG. As the lower camera 17, the stereo camera described with reference to FIGS. 5 and 6 may be used instead of the monocular camera. In this case, the same principle can be used to estimate the self position and the like by Visual SLAM. For example, SLAM using a laser range sensor is also applicable instead of Visual SLAM, and in this case, the laser range sensor is used instead of the lower camera 17 (Non-Patent Document 6).

The specific configuration of this prototype will be described below. This prototype has a barometric altimeter, a sonar, and a GPS sensor as the sensor section 14. Mainly, the downward camera 17 and visual SLAM processing by the SLAM processing circuit 18 provide highly reliable data such as the position of the aircraft. If not, the operation is switched to detection processing using these sensors of the sensor unit 14 . Data such as the aircraft position is transmitted from the SLAM processing circuit 18 to the main processing circuit 7a via a 3.3V UART (Universal Asynchronous Receiver/Transmitter) interface using one data line.

As the hardware configuration, as the circuit board of the SLAM processing circuit 18, NVIDIA Jetson TX2 (vision computer) and CTI Orbitty carrier board for NVIDIA Jetson TX2, as the stereo camera 3, ZED stereo camera (USB 3.0), and the lower camera 17 used IDS UI-1220SE mono grayscale camera (USB 2.0) and Theia MY110M lens for mono camera.

The operating power of the SLAM processing circuit 18 having the above configuration is basically required to be 2W and 9-14V. By pressing a power button (not shown) provided on the main body of the unmanned aerial vehicle 1, the unmanned aerial vehicle 1 is activated or stopped. can be switched. For example, when the power button is pressed to stop the operation of the main body of the unmanned aerial vehicle 1, a stop command signal is first transmitted from the main arithmetic circuit 7a to the SLAM processing circuit 18, and the SLAM processing circuit 18 stops operating. The machine stops working. A separate backup battery of sufficient capacity may be provided for the SLAM processing circuit 18 to allow the SLAM processing circuit 18 to shut down after the main battery is disconnected.

While this prototype is basically operated by an external input device such as a radio, the change processing according to the situation during flight (for example, when an obstacle at a short distance is detected, the above-mentioned distance control by the control signal generation circuit 8 etc. processing is performed and the external input signal is changed.) and change processing by command input from the outside (forcibly intervening in the flight by sending an emergency command such as a pause or a forced stop from the ground station, etc.) is possible.), etc., to override the input signal from the transmitter or the like. This prototype can operate in the following five modes.

1. Attitude control mode By using the external input command value indicated by the external input signal received from the external input device and the attitude information data obtained by the measurement of the sensor unit 14, the main computation circuit 7a executes the autonomous control program 9a. This is a semi-manual mode in which the attitude is autonomously controlled by generating a (combined) control command value in combination with the generated attitude control command value. To take off the unmanned aerial vehicle 1, simply push the "thrust" stick upward until the aircraft takes off, after which the aircraft can be steered according to external input signals while the attitude is stabilized by autonomous control. can. To land, simply push the "thrust" stick downwards until the aircraft lands. Takeoff and landing can be performed in all modes including the following modes, and the procedure is the same in each mode except GPS waypoint mode (takeoff and landing are performed autonomously) described later.
2. Vision assist mode
As already described, this mode uses information such as aircraft position, velocity, attitude, etc. obtained by Visual SLAM processing by the lower camera 17 and SLAM processing circuit 18 instead of the sensor unit 14 . By combining the external input command value indicated by the external input signal and the autonomous control command value generated by the main arithmetic circuit 7a executing the autonomous control program 9a using the information obtained by the Visual SLAM processing, ( This is a semi-manual mode in which control is performed by generating synthetic) control command values. In this control mode, when the operator releases his/her finger from the external input device, the unmanned aerial vehicle 1 remains at the current body position. To move the unmanned aerial vehicle 1 to the left, push the "roll" stick to the left. To stop, simply let go of the stick. To move the unmanned aerial vehicle 1 upward, push the "thrust" stick up. To stop, simply release the stick (the "thrust" stick is spring loaded and returns to the middle position).
3. Distance control mode
In this prototype, this mode is used together with the Vision Assist mode described in "2." above. In this mode, the distance is controlled so that a fixed distance is maintained. Flight controls in left/right and up/down directions can be used to “slide” the airframe along target elements in front of the unmanned aerial vehicle 1 . A target value as a fixed distance is set within a range of minimum 1 m and maximum 3 m using the distance setting knob 20 on the external input device 19 (FIG. 12).
4. GPS assist mode
This is a mode in which the attitude/position (at the time of hovering) is autonomously controlled based on GPS sensor data while basically operating with control signals from an external controller.
5. GPS waypoint mode
Using GPS waypoints set in advance as part of the flight plan information, in accordance with the flight plan given by the above-mentioned flight plan information, in a mode in which the flight plan route is autonomously flown using position data etc. from the GPS sensor. be.

Flight modes are selected using a mode switch (not shown) on the external input device. However, the distance control mode of "3." is disabled during takeoff and landing operations.

Also, before flying this prototype, setup work using the take-off pad 21 of FIG. 13 is performed for initialization of Visual SLAM processing. The setup procedure is as follows.
1. Make sure the external input device (wireless controller) is turned off.
2. Plug in the aircraft battery.
3. Press the "vision power" button on the aircraft.
a. The "vision power" LED starts flashing yellow.
b. Wait until the "vision power" LED turns solid green.
4. Place the aircraft on the take-off pad 21.
a. The stereo camera 3 is placed so as to face the direction of the arrow (forward) in FIG.
b. Also, the front two ends of the landing leg 5 are placed on the first mark 22 respectively.
5. Press the "Initialize" button on the back of the aircraft.
6. The aircraft is moved so that the front two ends of the landing legs 5 slide from the first mark 22 to the second mark 23 .
7. Confirm that the "Initialize" LED on the back of the aircraft has gone out. If the "Initialize" LED does not go out, repeat from step 4.

This setup operation consists of a first fixed position where the front two ends of the landing gear 5 of the fuselage are positioned at the first marks 22 respectively, and a second fixed position where the front two ends are positioned at the second marks 23 respectively. and the bottom camera 17 from a fixed position to obtain the first two images used for Visual SLAM processing. During the setup work, the initial setting picture 24 is captured from the first fixed position when the above "5." is performed, and the initial setting picture 24 is captured from the second fixed position when the above "6." A setting picture 24 is taken. The relative pose between the lower camera 17 and the observed 3D positions of the feature points can be calculated by finding the homographies between the cameras through the plane. Each marker (pattern) in the initialization picture 24 is of known size and the captured image can be used to determine the actual distance from the lower camera 17 to the take-off pad 21 . This actual distance can be compared to the distance from the plane of the take-off pad 21 obtained from the initial SLAM maps to set the scale between SLAM processing and the real world. Note that when a stereo camera is used as the lower camera 17, two images can be obtained by photographing the initial setting picture 24 from the first fixed position, so that the above operation "6." can be omitted.

INDUSTRIAL APPLICABILITY The present invention can be used to control any unmanned aerial vehicle used for any purpose including industrial use and hobby use.

1 Unmanned aerial vehicle 2 Main unit 3 Stereo camera 4 External environment camera 5 Landing leg 6 Measured value determination circuit (digital signal processing unit)
7a main arithmetic circuit 7b signal conversion circuit 8 control signal generation circuit 9a autonomous control program 9b distance control module 9c various databases 10 recording device 11 power supply system 12 communication antenna 13 communication circuit 14 sensor section C0 left camera C1 right camera A subject O ₀ , O ₁ optical center s ₀ , s ₁ imaging plane P ₀ , P ₁ image position 300 camera unit 301 lens 302 aperture 303 CMOS image sensor 304 CDS circuit 305 AGC circuit 306 A / D converter 307 frame memory 308 camera control unit (camera control circuit)
15a structure to be inspected 15b element to be inspected 16a equidistant surface 16b equidistant line 17 downward camera 18 SLAM processing circuit 19 external input device 20 distance setting knob 21 take-off pad 22 first mark 23 second mark 24 initial setting picture

Claims (11)

A photographing camera for photographing the target element as a distance sensor for measuring the distance between the unmanned aircraft flying under control using an external input signal and/or flight plan information generated in advance and the target element, and the photographed image a measurement determination circuit that uses the information to determine a measurement of the distance;
a control signal generation circuit that generates a control signal for controlling the flight of the unmanned aerial vehicle during flight according to the distance measurement value measured by the distance sensor;
The control signal generation circuit is
generating a control signal to move the unmanned aerial vehicle away from the target element when the measured value is less than a first reference value and the measured value decreases over time;
generating a control signal that does not involve control of the distance when the measured value is smaller than the first reference value and the measured value does not decrease over time;
generating a control signal for bringing the unmanned aerial vehicle closer to the target element when the measured value is greater than a second reference value equal to or greater than the first reference value and the measured value increases over time; death,
When the measured value is greater than the second reference value and the measured value does not increase over time, a control signal is generated that does not involve control of the distance.
configured as
Unmanned aerial vehicle flight controller. The unmanned aerial vehicle is an unmanned aerial vehicle that flies under control using at least the external input signal, the external input signal being a signal input in real time from an external input device during flight of the unmanned aerial vehicle, and 2. The flight control device according to claim 1, wherein the control signal is a signal obtained by changing the external input signal according to the measured value of the distance. The unmanned aerial vehicle is an unmanned aerial vehicle that flies under control using at least the flight plan information, and the flight plan information is flight plan information generated in advance before flight by a computer executing a program. The flight control device of claim 1. 4. A flight controller according to any one of the preceding claims, wherein the measurement determination circuit is integrated with the control signal generation circuit. 5. A flight control device according to any preceding claim, wherein the first reference value and the second reference value are equal. 6. The flight control device according to any one of claims 1 to 5, further comprising an external environment photographing camera that photographs a direction different from that of the photographing camera. 7. The flight control device according to any one of claims 1 to 6, further comprising a relative position measurement sensor for measuring the relative position of the unmanned aerial vehicle with respect to elements existing around the unmanned aerial vehicle. 8. A flight control system according to any one of claims 1 to 7, wherein the target element is a structure to be inspected. An unmanned aerial vehicle comprising a flight control device according to any one of claims 1 to 8. Photographing a target element and using the photographed image information to determine a measured distance between the target element and an unmanned aerial vehicle flying under control using an external input signal and/or pre-generated flight plan information. measuring the distance by
generating control signals for controlling flight of the unmanned aerial vehicle during flight in response to the distance measurement;
The step of generating the control signal includes:
generating a control signal to move the unmanned aerial vehicle away from the target element when the measured value is less than a first reference value and the measured value decreases over time;
generating a control signal that does not involve control of the distance when the measured value is smaller than the first reference value and the measured value does not decrease over time;
generating a control signal for bringing the unmanned aerial vehicle closer to the target element when the measured value is greater than a second reference value equal to or greater than the first reference value and the measured value increases over time; death,
When the measured value is greater than the second reference value and the measured value does not increase over time, a control signal is generated that does not involve control of the distance.
including
A flight control method for an unmanned aerial vehicle. Determining the measured distance between the target element and the unmanned aerial vehicle flying under control using external input signals and/or pre-generated flight plan information, using image information captured by the camera that captures the target element causing the measured value determination circuit to perform the step of
generating a control command value for controlling the flight of the unmanned aerial vehicle during flight according to the distance measurement,
generating a control command value for moving the unmanned aerial vehicle away from the target element when the measured value is smaller than a first reference value and the measured value decreases over time;
generating a control command value that does not involve control of the distance when the measured value is smaller than the first reference value and the measured value does not decrease over time;
When the measured value is greater than a second reference value equal to or greater than the first reference value and the measured value increases with time, a control command value for bringing the unmanned aerial vehicle closer to the target element is set. generate and
When the measured value is greater than the second reference value and the measured value does not increase over time, a control command value is generated that does not involve controlling the distance.
A program for causing a control signal generation circuit to perform a step including:

Images