JP2021047738A - Moving object, flight path control method and program - Google Patents

Abstract

To control a flight path of a moving object which performs aerial photography, and shoot for SLAM with a monaural camera.SOLUTION: A moving object is comprised of: a flight device for autonomously flying on a specified flight path; an imaging device for automatically taking images continuously during autonomous flight on the flight path; and a control device for controlling the flight device during autonomous flight so as to perform cross-movement for moving the flight path in a direction that intersects a direction of travel, while the control device estimates a self-position and creates an environmental map using the images taken by the imaging device taken during the cross-movement.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to moving objects, flight path control methods and programs.

In a moving body such as an unmanned aerial vehicle (UAV), a three-dimensional display can be realized by using a plurality of images acquired by an imaging device of the moving body (see Patent Document 1). Further, in order for such a moving body to efficiently create a three-dimensional map, a technique called SLAM (Simultaneous Localization and Mapping) has been proposed in which self-position estimation and environment map creation are performed at the same time.

Special Table 2018-530949

However, in the above-mentioned conventional technique, when the moving body is equipped with a stereo camera (binocular camera) when performing SLAM, a plurality of images can be taken at the same time, but due to the limitation of the machine body, a monaural camera (monaural camera) If only a monocular camera is installed, multiple images cannot be taken at the same time. Therefore, there is a demand for a technique capable of shooting for SLAM using only a monaural camera.

Therefore, the present disclosure proposes a moving body, a flight path control method, and a program capable of photographing for SLAM with a monaural camera.

According to the present disclosure, a flight device for autonomously flying on a designated flight path, an imaging device for automatically taking images continuously during autonomous flight of the flight path, and the above-mentioned flight path during autonomous flight. A control device that controls a flight device to perform cross-movement to move in a direction intersecting the direction of travel, and performs self-position estimation and environmental map creation using images taken by the image pickup device taken during the cross-movement. And a moving body comprising.

Further, according to the present disclosure, the moving body is made to fly autonomously along the designated flight path, and the image is continuously captured by using the image pickup device mounted on the moving body during the autonomous flight of the flight path. Automatically photographing, performing crossing movement to move the moving body in a direction intersecting the traveling direction during autonomous flight of the flight path, and capturing an image taken by the imaging device during the crossing movement. A method for controlling the flight path of a moving body including self-position estimation and environment mapping is provided.

Further, according to the present disclosure, an image is continuously captured by using a step of autonomously flying a moving body along a designated flight path and an imaging device mounted on the moving body during autonomous flight of the flight path. The step of automatically photographing, the step of performing the crossing movement of moving the moving body in the direction intersecting the traveling direction during the autonomous flight of the flight path, and the step of performing the crossing movement, and the captured image of the imaging device taken during the crossing movement are taken. A program is provided to allow a computer to perform the steps of self-position estimation and environment mapping using it.

It is a block diagram which shows the structural example of the moving body which concerns on embodiment of this disclosure. It is a flowchart for demonstrating the whole operation of the moving body which concerns on embodiment of this disclosure. It is a flowchart for demonstrating the operation at the time of SLAM photography of the moving body which concerns on embodiment of this disclosure. It is a conceptual diagram for explaining the parallax for SLAM. It is a conceptual diagram for demonstrating the lateral shift movement for SLAM of the moving body which concerns on embodiment of this disclosure. It is an image diagram which shows the example of the pattern which the body of the moving body which concerns on embodiment of this disclosure flies in a sine wave. It is an image diagram which shows the example of the pattern which the body of the moving body which concerns on embodiment of this disclosure flies in a rectangular wave (square wave). It is an image diagram which shows the example of the pattern in which the body of the moving body which concerns on embodiment of this disclosure flies in a triangular wave. It is an image diagram which shows the example of the pattern in which the body of the moving body which concerns on embodiment of this disclosure flies while drawing a sawtooth wave (sawtooth wave).

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same parts are designated by the same reference numerals, so that duplicate description will be omitted.

In addition, the present disclosure will be described according to the order of items shown below.
1. 1. Embodiment 1.1. Outline of mobile 1.2. Movement of moving object 1.2.1. Overall operation of the moving body 1.2.2. Operation of moving object during SLAM shooting 1.2.3. Exception handling 1.3. Lateral movement for SLAM 1.4. Difference from SLAM using a stereo camera 1.5. Modification example 2. Conclusion

1. 1. Embodiment First, the moving body according to the embodiment of the present disclosure will be described in detail with reference to the drawings below.

1.1. Outline of the Moving Body FIG. 1 is a block diagram showing a configuration example of the moving body 1 according to the embodiment of the present disclosure. The mobile body 1 is a small vehicle (so-called drone) that performs autonomous flight, such as an unmanned aerial vehicle (UAV). As shown in FIG. 1, the moving body 1 includes a flight control unit 2, a flight device 3, a flight path storage unit 4, a position information acquisition unit 5, an obstacle detection camera 6, and an aerial photography camera 7. A camera control unit 8, a skid signal generation unit 9, and a communication unit 10 are provided.

The flight control unit 2 (flight controller) is, for example, a processor such as a CPU (Central Processing Unit), a microcontroller, or an integrated circuit such as an ASIC (application specific integrated circuit) or an FPGA (field-programmable gate array). The flight device 3 is controlled according to the program stored in the memory of the above, and autonomous flight along the flight path is performed according to the flight path information stored in the flight path storage unit 4. Further, the flight control unit 2 performs feedback control so as to reach the target state based on the state of the position and attitude of the moving body 1 measured by various sensors mounted on the body of the moving body 1.

The flight device 3 is composed of, for example, a moving blade and a servomotor, or a propeller, a motor, and a control circuit. Examples of the aircraft of the moving body 1 having such a flight device 3 are a multicopter (MC: MultiCopter), a fixed-wing aircraft (FW: Fixed Wing), and a vertical take-off and landing aircraft (VTOL: Vertical Take-Off and Landing Aircraft). And so on.

The flight path storage unit 4 is, for example, a memory such as a RAM (Random Access Memory), and stores flight path information and various other information predetermined by an operator or the like. The flight path storage unit 4 may be a flash memory, a storage such as an SSD (Solid State Drive) or an HDD (hard disk), a storage medium (media) such as an optical disk, and a drive device (drive) for reading and writing the same.

The position information acquisition unit 5 is, for example, a GPS (Global Positioning System) receiver or the like, and uses GPS to acquire information regarding the current position of the mobile body 1. GPS is one of GNSS (Global Navigation Satellite System). That is, it is also possible to use GNSS other than GPS. Further, the current position of the moving body 1 may be specified by SLAM without using GPS.

The obstacle detection camera 6 is an image sensor made of, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, and is an imaging device that automatically continuously captures images during autonomous flight of the moving body 1. is there. The obstacle detection camera 6 is used to detect an obstacle or the like on the flight path. Obstacles on the flight path are, for example, terrain, living things, natural objects, structures, flying objects, flying objects, and the like. Specifically, mountains and cliffs, people and birds, trees, construction machinery, buildings, bridges (viaducts), steel towers, utility poles and electric wires, and other unmanned aerial vehicles (UAVs) can be obstacles. An object that is at least on the flight path and reaches the same altitude as the moving body 1 and interferes with the autonomous flight of the moving body 1 becomes an obstacle. In addition, if the existence is known in advance, it is possible to stay away from the flight by setting the flight path, but unexpected obstacles may occur on the flight path due to collapse or fallen trees due to an accident or disaster. .. The image captured by the obstacle detection camera 6 may be black-and-white / monochrome (grayscale) because it is sufficient if the terrain, structures, and the like can be identified by image processing. In the present embodiment, the obstacle detection camera 6 is a black-and-white image monaural camera (monocular camera).

The obstacle detection camera 6 employs a global shutter in which shutters of all pixels are simultaneously activated. The reason for adopting the global shutter for the obstacle detection camera 6 is that if the shutter timing is different for each pixel, for example, the upper left pixel and the lower right pixel of the captured image, the shutter timing is different. This is because the accuracy of the obstacle detection algorithm is lowered when the moving body 1 is moving because the scenery is being viewed. Therefore, the global shutter is used so that there is no time lag between the pixels in the image. However, if the speed of the moving body 1 is low, a camera with a rolling shutter can also be used.

An example of mounting the obstacle detection camera 6 will be described. For example, when the body of the moving body 1 is a vertical take-off and landing aircraft (VTOL), the obstacle detection camera 6 is mounted near the nose. The direction of the obstacle detection camera 6 is slightly downward so that the ground can be seen for tracking.

The aerial photography camera 7 is an image sensor made of, for example, a CCD or CMOS, and is an imaging device that automatically or in response to an operator operation during autonomous flight of the moving body 1. Since the captured image of the aerial camera 7 is viewed by humans, it is preferable to have color information. However, regardless of the quality of aerial photography, the output (black and white image) of the obstacle detection camera 6 can be used as an image taken by the aerial photography camera 7. Since the aerial photography camera 7 is not required to have the same accuracy as the obstacle detection camera 6 in terms of shooting pixels and shooting time, a rolling shutter that is cheaper than the global shutter is used. Instead, increase the resolution.

Each of the obstacle detection camera 6 and the aerial photography camera 7 and the camera control unit 8 are connected by, for example, USB (Universal Serial Bus). Each of the obstacle detection camera 6 and the aerial photography camera 7 is subjected to image shooting start, shooting stop, shooting setting, etc. from the camera control unit 8 in accordance with PTP (Picture Transfer Protocol), which is a USB communication standard (communication protocol). Receives various shooting control commands (commands).

The camera control unit 8 is, for example, a processor such as a CPU, a microcontroller, or an integrated circuit such as an ASIC or FPGA, and issues the above-mentioned shooting control command to the obstacle detection camera 6 and the sky according to a program stored in the internal memory. It is output to each of the photographing cameras 7. For example, the camera control unit 8 receives information (GPS signal, etc.) about the current position of the moving body 1 from the position information acquisition unit 5, and according to the above program, the shooting start position recorded in the flight path information and the GPS signal. When the current position based on is the same, a shooting start command is output to the aerial shooting camera 7. Further, when the shooting stop position recorded in the flight path information and the current position based on the GPS signal match, a shooting stop command is output to the aerial shooting camera 7. The camera control unit 8 may be integrated with the flight control unit 2.

Further, the camera control unit 8 determines the presence or absence of an obstacle on the flight path based on the captured image of the obstacle detection camera 6. That is, the camera control unit 8 also functions as an obstacle detection unit. In the present embodiment, the camera control unit 8 uses the obstacle detection camera 6 to perform shooting for SLAM (hereinafter referred to as SLAM shooting).

Further, the camera control unit 8 determines whether or not the aerial camera 7 is shooting based on the command output to the aerial camera 7 and / or the captured image of the aerial camera 7. When the aerial camera 7 is shooting, the camera control unit 8 stops SLAM shooting with the obstacle detection camera 6. For example, when the aerial camera 7 is shooting, the camera control unit 8 outputs a command to the flight control unit 2 and the skid signal generation unit 9 to stop the processing related to SLAM shooting. Even if the camera control unit 8 stops the SLAM shooting with the obstacle detection camera 6, the camera control unit 8 continues the shooting itself with the obstacle detection camera 6 in order to detect the obstacle.

The skid signal generation unit 9 is, for example, a processor such as a CPU, a microcontroller, or an integrated circuit such as an ASIC or FPGA, and is based on parameters stored in the flight path storage unit 4 according to a program stored in an internal memory. Generates a skid signal that adds skidding to the flight path. Sideslip is a cross movement that moves in a direction that intersects (for example, orthogonally) with the direction of travel of the flight path, and moves in a direction that intersects horizontally (left and right) with respect to the direction of travel of the flight path due to small turns or the like. It is a lateral shift movement that causes the vehicle to move. The reason for adding skidding to the flight path is to use "parallax" for SLAM shooting with a monaural camera. The details of the parallax will be described later in "1.3. Lateral movement for SLAM". The skid signal generation unit 9 may be integrated with the flight control unit 2.

The parameters are, for example, information regarding a strike-slip width SW (Sideslip Width), a strike-slip time ST (Sideslip Time), and a strike-slip pattern SP (Sideslip Pattern). The strike-slip pattern SP represents a pattern of a trajectory drawn by the body of the moving body 1 in the horizontal direction (front-back, left-right direction) when the moving body 1 is viewed from the vertical direction (vertical direction) during the flight of the moving body 1. The strike-slip pattern SP is, for example, a sine wave, a square wave (square wave), a triangular wave, or a sawtooth wave (sawtooth wave). The waveform of the strike-slip pattern SP is preferably a step wave (diagonal edge) rather than a pulse wave in consideration of the actual behavior of the airframe during the strike-slip movement during flight, and the corners may be rounded. For example, the square wave may be a trapezoidal wave or an S-shaped wave. In the present embodiment, a sine wave will be described as an example of the waveform of the strike-slip pattern SP. The lateral displacement SW is the displacement width from the central axis (original flight path) to the peak position (highest point) on one side of the waveform, and the lateral displacement time ST is the lateral displacement SW from the central axis (original flight path). It is the time required to move by the amount of. For example, when the waveform of the strike-slip pattern SP is a sine wave, the strike-slip width SW represents the amplitude. Further, a value obtained by multiplying the strike-slip time ST by 4 represents a unit time, and the reciprocal thereof represents a frequency.

The flight control unit 2 receives information on the direction of the obstacle and the distance to the obstacle from the camera control unit 8, and receives a skid signal from the skid signal generation unit 9 to add skidding to the flight path. Based on the skid signal, the flight control unit 2 controls the flight device 3 so that the moving body 1 shifts laterally in the direction of crossing the flight path horizontally (left and right) with respect to the traveling direction during flight. As a result, the moving body 1 flies so that the airframe draws a lateral displacement pattern SP in the horizontal direction when viewed from the vertical direction. The flight control unit 2 may give feedback from the trajectory information generated by using the GPS and the inertial sensor so that the trajectory of the moving body 1 has a predetermined lateral displacement width SW / lateral displacement pattern SP.

Further, when a certain time elapses after the skid signal generation unit 9 generates the skid signal, the camera control unit 8 sets the SLAM image flag on the captured image of the obstacle detection camera 6 at that time. The SLAM image flag is represented by, for example, a binary value of "0" or "1", and is a flag set on the captured image used for SLAM among the captured images of the obstacle detection camera 6. This fixed time is, for example, the time required for one-way lateral movement, from the end of one-way lateral movement to the setting of the SLAM image flag on the captured image (about several milliseconds to several tens of milliseconds). It is a value that takes into account. The time required for one-way strike-slip movement is the same as the strike-slip time ST because it is the movement from the central axis (original flight path) to the peak position on one side of the waveform at first, but the waveform after the second time. Since it moves between the peak positions on both sides of the above, the value is obtained by doubling the strike-slip time ST. The skid signal generation unit 9 outputs the SLAM image flag to the camera control unit 8 when the time required for the lateral slip movement in one direction elapses after the skid signal is generated. The camera control unit 8 receives an instruction of the SLAM image flag from the skid signal generation unit 9, and in response to the instruction of the SLAM image flag, sets the SLAM image flag to the captured image of the obstacle detection camera 6 at that time. Stand up.

The captured image of the obstacle detection camera 6 is captured at a fixed cycle regardless of the lateral shift movement, and for example, 30 images are captured in 1 second. Assuming that the lateral shift occurs every 3 seconds, the SLAM image flag is set on one of the 3 × 30 = 90 captured images. Here, the SLAM image flag is set for the captured image at the time when the lateral displacement movement in one direction is normally completed without being stopped or immediately after (at least before the lateral displacement pattern SP is inverted in the opposite phase). For example, an image flag for SLAM is set in the captured image at the peak position of the waveform of the strike-slip pattern SP. After that, every time the lateral shift movement is completed normally, the SLAM image flag is set in the captured image. Then, the camera control unit 8 executes SLAM using the captured image in which the SLAM image flag is set. The SLAM may be carried out in real time or after the flight is completed.

The communication unit 10 is, for example, an antenna for wireless communication, and is used for external transmission of captured images and various data, and reception of control signals and the like from the outside. The communication unit 10 may be a communication device for communicating via the Internet. For example, the flight control unit 2 may receive an instruction (control signal) from a remote controller on the ground via the communication unit 10 and perform flight control based on the instruction. Further, the camera control unit 8 can receive an instruction from the remote controller on the ground via the communication unit 10 and output a shooting control command to the camera based on the instruction. Further, the camera control unit 8 can also transmit the image captured by the camera to a communication device such as a PC (personal computer) on the ground via the communication unit 10. The remote controller on the ground may send an instruction to the moving body 1 according to the operation of the operator, or may automatically send the instruction by program control.

1.2. Movement of moving object 1.2.1. Overall operation of the moving body Next, the operation of the moving body according to the embodiment of the present disclosure will be described. FIG. 2 is a flowchart for explaining the overall operation of the moving body 1 according to the embodiment of the present disclosure.

As shown in FIG. 2, the operator turns on the power of the moving body 1 (step S101), and sets the parameter (step S102) and the flight path (step S103) in the moving body 1. As a result, the moving body 1 stores the parameters and the flight route information in the flight path storage unit 4.

The flight control unit 2 controls the flight device 3 and causes autonomous flight along the flight path according to the flight path information stored in the flight path storage unit 4 (step S104).

The flight control unit 2 outputs a command to the skid signal generation unit 9 with the start of the autonomous flight of the moving body 1, and performs SLAM imaging with the obstacle detection camera 6 (step S105).

The flight control unit 2 determines whether or not the vehicle has arrived at the destination based on the flight route information stored in the flight path storage unit 4 and the information regarding the current position of the moving body 1 from the position information acquisition unit 5. Determine (step S106). The destination may be the same as the departure point. If the flight control unit 2 determines that it has not arrived at the destination (No in step S106), the flight control unit 2 continues flight control and continues SLAM imaging with the obstacle detection camera 6 (returns to step S105). ).

When the flight control unit 2 determines that the aircraft has arrived at the destination (Yes in step S106), the flight control unit 2 outputs a command to the skid signal generation unit 9, ends SLAM shooting with the obstacle detection camera 6, and flies. The control ends (step S107).

1.2.2. Operation of the moving body during SLAM imaging Next, the details of the operation during SLAM imaging shown in step S105 of FIG. 2 will be described. FIG. 3 is a flowchart for explaining the operation of the moving body 1 according to the embodiment of the present disclosure at the time of SLAM photographing.

As shown in FIG. 3, the flight control unit 2 outputs a command to the skid signal generation unit 9 with the start of the autonomous flight of the moving body 1 (step S104 in FIG. 2), and the obstacle detection camera 6 is used. SLAM shooting is started (step S201).

The skid signal generation unit 9 generates a skid signal that adds skidding to the flight path from the parameters stored in the flight path storage unit 4, and outputs the skid signal to the flight control unit 2 (step S202).

The flight control unit 2 receives the skid signal from the skid signal generation unit 9, and based on the skid signal, causes the moving body 1 to laterally move in a direction horizontally intersecting the traveling direction during flight. The flight device 3 is controlled (step S203).

The flight control unit 2 and / or the camera control unit 8 determines whether or not exception handling has occurred (step S204). The details of exception handling will be described in "12.3. Exception handling" described later. If it is not determined that the exception handling has occurred (No in step S204), the flight control unit 2 and / or the camera control unit 8 continues the current operation (shift to step S205).

The camera control unit 8 determines whether or not the instruction of the SLAM image flag has been received from the skid signal generation unit 9 (step S205). For example, the skid signal generation unit 9 outputs an instruction of the SLAM image flag to the camera control unit 8 when the time required for the lateral slip movement in one direction elapses after the skid signal is generated. If the camera control unit 8 has not received the SLAM image flag instruction from the skid signal generation unit 9 (No in step S205), the camera control unit 8 continues the current operation until it receives the SLAM image flag instruction (No). Return to step S203).

When the camera control unit 8 receives the instruction of the SLAM image flag from the skid signal generation unit 9 (Yes in step S205), the camera control unit 8 responds to the instruction of the SLAM image flag and determines the obstacle detection camera 6 at that time. An image flag for SLAM is set in the captured image of (step S206). As a result, the camera control unit 8 sets the SLAM image flag on the captured image of the obstacle detection camera 6 at that time when a certain time elapses after the skid signal generation unit 9 generates the skid signal.

After outputting the instruction of the image flag for SLAM to the camera control unit 8, the skid signal generation unit 9 generates a skid signal having a phase opposite to the current skid signal from the parameters stored in the flight path storage unit 4. Output to the flight control unit 2 (step S207). The flight control unit 2 receives the skid signal of the opposite phase from the skid signal generation unit 9, and controls the flight device 3 so as to laterally move in the opposite direction to the conventional one based on the skid signal of the opposite phase (step). Return to S203).

If it is determined that exception handling has occurred (Yes in step S204), the flight control unit 2 and / or the camera control unit 8 outputs a command to the skid signal generation unit 9 to detect obstacles. The SLAM shooting in step 6 is finished (step S208). Further, the flight control unit 2 ends the lateral slip movement and causes the moving body 1 to perform a normal flight (step S209). Then, the flight control unit 2 and / or the camera control unit 8 performs an operation according to the exception handling that has occurred.

The flight control unit 2 and / or the camera control unit 8 determines whether or not the exception handling is completed (step S210). If it is not determined that the exception handling is completed (No in step S210), the flight control unit 2 causes the moving body 1 to continue the normal flight (returns to step S209). When it is determined that the exception handling is completed (Yes in step S210), the flight control unit 2 and / or the camera control unit 8 outputs a command to the skid signal generation unit 9, and the obstacle detection camera 6 displays a command. SLAM shooting is started again (returning to step S201).

1.2.3. Exception handling Next, the details of the exception handling shown in step S204 of FIG. 3 and the like will be described. The exception processing in the present embodiment is, for example, detection of an obstacle by the obstacle detection camera 6 (exception processing 1), aerial photography by the aerial photography camera 7 (exception processing 2), and an operator's instruction via the communication unit 10. (Exception handling 3) and the like. Each case will be described in detail below with reference to step S204 and subsequent steps in FIG.

[Exception handling 1: Obstacle detection by obstacle detection camera 6]
The camera control unit 8 determines the presence or absence of an obstacle on the flight path based on the captured image of the obstacle detection camera 6 (step S204). For example, the camera control unit 8 executes a detection algorithm once every few images (sometimes every time) when an image is input from before the start of the lateral shift movement to after the end, and an obstacle. Detects obstacles if is present. The time required from executing the detection algorithm to detecting the obstacle is less than 1 second. If it is not determined that there is an obstacle (No in step S204), the flight control unit 2 and the camera control unit 8 continue the current operation (shift to step S205).

If it is determined that there is an obstacle (Yes in step S204), the camera control unit 8 outputs a command to the skid signal generation unit 9 to end the SLAM shooting with the obstacle detection camera 6 (step). S208). Further, the flight control unit 2 ends the lateral slip movement and causes the moving body 1 to perform a normal flight (step S209). At this time, the camera control unit 8 generates information regarding the direction of the obstacle and the distance to the obstacle, and outputs the information to the flight control unit 2. The flight control unit 2 controls the flight device 3 based on the information regarding the direction of the obstacle and the distance to the obstacle, and avoids the obstacle. The reason for ending SLAM shooting and lateral movement and returning to normal flight when there is an obstacle is that the behavior of the moving body 1 is restricted during lateral movement, making it difficult to avoid obstacles, so during normal flight. This is to avoid obstacles. In addition, since effective SLAM imaging cannot be performed during normal flight without lateral shift movement, SLAM imaging is stopped.

The camera control unit 8 determines whether or not the obstacle on the flight path has been avoided based on the captured image of the obstacle detection camera 6 (step S210). If it is not determined that the obstacle on the flight path has been avoided (No in step S210), the flight control unit 2 causes the moving body 1 to continue the normal flight (returns to step S209). When it is determined that the obstacle on the flight path has been avoided (Yes in step S210), the camera control unit 8 outputs a command to the skid signal generation unit 9, and SLAM shooting is performed by the obstacle detection camera 6. Is started again (returning to step S201).

[Exception handling 2: Aerial photography with aerial photography camera 7]
The camera control unit 8 determines whether or not the aerial camera 7 is shooting based on the command output to the aerial camera 7 and / or the captured image of the aerial camera 7 (step S204). If it is not determined that the aerial camera 7 is shooting (No in step S204), the flight control unit 2 and the camera control unit 8 continue the current operation (shift to step S205).

When it is determined that the aerial camera 7 is shooting (Yes in step S204), the camera control unit 8 outputs a command to the skid signal generation unit 9 and SLAM in the obstacle detection camera 6. Shooting is finished (step S208). Further, the flight control unit 2 ends the lateral slip movement and causes the moving body 1 to perform a normal flight (step S209). The reason for ending SLAM shooting and lateral movement and returning to normal flight when the aerial camera 7 is shooting is that the behavior of the moving body 1 is unstable during the lateral movement, and the sky by the aerial camera 7 This is because aerial photography is performed by the aerial photography camera 7 during normal flight because it becomes difficult to take pictures. In addition, since effective SLAM imaging cannot be performed during normal flight without lateral shift movement, SLAM imaging is stopped.

The camera control unit 8 determines whether or not the aerial photography by the aerial photography camera 7 is completed based on the command output to the aerial photography camera 7 and / or the captured image of the aerial photography camera 7 (step). S210). If it is not determined that the aerial photography by the aerial photography camera 7 is completed (No in step S210), the flight control unit 2 causes the moving body 1 to continue the normal flight (returns to step S209). When it is determined that the aerial photography by the aerial photography camera 7 is completed (Yes in step S210), the camera control unit 8 outputs a command to the skid signal generation unit 9, and the obstacle detection camera 6 is used. SLAM shooting is started again (returning to step S201).

[Exception handling 3: Reception of operator's instruction via communication unit 10]
The flight control unit 2 and / or the camera control unit 8 determines whether or not an operator instruction (control signal) has been received from the remote controller on the ground via the communication unit 10 (step S204). The contents of the operator's instructions are, for example, flight control according to the operator's operation, manual shooting according to the operator's operation, and the like. If it is not determined that the operator's instruction has been received (No in step S204), the flight control unit 2 and the camera control unit 8 continue the current operation (shift to step S205). In the present embodiment, even if the instruction is given from the remote controller on the ground, the current operation is continued when the operator does not manually operate the moving body 1 such as changing the setting of the flight path for autonomous flight. The flight route setting change is, for example, a change in the location of a destination, a deletion / addition of a part of a plurality of destinations, or the like. The destination is, for example, a destination, a WP (waypoint), or the like.

When it is determined that the operator's instruction has been received (Yes in step S204), the flight control unit 2 and / or the camera control unit 8 outputs a command to the skid signal generation unit 9, and the obstacle detection camera 6 SLAM shooting in (step S208) is completed. Further, the flight control unit 2 ends the lateral slip movement and causes the moving body 1 to perform a normal flight (step S209). At this time, the flight control unit 2 and / or the camera control unit 8 switches from the automatic mode to the manual mode. The reason for ending SLAM shooting and lateral movement and returning to normal flight when instructed by the operator is that the moving body 1 moves laterally when the operator attempts to control the moving body 1 or take a camera image by manual operation. This is because the operator's operation is not reflected well, or the operator feels uncomfortable with unintended behavior.

The flight control unit 2 and / or the camera control unit 8 determines whether or not the received operator's instruction has been completed (step S210). If it is not determined that the received operator's instruction is completed (No in step S210), the flight control unit 2 causes the moving body 1 to continue the normal flight (returns to step S209). When it is determined that the received operator's instruction is completed (Yes in step S210), the flight control unit 2 and / or the camera control unit 8 outputs a command to the skid signal generation unit 9 for obstacle detection. SLAM shooting with the camera 6 is started again (returning to step S201). At this time, the flight control unit 2 and / or the camera control unit 8 switches from the manual mode to the automatic mode.

1.3. Lateral movement for SLAM The moving body 1 according to the embodiment of the present disclosure controls the flight path and carries out SLAM with a monaural camera. FIG. 4 is a conceptual diagram for explaining parallax for SLAM. FIG. 5 is a conceptual diagram for explaining the lateral displacement movement of the moving body 1 according to the embodiment of the present disclosure for SLAM. As shown in FIG. 4, when the moving body 1 shoots an object OBJ (Object) in SLAM, it shoots at two points, a first shooting point P1 and a second shooting point P2, and between the two points. The position and orientation of the object OBJ and the depth D (Depth) to the object OBJ are obtained by using the parallax of. Here, the distance between two points is represented by the baseline length L (Baseline Length). The parallax is represented by an angle θ. Depth D represents the distance to the object OBJ. For example, if the angle θ is constant, the longer the baseline length L, the longer the depth D, and SLAM can recognize even a distant object OBJ.

When performing SLAM, if the moving body 1 is equipped with a stereo camera (binocular camera), it is possible to shoot from multiple viewpoints at the same time, but due to the restrictions of the aircraft, only a monaural camera (monocular camera) is available. If it is not installed, you cannot shoot from multiple viewpoints at the same time.

When SLAM is performed with a monaural camera without using a stereo camera, for example, as shown in FIG. 5, the moving body 1 travels in a straight line forward in order to acquire the parallax shown in FIG. By alternately taking paths that are shifted to the left and right with respect to the direction of travel, a shift in position corresponding to parallax is created, and the flight is made so as to enable accurate SLAM. For example, during autonomous flight of the moving body 1, even if the moving body 1 is flying straight along the flight path, it moves laterally with a width of 1 m to 2 m (lateral shift width SW) for SLAM. As shown in FIG. 5, the lateral displacement width SW is the deviation width of only one side from the central axis (original flight path), and the value obtained by doubling the lateral displacement width SW is the baseline length L. In addition, since the angle of view (image quality) of the aerial camera 7 is affected during shooting with the aerial camera 7 (aerial shooting), the status of whether or not the aerial camera 7 is shooting is continuously displayed. Judgment is made, it is detected that the aerial photography camera 7 is not shooting, and the lateral shift movement is controlled.

In SLAM using a monaural camera, pixels having features in each image are obtained for images captured by the camera continuously, and for each of the pixels (plurality of pixels) on the images that are continuous in time. By calculating the correlation, the position and orientation of the object OBJ between the images can be obtained, and at the same time, the depth D to the object OBJ represented by the pixels can be relatively obtained by the principle of triangulation.

Since it is based on triangulation, if the distance from the image to the image is not sufficient in the direction intersecting the object, the accuracy of the depth D deteriorates. Therefore, by moving in a direction that intersects the direction of travel horizontally during flight, the depth D to the object can be obtained more accurately, and if the object is an obstacle, have a sufficient distance. You will be able to take evasive movements.

Further, the moving body 1 according to the embodiment of the present disclosure generates a skid signal in real time and applies it to an output signal for flight control. The reason for generating the skid signal in real time is that when the skid pattern is generated in advance in the form of a flight path, the frequency of strike slip occurrence seems to be much higher than the number of WPs (waypoints) of the flight path. This is because the file size of flight route information becomes large and the management effort increases. For example, if a strike-slip of 2 m is always performed at intervals of 10 seconds, 60 times occur in a flight time of 10 minutes, and if all of them are represented by WP, 60 WPs are required. In addition, it is difficult to deal with the effects of disturbances such as gusts.

Further, when the assumed flight speed cannot be obtained, the moving body 1 according to the embodiment of the present disclosure generates a skid signal from the strike-slip width SW, the strike-slip pattern SP, and the flight speed during flight, and is used for flight control. It is applied to the output signal of. That is, the skid signal is generated by adding the velocity to the input parameter. For example, when the speed is very low, it may not be possible to achieve the strike-slip width SW within the set strike-slip time ST even if the side slips as rapidly as possible. In this case, the parameters may be divided into a low speed range and a high speed range, and the strike-slip width SW, the strike-slip time ST, and the strike-slip pattern SP may be set for each of the low-speed range and the high-speed range. .. Further, the strike-slip width SW, the strike-slip time ST, and the strike-slip pattern SP may be set for each predetermined speed.

Here, with reference to FIGS. 6A to 6D, an example of a lateral displacement pattern SP drawn by the body of the moving body 1 in the horizontal direction will be described. FIG. 6A is an image diagram showing an example of a pattern in which the body of the moving body 1 flies in a sine wave. FIG. 6B is an image diagram showing an example of a pattern in which the body of the moving body 1 flies in a rectangular wave (square wave). FIG. 6C is an image diagram showing an example of a pattern in which the body of the moving body 1 flies in a triangular wave. FIG. 6D is an image diagram showing an example of a pattern in which the body of the moving body 1 flies in a sawtooth wave (sawtooth wave). The waveform of the strike-slip pattern SP is preferably a step wave (diagonal edge) rather than a pulse wave, and may have rounded corners. For example, the square wave may be a trapezoidal wave or an S-shaped wave.

1.4. Differences from SLAM using a stereo camera Next, the differences from SLAM using a stereo camera will be described. Compared to the baseline length of the stereo camera that can be mounted on the body of the moving body 1, the baseline length that can be achieved by lateral movement (lateral movement) is not limited in length, so it is possible to detect objects (obstacles). The distance can be farther than that of a stereo camera. That is, the baseline length of the stereo camera is limited to the width between the cameras (between the lenses), but in the case of lateral shift movement, the degree of freedom is higher than the width between the cameras of the stereo camera.

In addition, SLAM using a stereo camera also has restrictions on the airframe. The baseline length (distance between cameras) of a stereo camera required to detect an obstacle is 2.5 cm if the obstacle is 10 m away, 25 cm if it is 100 m away, and several meters beyond that (a few meters away). For example, 2.5m is required for 1km ahead). There is a need to reduce the weight of the aircraft, and it is difficult to maintain the position and attitude of cameras that are several meters apart with a moving object that flies at a speed of several tens of kilometers per hour while being blown by the wind outdoors. Further, for example, the size of a fixed-wing drone for logistics as a platform depends on the flight time and the maximum speed, but when the payload is 1 kg, the wingspan is about 2 m. Therefore, the baseline length (distance between cameras) of a stereo camera is limited to the wingspan or less. If the mobile body 1 and the flight path control method thereof according to the embodiment of the present disclosure are adopted, the restrictions of the airframe can be released.

1.5. Modification example Performing SLAM using a stereo camera on a moving object during normal flight (during straight flight along the flight path) has restrictions as described above, but one or both of the stereo cameras The method described in the present disclosure may be applied by regarding an image synthesized by an operation such as removing a difference caused by an object at a short distance from the camera as an image taken by a monaural camera. That is, the stereo camera may be used as a pseudo monaural camera. In this case, a stereo camera can be used as the obstacle detection camera 6.

Further, when the aircraft of the moving body 1 according to the embodiment of the present disclosure is a vertical takeoff and landing aircraft (VTOL), the SLAM is initialized not only during straight-ahead travel but also during vertical ascent immediately after takeoff and in a hovering state. It may be laterally displaced in the left-right direction with respect to the nose direction (camera direction) for reinitialization.

Further, in the above description, the skid signal generation unit 9 outputs the SLAM image flag to the camera control unit 8 when a certain time has elapsed since the skid signal was generated. However, in reality, the camera control unit 8 also receives the skid signal from the skid signal generation unit 9 at the same time as the flight control unit 2, and when a certain time elapses after receiving the skid signal, the obstacle at that time is present. The SLAM image flag may be set in the captured image of the detection camera 6. That is, the camera control unit 8 may count a certain period of time.

Further, in the above description, the moving body 1 is laterally displaced in the direction horizontally intersecting the traveling direction during autonomous flight, but similarly, the moving body 1 is moved up and down with respect to the traveling direction. It may be "moved vertically" in the direction of intersection. For example, the lateral shift movement and the vertical shift movement may be alternately performed at regular intervals. Further, the vertical shift movement may be performed when passing through the central axis (original flight path) in the middle of the lateral shift movement. Although lateral displacement movement is indispensable, it may be possible to select whether or not vertical displacement movement is performed. The parameters in the case of the vertical shift movement are, for example, information regarding the vertical slip width DW (Dip-slip Width), the vertical slip time DT (Dip-slip Time), and the vertical slip pattern DP (Dip-slip Pattern). The vertical shift pattern DP is, for example, a sine wave, a rectangular wave, a triangular wave, or a sawtooth wave, as in the case of the lateral shift pattern SP. Therefore, the moving body 1 flies vertically with respect to the traveling direction in a pattern as shown in FIGS. 6A to 6D by moving vertically during autonomous flight. The height and height difference of the object can be recognized by taking a SLAM image of the moving body 1 by moving it vertically. This makes it possible to create a highly accurate three-dimensional map in creating an environmental map.

Further, the operation related to SLAM of the mobile body 1 according to the embodiment of the present disclosure may be executed by a computer operating according to the program. For example, a CPU or the like mounted on the mobile body 1 may read and execute a program that defines operations related to SLAM to function as a flight control unit 2, a camera control unit 8, and a skid signal generation unit 9. .. Further, the above program may be a flight simulation program that reproduces the operation of the mobile body 1 according to the embodiment of the present disclosure regarding SLAM on a computer.

2. Conclusion Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can come up with various modifications or modifications within the scope of the technical ideas described in the claims. Of course, it is understood that the above also belongs to the technical scope of the present disclosure.

In addition, the effects described herein are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

The present technology can also have the following configurations.
(1)
A flight device for autonomous flight on a specified flight path, and
An imaging device that automatically captures images continuously during autonomous flight on the flight path,
A cross movement is performed in which the flight path is controlled to move in a direction intersecting the traveling direction during autonomous flight, and self-position estimation is performed using an image captured by the image pickup device taken during the cross movement. And a control device that creates an environmental map,
A mobile body equipped with.
(2)
The control device controls the flight device to move it laterally in a direction that intersects the traveling direction of the flight path horizontally, and alternately takes a path shifted to the left and right, which corresponds to the parallax of the image pickup device. The moving body according to (1), which creates a displacement of the position to be operated.
(3)
The control device obtains pixels having features in each captured image with respect to the captured image of the imaging device, calculates a correlation for each of the pixels on the captured images that are continuous in time, and inter-captured images. The moving body according to (1) or (2), wherein at the same time as obtaining the position and orientation of the above, the depth to the object represented by the pixel is relatively obtained by the principle of triangular surveying.
(4)
When the control device detects an obstacle on the flight path based on the captured image of the image pickup device, the control device stops the crossing movement by the flight device, according to any one of (1) to (3). The described mobile body.
(5)
In addition to the imaging device, an imaging device for aerial photography that performs aerial photography is further provided.
The moving body according to any one of (1) to (4), wherein the control device stops the crossing movement by the flight device when the aerial photography imaging device starts aerial photography.
(6)
Further equipped with a communication device for receiving operator instructions,
The moving body according to any one of (1) to (5), wherein when the control device receives an instruction from the operator via the communication device, the crossing movement by the flight device is stopped.
(7)
The control device continues the cross-movement by the flight device when the operator's instruction changes the setting of the flight path, and stops the cross-movement by the flight device when the operator's instruction is a manual operation. , (6).
(8)
When the crossing movement by the flight device in one direction is normally completed without being stopped, the control device sets a flag for the captured image of the image pickup device at that time, from (1) to (7). ) Is described in any one of the paragraphs.
(9)
The moving body according to (8), wherein the control device performs the self-position estimation and the environment map creation using the captured image of the image pickup device with the flag set.
(10)
The moving body according to (8) or (9), wherein the control device reverses the moving direction of the crossing movement by the flying device after setting the flag.
(11)
The item according to any one of (1) to (10), wherein the control device controls crossing movement by the flight device based on parameters defining a movement width SW, a movement time ST, and a movement pattern SP. Mobile body.
(12)
The moving body according to (11), wherein the control device controls crossing movement by the flight device according to a control signal in which the flight speed of the flight device is added to the parameters.
(13)
The moving body according to (11) or (12), wherein the moving pattern SP is any one of a sine wave, a square wave, a triangular wave, and a sawtooth wave.
(14)
The control device controls the flight device during autonomous flight to move the flight path vertically in a direction intersecting the direction of travel, and captures an image taken by the image pickup device during the vertical shift movement. The moving body according to any one of (1) to (13), wherein the environmental map is created by using the moving body.
(15)
The moving body according to any one of (1) to (14), wherein the imaging device is a monocular camera.
(16)
To make a moving object autonomously fly along a specified flight path,
During autonomous flight of the flight path, continuous and automatic image capture using an imaging device mounted on the moving body, and
Performing cross-movement in which the moving body is moved in a direction intersecting the traveling direction during autonomous flight of the flight path, and
Self-position estimation and environment map creation are performed using the images taken by the image pickup device taken during the crossing movement, and
A method of controlling the flight path of a moving body including.
(17)
Steps to autonomously fly a moving object along a specified flight path,
A step of automatically continuously taking an image using an image pickup device mounted on the moving body during autonomous flight of the flight path, and a step of automatically taking an image.
A step of performing cross-movement in which the moving body is moved in a direction intersecting the traveling direction during autonomous flight of the flight path, and
A step of performing self-position estimation and environment map creation using the captured image of the imaging device taken during the crossing movement, and
A program that lets your computer run.

1 Mobile 2 Flight control unit 3 Flight equipment 4 Flight path storage unit 5 Position information acquisition unit 6 Obstacle detection camera 7 Aerial photography camera 8 Camera control unit 9 Side slip signal generation unit 10 Communication unit

Claims (17)

A flight device for autonomous flight on a specified flight path, and
An imaging device that automatically captures images continuously during autonomous flight on the flight path,
A cross movement is performed in which the flight path is controlled to move in a direction intersecting the traveling direction during autonomous flight, and self-position estimation is performed using an image captured by the image pickup device taken during the cross movement. And a control device that creates an environmental map,
A mobile body equipped with. The control device controls the flight device to move it laterally in a direction that intersects the traveling direction of the flight path horizontally, and alternately takes a path shifted to the left and right, which corresponds to the parallax of the image pickup device. The moving body according to claim 1, which creates a displacement of the position to be operated. The control device obtains pixels having features in each captured image with respect to the captured image of the imaging device, calculates a correlation for each of the pixels on the captured images that are continuous in time, and inter-captured images. The moving body according to claim 1, wherein the position and the orientation of the image are obtained, and at the same time, the depth to the object represented by the pixel is relatively obtained by the principle of triangulation. The moving body according to claim 1, wherein when the control device detects an obstacle on the flight path based on the captured image of the imaging device, the crossing movement by the flight device is stopped. In addition to the imaging device, an imaging device for aerial photography that performs aerial photography is further provided.
The moving body according to claim 1, wherein the control device stops the crossing movement by the flight device when the aerial photography imaging device starts aerial photography. Further equipped with a communication device for receiving operator instructions,
The moving body according to claim 1, wherein when the control device receives an instruction from the operator via the communication device, the crossing movement by the flight device is stopped. The control device continues the cross-movement by the flight device when the operator's instruction changes the setting of the flight path, and stops the cross-movement by the flight device when the operator's instruction is a manual operation. , The moving body according to claim 6. The control device according to claim 1, wherein when the crossing movement by the flight device in one direction is normally completed without being stopped, a flag is set for the captured image of the image pickup device at that time. Mobile body. The moving body according to claim 8, wherein the control device performs the self-position estimation and the environment map creation using the captured image of the image pickup device with the flag set. The moving body according to claim 8, wherein the control device reverses the moving direction of the crossing movement by the flying device after setting the flag. The moving body according to claim 1, wherein the control device controls crossing movement by the flight device based on parameters defining a movement width, a movement time, and a movement pattern. The moving body according to claim 11, wherein the control device controls crossing movement by the flight device according to a control signal in which the flight speed of the flight device is added to the parameters. The moving body according to claim 11, wherein the moving pattern is any one of a sine wave, a square wave, a triangular wave, and a sawtooth wave. The control device controls the flight device during autonomous flight to move the flight path vertically in a direction intersecting the direction of travel, and captures an image taken by the image pickup device during the vertical shift movement. The moving body according to claim 1, wherein the environmental map is created by using the moving body. The moving body according to claim 1, wherein the imaging device is a monocular camera. To make a moving object autonomously fly along a specified flight path,
During autonomous flight of the flight path, continuous and automatic image capture using an imaging device mounted on the moving body, and
Performing cross-movement in which the moving body is moved in a direction intersecting the traveling direction during autonomous flight of the flight path, and
Self-position estimation and environment map creation are performed using the images taken by the image pickup device taken during the crossing movement, and
A method of controlling the flight path of a moving body including. Steps to autonomously fly a moving object along a specified flight path,
A step of automatically continuously taking an image using an image pickup device mounted on the moving body during autonomous flight of the flight path, and a step of automatically taking an image.
A step of performing cross-movement in which the moving body is moved in a direction intersecting the traveling direction during autonomous flight of the flight path, and
A step of performing self-position estimation and environment map creation using the captured image of the imaging device taken during the crossing movement, and
A program that lets your computer run.

Images