WO2021149546A1

WO2021149546A1 - Information processing device

Info

Publication number: WO2021149546A1
Application number: PCT/JP2021/000781
Authority: WO
Inventors: ▲高▼橋　誠; 鷹見　忠雄; 寛河上; 香緒莉新畑
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2020-01-21
Filing date: 2021-01-13
Publication date: 2021-07-29
Also published as: JP7461384B2; JPWO2021149546A1

Abstract

A sensor obtainment unit 101 obtains a captured image captured by a stereo camera 17 vertically below a host machine. A distance calculation unit 102 calculates a distance for each of pixels in the obtained captured image. A danger level determination unit 103 determines a danger level for landing in each of a plurality of segmented regions vertically below the host machine, on the basis of the calculated distance for each of the pixels. A flight instruction unit 104 repeatedly makes an instruction to move the host machine in a direction and an altitude according to an arrangement of the segmented regions in which the determined danger level is at least a first standard. When the altitude drops below a predetermined altitude, an obstruction detection unit 107 performs processing for detecting an obstruction from the obtained captured image. When no obstruction is detected, or when a landing condition is met despite an obstruction being detected, a landing instruction unit 109 instructs a flight control unit 105 to land the host machine based on the calculated distance for each of the pixels, and causes landing control to start.

Description

Information processing device

The present invention relates to a technique for landing an air vehicle.

As a technique for landing an air vehicle, Patent Document 1 states that a GPS mounted on an unmanned airplane is used to fly to near the landing point, and then a target mark placed at the landing point is captured by an imaging device, aiming at that. A technique for landing and a technique for landing so that the difference between the position information measured by the GPS of the takeoff and landing platform and the position information of the drone are made zero are disclosed.

JP-A-2018-190362

When landing an aircraft such as a drone, if it collides with an obstacle (human, vehicle, building, etc.) near the landing point, the aircraft will be damaged as well as the obstacle (especially in the case of humans, injuries). ) May also lead to.
Therefore, an object of the present invention is to reduce the possibility of colliding with an obstacle when the flying object lands.

In order to achieve the above object, the present invention relates to an acquisition unit that acquires an image taken by a stereo camera provided on the flying object, and a distance from the stereo camera to a point indicated by each pixel of the acquired image. A calculation unit that calculates the distance, a detection unit that detects an obstacle from the acquired image, and a calculated distance if the position of the obstacle satisfies the landing condition even if the obstacle is detected. Provided is an information processing apparatus including a landing instruction unit for instructing the landing of the aircraft based on the above.

According to the present invention, it is possible to reduce the possibility of colliding with an obstacle when the flying object lands.

The figure which shows an example of the hardware configuration of the drone which concerns on 1st Example Diagram showing the functional configuration realized by the drone Diagram showing an example of a divided area Diagram showing an example of a weight table Diagram showing an example of the risk table The figure which shows the example of the relationship between the arrangement of the division area and the movement direction. Diagram showing an example of the damage degree table Diagram showing an example of the determined damage level The figure which shows an example of the operation procedure in a landing process Diagram showing an example of the overall configuration of the landing support system according to the second embodiment Diagram showing an example of the hardware configuration of the server device Diagram showing the functional configuration realized in the embodiment The figure which shows an example of the operation procedure in the landing process of an Example.

[1] First Example FIG. 1 shows an example of the hardware configuration of the drone 10 according to the first embodiment. The drone 10 is physically configured as a computer device including a processor 11, a memory 12, a storage 13, a communication device 14, a flight device 15, a sensor device 16, a stereo camera 17, a bus 18, and the like. May be done. In the following description, the word "device" can be read as a circuit, a device, a unit, or the like.

Further, each device may be included one or more, or some devices may not be included. The processor 11 operates, for example, an operating system to control the entire computer. The processor 11 may be configured by a central processing unit (CPU: Central Processing Unit) including an interface with a peripheral device, a control device, an arithmetic unit, a register, and the like.

For example, the baseband signal processing unit and the like may be realized by the processor 11. Further, the processor 11 reads a program (program code), a software module, data, and the like from at least one of the storage 13 and the communication device 14 into the memory 12, and executes various processes according to the read program and the like. As the program, a program that causes a computer to execute at least a part of the operations described in the above-described embodiment is used.

Although it has been explained that the various processes described above are executed by one processor 11, they may be executed simultaneously or sequentially by two or more processors 11. The processor 11 may be implemented by one or more chips. The program may be transmitted from the network via a telecommunication line. The memory 12 is a computer-readable recording medium.

The memory 12 may be composed of at least one such as a ROM (ReadOnlyMemory), an EPROM (ErasableProgrammableROM), an EPROM (ElectricallyErasableProgrammableROM), and a RAM (RandomAccessMemory). The memory 12 may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory 12 can store a program (program code), a software module, or the like that can be executed to implement the wireless communication method according to the embodiment of the present disclosure.

The storage 13 is a computer-readable recording medium, and is, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, an optical magnetic disk (for example, a compact disk, a digital versatile disk, or a Blu-ray). It may consist of at least one (registered trademark) disk), smart card, flash memory (eg, card, stick, key drive), floppy (registered trademark) disk, magnetic strip, and the like.

The storage 13 may be called an auxiliary storage device. The storage medium described above may be, for example, a database, server or other suitable medium containing at least one of the memory 12 and the storage 13. The communication device 14 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network.

For example, the above-mentioned transmission / reception antenna, amplifier unit, transmission / reception unit, transmission line interface, and the like may be realized by the communication device 14. The transmission / reception unit may be physically or logically separated from each other in the transmission unit and the reception unit. Further, each device such as the processor 11 and the memory 12 is connected by a bus 18 for communicating information. The bus 18 may be configured by using a single bus, or may be configured by using a different bus for each device.

The flight device 15 is a device provided with a motor, a rotor, etc., to fly its own aircraft. The flight device 15 can move its own aircraft in all directions and make its own aircraft stationary (hovering) in the air. The sensor device 16 is a device having a sensor group for acquiring information necessary for flight control. The sensor device 16 includes, for example, a position sensor that measures the position (latitude and longitude) of the own machine.

Further, the sensor device 16 includes a direction sensor for measuring the direction in which the own machine is facing (the direction in which the drone is facing the front direction of the own machine and the determined front direction is facing), and the own machine. It is equipped with an altitude sensor that measures altitude. Further, the sensor device 16 includes a speed sensor for measuring the speed of the own machine and an inertial measurement sensor (IMU (Inertial Measurement Unit)) for measuring the angular speed of three axes and the acceleration in three directions.

The stereo camera 17 is a camera provided with a plurality of cameras and capable of recording information in the depth direction of the object by photographing the object from a plurality of directions with the plurality of cameras. The stereo camera 17 includes two cameras in this embodiment. Each camera includes an optical system including a lens and an image sensor. The stereo camera 17 is provided so as to take a picture vertically below the own machine.

In addition, the above-mentioned device includes hardware such as a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). It may be configured. Further, in the above-mentioned device, a part or all of each functional block may be realized by the hardware. For example, the processor 11 may be implemented using at least one of the hardware.

Each function in the drone 10 is such that the processor 11 performs an operation by loading predetermined software (program) on the hardware such as the processor 11 and the memory 12, and controls the communication by the communication device 14, or the memory 12 and the memory 12. It is realized by controlling at least one of reading and writing of data in the storage 13.

FIG. 2 shows the functional configuration realized by the drone 10. The drone 10 is photographed by the sensor acquisition unit 101, the distance calculation unit 102, the risk determination unit 103, the flight instruction unit 104, the flight control unit 105, the damage scale determination unit 106, the obstacle detection unit 107, and the like. It includes an instruction unit 108 and a landing instruction unit 109. The sensor acquisition unit 101 acquires the measured value of the sensor provided in the own aircraft (drone 10) which is an air vehicle.

In this embodiment, the sensor acquisition unit 101 acquires an image vertically below the own machine taken by the stereo camera 17 provided in the own machine as the measured value of the image sensor. In other words, the sensor acquisition unit 101 acquires a pixel value indicating an image vertically below the own machine output by the image sensor of the stereo camera 17 as a measured value of the sensor. The sensor acquisition unit 101 is an example of the “acquisition unit” of the present invention. The sensor acquisition unit 101 supplies the acquired image to the distance calculation unit 102.

The distance calculation unit 102 calculates the distance from the stereo camera 17 (hereinafter referred to as "distance for each pixel") to the point where the object indicated by each pixel of the image acquired by the sensor acquisition unit 101 exists. The distance calculation unit 102 is an example of the "calculation unit" of the present invention. Since the image captured by the stereo camera 17 captures the same point from a plurality of angles, the distance from the stereo camera 17 to the point where the object reflected in each pixel exists can be calculated.

Here, the captured image does not necessarily show the ground, but may show any object such as a building, a natural object, a water surface, or an animal. In this embodiment, the distance from the stereo camera 17 can be regarded as the distance from the drone 10. As a method for calculating the distance for each pixel, a well-known technique such as that described in JP-A-11-230745 may be used.

When the distance calculation unit 102 calculates the distance for each pixel, the distance calculation unit 102 supplies the distance information indicating the calculated distance for each pixel to the risk determination unit 103. Based on the measured values acquired by the sensor acquisition unit 101, the risk level determination unit 103 determines the risk level at the time of landing of each of the plurality of divided areas that divide the above-ground area vertically below the own aircraft.

The degree of danger at the time of landing is the high possibility that it will not be possible to land safely. A safe landing is a landing without any damage. In other words, the degree of danger is, for example, the drone 10 itself due to contact with some object (including humans and other animals) during landing or disturbance of the attitude at the time of landing (including the case of falling as a result). It represents the high possibility of damage or damage (including injury) to an object that the drone 10 comes into contact with.

FIG. 3 shows an example of the divided area. In this embodiment, the risk determination unit 103 divides four square areas having the own machine as the apex and having the same side length into divided areas B1, B2, and B3 when the own machine is viewed vertically downward from the sky. , Used as B4. The risk level determination unit 103 determines the risk level based on the ratio of the unsuitable area that is not suitable for landing in the divided area, which is calculated based on the measured value acquired by the sensor acquisition unit 101.

Unsuitable areas are areas such as commercial areas, residential areas, industrial areas, and parks where people gather and are more likely to injure people when falling than other areas. In addition, the unsuitable area is an area where there is a higher possibility that the drone 10 will be damaged by contacting or falling over some object when trying to land, such as a forest, slope, utility pole, vehicle, etc., as compared with other areas. You may.

In this embodiment, the risk determination unit 103 first recognizes an object existing on the ground from the distance indicated by the supplied distance information, that is, the distance for each pixel calculated by the distance calculation unit 102. Objects such as buildings, vehicles, people, animals, utility poles, trees, forests, slopes and flatlands may exist on the ground. The risk determination unit 103 calculates the height of the object indicated by each pixel from the ground from the distance between each pixel included in the portion representing the object in the image and the own machine and the altitude of the own machine.

If the height of each pixel representing an object is known, the outline of the object in the three-dimensional space will be represented. The risk determination unit 103 stores a pattern of shape and size for each type of object, and recognizes that the object corresponds to the pattern when the similarity with the stored pattern exceeds the threshold value. In this way, the risk determination unit 103 recognizes an object existing on the ground from the pixel values acquired by the sensor acquisition unit 101.

Then, the risk level determination unit 103 determines the risk level based on the ratio of the unsuitable area calculated by weighting the area occupied by the recognized object according to the attribute of the object. The risk determination unit 103 uses a weight table in which the attributes of the object and the weights are associated with each other.
FIG. 4 shows an example of a weight table. In the example of FIG. 4, "1.0" is for "slope", "1.5" is for "buildings, utility poles, trees", "2.0" is for "vehicles, people, animals", and so on. Object attributes and weights are associated.

When the risk determination unit 103 recognizes an object existing in the divided region, it calculates for all the recognized objects by multiplying the number of pixels of the object by a weight according to the attribute of the object. Then, the risk determination unit 103 calculates the value obtained by dividing the total value of the calculated values by the number of pixels of the entire divided region as the ratio of the unsuitable region. The risk level determination unit 103 determines the risk level using, for example, a risk level table in which the ratio of the unsuitable region and the risk level are associated with each other.

FIG. 5 shows an example of a risk table. In the example of FIG. 5, the risk levels of "danger Lv1", "danger Lv2", and "danger Lv3" are associated with the proportions of the unsuitable regions "less than Th11", "Th11 or more and less than Th12", and "Th12 or more". ing. Th11 and Th12 are thresholds for the proportion of unsuitable regions, such as 30% and 60%. The higher the value, the higher the risk. The risk determination unit 103 determines, for example, the risk of the divided region in which the ratio of the unsuitable region is Th11 or more and less than Th12 as danger Lv2.

When the risk level determination unit 103 determines the risk level of each divided region, the risk level determination unit 103 supplies the determination result to the flight instruction unit 104. The flight instruction unit 104 gives an instruction to move the own aircraft (drone 10) in a direction corresponding to the arrangement of the divided regions in which the degree of danger determined by the degree of danger determination 103 is equal to or higher than a predetermined reference.

The flight instruction unit 104 gives the above instructions to the flight control unit 105. The flight control unit 105 controls the flight of its own aircraft (drone 10) based on various values (position, altitude, direction, etc.) measured by the sensor device 16. When the flight instruction unit 104 gives an instruction to move the flight control unit 105, the flight control unit 105 controls to move the aircraft in the instructed direction.

FIG. 6 shows an example of the relationship between the arrangement of the divided areas and the moving direction. In FIG. 6, it is assumed that the standard risk level is set to Lv3. When there are three division regions of danger Lv3 as shown in FIG. 6A, the flight instruction unit 104 instructs the direction A1 toward the center of the division region where the remaining one risk level is less than the reference as the movement direction. In this embodiment, the flight instruction unit 104 instructs the movement to the center C1 of the division region B1 in the direction A1.

When the flight instruction unit 104 has two divided regions of danger Lv3 arranged side by side as shown in FIG. 6B, the movement direction is the direction A2 toward the boundary between the divided regions where the remaining two danger levels are less than the standard. Instruct. In this case, in this embodiment, the flight instruction unit 104 instructs the movement to the center C2 of the sides of the divided regions B1 and B2 in the direction A2. The flight instruction unit 104 may instruct the movement to the apex C3 on the diagonal line of the division region B1 in the case of FIG. 6A.

Further, in the case of FIG. 6B, the movement to the end C4 on the opposite side of the sides of the divided regions B1 and B2 may be instructed. However, in the following description, it is assumed that the movement to the position corresponding to the centers C1 and C2 is instructed. When the flight instruction unit 104 has only one division region of danger Lv3 as shown in FIG. 6C, the movement direction is the direction A3 toward the center of the division region facing the division region across the drone 10. Instruct.

Further, when the flight instruction unit 104 has only two division regions of danger Lv3 and is not adjacent to each other as shown in FIG. 6D, the center of any of the two division regions adjacent to the division regions. The direction A4-1 or the direction A4-2 toward the direction A4-1 is indicated as the moving direction. The flight instruction unit 104 may randomly select either the direction A4-1 or the direction A4-2, or may compare the risk levels of the corresponding divided regions and select the smaller one.

Further, if the risk levels of the two divided regions are the same, the flight instruction unit 104 may compare the ratios of the unsuitable regions and select the smaller one. Further, when the flight instruction unit 104 has two divided regions of danger Lv3 arranged side by side, the flight instruction unit 104 may instruct the direction toward the boundary between the divided regions whose remaining two danger levels are less than the reference as the moving direction. If the risk levels of the remaining two divided areas are less than the standard as shown in FIG. 6 (e) but the levels are different, the direction A5 toward the center of the divided areas with the lower risk is designated as the moving direction. May be good.

Depending on the area where the drone 10 is flying, it is possible that all four division areas are dangerous Lv3. In that case, the flight instruction unit 104 attempts the following two flight instructions. When there is no divided region in which the risk level determined by the risk level determination unit 103 is less than the first reference, the flight instruction unit 104 first instructs the flight instruction unit 104 to increase the altitude and then acquire the pixel value again.

Specifically, the flight instruction unit 104 instructs the flight control unit 105 to raise the altitude, and instructs the sensor acquisition unit 101 to reacquire the pixel value. When the altitude is increased according to the instruction and the pixel value is acquired, the risk level determination unit 103 determines the risk level of the divided region that shows a wider range than before the altitude was increased. As a result, there is a possibility that there is a divided area where the risk level is less than the first criterion, so that it is possible to attempt landing after moving to an area where a safer landing is possible.

The flight instruction unit 104 may repeatedly give the above-mentioned instruction to raise the altitude when the division area whose risk level is less than the first criterion is not found. However, if the division area where the risk level is less than the first criterion is not found even after repeating the instruction a fixed number of times (for example, the number of times when the altitude at which flight is prohibited) is repeated, the flight instruction unit 104 fails to land. Instruct the movement in consideration of the damage at the time.

The damage scale determination unit 106 determines the degree of damage indicating the magnitude of damage when landing fails for each of the plurality of divided regions. The damage scale determination unit 106 is supplied with distance information indicating the distance for each pixel from the distance calculation unit 102. The damage scale determination unit 106 recognizes an object existing on the ground from the distance for each pixel indicated by the supplied distance information, for example, by a method using the same pattern as the risk determination unit 103. The damage scale determination unit 106 identifies the attributes of each divided region from the recognized object.

The damage scale determination unit 106 specifies, for example, a divided area in which trees and slopes are included in a certain proportion or more as "forest". In addition, the damage scale determination unit 106 identifies the divided area containing crops and agricultural machinery (tractors, etc.) in a certain proportion or more as "agricultural land", and the factory equipment (pipeline, warehouse, etc.) has a certain proportion. The divided area included above is specified as an "industrial area". In addition, the damage scale determination unit 106 identifies the divided area containing a certain percentage or more of single-family homes and collective housing as a "residential area", and identifies the divided area containing a certain percentage or more of arcades and signboards as a "commercial area". do.

The method of specifying the attribute of the divided area is not limited to this. For example, the sensor acquisition unit 101 acquires position information indicating the position of the own machine as a measured value of the sensor and supplies it to the damage scale determination unit 106.
Then, the damage scale determination unit 106 stores the map information indicating the attribute for each area, and specifies the attribute of the position indicated by the supplied position information. As described above, the damage scale determination unit 106 may use the measured value of the sensor different from that used by the risk determination unit 103 and the like.

Since the distance for each pixel is also a value calculated based on the measured value, in each case, the damage scale determination unit 106 is based on the measured value acquired by the sensor acquisition unit 101 (directly like the position information). The degree of damage is determined based on the case (including the case based on the case and the case based indirectly such as the distance per pixel). The damage scale determination unit 106 determines the damage degree using, for example, a damage degree table in which the attributes of the divided area and the damage degree are associated with each other.

FIG. 7 shows an example of the damage degree table. In the example of FIG. 7, the attributes of the divided area are "damage Lv1" for "forest, agricultural land", "damage Lv2" for "industrial land", and "damage Lv3" for "residential area, commercial land". And the degree of damage are associated with each other. The damage scale determination unit 106 determines the damage degree associated with the attribute of the divided area specified as described above in the damage degree table as the damage degree of the divided area.

When the damage scale determination unit 106 determines the degree of damage for each divided area, the damage scale determination unit 106 supplies the determination result to the flight instruction unit 104. The flight instruction unit 104 moves its own aircraft in the direction of the division area having the smallest damage degree determined by the damage scale determination unit 106 when there is no division area in which the risk degree determined by the danger degree determination unit 103 is less than the standard. Instruct to let it descend.

FIG. 8 shows an example of the determined damage degree. In the example of FIG. 8, the degree of danger of the divided areas B1 to B4 is all Lv3, but the degree of damage is determined to be Lv1 for the divided area B1, Lv2 for the divided area B3, and Lv3 for the divided areas B2 and B4. In this case, the flight instruction unit 104 instructs the direction A6 toward the center of the division region B1 having the least damage degree as the movement direction. By instructing the moving direction in consideration of the degree of damage in this way, even if the landing of the drone 10 fails, the damage caused by it can be minimized.

When the division area where the risk level determined by the risk level determination unit 103 is less than a predetermined standard is in the direction of movement of the own aircraft (drone 10), the flight instruction unit 104 makes a predetermined distance descent. Instruct. In this embodiment, it is assumed that the danger Lv2 is used as the second criterion. In the examples of FIGS. 6A to 6E, the flight instruction unit 104 moves to the divided region of danger Lv1, so that the flight instruction unit 104 also instructs the descent of a predetermined distance.

On the other hand, when the divided region in the moving direction A6 is danger Lv2 as shown in FIG. 6 (f), the flight instruction unit 104 does not instruct the descent but only instructs the movement. The flight control unit 105 controls the aircraft to move in the horizontal direction when only the movement is instructed, and controls the aircraft to move diagonally downward when instructed to move and descend. In the latter case, the flight control unit 105 may control the aircraft so as to move in the horizontal direction and then descend.

The flight instruction unit 104 repeats the above-mentioned movement and descent instructions, and when the altitude of the own aircraft (drone 10) becomes less than a predetermined altitude, the flight instruction unit 105 causes the flight control unit 105 to perform landing control based on the measurement result by the distance measuring means. Instruct. The distance measuring means is a means for measuring the distance to the ground. In this embodiment, the distance calculating unit 102 is used as the distance measuring means, and the distance calculated by the distance calculating unit 102 is used as the measurement result.

The distance calculation unit 102 also supplies distance information indicating the distance for each pixel to the obstacle detection unit 107. The obstacle detection unit 107 detects an obstacle from the image taken by the stereo camera 17 acquired by the sensor acquisition unit 101. The obstacle detection unit 107 is an example of the "detection unit" of the present invention. An obstacle is an object that interferes with the landing of the drone 10, and may be a building, a natural object, an animal (including a human being), or the like.

The obstacle detection unit 107 detects an obstacle when the shape indicated by the calculated distance for the pixel that may indicate the obstacle is within the allowable range as the obstacle. The shape indicated by the distance is the same as the contour of the object used when the risk determination unit 103 recognizes the object. Similar to the risk level determination unit 103, the obstacle detection unit 107 stores a pattern of shape and size for each type of obstacle.

Then, the obstacle detection unit 107 determines that the shape is within the allowable range as an obstacle when the similarity with the stored pattern exceeds the threshold value, and detects it as an obstacle corresponding to the pattern. When the obstacle detection unit 107 detects an obstacle, it supplies pixel information indicating a pixel in which the detected obstacle is captured to the photographing instruction unit 108. When an obstacle is detected by the obstacle detection unit 107, the shooting instruction unit 108 instructs the shooting from another direction of the obstacle. The shooting instruction unit 108 is an example of the “shooting instruction unit” of the present invention.

The shooting instruction unit 108 determines the position of the obstacle on the ground detected from the position of the pixel indicated by the supplied pixel information, and causes the flight control unit 105 to move the aircraft within a range including the position in the angle of view. Instruct. Then, the shooting instruction unit 108 instructs the stereo camera 17 to shoot after moving, thereby instructing the stereo camera 17 to shoot from another direction of the obstacle. The obstacle detection unit 107 adds an image taken according to the shooting instruction of the shooting instruction unit 108, and detects the obstacle again.

By performing re-imaging and re-detection as described above, obstacles photographed from a plurality of angles can be detected, and the area in space indicated by the obstacles can be detected as compared with the detection from only one image. It can be grasped more accurately. The obstacle detection unit 107 supplies the landing instruction unit 109 with pixel information indicating the pixels in which the obstacle detected in this way is captured. The landing instruction unit 109 instructs the flight control unit 105 to land the own aircraft (drone 10) based on the distance for each pixel calculated by the distance calculation unit 102. The landing instruction unit 109 is an example of the "landing instruction unit" of the present invention.

If there are no obstacles in the landing direction, the landing instruction unit 109 will instruct the aircraft to land by lowering the altitude as it is. Further, even if an obstacle is detected in the landing direction, the landing instruction unit 109 of the own aircraft based on the calculated distance for each pixel if the position of the obstacle satisfies the landing condition. Instruct the flight control unit 105 to land. The landing instruction unit 109 determines that the landing condition is satisfied when, for example, the obstacle detected by the obstacle detection unit 107 is not included in the predetermined range of the captured image.

The predetermined range is, for example, a range in the shape of a circle having a predetermined radius from the center of the captured image. The shape of the range is not limited to a circle, and may be an ellipse, a square, a quadrangle, or the like, or may be deviated from the center of the image. In the following, these ranges will be referred to as "ranges for landing judgment". This makes it possible to land, for example, if an obstacle appears near the edge of the captured image.

Further, even if the obstacle detected by the obstacle detection unit 107 is included in the landing determination range, the landing instruction unit 109 moves the obstacle to the landing determination range before landing of the own aircraft. If it is no longer included, it is judged that the landing conditions are met. Moving obstacles are, for example, vehicles or animals (including humans). When the landing is instructed, the flight control unit 105 lowers the altitude of the aircraft and ends the flight control when it lands on the ground or an object on the ground.

The drone 10 performs a landing process for landing its own aircraft based on the above configuration.
FIG. 9 shows an example of the operation procedure in the landing process. The operation procedure of FIG. 9 is started, for example, when the drone 10 flies on the planned flight path. First, the drone 10 (sensor acquisition unit 101) acquires an image taken vertically below the own machine by the stereo camera 17 as a measured value of the image sensor (step S11).

Next, the drone 10 (distance calculation unit 102) calculates the distance to the point indicated by each pixel of the acquired captured image (step S12). Subsequently, the drone 10 (risk level determination unit 103) determines the risk level at the time of landing of each of the plurality of divided regions vertically below the own aircraft based on the calculated distance for each pixel (step S13). ). Next, the drone 10 (flight instruction unit 104) gives an instruction to move the aircraft in the direction and altitude according to the arrangement of the divided regions whose determined risk level is the first reference or higher (step S14).

Subsequently, the drone 10 determines whether or not the altitude is below the predetermined altitude (step S15), and if it is determined that the altitude is not (NO), the drone 10 returns to step S11 to continue the operation and falls below the predetermined altitude. If it is determined that the result is (YES), the procedure proceeds to the next operation procedure. First, the drone 10 (sensor acquisition unit 101) acquires a photographed image vertically below the own machine in the same manner as in step S11 (step S21).

Next, the drone 10 (distance calculation unit 102) calculates the distance to the point indicated by each pixel of the acquired captured image (step S22). Subsequently, the drone 10 (obstacle detection unit 107) performs a process of detecting an obstacle from the acquired captured image based on the calculated distance for each pixel (step S23). If an obstacle is detected in step S23 (YES), the drone 10 determines whether or not the landing condition is satisfied (step S24).

If it is determined in step S24 that the landing condition is not satisfied (NO), the drone 10 returns to step S21 and continues the operation, and if it is determined that the landing condition is satisfied (YES), the drone 10 proceeds to the next operation procedure. When no obstacle is detected in step S23 (NO) and when the landing condition is satisfied (YES) in step S24, the drone 10 (landing instruction unit 109) is based on the calculated distance for each pixel. The flight control unit 105 is instructed to land the own aircraft, and landing control is started (step S25).

In this embodiment, as described above, it is possible to search for a divided region having a lower risk than the others among a plurality of divided regions, that is, a place where the flying object has a relatively high possibility of landing safely. Further, in this embodiment, since not only the movement but also the descent is instructed, the range of the safe division region can be gradually narrowed down. Then, by moving to the divided region searched in this way and then performing the landing control, the aircraft can land safely as compared with the case where the landing control is performed regardless of the risk of the divided region.

Further, when landing control is performed based on the measurement result by the distance measuring means as described above, the lower the accuracy of the measurement result, the more undesired behavior (for example, movement to avoid obstacles more than necessary) may occur. In this embodiment, since the landing control is performed after the drone 10 has descended to less than a predetermined altitude, it is possible to suppress the occurrence of undesired behavior due to low measurement accuracy. Further, in this embodiment, the degree of risk is determined based on the proportion of unsuitable areas that are not suitable for landing. As a result, it is possible to determine the degree of risk with higher accuracy than when the ratio of the unsuitable region is not taken into consideration.

Further, in this embodiment, even if the drone 10 detects an obstacle, the landing control is performed if the landing conditions are satisfied. As a result, it is possible to reduce the possibility of colliding with an obstacle when the flying object lands, and to prevent a situation in which, for example, a slight obstacle prevents the aircraft from landing indefinitely and causes the battery to run out. Further, for example, an image taken by a stereo camera 17 may be used for stereoscopic viewing or the like, and an obstacle can be detected by using an image taken for another purpose as such.

[2] Second Example The second embodiment of the present invention will be described below focusing on the differences from the first embodiment. In the first embodiment, the drone 10 itself performed all the processing related to landing, but in the second embodiment, the drone 10 and the server device share the processing.
FIG. 10 shows an example of the overall configuration of the landing support system 1 according to the second embodiment. The landing support system 1 is a system that supports the landing of an aircraft such as the drone 10.

The landing support system 1 includes a network 2, a drone 10, and a server device 20. The network 2 is a communication system including a mobile communication network, the Internet, and the like, and relays data exchange between devices accessing the own system. The server device 20 accesses the network 2 by wire communication (may be wireless communication), and the drone 10 accesses the network 2 by wireless communication.

FIG. 11 shows an example of the hardware configuration of the server device 20. The server device 20 may be physically configured as a computer device including a processor 21, a memory 22, a storage 23, a communication device 24, a bus 25, and the like. The hardware of the same name shown in FIG. 1 such as the processor 21 is the same type of hardware as in FIG. 1, although there are differences in performance, specifications, and the like.

FIG. 12 shows the functional configuration realized in this embodiment. The drone 10 includes a flight control unit 105 and an image capturing unit 110. The server device 20 includes a sensor acquisition unit 101, a distance calculation unit 102, a risk level determination unit 103, a flight instruction unit 104, a damage scale determination unit 106, an obstacle detection unit 107, and a shooting instruction unit 108. It is provided with a landing instruction unit 109. The image capturing unit 110 includes a stereo camera 17 and captures an image vertically below the own machine.

The image capturing unit 110 transmits captured image data indicating the captured image to the server device 20. The sensor acquisition unit 101 of the server device 20 acquires the captured image indicated by the transmitted captured image data as the measured value of the image sensor. After that, the sensor acquisition unit 101 to the landing instruction unit 109 operate in the same manner as in the first embodiment to support the landing of the drone 10. The difference from the first embodiment is that the instructions by the flight instruction unit 104, the shooting instruction unit 108, and the landing instruction unit 109 are given by transmitting the instruction data to the drone 10.

FIG. 13 shows an example of the operation procedure in the landing process of this embodiment. The operation procedure of FIG. 13 is started, for example, when the drone 10 flies on the planned flight path. First, the drone 10 (imaging unit 110) takes an image vertically below the own machine and measures the altitude of the own machine (step S31). Next, the drone 10 (image capturing unit 110) transmits the captured image and captured image data indicating the altitude of the own machine to the server device 20 (step S32).

The server device 20 determines whether or not the altitude of the drone 10 indicated by the transmitted captured image data is less than a predetermined altitude (step S33). If it is determined in step S33 that the altitude is not lower than the predetermined altitude (NO), the server device 20 (sensor acquisition unit 101) acquires an image taken vertically below the own device by the stereo camera 17 as a measured value of the image sensor. (Step S41).

Next, the server device 20 (distance calculation unit 102) calculates the distance to the point indicated by each pixel of the acquired captured image as the distance for each pixel (step S42). Subsequently, the server device 20 (risk degree determination unit 103) determines the risk degree of the plurality of divided regions based on the calculated distance for each pixel (step S43). Next, the server device 20 (flight instruction unit 104) generates instruction data indicating an instruction to move the own aircraft in the direction and altitude according to the arrangement of the divided areas whose determined risk level is the first reference or higher ( Step S44), transmission to the drone 10 (step S45).

The drone 10 (flight control unit 105) controls the flight of its own aircraft according to the instructions indicated by the transmitted instruction data (step S46). After that, the operations from steps S31 to S46 are repeated until it is determined in step S33 that the altitude is below a predetermined altitude (YES). If it is determined in step S33 that the altitude is below a predetermined altitude (YES), first, the operation procedure of steps S51, S52, S53, and S54, which is the same operation procedure as in steps S31, S32, S41, and S42, is performed.

Next, the server device 20 (obstacle detection unit 107) performs a process of detecting an obstacle from the acquired captured image based on the distance for each pixel calculated in step S54 (step S55). In the example of FIG. 13, it is assumed that an obstacle is detected. The server device 20 determines whether or not the landing condition is satisfied (step S56), and if it determines that the landing condition is not satisfied (NO), the server device 20 returns to step S52 to continue the operation and satisfies the landing condition (YES). If it is determined, proceed to the next operation procedure.

If it is determined in step S56 that the landing condition is satisfied (YES), the server device 20 (landing instruction unit 109) instructs the calculated distance for each pixel and the landing of the own aircraft based on the distance. Data is generated (step S57) and transmitted to the drone 10 (step S58). The drone 10 (flight control unit 105) controls the landing of its own aircraft according to the instructions indicated by the transmitted instruction data (step S59).

In this embodiment, since the server device 20 performs processing such as calculation of the distance for each pixel, determination of the degree of danger, determination of the scale of damage, and detection of obstacles, the processing of the drone 10 is performed as compared with the first embodiment. The load is reduced. As a result, computer resources such as the processor 11 mounted on the drone 10 can be reduced in weight and size. On the other hand, in the case of the first embodiment, since communication is not required for flight instructions and the like, landing can be supported without being affected by the communication status.

[3] Modification example The above-described embodiment is merely an example of the embodiment of the present invention, and may be modified as follows. Further, the examples and the modified examples may be combined as necessary. When combining the examples and each variant, prioritize each variant (prioritize which one to prioritize when conflicting events occur when each variant is implemented). May be good.

[3-1] Sensor acquisition unit 101 The sensor acquisition unit 101 has acquired the measured values of the image sensor of the stereo camera 17 in the above embodiment, but the present invention is not limited to this. For example, the sensor acquisition unit 101 acquires position information indicating the position of the own machine measured by the position sensor included in the sensor device 16 of the own machine as a measured value of the sensor.

The sensor acquisition unit 101 supplies the acquired position information to the risk determination unit 103. The risk level determination unit 103 determines the risk level based on the supplied position information. The risk determination unit 103 stores, for example, a map showing the attributes of an object existing on the ground for each area. The attribute of the object is, for example, the attribute shown in the weight table of FIG.

First, the risk determination unit 103 sets a predetermined area including the position indicated by the position information acquired by the sensor acquisition unit 101 as the ground area on the stored map. Then, the risk level determination unit 103 determines the risk level based on the ratio of the unsuitable area calculated by weighting the area of the unsuitable area occupied by the object existing in the divided area obtained by dividing the ground area according to the attribute of the object. judge.

As described in the first embodiment, the risk level determination unit 103 recognizes the object, weights it using the weighting table of FIG. 4, and determines the risk level using the riskiness table of FIG. In this modified example, since the risk level is determined without using the captured image, the risk level can be determined even in a situation where the ground cannot be clearly photographed from the sky due to fog, for example. On the other hand, in the case of the first embodiment, an obstacle can be detected by using an image taken for another purpose (such as taking an image for stereoscopic viewing), for example.

In this modified example, as described in the first embodiment, the division region in which the risk level determined by the risk level determination unit 103 is less than the first reference (in the example of FIG. 6, the risk levels Lv1 and Lv2 are divided). If there is no area), the risk determination unit 103 enlarges the above-ground area on the stored map and determines the risk again. The above-ground area is an area before being divided into divided areas, and in the example of FIG. 6, it is an area in which divided areas B1, B2, B3, and B4 are combined.

By expanding the above-ground area, the expanded part may include a low-risk area. As a result, there is a possibility that there is a divided area where the risk level is less than the first criterion, so that it is possible to attempt landing after moving to an area where a safer landing is possible. If the risk level determination unit 103 does not find a divided region whose risk level is less than the first criterion even after re-determining once, the risk level may be repeatedly re-determined by expanding the above-ground area. By doing so, it is possible to more surely find a divided region whose risk level is less than the first criterion.

[3-2] Aircraft In the examples, a rotary-wing aircraft type air vehicle was used as the air vehicle, but the vehicle is not limited to this. The air vehicle may be, for example, an airplane type air vehicle or a helicopter type air vehicle. Further, it may be a vertical take-off and landing aircraft called VTOL (Vertical Take-Off and Landing Aircraft).

[3-3] Landing conditions The landing instruction unit 109 may use landing conditions different from those in the first embodiment. For example, in the first embodiment, the landing instruction unit 109 determines that the landing condition is satisfied when the detected obstacle is not included in the predetermined range of the captured image. It may be changed according to the type of the flying object.

For example, in the case of the rotary-wing aircraft type air vehicle or helicopter type air vehicle described in the first embodiment, the landing instruction unit 109 descends from substantially vertically upward to vertically downward, so a circular range is used. .. Further, in the case of a VTOL aircraft, the landing instruction unit 109 often descends while gliding diagonally in order to stably descend, so an elliptical range is used. Further, in the case of an airplane-type flying object, the landing instruction unit 109 descends diagonally at an angle close to horizontal, so a longer elliptical range is used.

For example, when an airplane-type flying object uses a circular range, it flies at a height that makes contact with an obstacle before landing in that range, so there is a risk of contacting an obstacle outside the range. Therefore, in the case of an airplane-type flying object, by using a longer elliptical range, it is possible to land through a space above a predetermined range where there is no risk of contact with an obstacle. In this way, by changing the length of a predetermined range according to the angle of the flight path when the flying object descends, landing regardless of the type of the flying object, as compared with the case where a circular range is always used, for example. The risk of contact with obstacles at times can be reduced.

Further, the landing instruction unit 109 may change a predetermined range according to the weather conditions at the landing point. In this case, the landing instruction unit 109 inquires of the system of the operator that provides the weather condition for each region on the Internet, for example, the weather condition of the region including the landing point. The landing instruction unit 109 uses a range as a predetermined range, which is so small that the weather condition obtained in response to the inquiry is suitable for the flight of the drone 10.

That is, the landing instruction unit 109 increases the predetermined range as the weather conditions are bad and the drone 10 is not suitable for flight. If the weather conditions are bad (for example, the wind is strong), the risk of the drone 10 being blown by the wind while descending and coming into contact with a nearby obstacle increases. Therefore, by increasing the predetermined range as the weather condition is worse, the risk of contact with an obstacle at the time of landing can be reduced as compared with the case where the range is kept constant regardless of the weather condition.

[3-4] Divided region In the embodiment, four square divided regions viewed from above vertically are used, but the present invention is not limited to this. For example, the shape of each divided region may be rectangular. Further, for example, three regular hexagonal division regions may be used, or six regular triangle division regions may be used. Further, although the shapes and sizes of the divided regions were the same, they do not have to be exactly the same and may be slightly different. In either case, it suffices that the arrangement of the divided areas having a risk level of the first criterion or higher is associated with the moving direction toward the area where the safest landing can be expected at the time of the arrangement.

[3-5] Instruction method The flight instruction unit 104, the shooting instruction unit 108, and the landing instruction unit 109 give instructions to the drone 10 in the embodiment, but the present invention is not limited to this, and the drone 10 is operated by, for example, a radio. If so, the operator may be instructed. In this case, each indicator transmits data indicating, for example, the moving direction, the altitude after the movement, the shooting position, and the character string instructing the switching to the landing control to the radio, and those character strings are displayed on the display surface of the radio. It is advisable to give instructions by displaying.

[3-6] Danger level The risk level determination unit 103 determines the risk level in three stages in the embodiment, but it may be in two stages or in four or more stages. In addition to being represented by a numerical value such as "Lv1", the degree of danger is represented by a character string such as "Large", "Medium", "Small", "A", "B", and "C". You may. In short, it is sufficient if the information indicates the high possibility that the drone 10 cannot land safely.
The first criterion used when the flight instruction unit 104 determines the moving direction and the second criterion used when determining the presence or absence of descent may be determined according to the way of expressing the degree of danger.

[3-7] Method of Determining Movement Direction The flight instruction unit 104 may determine the movement direction by a method different from that of the embodiment. The flight instruction unit 104 may determine the moving direction in consideration of the balance between the degree of danger and the degree of damage, for example.
For example, when there is a division area B1 of damage Lv1 in danger Lv2 and a division area B2 of damage Lv3 in danger Lv1, the flight instruction unit 104 may set the division area B2 having the smaller risk as the movement direction, for example. The division area B1 in which the total value of both levels is small may be set as the moving direction.

[3-8] Distance measuring means The distance measuring means used in landing control is not limited to a stereo camera. For example, a means for measuring the distance to an object using infrared rays, ultrasonic waves, millimeter waves, or the like may be used as the distance measuring means.

[3-9] Device for Realizing Each Function The device for realizing each function shown in FIGS. 2 and 12 is not limited to the above-mentioned device. For example, the computer resources provided by the cloud service may realize the functions realized by the server device 20. In either case, it suffices that each function shown in FIG. 2 or FIG. 12 is realized in the entire landing support system 1.

[3-10] Category of Invention The present invention can be regarded as an information processing system such as a landing support system 1 including the above-mentioned information processing devices such as the drone 10 and the server device 20. .. Further, the present invention can be regarded as an information processing method for realizing the processing performed by the information processing device, and also as a program for operating a computer that controls the information processing device. The program regarded as the present invention may be provided in the form of a recording medium such as an optical disk in which the program is stored, or may be downloaded to a computer via a network such as the Internet, and the downloaded program may be installed and used. It may be provided in the form of

[3-11] Functional Blocks The block diagram used in the description of the above embodiment shows blocks for functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Further, the method of realizing each functional block is not particularly limited.

That is, each functional block may be realized by using one device that is physically or logically connected, or directly or indirectly (for example, by two or more devices that are physically or logically separated). , Wired, wireless, etc.) and may be realized using these plurality of devices. The functional block may be realized by combining the software with the one device or the plurality of devices.

Functions include judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, solution, selection, selection, establishment, comparison, assumption, expectation, and assumption. There are broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, assigning, etc., but only these. I can't. For example, a functional block (constituent unit) that makes transmission function is called a transmitting unit (transmitting unit) or a transmitter (transmitter). As described above, the method of realizing each of them is not particularly limited.

[3-12] Handling of input / output information and the like The input and output information and the like may be stored in a specific location (for example, a memory) or may be managed using a management table. Input / output information and the like can be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

[3-13] Judgment method Judgment may be performed by a value represented by 1 bit (0 or 1), may be performed by a boolean value (Boolean: true or false), or may be a numerical value. (For example, comparison with a predetermined value) may be performed.

[3-14] Processing procedure, etc. The order of the processing procedures, sequences, flowcharts, etc. of each aspect / embodiment described in the present disclosure may be changed as long as there is no contradiction. For example, the methods described in the present disclosure present elements of various steps using exemplary order, and are not limited to the particular order presented.

[3-15] Handling of input / output information, etc. The input / output information, etc. may be saved in a specific location (for example, memory) or may be managed by a management table. Input / output information and the like can be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

[3-16] Software Software, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name, is an instruction, instruction set, code, code segment, program code, program. , Subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc. should be broadly interpreted.

Further, software, instructions, information, etc. may be transmitted and received via a transmission medium. For example, a website, where the software uses at least one of wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technology (infrared, microwave, etc.). When transmitted from a server, or other remote source, at least one of these wired and wireless technologies is included within the definition of transmission medium.

[3-17] Information, Signals The information, signals, etc. described in the present disclosure may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.

[3-18] "Judgment", "Decision"
The terms "determining" and "determining" as used in this disclosure may include a wide variety of actions. "Judgment" and "decision" are, for example, judgment (judging), calculation (calculating), calculation (computing), processing (processing), derivation (deriving), investigating (investigating), search (looking up, search, inquiry). (For example, searching in a table, database or another data structure), ascertaining may be regarded as "judgment" or "decision".

Also, "judgment" and "decision" are receiving (for example, receiving information), transmitting (for example, transmitting information), input (input), output (output), and access. (Accessing) (for example, accessing data in memory) may be regarded as "judgment" or "decision". In addition, "judgment" and "decision" mean that the things such as solving, selecting, choosing, establishing, and comparing are regarded as "judgment" and "decision". Can include. That is, "judgment" and "decision" may include considering some action as "judgment" and "decision". Further, "judgment (decision)" may be read as "assuming", "expecting", "considering" and the like.

[3-19] Meaning of "based on" The description "based on" used in this disclosure does not mean "based on only" unless otherwise specified. In other words, the statement "based on" means both "based only" and "at least based on".

[3-20] "Different" In the present disclosure, the term "A and B are different" may mean "A and B are different from each other". The term may mean that "A and B are different from C". Terms such as "separate" and "combined" may be interpreted in the same way as "different".

[3-21] "and", "or" In the present disclosure, for configurations that can be implemented by either "A and B" or "A or B", the configuration described in one expression is described in the other expression. It may be used as a configured configuration. For example, when "A and B" are described, they may be used as "A or B" as long as they are not inconsistent with other descriptions and can be implemented.

[3-22] Variations of Aspects, etc. Each aspect / embodiment described in the present disclosure may be used alone, in combination, or switched with execution. Further, the notification of predetermined information (for example, the notification of "being X") is not limited to the explicit one, but is performed implicitly (for example, the notification of the predetermined information is not performed). May be good.

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure may be implemented as an amendment or modification without departing from the purpose and scope of the present disclosure, which is determined by the description of the scope of claims. Therefore, the description of the present disclosure is for the purpose of exemplary explanation and does not have any limiting meaning to the present disclosure.

1 ... Landing support system, 10 ... Drone, 17 ... Stereo camera, 20 ... Server device, 101 ... Sensor acquisition unit, 102 ... Distance calculation unit, 103 ... Danger level determination unit, 104 ... Flight instruction unit, 105 ... Flight control unit , 106 ... Damage scale determination unit, 107 ... Obstacle detection unit, 108 ... Shooting instruction unit, 109 ... Landing instruction unit, 110 ... Imaging unit.

Claims

An acquisition unit that acquires images taken by a stereo camera installed on the aircraft, and
A calculation unit that calculates the distance from the stereo camera to the point indicated by each pixel of the acquired image, and a calculation unit.
A detection unit that detects obstacles from the acquired image, and
An information processing device including a landing instruction unit that instructs the landing of the flying object based on the calculated distance if the position of the obstacle satisfies the landing condition even if the obstacle is detected.
The information processing device according to claim 1, wherein the detection unit determines whether or not the obstacle is an obstacle based on the degree of similarity between the detected shape of the obstacle and the shape registered in advance.
A shooting instruction unit for instructing shooting from another direction of the obstacle when the obstacle is detected is provided.
The information processing device according to claim 1 or 2, wherein the detection unit adds an image taken according to the shooting instruction and detects the obstacle again.
The indicator according to any one of claims 1 to 3, wherein the indicating unit determines that the landing condition is satisfied when the detected obstacle is not included in the predetermined range in the captured image. Information processing device.
Even if the detected obstacle is included in the range, the indicator determines that the landing condition is satisfied if the obstacle is not included in the range before the landing of the flying object due to movement. The information processing apparatus according to claim 4.
The information processing device according to claim 4 or 5, wherein the indicating unit changes the range according to the type of the flying object.
The information processing device according to any one of claims 4 to 6, wherein the indicating unit changes the range according to the weather conditions at the landing point.