JP2020095519A - Shape estimation device, shape estimation method, program, and recording medium - Google Patents

Abstract

To provide a shape estimation device capable of improving a real time property, when estimating the three-dimensional shape of a ground surface.SOLUTION: The shape estimation device generates a grid having a plurality of grid points in an imaging range imaged by an imaging apparatus, acquires a first captured image and a second captured image, to acquire the information of a three-dimensional position and an imaging direction, and assumes the plurality of attitudes of a first grid point at the grid points. Further, the shape estimation device derives a first assumed projection position where the first grid point is projected to the first captured image and a second assumed projection position where the first grid point is projected to the second captured image on the basis of the three-dimensional position and the imaging direction by the imaging apparatus, for the plurality of assumed attitudes and estimates the attitude of the first grid point on the basis of similarity between the first assumed projection position in the first captured image and the second assumed projection position in the second captured image, for the plurality of assumed attitudes, to estimate the three-dimensional shape of the imaging range on the basis of the estimated attitude of the first grid point.SELECTED DRAWING: Figure 6

Description

The present disclosure relates to a shape estimation device, a shape estimation method, a program, and a recording medium that estimates a three-dimensional shape based on a plurality of captured images.

Conventionally, a platform (unmanned aerial vehicle) that performs imaging while passing through a preset fixed path is known. The platform receives an imaging instruction from the ground base and images the imaging target. When capturing an image of an imaging target, this platform tilts the imaging device of the platform to capture an image while flying on a fixed path, depending on the positional relationship between the platform and the imaging target (see Patent Document 1).

JP, 2010-61216, A

The three-dimensional shape of the ground can be estimated using a plurality of images captured by the drone of Patent Document 1. However, the calculation of the three-dimensional shape estimation is complicated, the calculation cost is high, and it can be done only offline. For example, in the conventional processing of three-dimensional shape estimation, it is necessary to sequentially perform raw point generation (for example, sfm: Structure from Motion), dense point generation (for example, mvs: Multi-View Stereo), mesh generation, texture generation, and the like. The processing load for three-dimensional shape estimation is high. Therefore, it is difficult for the drone to estimate the three-dimensional shape of the ground in real time while capturing an image.

In one aspect, a shape estimation device that estimates a three-dimensional shape based on a plurality of captured images captured by a flying object, including a processing unit that performs processing relating to the estimation of the three-dimensional shape, and the processing unit is a flying object. Generate a grid having a plurality of grid points in an imaging range imaged by the imaging device, and obtain a first captured image and a second captured image captured by the imaging device, and obtain a first captured image and a second captured image. The information about the three-dimensional position of the image pickup device and the image pickup direction by the image pickup device when the two picked-up images are picked up is obtained, a plurality of altitudes of the first grid point in the plurality of grid points are assumed, and the plurality of assumed altitudes are A first hypothetical projection position where the first grid point is projected on the first captured image, and the first grid point is the second captured image based on the three-dimensional position of the image capturing device and the image capturing direction of the image capturing device. A second hypothetical projection position projected on the image, and a plurality of hypothetical altitudes, a first hypothetical projection position in the first captured image and a second hypothetical projected position in the second captured image. Of the three-dimensional imaging range based on the estimated altitude of the first grid point based on the similarity of a plurality of assumed altitudes. The shape may be estimated.

The processing unit performs three-dimensional reconstruction based on the first captured image and the second captured image, acquires point cloud data, generates surface data based on the point cloud data, and generates a first surface data based on the surface data. The initial altitude of one grid point may be derived, and the altitude of the first grid point may be assumed in order from the initial altitude.

The first captured image may be one or more existing captured images. The second captured image may be one or more latest captured images. The processing unit acquires the first point cloud data generated based on the first captured image, generates the second point cloud data based on the second captured image, and generates the first point cloud data. Surface data may be generated based on and the second point cloud data.

The processing unit calculates an error score indicating an error between the first assumed projection position and the second assumed projection position based on the degree of similarity, and estimates the altitude of the first grid point based on the error score. You may.

The processing unit may calculate the error score based on the interval between the first grid point and the adjacent second grid point in the grid.

The processing unit may calculate the error score based on the smoothness between the first grid point and the second grid point in the grid.

The first grid points may be all grid points in the grid.

The first grid points may be some grid points in the grid.

The processing unit may set the interval between the points of the grid.

In one aspect, the method is a shape estimation method for estimating a three-dimensional shape based on a plurality of captured images captured by an air vehicle, and when performing processing related to the estimation of the three-dimensional shape, the image is captured by an imaging device included in the air vehicle. Generating a grid having a plurality of grid points in an image capturing range, acquiring a first captured image and a second captured image captured by the image capturing device, a first captured image and a second captured image A step of acquiring information on a three-dimensional position of the image pickup device and an image pickup direction by the image pickup device when an image is taken; a step of assuming a plurality of altitudes of a first grid point among a plurality of grid points; and a plurality of assumed altitudes Regarding the three-dimensional position of the imaging device and the imaging direction of the imaging device, the first assumed projection position where the first grid point is projected on the first captured image, and the first grid point is the second projection position. Deriving a second hypothetical projection position projected on the captured image, a plurality of hypothesized altitudes, a first hypothetical projection position on the first captured image, and a second hypothesis on the second captured image. A step of deriving a similarity to the projection position, a step of estimating the altitude of the first grid point based on the similarities of a plurality of assumed altitudes, and a step of estimating the altitude of the first grid point And estimating the three-dimensional shape of the imaging range.

The step of assuming a plurality of altitudes of the first grid points includes three-dimensional reconstruction based on the first captured image and the second captured image to obtain point cloud data, and a plane based on the point cloud data. It may include a step of generating data, a step of deriving an initial altitude of the first grid point based on the surface data, and a step of sequentially assuming the altitude of the first grid point from the initial altitude.

The first captured image may be one or more existing captured images. The second captured image may be one or more latest captured images. The step of generating the point cloud data includes a step of acquiring the first point cloud data generated based on the first captured image and a step of generating the second point cloud data based on the second captured image. And steps. The step of generating surface data may include a step of generating surface data based on the first point cloud data and the second point cloud data.

The step of estimating the altitude of the first grid point includes the steps of calculating an error score indicating an error between the first assumed projection position and the second assumed projection position based on the similarity, and based on the error score. , Estimating the altitude of the first grid point.

The step of calculating the error score may include the step of calculating the error score based on the interval between the first grid point and the adjacent second grid point in the grid.

The step of calculating the error score may include the step of calculating the error score based on the smoothness between the first grid points and the second grid points in the grid.

The first grid points may be all grid points in the grid.

The first grid points may be some grid points in the grid.

The step of generating a grid may include the step of setting the spacing between points of the grid.

In one aspect, the program causes a shape estimation device that estimates a three-dimensional shape based on a plurality of captured images captured by the flying object to be captured by an imaging device included in the flying object when performing processing related to the estimation of the three-dimensional shape. Generating a grid having a plurality of grid points in an image capturing range, acquiring a first captured image and a second captured image captured by an image capturing device, a first captured image and a second captured image. A step of acquiring information on a three-dimensional position of the image pickup apparatus and an image pickup direction of the image pickup apparatus when the picked-up image is picked up, a step of assuming a plurality of altitudes of the first grid points in the plurality of grid points, and a plurality of assumptions are made. Regarding the altitude, based on the three-dimensional position of the imaging device and the imaging direction by the imaging device, the first assumed projection position at which the first grid point is projected on the first captured image and the first grid point at the second Deriving a second hypothetical projection position projected on the captured image of the first hypothetical projection position, and a plurality of hypothesized altitudes, the first hypothetical projection position on the first pictorial image and the second hypothetical projection position on the second pictorial image. The step of deriving the similarity with the assumed projection position, the step of estimating the altitude of the first grid point based on the similarities for the plurality of assumed altitudes, and the step of estimating the altitude of the first grid point And a step of estimating a three-dimensional shape of the imaging range based on the program.

In one aspect, the recording medium is configured such that a shape estimation device that estimates a three-dimensional shape based on a plurality of captured images captured by a flying object is used by the imaging device included in the flying object when performing processing related to the estimation of the three-dimensional shape. A step of generating a grid having a plurality of grid points in an image capturing range, a step of acquiring a first captured image and a second captured image captured by an image capturing device, a first captured image and a second captured image A plurality of hypotheses, a step of acquiring information on a three-dimensional position of the image pickup apparatus and an image pickup direction of the image pickup apparatus when the picked-up image is captured, a step of assuming a plurality of altitudes of the first grid points in the plurality of grid points, For the altitude, the first assumed projection position where the first grid point is projected on the first captured image and the first grid point are determined based on the three-dimensional position of the image capturing device and the image capturing direction of the image capturing device. Deriving a second hypothetical projection position projected on the second picked-up image, and a plurality of hypothesized altitudes, the first hypothetical projection position on the first picked-up image and the second hypothetical position on the second picked-up image. Of deriving the similarity with the assumed projection position of, the step of estimating the altitude of the first grid point based on the similarities of the plurality of assumed altitudes, and the estimated altitude of the first grid point And a step of estimating a three-dimensional shape of an imaging range based on the above, and a computer-readable recording medium recording a program for executing the step.

The above summary of the invention does not enumerate all the features of the present disclosure. Further, a sub-combination of these feature groups can also be an invention.

Schematic diagram showing a first configuration example of the flight system in the first embodiment Schematic diagram showing a second configuration example of the flight system in the first embodiment. The figure which shows an example of the specific appearance of an unmanned aerial vehicle. Block diagram showing an example of the hardware configuration of an unmanned aerial vehicle Block diagram showing an example of the hardware configuration of the terminal Flowchart showing the terrain estimation operation procedure Diagram showing an example of initial grid generation Diagram showing an example of sparse point cloud generation Diagram showing an example of generating a surface surrounded by sparse point clouds Diagram showing an example of initial grid value generation Figure explaining the search example of grid altitude Diagram illustrating an example of grid shape estimation

Hereinafter, the present disclosure will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Not all combinations of the features described in the embodiments are essential to the solution means of the invention.

The claims, the description, the drawings and the abstract contain the subject matter of copyright protection. The copyright owner has no objection to the reproduction of any of these documents by anyone as it appears in the Patent Office file or record. However, in all other cases, all copyrights are reserved.

In the following embodiments, an unmanned aerial vehicle (UAV) is exemplified as a flying body. Unmanned aerial vehicles include aircraft moving in the air. In the drawings attached to this specification, an unmanned aerial vehicle is referred to as a "UAV". A flight system is exemplified as the shape estimation system. Although an unmanned aerial vehicle is mainly exemplified as the shape estimation device, it may be a terminal. The terminal may include a smartphone, a tablet terminal PC (Personal Computer), or another device. The shape estimation method defines the operation of the shape estimation device. Further, the recording medium has recorded therein a program (for example, a program for causing the shape estimation device to execute various processes).

(Embodiment 1)
FIG. 1 is a diagram illustrating a first configuration example of a flight system 10 according to the first embodiment. The flight system 10 includes an unmanned aerial vehicle 100 and a terminal 80. The unmanned aerial vehicle 100 and the terminal 80 can communicate with each other by wire communication or wireless communication (for example, wireless LAN (Local Area Network)). FIG. 1 illustrates that the terminal 80 is a mobile terminal (for example, a smartphone or a tablet terminal).

The flight system may include an unmanned aerial vehicle, a transmitter (propo), and a mobile terminal. When equipped with a transmitter, the left and right control rods located on the front of the transmitter allow the user to control the flight of the unmanned aerial vehicle. Further, in this case, the unmanned aerial vehicle, the transmitter, and the mobile terminal can communicate with each other by wire communication or wireless communication.

FIG. 2 is a schematic diagram showing a second configuration example of the flight system 10 in the first embodiment. FIG. 2 exemplifies that the terminal 80 is a PC. The function of the terminal 80 may be the same in any of FIG. 1 and FIG.

FIG. 3 is a diagram showing an example of a specific appearance of unmanned aerial vehicle 100. FIG. 3 shows a perspective view when unmanned aerial vehicle 100 flies in movement direction STV0. The unmanned aerial vehicle 100 is an example of a moving body.

As shown in FIG. 3, the roll axis (see the x-axis) is set in the direction parallel to the ground and along the movement direction STV0. In this case, a pitch axis (see y-axis) is set in a direction parallel to the ground and perpendicular to the roll axis, and a yaw axis (z-axis) is set in a direction perpendicular to the ground and perpendicular to the roll axis and the pitch axis. (See) is set.

The unmanned aerial vehicle 100 is configured to include a UAV body 102, a gimbal 200, an image capturing section 220, and a plurality of image capturing sections 230.

The UAV body 102 includes a plurality of rotor blades (propellers). The UAV body 102 causes the unmanned aerial vehicle 100 to fly by controlling the rotation of a plurality of rotor blades. The UAV body 102 flies the unmanned aerial vehicle 100 using, for example, four rotors. The number of rotor blades is not limited to four. Further, unmanned aerial vehicle 100 may be a fixed-wing aircraft having no rotary wing.

The image capturing unit 220 is a camera for capturing an image of a subject included in a desired image capturing range (for example, a state of the sky as an aerial image capturing target, a landscape such as a mountain or a river, or a building on the ground).

The plurality of imaging units 230 are sensing cameras that capture images of the surroundings of the unmanned aerial vehicle 100 in order to control the flight of the unmanned aerial vehicle 100. The two imaging units 230 may be provided on the front surface that is the nose of the unmanned aerial vehicle 100. Further, the other two imaging units 230 may be provided on the bottom surface of unmanned aerial vehicle 100. The two image capturing units 230 on the front side may form a pair and function as a so-called stereo camera. The two imaging units 230 on the bottom side may also be paired and may function as a stereo camera. Three-dimensional space data around the unmanned aerial vehicle 100 may be generated based on the images captured by the plurality of imaging units 230. The number of imaging units 230 included in unmanned aerial vehicle 100 is not limited to four. The unmanned aerial vehicle 100 may include at least one imaging unit 230. The unmanned aerial vehicle 100 may include at least one imaging unit 230 on each of a nose, a stern, a side surface, a bottom surface, and a ceiling surface of the unmanned aerial vehicle 100. The angle of view that can be set by the imaging unit 230 may be wider than the angle of view that can be set by the imaging unit 220. The image capturing section 230 may include a single focus lens or a fisheye lens.

FIG. 4 is a block diagram showing an example of the hardware configuration of unmanned aerial vehicle 100. The unmanned aerial vehicle 100 includes a UAV control unit 110, a communication interface 150, a memory 160, a storage 170, a gimbal 200, a rotary wing mechanism 210, an imaging unit 220, an imaging unit 230, a GPS receiver 240, The configuration includes an inertial measurement unit (IMU: Inertial Measurement Unit) 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, and a laser measuring device 290.

The UAV control unit 110 is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). The UAV control unit 110 performs signal processing for integrally controlling the operation of each unit of the unmanned aerial vehicle 100, data input/output process with other units, data calculation process, and data storage process.

The UAV control unit 110 controls the flight of the unmanned aerial vehicle 100 according to the program stored in the memory 160. The UAV controller 110 may control flight. The UAV control unit 110 may take an aerial image.

UAV control unit 110 acquires position information indicating the position of unmanned aerial vehicle 100. The UAV control unit 110 may acquire, from the GPS receiver 240, position information indicating the latitude, longitude, and altitude at which the unmanned aerial vehicle 100 exists. The UAV control unit 110 acquires from the GPS receiver 240 latitude/longitude information indicating the latitude and longitude where the unmanned aerial vehicle 100 exists, and altitude information indicating the altitude at which the unmanned aerial vehicle 100 exists from the barometric altimeter 270 as position information. Good. The UAV control unit 110 may acquire the distance between the ultrasonic wave emission point of the ultrasonic wave sensor 280 and the ultrasonic wave reflection point as altitude information.

The UAV control unit 110 may acquire the orientation information indicating the orientation of the unmanned aerial vehicle 100 from the magnetic compass 260. The orientation information may be indicated by an orientation corresponding to the orientation of the nose of unmanned aerial vehicle 100, for example.

The UAV control unit 110 may acquire position information indicating a position where the unmanned aerial vehicle 100 should exist when the imaging unit 220 images the imaging range to be imaged. The UAV control unit 110 may acquire the position information indicating the position where the unmanned aerial vehicle 100 should exist from the memory 160. The UAV control unit 110 may acquire position information indicating the position where the unmanned aerial vehicle 100 should exist from another device via the communication interface 150. The UAV control unit 110 may refer to the three-dimensional map database to identify a position where the unmanned aerial vehicle 100 can exist, and acquire the position as position information indicating a position where the unmanned aerial vehicle 100 should exist.

The UAV control unit 110 may acquire imaging range information indicating the imaging ranges of the imaging unit 220 and the imaging unit 230. The UAV control unit 110 may acquire angle-of-view information indicating the angle of view of the image-capturing units 220 and 230 from the image-capturing units 220 and 230 as a parameter for specifying the image-capturing range. The UAV control unit 110 may acquire information indicating the image pickup directions of the image pickup units 220 and 230 as a parameter for specifying the image pickup range. The UAV control unit 110 may acquire posture information indicating the state of the posture of the image capturing unit 220 from the gimbal 200, for example, as information indicating the image capturing direction of the image capturing unit 220. The attitude information of the imaging unit 220 may indicate the rotation angles of the pitch axis and the yaw axis of the gimbal 200 from the reference rotation angle.

The UAV control unit 110 may acquire position information indicating the position where the unmanned aerial vehicle 100 exists as a parameter for specifying the imaging range. The UAV control unit 110 defines an imaging range indicating a geographical range imaged by the imaging unit 220 based on the angle of view and the imaging direction of the imaging unit 220 and the imaging unit 230, and the position where the unmanned aerial vehicle 100 exists, The imaging range information may be acquired by generating the imaging range information.

The UAV control unit 110 may acquire the imaging range information from the memory 160. The UAV control unit 110 may acquire the imaging range information via the communication interface 150.

The UAV control unit 110 controls the gimbal 200, the rotary wing mechanism 210, the imaging unit 220, and the imaging unit 230. The UAV control unit 110 may control the imaging range of the imaging unit 220 by changing the imaging direction or the angle of view of the imaging unit 220. The UAV controller 110 may control the imaging range of the imaging unit 220 supported by the gimbal 200 by controlling the rotation mechanism of the gimbal 200.

The imaging range refers to a geographical range in which an image is captured by the image capturing unit 220 or the image capturing unit 230. The imaging range is defined by latitude, longitude, and altitude. The imaging range may be a range in three-dimensional spatial data defined by latitude, longitude, and altitude. The imaging range may be a range in the two-dimensional space data defined by latitude and longitude. The imaging range may be specified based on the angle of view and the imaging direction of the imaging unit 220 or the imaging unit 230, and the position where the unmanned aerial vehicle 100 exists. The imaging directions of the imaging unit 220 and the imaging unit 230 may be defined by the azimuth and the depression angle facing the front where the imaging lenses of the imaging unit 220 and the imaging unit 230 are provided. The imaging direction of the imaging unit 220 may be a direction specified by the heading of the unmanned aerial vehicle 100 and the attitude of the imaging unit 220 with respect to the gimbal 200. The imaging direction of the imaging unit 230 may be a direction specified by the heading of the unmanned aerial vehicle 100 and the position where the imaging unit 230 is provided.

The UAV control unit 110 may identify the environment around the unmanned aerial vehicle 100 by analyzing the plurality of images captured by the plurality of image capturing units 230. The UAV control unit 110 may control the flight, for example, by avoiding an obstacle based on the environment around the unmanned aerial vehicle 100.

The UAV control unit 110 may acquire stereoscopic information (three-dimensional information) indicating the three-dimensional shape (three-dimensional shape) of an object existing around the unmanned aerial vehicle 100. The object may be, for example, a part of a landscape such as a building, a road, a car or a tree. The stereoscopic information is, for example, three-dimensional space data. The UAV control unit 110 may acquire stereoscopic information by generating stereoscopic information indicating the stereoscopic shape of an object existing around the unmanned aerial vehicle 100 from each image obtained from the plurality of imaging units 230. The UAV control unit 110 may acquire the stereoscopic information indicating the stereoscopic shape of the object existing around the unmanned aerial vehicle 100 by referring to the three-dimensional map database stored in the memory 160 or the storage 170. The UAV control unit 110 may acquire three-dimensional information regarding the three-dimensional shape of an object existing around the unmanned aerial vehicle 100 by referring to a three-dimensional map database managed by a server existing on the network.

The UAV control unit 110 controls the flight of the unmanned aerial vehicle 100 by controlling the rotary wing mechanism 210. That is, the UAV control unit 110 controls the position of the unmanned aerial vehicle 100 including latitude, longitude, and altitude by controlling the rotary wing mechanism 210. The UAV control unit 110 may control the imaging range of the imaging unit 220 by controlling the flight of the unmanned aerial vehicle 100. The UAV control unit 110 may control the angle of view of the image capturing unit 220 by controlling the zoom lens included in the image capturing unit 220. The UAV control unit 110 may use the digital zoom function of the imaging unit 220 to control the angle of view of the imaging unit 220 by digital zoom.

When the imaging unit 220 is fixed to the unmanned aerial vehicle 100 and the imaging unit 220 cannot be moved, the UAV control unit 110 moves the unmanned aerial vehicle 100 to a specific position at a specific date and time to perform desired imaging in a desired environment. The range may be captured by the image capturing section 220. Alternatively, even when the image capturing unit 220 does not have a zoom function and the angle of view of the image capturing unit 220 cannot be changed, the UAV control unit 110 moves the unmanned aerial vehicle 100 to a specific position at the specified date and time, thereby making it Under the above environment, the image capturing section 220 may capture a desired image capturing range.

The communication interface 150 communicates with the terminal 80. The communication interface 150 may perform wireless communication by any wireless communication method. The communication interface 150 may perform wired communication by any wired communication method. The communication interface 150 may transmit the aerial image and additional information (metadata) regarding the aerial image to the terminal 80.

In the memory 160, the UAV control unit 110 includes a gimbal 200, a rotary wing mechanism 210, an imaging unit 220, an imaging unit 230, a GPS receiver 240, an inertial measurement device 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, and a laser. It stores programs and the like necessary for controlling the measuring device 290. The memory 160 may be a computer-readable recording medium such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and It may include at least one of flash memories such as a USB (Universal Serial Bus) memory. Memory 160 may be removable from unmanned aerial vehicle 100. The memory 160 may operate as a working memory.

The storage 170 may include at least one of HDD (Hard Disk Drive), SSD (Solid State Drive), SD card, USB memory, and other storage. The storage 170 may hold various information and various data. Storage 170 may be removable from unmanned aerial vehicle 100. The storage 170 may record an aerial image.

The memory 160 or the storage 170 may hold information on the aerial shooting position and the aerial shooting route generated by the terminal 80 or the unmanned aerial vehicle 100. The information on the aerial shooting position and the aerial shooting route is one of the aerial shooting parameters related to the aerial shooting scheduled by the unmanned aerial vehicle 100 or the flight parameters related to the flight scheduled by the unmanned aerial vehicle 100. May be set by. This setting information may be held in the memory 160 or the storage 170.

The gimbal 200 may rotatably support the imaging unit 220 around a yaw axis, a pitch axis, and a roll axis. The gimbal 200 may change the imaging direction of the imaging unit 220 by rotating the imaging unit 220 around at least one of the yaw axis, the pitch axis, and the roll axis.

The rotary blade mechanism 210 includes a plurality of rotary blades and a plurality of drive motors that rotate the plurality of rotary blades. The rotary wing mechanism 210 causes the unmanned aerial vehicle 100 to fly by being controlled in rotation by the UAV control unit 110. The number of the rotary blades 211 may be, for example, four or any other number. Further, unmanned aerial vehicle 100 may be a fixed-wing aircraft having no rotary wing.

The imaging unit 220 images a subject in a desired imaging range and generates captured image data. The image data (for example, an aerial image) obtained by the image pickup of the image pickup unit 220 may be stored in the memory included in the image pickup unit 220 or the storage 170. The image capturing section 220 is an example of an image capturing section.

The imaging unit 230 images the surroundings of the unmanned aerial vehicle 100 and generates captured image data. The image data of the image capturing unit 230 may be stored in the storage 170.

The GPS receiver 240 receives a plurality of signals indicating the time transmitted from a plurality of navigation satellites (that is, GPS satellites) and the position (coordinates) of each GPS satellite. The GPS receiver 240 calculates the position of the GPS receiver 240 (that is, the position of the unmanned aerial vehicle 100) based on the received signals. The GPS receiver 240 outputs the position information of the unmanned aerial vehicle 100 to the UAV control unit 110. The position information of the GPS receiver 240 may be calculated by the UAV control unit 110 instead of the GPS receiver 240. In this case, the UAV control unit 110 is input with the information indicating the time and the position of each GPS satellite included in the plurality of signals received by the GPS receiver 240.

Inertial measurement device 250 detects the attitude of unmanned aerial vehicle 100 and outputs the detection result to UAV control unit 110. The inertial measurement device 250 detects, as the attitude of the unmanned aerial vehicle 100, three-axis accelerations in the front-rear, left-right, and up-down directions of the unmanned aerial vehicle 100 and angular velocities in the three-axis directions of the pitch axis, roll axis, and yaw axis. You may.

The magnetic compass 260 detects the heading of the nose of the unmanned aerial vehicle 100, and outputs the detection result to the UAV control unit 110.

Barometric altimeter 270 detects the altitude at which unmanned aerial vehicle 100 flies, and outputs the detection result to UAV controller 110.

The ultrasonic sensor 280 emits an ultrasonic wave, detects the ultrasonic wave reflected by the ground or an object, and outputs the detection result to the UAV control unit 110. The detection result may indicate the distance from the unmanned aerial vehicle 100 to the ground, that is, the altitude. The detection result may indicate the distance from the unmanned aerial vehicle 100 to the object (subject).

Laser measuring instrument 290 irradiates an object with laser light, receives reflected light reflected by the object, and measures the distance between unmanned aerial vehicle 100 and the object (subject) by the reflected light. The distance measurement method using laser light may be, for example, a time-of-flight method.

FIG. 5 is a block diagram showing an example of the hardware configuration of the terminal 80. The terminal 80 includes a terminal control unit 81, an operation unit 83, a communication unit 85, a memory 87, a display unit 88, and a storage 89. Terminal 80 may be carried by a user who desires flight control of unmanned aerial vehicle 100.

The terminal control unit 81 is configured by using, for example, a CPU, MPU or DSP. The terminal control unit 81 performs signal processing for centrally controlling the operation of each unit of the terminal 80, data input/output processing with other units, data arithmetic processing, and data storage processing.

The terminal control unit 81 may acquire data or information from the unmanned aerial vehicle 100 via the communication unit 85. The terminal control unit 81 may acquire data or information (for example, various parameters) input via the operation unit 83. The terminal control unit 81 may acquire the data and information held in the memory 87. The terminal control unit 81 may cause the unmanned aerial vehicle 100 to transmit data and information (for example, position, speed, flight route information) via the communication unit 85. The terminal control unit 81 may send data or information to the display unit 88 and cause the display unit 88 to display display information based on the data or information.

The operation unit 83 receives and acquires data and information input by the user of the terminal 80. The operation unit 83 may include input devices such as buttons, keys, a touch screen, and a microphone. Here, it is illustrated that the operation unit 83 and the display unit 88 are mainly configured by a touch panel. In this case, the operation unit 83 can accept touch operations, tap operations, drag operations, and the like. The operation unit 83 may receive information on various parameters. The information input by operation unit 83 may be transmitted to unmanned aerial vehicle 100.

The communication unit 85 wirelessly communicates with the unmanned aerial vehicle 100 by various wireless communication systems. The wireless communication method of this wireless communication may include communication via a wireless LAN, Bluetooth (registered trademark), or a public wireless line, for example. The communication unit 85 may perform wired communication by any wired communication method.

The memory 87 has, for example, a ROM that stores a program that defines the operation of the terminal 80 and data of setting values, and a RAM that temporarily stores various information and data used during processing by the terminal control unit 81. You may. The memory 87 may include a memory other than a ROM and a RAM. The memory 87 may be provided inside the terminal 80. The memory 87 may be provided so as to be removable from the terminal 80. The program may include an application program.

The display unit 88 is configured using, for example, an LCD (Liquid Crystal Display), and displays various information and data output from the terminal control unit 81. The display unit 88 may display various data and information relating to the execution of the application.

The storage 89 accumulates and holds various data and information. The storage 89 may be a HDD, SSD, SD card, USB memory, or the like. The storage 89 may be provided inside the terminal 80. The storage 89 may be detachably provided from the terminal 80. The storage 89 may store an aerial image acquired from the unmanned aerial vehicle 100 and its additional information. The additional information may be held in the memory 87.

Next, the operation of the flight system 10 will be described. The unmanned aerial vehicle 100 or the terminal 80 of the flight system 10 performs a process for estimating the three-dimensional shape of the ground (terrain estimation) based on the plurality of captured images captured by the unmanned aerial vehicle 100. Note that the terrain here may include a wide range of shapes of objects (for example, the ground, buildings, and other objects) reflected in the imaging range when the imaging unit 220 images the ground direction.

FIG. 6 is a flowchart showing an example of a terrain estimation processing procedure. As an example, this process may be performed by the UAV control unit 110 of the unmanned aerial vehicle 100 executing a program stored in the memory 160. In addition, the terminal 80 may perform an operation that assists the terrain estimation processing. For example, the terminal 80 may receive a user operation or display information to the user.

The UAV control unit 110 acquires a flight range and various parameters (S1). In this case, the user may input the flight range and parameters into the terminal 80. The terminal control unit 81 may receive a user input via the operation unit 83 and acquire the input flight range and parameters. The terminal control unit 81 may transmit the flight range and parameters to the terminal 80 via the communication unit 85.

The terminal control unit 81 may acquire the map information from the external server via the communication unit 85. The information on the flight range may be obtained by the user inputting the positions (latitude, longitude) of the four corners of the rectangle in the map information when setting the flight range to a rectangular range, for example. Further, the flight range information may be obtained by the user inputting the radius of a circle centered on the flight position when the flight range is set to a circular range. In addition, the flight range information may be obtained based on the map information by the user inputting information such as a region and a specific place name (for example, Tokyo). Further, the terminal control unit 81 may acquire the flight range held in the memory 87 or the storage 89 from the memory 87 or the storage 89. The flight range may be the range in which unmanned aerial vehicle 100 is scheduled to fly.

The parameters may be imaging parameters related to imaging by the imaging unit 220 or flight parameters related to flight of the unmanned aerial vehicle 100. The imaging parameters include an imaging position, an imaging date and time, a distance to a subject, an imaging angle of view, an attitude of the unmanned aerial vehicle 100 at the time of imaging, an imaging direction, an imaging condition, camera parameters (shutter speed, exposure value, imaging mode, etc.). Good. The flight parameter may include a flight position (three-dimensional position or two-dimensional position), flight altitude, flight speed, flight acceleration, flight path, flight date and time, and the like. The terminal controller 81 may acquire various parameters held in the memory 87 or the storage 89 from the memory 87 or the storage 89. The terminal control unit 81 may transmit the acquired flight range and various parameters to the unmanned aerial vehicle 100 via the communication unit 85.

In the unmanned aerial vehicle 100, the UAV control unit 110 may acquire the flight range and various parameters from the terminal 80 via the communication interface 150. Note that the UAV control unit 110 itself acquires the flight range and various parameters from the external server via the communication interface 150, the memory 160, and each sensor (for example, the GPS receiver 240, etc.) in the unmanned aerial vehicle 100. It may be obtained or derived from the inertial measurement device 250). The UAV control unit 110 may determine the flight route and the imaging position based on the flight range and various parameters.

The UAV control unit 110 generates an initial grid (grid gd in the initial state) as a mesh pattern (S2).

In the grid gd, the terrain in the image capturing range whose shape is to be estimated is virtually represented. Therefore, the grid gd is set to the same range as the imaging range or a range included in the imaging range. When the altitude of each point in the imaging range in the real space is assumed or estimated, the altitude of the grid point in the grid gd in the virtual space is assumed or estimated. The grid gd may have a grid shape, a triangle, another polygon, or another shape, and includes a grid point gp that is the apex of the grid gd.

All the altitudes of the grid points gp may be 0 in the initial state. The initial altitude of the grid point gp may be an arbitrary altitude value set by the user via the operation unit 83 of the terminal 80. In this case, the UAV control unit 110 may acquire the altitude value from the terminal 80 via the communication interface 150.

The interval (grid interval) between the grid points gp may be a predetermined value or may be arbitrarily set by the UAV control unit 110. The grid spacing may be 1 m, 2 m, etc. For example, the terminal control unit 81 of the terminal 80 may specify the grid interval via the operation unit 83 and transmit the grid interval information to the unmanned aerial vehicle 100 via the communication unit 85. The UAV control unit 110 may acquire and set the information on the grid interval via the communication interface 150.

Further, when the information on the rough shape of the imaging range is prepared, the UAV control unit 110 may acquire the information on the rough shape. The UAV control unit 110 may sample the outline shape at the grid points gp, obtain the altitude at each grid point gp, and set the altitude at the grid point gp. For example, the UAV control unit 110 may interlock with a map server on a network connected via the communication interface 150 and acquire the topographical information of the image capturing range as the information of the rough shape. The topographical information may include position information (latitude, longitude, altitude) of each grid point gp in the imaging range. The topographical information may include information on the shape of the ground surface such as buildings, mountains, forests, and steel towers, and information on objects.

The UAV control unit 110 controls flight according to the determined flight path, causes the image capturing unit 220 to capture an image (for example, aerial image capturing), and stores the captured image in the storage 170 (S3). This imaging is performed at each imaging position along the flight path, and a plurality of captured images may be held in the storage 170. Further, the UAV control unit 110 may cause the storage 170 to hold additional information regarding the captured image. This additional information may include the same information as the above-mentioned various parameters (for example, flight parameters and imaging parameters). Therefore, the UAV control unit 110 may acquire information such as the image capturing position of the image capturing unit 220 (that is, the unmanned aerial vehicle 100) when capturing the captured image, the attitude at the time of image capturing, the image capturing direction, and the like.

The UAV control unit 110 acquires a plurality of captured images from the image capturing unit 220 or the storage 170. The UAV control unit 110 may extract a plurality of feature points included in each captured image. The feature point may be a point at any position on the captured image. The UAV control unit 110 may perform matching processing on the plurality of feature points included in each captured image so that the same feature points correspond to each other, and generate corresponding points as corresponding feature points. The UAV control unit 110 may generate a sparse point group including a plurality of corresponding points. The UAV control unit 110 may generate a sparse point cloud according to, for example, sfm. The number of sparse point groups may be, for example, several hundred points per image. The data of the points included in the sparse point group may include data indicating a three-dimensional position. In this way, the UAV control unit 110 generates a sparse point cloud based on a plurality of captured images (S5).

Note that the sparse point group does not have to match the grid point gp of the grid gd. The position of the sparse point group on the two-dimensional plane (the position that does not consider the altitude in the real space) does not have to match the position of the grid point gp on the two-dimensional plane (the position that does not consider the grid altitude). This is because the altitude is not strictly defined.

The UAV control unit 110 may store the generated feature points, corresponding points, and sparse point group data in the memory 160 or the storage 170. As a result, the UAV control unit 110 may generate a sparse point group based on a newly acquired captured image when generating a sparse point group from the next time onward, and the sparse point group before this time is stored in the memory 160. Alternatively, it may be acquired from the storage 170. For example, a new feature point, corresponding point, and sparse point cloud data may be generated by capturing an image of a place that was not captured by the image capturing before this time and the next time. In addition, even if it is the place where the image was taken in the previous image, but the imaging direction and image angle in the previous image were different and the amount of information about the same place was insufficient, new feature points and corresponding points , Sparse point cloud data may be generated. That is, the UAV control unit 110 may add the sparse point cloud based on the new captured image to the sparse point cloud of the existing captured image. In this case, the unmanned aerial vehicle 100 can reduce the processing load for generating sparse point cloud data, reduce the processing time, and improve the real-time property. Therefore, the UAV control unit 110 may sequentially repeat the image capturing, the sparse point cloud generation for the captured image, the new image capturing, and the sparse point cloud generation for the new image.

The UAV control unit 110 uses a meshing technique such as Delaunay triangulation or PSR (Poisson Surface Reconstruction) to form a surface sf having a plurality of points (for example, three points) included in a sparse point group as vertices. To generate. That is, the UAV control unit 110 generates the shape of the surface sf including a plurality of points included in the sparse point group based on the sparse point group. Further, the surface sf may correspond to the shape of the surface that is the estimation target of the shape estimation. The UAV control unit 110 corrects the grid points gp (grid altitude of each grid point gp) so that the surface sf formed by the sparse point group includes the grid points gp in the grid gd, and the grid initial values are set as the grid initial values. Yes (S6). Note that the grid initial value at the grid point gp (the initial value of the grid altitude of the grid) does not always indicate the altitude corresponding to the terrain. The altitude corresponding to the terrain may be searched while changing the altitude at the grid point gp with the grid initial value as the starting point.

The UAV control unit 110 assumes the grid altitude while changing the grid altitude along the search axis (axis perpendicular to the grid plane indicating the ground) for searching the grid altitude with the grid initial value as the starting point. Therefore, the UAV controller 110 generates the hypothetical grid point PS having the hypothesized grid height. The grid altitude corresponds to, for example, the altitude of the estimation target or the terrain corresponding to the grid point. The UAV control unit 110 may derive (for example, calculate) the degree of similarity of images (pixels) with respect to each hypothetical projection position where the same hypothetical grid point PS is projected in a plurality of captured images. The UAV control unit 110 estimates the optimum (probably close to the ground) grid point ps among the plurality of hypothetical grid points PS on the search axis Ax1 based on the similarity of the pixels. The UAV control unit 110 may estimate, as the grid point gp, a hypothetical grid point PS at which the degree of similarity is greater than or equal to the threshold value th1 (for example, maximum). Note that the UAV control unit 110 does not estimate the grid points (grid altitude) based on the similarity of a single pixel, but determines the grid points based on the similarity of a predetermined pixel range including a plurality of adjacent pixels. It may be estimated.

The search method for searching the altitude of the grid point gp along the search axis Ax1 by the UAV control unit 110 may be a full search method or an optimum method (for example, a binary search method or an LM search method). The UAV control unit 110 may calculate the similarity of each hypothetical projection position in each captured image using NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), or the like. ..

Note that the similarity of the pixels at each assumed projection position in each captured image is used as an index for estimating the grid point gp (grid altitude), and the UAV control unit 110 uses the other indices as the basis. , The optimum grid point gp may be estimated. For example, the UAV control unit 110 may assume an error score that results in a low score when the degree of similarity is high, and may estimate an optimum grid point based on the error score. In this case, the UAV control unit 110 may estimate the optimum grid point such that the error score is equal to or less than the threshold th2 (for example, the minimum).

The UAV control unit 110 searches the altitude of the grid point gp for each grid point gp in the grid gd corresponding to the imaging range. The grid points gp here may be all grid points gp in the grid gd or some grid points gp.

In this way, the UAV control unit 110 estimates the altitude of the grid point gp. The UAV control unit 110 estimates the grid shape by estimating the altitude of each grid point gp (S7). The grid shape is the shape of the grid gd that reflects the result of the estimation of the actual altitude of each grid point gp, and corresponds to the shape of the ground in the imaging range.

The UAV control unit 110 determines whether or not the imaging (for example, aerial photography) by the imaging unit 220 included in the unmanned aerial vehicle 100 is completed (S8). The UAV control unit 110 may determine that the aerial shooting is completed, for example, when the unmanned aerial vehicle 100 flies according to a scheduled flight path and the image capturing at the scheduled image capturing position ends. .. The UAV control unit 110 determines that the aerial photography is completed when the user gives an instruction to end the aerial photography via the operation unit 83 of the terminal 80 and the information of this instruction is acquired via the communication unit 85. You may. If the aerial shooting is not completed, the UAV control unit 110 returns to S3 and continues the aerial shooting. When the aerial shooting is completed, the UAV control unit 110 ends the processing of FIG.

The unmanned aerial vehicle 100 repeatedly performs imaging of the ground direction and estimation of the grid shape. Therefore, as the number of iterations increases, the number of generated sparse point clouds increases. Therefore, the terrain shape obtained as the estimation of the grid shape gradually approaches the actual terrain shape. Even in this case, the UAV control unit 110 adds the sparse point group extracted from the newly added captured image to the sparse point group extracted from the existing captured image to create a new sparse point group. Can be generated. Therefore, the unmanned aerial vehicle 100 can suppress an increase in the amount of calculation each time a captured image is acquired, and can realize estimation of a grid shape with high real-time property, that is, terrain estimation.

Note that the UAV control unit 110 may transmit the grid shape information estimated in S7 to the terminal 80 via the communication interface 150. In the terminal 80, the grid shape information estimated via the communication unit 85 may be acquired and displayed via the display unit 88. This allows the user to confirm the estimated grid shape, that is, the estimated ground shape in the flight range.

Note that the grid shape estimation may be performed based on the captured image captured by the image capturing unit 230 instead of the captured image captured by the image capturing unit 220.

It should be noted that at least a part of the terrain estimation processing of FIG. 6 may be executed by the terminal 80. For example, the terminal control unit 81 acquires a captured image from the unmanned aerial vehicle 100 in S3, performs sparse point cloud generation (S5), grid initial value generation (S6), grid shape estimation (S7), and the like. Good. In addition, the terminal control unit 81 may acquire, from the unmanned aerial vehicle 100, information on whether or not the aerial shooting is completed in S9. In addition, the unmanned aerial vehicle 100 and the terminal 80 may share the terrain estimation processing.

Next, the processing in each step of FIG. 6 will be supplemented.

FIG. 7 is a diagram showing an example of generating an initial grid. The shape of the initial grid may be determined based on the imaging range. In FIG. 7, the shape of the initial grid is a planar rectangle. In FIG. 7, a grid gd is generated by combining quadrangles having four adjacent grid points gp as vertices. Other shapes (for example, triangles) may be generated by the adjacent grid points gp, and the other shapes may be combined to generate the grid gd. That is, the unit grid having the apex at each grid point gp in the grid gd may be a triangle, a quadrangle, or any other shape.

FIG. 8 is a diagram illustrating an example of generating a sparse point cloud. The sparse point cloud may be generated by the UAV control unit 110 based on the feature points included in the captured image. In FIG. 8, four sparse point groups (points P1 to P4) are generated. Of the four sparse point groups (points P1 to P4), the sparse point P2 has the lowest altitude and the sparse points P3 and P4 have the highest altitude.

FIG. 9 is a diagram showing a generation example of the surfaces sf1 to sf3 surrounded by the sparse point group (points P1 to P4). The surface sf (sf1, sf2, sf3,...) Representing the shape of the sparse point cloud may be generated by the meshing technique described above. The surface sf1 is a surface surrounded by the sparse points P1, P2, and P4. The surface sf2 is a surface surrounded by the sparse points P1, P2, and P3. The surface sf3 is a surface surrounded by the sparse points P2, P3, and P4. The entire surface represented by the three surfaces sf1 to sf3 has a shape in which the center (sparse point P2) is recessed.

FIG. 10 is a diagram illustrating an example of generating the grid initial value gpa. The altitude of the grid points gp is corrected so that the surfaces sf1 to sf3 surrounded by the sparse point group (points P1 to P4) include the grid points gp. For example, the point where the altitude is 0 in the initial state is corrected according to the surface, or the altitude of the grid point gp estimated in the previous loop of the process of FIG. 6 is based on the sparse point group generated this time. It may be corrected according to the surface sf. The grid point gp having the altitude after this correction becomes the grid initial value gpa at the time of this loop. The same correction may be performed on all grid points gp to generate a grid initial value gpa for each grid point.

In FIG. 10, the grid initial values in which the altitude of each grid point has been corrected are grid initial values gpa1, gpa2, gpa3, gpa4, gpa5. The altitude-corrected grid point gp has a shape that is bulged upward as compared with the initial grid of altitude=0 at each grid point gp.

FIG. 11 is a diagram illustrating an example of searching the altitude of the grid point gp (grid altitude). An example of a grid altitude search using the captured images GM1 and GM2 is shown. In FIG. 11, the unmanned aerial vehicle 100 in flight picks up a plurality of picked-up images GM1 and GM2, so that the picked-up position and the picked-up direction are different. In addition, in FIG. 11, it is assumed that the actual ground SG passes through the vicinity of the assumed grid point PS2 of the grid height h2.

The UAV control unit 110 searches for an optimum grid altitude while changing the grid altitude along the search axis ax1 with the grid initial value gpa as a starting point. In FIG. 11, the grid initial value gpa is the hypothetical grid point PS1 having the highest grid altitude among the assumed grid altitudes, and the search is performed while sequentially lowering the grid altitude. The altitude of the grid initial value gpa is the assumed grid point PS6 having the lowest grid altitude among the assumed grid altitudes, and the search may be performed while sequentially increasing the grid altitude. In addition, a hypothetical grid point (for example, a hypothetical grid point PS3) having a grid altitude of an intermediate portion instead of an end portion on the search axis ax1 may be the initial grid altitude.

First, the UAV control unit 110 assumes that the grid altitude is h1. In this case, in the captured image GM1, the position sg1a (pixel) on the ground is drawn at the assumed projection position PS1a where the assumed grid point PS1 of the grid height h1 is projected. In the captured image GM2, the position sg1b (pixel) on the ground is drawn at the assumed projection position PS1b where the assumed grid point PS1 at the grid height h1 is projected. Therefore, in the two captured images GM1 and GM2, the same hypothetical grid points PS1 that are assumed to be the ground are not drawn at the hypothetical projection positions PS1a and PS1b, but different actual ground SG positions sg1a and sg1b are drawn. To be done. Therefore, the degree of similarity (pixel similarity) of the images drawn at the hypothetical projection positions PS1a and PS1b becomes low.

Next, the UAV control unit 110 assumes that the grid altitude is h2. In this case, in the captured image GM1, the position sg2a (pixel) on the ground is drawn at the assumed projection position PS2a where the assumed grid point PS2 of the grid height h2 is projected. In the captured image GM1, the position sg2b (pixel) on the ground is drawn at the assumed projection position PS2b where the assumed grid point PS2 at the grid height h2 is projected. Therefore, in the two captured images GM1 and GM2, the positions sg2a and sg2b of the actual ground SG close to the same hypothetical grid point PS2 assumed to be the ground are drawn at the hypothetical projection positions PS2a and PS2b. Therefore, the similarity (pixel similarity) of the images drawn at the assumed projection positions PS2a and PS2b when the grid height h2 is assumed is higher than when the grid altitude h1 is assumed.

Therefore, the UAV control unit 110 calculates the similarity between the images drawn at the hypothetical projection positions (PS1a and PS1b or PS2a and PS2b) in the plurality of captured images GM1 and GM2, and based on the calculated similarity, the optimum It is possible to estimate grid points at grid altitudes (estimated as the actual ground). For example, the grid altitude corresponding to the similarity of the threshold th1 or more (for example, the maximum) among the plurality of derived similarities may be estimated as the altitude of the grid point gp.

Note that both of the captured images GM1 and GM2 may be images captured by the imaging unit 220 this time instead of the existing captured images accumulated in the storage 170. The captured image GM1 may be an existing captured image stored in the storage 170, and the captured image GM2 may be the latest captured image captured by the image capturing unit 220 this time. Further, there may be three or more captured images for estimating the grid altitude. There may be a plurality of existing captured images. There may be a plurality of latest captured images.

FIG. 12 is a diagram illustrating an example of grid shape estimation. In FIG. 12, the UAV control unit 110 estimates that the hypothetical grid point PS2 moved upward with respect to the grid initial value gpa is the optimum (most probable) grid point gp based on the similarity of the pixels. To do. The UAV control unit 110 may calculate the similarity between a plurality of images captured by the image capturing unit 220, and estimate the grid altitude at which the similarity is highest as the optimized grid point gpp altitude. In FIG. 12, the grid altitudes may be adjusted within the search ranges za to ze, respectively.

As described above, in the flight system 10 of the first embodiment, the UAV control unit 110 may derive the grid initial value based on the sparse point cloud. The UAV control unit 110 assumes a plurality of grid altitudes with the grid initial value as a starting point, and optimizes the grid altitude based on the similarity of the pixels at the hypothetical projection position where the hypothetical grid point PS having the hypothesized grid altitude is projected. Can be converted. That is, the UAV control unit 110 can re-project the grid points gp on different captured images and calculate the degree of similarity of each re-projected location (provisional projection position). The UAV control unit 110 can recognize the imaging range included in the captured image based on the imaging position and the imaging direction. Therefore, the UAV control unit 110 can specify the assumed projection position in the captured image. The UAV control unit 110 can estimate the actual grid altitude (true value) based on the above similarity.

When generating the sparse point group, the UAV control unit 110 does not need to generate the sparse point group by using all the captured images. The UAV control unit 110 may accumulate information of the sparse point cloud that has been generated once, and may generate a sparse point cloud for a new captured image. The UAV control unit 110 can add a new sparse point cloud to the accumulated sparse point cloud and can use it as data for generating the surface sf. Therefore, the unmanned aerial vehicle 100 can simplify the generation processing of the sparse point cloud having a large processing load and a relatively long calculation time, can reduce the processing load of the UAV control unit 110, and can shorten the processing time. Therefore, the unmanned aerial vehicle 100 can improve the real-time property when estimating the three-dimensional shape of the ground.

As described above, the unmanned aerial vehicle 100 (an example of a shape estimation device) estimates the three-dimensional shape of the terrain based on the plurality of captured images captured by the unmanned aerial vehicle 100 (aircraft). The unmanned aerial vehicle 100 includes a UAV control unit 110 (an example of a processing unit) that performs processing related to estimation of the three-dimensional shape of the terrain. The UAV control unit 110 may generate a grid gd having a plurality of grid points gp in an imaging range imaged by the imaging unit 220 (imaging device) included in the unmanned aerial vehicle 100. The UAV control unit 110 may acquire the captured image GM1 (an example of a first captured image) and the captured image GM2 (an example of a second captured image) captured by the image capturing unit 220. The UAV control unit 110 may acquire information about the three-dimensional position of the image capturing unit 220 when capturing the captured images GM1 and GM2 and the image capturing direction (corresponding to the posture of the image capturing unit 220) by the image capturing unit 220. The UAV control unit 110 may assume a plurality of altitudes of hypothetical grid points PS (an example of the first grid points) at a plurality of grid points gp. The altitude of the assumed grid point PS corresponds to, for example, the altitude of the estimation target or the terrain corresponding to the grid point. The UAV control unit 110, based on the three-dimensional position of the image capturing unit 220 and the image capturing direction of the image capturing unit 220, for a plurality of assumed altitudes, the assumed projection position PS1a (the first projected position PS1a) at which the assumed grid point PS is projected on the captured image GM1. 1) and an assumed projection position PS1b projected on the captured image GM2 (an example of a second assumed projection position) may be derived (for example, calculated). The UAV control unit 110 may derive the degree of similarity between the assumed projection position PS1a in the captured image GM1 and the assumed projection position PS1b in the captured image GM2 for a plurality of assumed altitudes. The UAV control unit 110 estimates the altitude of the grid point gp based on the similarities of a plurality of assumed altitudes. The UAV control unit 110 may estimate the three-dimensional shape of the imaging range based on the estimated altitude of the grid point gp.

Accordingly, the unmanned aerial vehicle 100 calculates the similarity for each assumed projection position in each captured image while changing the altitude, and optimizes the similarity (for example, by deriving the maximum similarity) to generate a grid. The altitude of the point gp can be estimated, and the calculation when performing the three-dimensional shape estimation can be simplified. Therefore, the unmanned aerial vehicle 100 does not need to sequentially perform all of the conventional processes such as raw point generation (for example, sfm), dense point generation (for example, MVS: Multi-View Stereo), mesh generation, and texture generation. Therefore, the unmanned aerial vehicle 100 can reduce the processing load for performing the three-dimensional shape estimation process, and can perform the three-dimensional shape estimation online. Further, since the unmanned aerial vehicle 100 can improve the real-time property, it is possible to obtain the result of the estimation processing of the three-dimensional shape on the spot.

For example, when the degree of similarity is large, each pixel (pixel range) in which the same position (the assumed grid point PS) is projected on a plurality of captured images is the most similar, and therefore these pixels represent the same target. There is a high possibility that The unmanned aerial vehicle 100 can estimate the altitude of the hypothetical grid point PS in this case as the altitude of the position in the real space corresponding to the image position in the captured image in which this pixel exists.

In addition, the UAV control unit 110 may generate sparse point cloud data (an example of point cloud data) based on the captured images GM1 and GM2 by a method such as sfm (Structure from Motion). The UAV control unit 110 may generate the surface sf (an example of surface data) based on the sparse point cloud data. The UAV control unit 110 may derive the initial grid height h1 (grid initial value gpa) of the assumed grid point PS (an example of the initial height of the first grid point) based on the surface sf. The UAV control unit 110 may assume the altitude of the assumed grid point PS in order from the grid altitude h1.

Accordingly, the unmanned aerial vehicle 100 can set the grid altitude h1 of the grid point gp (the assumed grid point PS) based on the surface sf based on the sparse point cloud data, and the search time for deriving the altitude of the grid point gp can be set. Can be shortened. Further, the unmanned aerial vehicle 100 can easily search for the true value even if the initial altitude of the grid point gp deviates significantly from the true value (actual altitude).

In addition, the first captured image may be one or more existing captured images (for example, captured images captured before the previous time). The second captured image may be one or more latest captured images (for example, the captured image captured this time). The UAV control unit 110 acquires the first point cloud data generated based on the first captured image, generates the second point cloud data based on the second captured image, and then generates the first point cloud data. The surface sf may be generated based on the group data and the second point cloud data.

As a result, the unmanned aerial vehicle 100 does not need to calculate the existing point cloud data by using the existing image (for example, the first captured image), and uses the existing image (for example, the second captured image). The calculation for generating the point cloud data based on the above is sufficient. Therefore, the unmanned aerial vehicle 100 can quickly estimate the three-dimensional shape using a plurality of captured images, and can improve the real-time property.

The grid points gp may be all grid points in the grid gd. As a result, the unmanned aerial vehicle 100 can quickly estimate, for example, the three-dimensional shape of the entire imaging range with high real-time property.

The grid points gp may be some grid points gp in the grid gd. As a result, the unmanned aerial vehicle 100 can quickly estimate the three-dimensional shape of a specific range in which an object exists in the imaging range with high real-time property and reduce the calculation cost.

Further, the UAV control unit 110 may set the interval between the grid points gp of the grid gd. For example, the terminal control unit 81 may acquire operation information for setting the interval between the grid points gp via the operation unit 83, and transmit this operation information via the communication unit 85. The UAV control unit 110 may acquire the operation information via the communication interface 150 and set the interval between the grid points gp according to the operation information.

As a result, the unmanned aerial vehicle 100 can set the fineness of the grid gd (the number of grid points gp) and can realize the fineness of the three-dimensional shape estimation desired by the user.

(Embodiment 2)
In the first embodiment, a plurality of altitudes are assumed in the search target of the grid altitudes, and the grid altitudes are estimated based on the similarity of pixels (pixel ranges) at the assumed projection positions in the plurality of captured images for the plurality of altitudes. I assumed that. On the other hand, in the first embodiment, the influence of the altitude estimation on the search target grid by the grid other than the search target is not taken into consideration. The second embodiment shows a case where the grid shape is estimated in consideration of the adjacency relationship of the grids. Although the UAV control unit 110 takes the initiative in estimating the grid shape in the present embodiment, the terminal control unit 81 may take the initiative in estimating the grid shape.

The flight system of the second embodiment has substantially the same configuration as that of the first embodiment. The same reference numerals are used for the same components as in the first embodiment, and the description thereof will be omitted.

The UAV control unit 110 optimizes the altitude of each grid point gp based on the plurality of captured images. In this case, the UAV control unit 110 may calculate the error score E(X) based on the degree of similarity derived in the first embodiment. The UAV control unit 110 may estimate the altitude of each grid point gp based on the calculation result of the error score E(X). The error score E(X) corresponds to the error in the position in the real space indicated by each assumed projection position in each captured image when the assumed grid point PS is projected on a plurality of captured images. If the position error in the real space indicated by each assumed projection position is large, the degree of similarity of pixels at each assumed projection position in each captured image becomes low. When the position in the real space indicated by each assumed projection position is small, the similarity of the pixels at the assumed projection position in each captured image becomes high. The error score E(x) may be represented by Expression (1), for example.

In the calculation of the error score E(x), the UAV control unit 110 assumes a grid point gp in which the altitude x _s (for example, h1, h2,...) Is sequentially changed at each grid point gp, and the assumed grid point PS To generate. The UAV control unit 110 may calculate the error score E(x) by adding the cost of the first term on the right side and the influence degree of the second term on the right side. The first term of Expression (1) represents the cost of a single grid point gp (also referred to as grid point s) to be searched. The second term represents the degree of influence on the grid point gp by the grid point gp2 (also referred to as the grid point t) adjacent to the search target grid point gp. The set V is a set of grid points gp. The set E is a set of sides connecting the grid points s and t that are adjacent to each other in the grid gd.

In Expression (1), x _s represents the altitude of the grid point s. x _t represents the altitude of the grid point t. w _s represents a function based on the distance from the current imaging position (that is, the position of the unmanned aerial vehicle 100) to the position in the real space corresponding to the grid point s. w _st represents a function based on the distance between the grid point s and the grid point t. w _s and w _st each have a smaller value as the distance is longer, and the value of the term (first term or second term) in which w _s and w _st exist becomes smaller and is given to the error score E(X). The effect is small.

π_ｉ（ｘ_ｓ）は、ｉ位置で無人航空機１００の撮像部２２０によって撮像された撮像画像における、グリッド点ｓの画素位置（仮想投影位置に相当）を示す。π_ｊ（ｘ_ｓ）は、ｊ位置で無人航空機１００の撮像部２２０によって撮像された撮像画像における、グリッド点ｓの画素位置（仮想投影位置に相当）を示す。ＮＣＣは、画像間の類似度を表す関数の一例であり、画素位置π_ｉ（ｘ_ｓ）の画像と画素位置π_ｊ（ｘ_ｓ）の画像の類似度を表す。画像間の再投影位置の類似度が高い場合、φ_ｓ（ｘ_ｓ）は、小さな値となり、つまり誤差スコアＥ（Ｘ）が小さくなる。一方、画像間の再投影位置の類似度が低い場合、φ_ｓ（ｘ_ｓ）は、大きな値となり、つまり誤差スコアＥ（Ｘ）が大きくなる。 Further, φ _s (x _s ) indicates the similarity of the reprojection position (virtual projection position) of the hypothetical grid point PS between the plurality of captured images, and is represented by Expression (2).

π _i (x _s ) indicates the pixel position (corresponding to the virtual projection position) of the grid point s in the captured image captured by the imaging unit 220 of the unmanned aerial vehicle 100 at the i position. π _j (x _s ) indicates the pixel position (corresponding to the virtual projection position) of the grid point s in the captured image captured by the imaging unit 220 of the unmanned aerial vehicle 100 at the j position. NCC is an example of a function representing the similarity between images, and represents the similarity between the image at the pixel position π _i (x _s ) and the image at the pixel position π _j (x _s ). When the reprojection position similarity between images is high, φ _s (x _s ) has a small value, that is, the error score E(X) is small. On the other hand, when the reprojection position similarity between images is low, φ _s (x _s ) has a large value, that is, the error score E(X) is large.

X _st indicates the cost of the adjacency relationship, and may include, for example, information on the smoothness between adjacent grid points. X _st (x _s , x _t ) includes the cost between two points of the grid point s and the grid point t that is in the adjacent relationship, and is represented by the equation (3). In the case of Formula (3), the distance (norm) between grid points s and t is shown.

The smoothness information may be held in the memory 160 or may be acquired by the UAV control unit 110 from an external server. The information on the smoothness may include information indicating that the smoothness is low, for example, in the case of a land where the topography is extremely hungry (mountainous region), and when the topography is a plain or water surface (sea surface, river surface), It may include information indicating that the smoothness is high.

The argmin function, which is X shown in Expression (4), is a function that passes x, which is a function, which minimizes the error score E(X), as an argument. The function value of the argmin function is a function for minimizing the reprojection error of the search target grid point s on each captured image. An MRF optimization method such as a tree weight redistribution message transmission method (TRW-S) is used as an optimization solution method for the argmin function (that is, an optimization solution method for the grid shape).

The UAV control unit 110 derives the altitude x _s of the grid point s such that the error score E(x) is less than the threshold th3 (for example, the minimum) according to the equation (4). Accordingly, the UAV control unit 110 can optimize the altitude of each grid point gp based on the plurality of captured images captured by the image capturing unit 220.

For example, when an obstacle is reflected in the captured image, the obstacle may be reflected in the assumed projection position in the projection of the assumed grid point PS in each captured image. In this case, the pixel value of the assumed projection position is Change. Therefore, the obstacle may be erroneously estimated as a part of the terrain. On the other hand, in the present embodiment, the UAV control unit 110 estimates the grid shape in consideration of not only the grid point s to be searched but also the relationship with the grid point t located adjacent to the grid point s. Therefore, the unmanned aerial vehicle 100 can generate the ground shape with higher accuracy by considering the situation of grid points around the search target even when image noise such as an obstacle is present.

As described above, in the present embodiment, as compared with the first embodiment, the relationship with the grid points t around the grid point s to be searched is taken into consideration, so that the unmanned aerial vehicle 100, for example, has an obstacle in the captured image. Even when the image is reflected, it is possible to prevent the obstacle from being mistakenly estimated as a part of the terrain.

As described above, the UAV control unit 110 determines the pixels of the hypothetical projection position PS1a and the hypothetical projection position PS1b based on the similarity (for example, φ _s (x _s )) between the hypothetical projection position PS1a and the hypothetical projection position PS1b. An error score E(X) indicating a value error may be calculated. The UAV control unit 110 may estimate the altitude of the grid point gp (grid point s) based on the error score E(X).

Accordingly, the unmanned aerial vehicle 100 can easily use the error score based on the similarity between the hypothetical projected positions PS1a and PS2b at which the same position (for example, the hypothetical grid point PS1) is projected on the captured images GM1 and GM2. The altitude of grid points can be estimated.

In addition, the UAV control unit 110 is based on the interval (for example, X _st ) between the grid point s (for example, the first grid) to be searched in the grid gd and the adjacent grid point t (for example, the second grid). Then, the error score E(x) may be calculated.

As a result, the unmanned aerial vehicle 100 can adjust the degree of influence of the three-dimensional reconstruction on the shape adjustment by taking into consideration the distance (distance) between the grid point s to be searched and the adjacent grid point t. For example, in the unmanned aerial vehicle 100, the longer the distance between the grid points s and t, the smaller the influence of the three-dimensional reconstruction on the shape adjustment. This is because if the spacing between the grid points gp becomes wider, the relationship between the adjacent grid points s and t becomes smaller due to the undulation of the ground shape or the like. In addition, when the distance is more than a certain degree (for example, the threshold value th4 or more), the second term of the error score E(x) (the latter term of the right side of the equation (1)) is set to a value of 0, and the adjacent grid points t are used. The influence on the grid points s may be eliminated.

Further, the UAV control unit 110 may calculate the error score E(x) based on the smoothness between the grid points s and t.

The relationship between adjacent grid points s and t differs depending on whether the topography between the grid points s and t is smooth. The UAV control unit 110 may calculate the error score E(x) by correcting the above-mentioned similarity by taking the smoothness into consideration. The unmanned aerial vehicle 100 can increase the corrected similarity (lower the error score E(X)) when, for example, the terrain between adjacent grid points s and t is flat terrain, and the adjacent grid points When the terrain between s and t is a terrain with rich undulations, the corrected similarity can be low (the error score E(X) is large). Therefore, the flatter the distance between the adjacent grid points s and t, the smaller the error score E(X), and the unmanned aerial vehicle 100 can easily estimate the grid height to be searched with a small error.

Although the present disclosure has been described using the embodiments, the technical scope of the present disclosure is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be added to the above-described embodiment. It is also apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present disclosure.

The execution order of each process such as operations, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, the specification, and the drawings is "preceding" or "prior to prior". It is possible to realize in any order unless the output of the previous process is used in the subsequent process. The operation flow in the claims, the description, and the drawings, even if it is described by using "first", "next", etc. for convenience, means that it is essential to carry out in this order. is not.

10 Flight system 80 Terminal 81 Terminal control unit 83 Operation unit 85 Communication unit 87 Memory 88 Display unit 89 Storage 100 Unmanned aerial vehicle 110 UAV control unit 150 Communication interface 160 Memory 170 Storage 200 Gimbal 210 Rotary wing mechanism 220, 230 Imaging unit 240 GPS reception Machine 250 Inertia measuring device 260 Magnetic compass 270 Barometric altimeter 280 Ultrasonic sensor 290 Laser measuring device GM1, GM2 Captured image gd Grid gp Grid points P1 to P4 Sparse points PS1, PS2, PS3, PS4 Assumed grid points PS1a, PS1b, PS2a , PS2b Assumed projection positions sf1 to sf3 Surfaces za to ze Search range

Claims (20)

A shape estimating device for estimating a three-dimensional shape based on a plurality of picked-up images picked up by a flying object,
A processing unit that performs processing relating to the estimation of the three-dimensional shape,
The processing unit is
Generating a grid having a plurality of grid points in an imaging range imaged by an imaging device included in the flying object,
Acquiring a first captured image and a second captured image captured by the image capturing device,
Acquiring information on the three-dimensional position of the imaging device and the imaging direction of the imaging device when the first captured image and the second captured image are captured,
Assuming a plurality of altitudes of the first grid point in the plurality of grid points,
A plurality of assumed altitudes, based on a three-dimensional position of the imaging device and an imaging direction by the imaging device, a first hypothetical projection position at which the first grid points are projected on the first captured image; Deriving a second hypothetical projection position at which the first grid point is projected on the second captured image,
For a plurality of assumed altitudes, the degree of similarity between the first assumed projection position in the first captured image and the second assumed projection position in the second captured image is derived,
Estimating the altitude of the first grid point based on the similarities for a plurality of assumed altitudes;
Estimating a three-dimensional shape of the imaging range based on the estimated altitude of the first grid point;
Shape estimation device. The processing unit is
Generating sparse point cloud data based on the first captured image and the second captured image,
Generate surface data based on the sparse point cloud data,
Deriving an initial altitude of the first grid point based on the surface data,
Assuming the altitudes of the first grid points in order from the initial altitude,
The shape estimation device according to claim 1. The first captured image is one or more existing captured images,
The second captured image is one or more latest captured images,
The processing unit is
Acquiring first point cloud data generated based on the first captured image,
Generate second point cloud data based on the second captured image,
Generating the surface data based on the first point cloud data and the second point cloud data,
The shape estimation device according to claim 2. The processing unit is
An error score indicating an error between the first assumed projection position and the second assumed projection position is calculated based on the similarity,
Estimating the altitude of the first grid point based on the error score;
The shape estimation device according to claim 1. The processing unit calculates the error score based on a distance between the first grid point and an adjacent second grid point in the grid.
The shape estimation device according to claim 4. The processing unit calculates the error score based on a smoothness between the first grid point and a second grid point in the grid.
The shape estimation device according to claim 4 or 5. The first grid points are all grid points in the grid,
The shape estimation device according to any one of claims 1 to 6. The first grid points are some grid points in the grid,
The shape estimation device according to any one of claims 1 to 6. The processing unit sets an interval between points of the grid,
The shape estimation device according to claim 1. A shape estimation method for estimating a three-dimensional shape based on a plurality of picked-up images picked up by an air vehicle,
Generating a grid having a plurality of grid points in an imaging range imaged by an imaging device included in the flying object,
Acquiring a first captured image and a second captured image captured by the image capturing device;
A step of acquiring information about a three-dimensional position of the imaging device and an imaging direction of the imaging device when the first captured image and the second captured image are captured,
Assuming a plurality of altitudes of a first grid point at said plurality of grid points,
A plurality of assumed altitudes, based on a three-dimensional position of the imaging device and an imaging direction by the imaging device, a first hypothetical projection position at which the first grid points are projected on the first captured image; Deriving a second hypothetical projection position at which the first grid point is projected on the second captured image,
Deriving a degree of similarity between the first assumed projection position in the first captured image and the second assumed projection position in the second captured image for a plurality of assumed altitudes,
Estimating the altitude of the first grid point based on the similarities for a plurality of assumed altitudes;
Estimating a three-dimensional shape of the imaging range based on the estimated altitude of the first grid point.
Shape estimation method. The step of assuming a plurality of altitudes of the first grid points includes
Generating point cloud data based on the first captured image and the second captured image;
Generating surface data based on the point cloud data,
Deriving an initial altitude of the first grid point based on the surface data,
Assuming the altitudes of the first grid points in order from the initial altitude.
The shape estimation method according to claim 10. The first captured image is one or more existing captured images,
The second captured image is one or more latest captured images,
The step of generating the point cloud data includes
Acquiring first point cloud data generated based on the first captured image;
Generating second point cloud data based on the second captured image,
The step of generating the surface data includes a step of generating the surface data based on the first point cloud data and the second point cloud data.
The shape estimation method according to claim 11. The step of estimating the altitude of the first grid point comprises:
Calculating an error score indicating an error in pixel value between the first hypothetical projected position and the second hypothetical projected position based on the similarity;
Estimating the altitude of the first grid point based on the error score.
The shape estimation method according to claim 10. The step of calculating the error score includes the step of calculating the error score based on a distance between the first grid point and an adjacent second grid point in the grid.
The shape estimation method according to claim 13. The step of calculating the error score includes the step of calculating the error score based on the smoothness between the first grid points and the second grid points in the grid.
The shape estimation method according to claim 13. The first grid points are all grid points in the grid,
The shape estimation method according to any one of claims 10 to 15. The first grid points are some grid points in the grid,
The shape estimation method according to any one of claims 10 to 15. The step of generating the grid includes the step of setting an interval between points of the grid,
The shape estimation method according to any one of claims 10 to 17. A shape estimation device that estimates a three-dimensional shape based on a plurality of captured images captured by a flying object,
Generating a grid having a plurality of grid points in an imaging range imaged by an imaging device included in the flying object,
Acquiring a first captured image and a second captured image captured by the image capturing device;
A step of acquiring information about a three-dimensional position of the imaging device and an imaging direction of the imaging device when the first captured image and the second captured image are captured,
Assuming a plurality of altitudes of a first grid point at said plurality of grid points,
A plurality of assumed altitudes, based on a three-dimensional position of the imaging device and an imaging direction by the imaging device, a first hypothetical projection position at which the first grid points are projected on the first captured image; Deriving a second hypothetical projection position at which the first grid point is projected on the second captured image,
Deriving a degree of similarity between the first assumed projection position in the first captured image and the second assumed projection position in the second captured image for a plurality of assumed altitudes,
Estimating the altitude of the first grid point based on the similarities for a plurality of assumed altitudes;
A step of estimating a three-dimensional shape of the imaging range based on the estimated altitude of the first grid point. A shape estimation device that estimates a three-dimensional shape based on a plurality of captured images captured by a flying object,
Generating a grid having a plurality of grid points in an imaging range imaged by an imaging device included in the flying object,
Acquiring a first captured image and a second captured image captured by the image capturing device;
A step of acquiring information about a three-dimensional position of the imaging device and an imaging direction of the imaging device when the first captured image and the second captured image are captured,
Assuming a plurality of altitudes of a first grid point at said plurality of grid points,
A plurality of assumed altitudes, based on a three-dimensional position of the imaging device and an imaging direction by the imaging device, a first hypothetical projection position at which the first grid points are projected on the first captured image; Deriving a second hypothetical projection position at which the first grid point is projected on the second captured image,
Deriving a degree of similarity between the first assumed projection position in the first captured image and the second assumed projection position in the second captured image for a plurality of assumed altitudes,
Estimating the altitude of the first grid point based on the similarities for a plurality of assumed altitudes;
Estimating a three-dimensional shape of the imaging range based on the estimated altitude of the first grid point;
A computer-readable recording medium in which a program for executing is recorded.

Images