WO2020119572A1 - 形状推断装置、形状推断方法、程序以及记录介质 - Google Patents

形状推断装置、形状推断方法、程序以及记录介质 Download PDF

Info

Publication number
WO2020119572A1
WO2020119572A1 PCT/CN2019/123314 CN2019123314W WO2020119572A1 WO 2020119572 A1 WO2020119572 A1 WO 2020119572A1 CN 2019123314 W CN2019123314 W CN 2019123314W WO 2020119572 A1 WO2020119572 A1 WO 2020119572A1
Authority
WO
WIPO (PCT)
Prior art keywords
grid
camera
assumed
grid point
point
Prior art date
Application number
PCT/CN2019/123314
Other languages
English (en)
French (fr)
Inventor
顾磊
沈思傑
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to CN201980008243.0A priority Critical patent/CN111788457A/zh
Publication of WO2020119572A1 publication Critical patent/WO2020119572A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • G01C11/04—Interpretation of pictures
    • G01C11/06—Interpretation of pictures by comparison of two or more pictures of the same area
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images

Definitions

  • the present disclosure relates to a shape inference device, a shape inference method, a program, and a recording medium for inferring a three-dimensional shape from a plurality of captured images.
  • a platform for taking pictures while traveling along a predetermined fixed path.
  • the platform receives camera instructions from the ground base and takes pictures of the objects.
  • the imaging device of the platform is tilted according to the positional relationship between the platform and the imaged object while flying along a fixed path (see Patent Document 1).
  • Patent Document 1 Japanese Patent Application Publication No. 2010-61216
  • the three-dimensional shape of the ground can be inferred.
  • the calculation of three-dimensional shape inference is more complicated and the calculation cost is higher, which can only be processed offline.
  • prime points for example, sfm: Structure from Motion
  • dense points for example, mvs: Multi-View Stereo
  • generate grids generate textures, etc.
  • the processing load of the three-dimensional shape is large. Therefore, it is difficult for a drone to infer the three-dimensional shape of the ground in real time while taking images.
  • the shape inference device is a shape inference device for inferring a three-dimensional shape based on a plurality of captured images captured by the flying body, and may include a processing unit that performs a related process of three-dimensional shape inference.
  • the provided camera device generates a grid with a plurality of grid points within the imaging range of the camera; acquires the first camera image and the second camera image captured by the camera device; acquires the camera device when the first camera image and the second camera image are captured
  • the three-dimensional position of the camera and the imaging direction information of the camera device assuming multiple heights of the first grid point among multiple grid points; based on the assumed multiple heights, the first grid is derived from the three-dimensional position of the camera device and the imaging direction of the camera device
  • the processing unit can perform three-dimensional reconstruction based on the first and second captured images, obtain point cloud data, generate surface data from the point cloud data, and then derive the initial height of the first grid point from the surface data.
  • One grid height One grid height.
  • the first captured image may be more than one existing captured image.
  • the second camera image may be more than one latest camera image.
  • the processing unit may acquire the first point cloud data generated from the first captured image, generate the second point cloud data from the second captured image, and generate the surface data from the first point cloud data and the second point cloud data.
  • the processing unit may calculate an error score indicating the error of the first assumed projection position and the second assumed projection position based on the similarity, and infer the height of the first lattice point based on the error score.
  • the processing unit may calculate the error score according to the interval between the first grid point and the adjacent second grid point in the grid.
  • the processing unit may calculate the error score based on the smoothness between the first grid point and the second grid point in the grid.
  • the first grid point may be all grid points in the grid.
  • the first grid point may be a partial grid point in the grid.
  • the processing unit can set the interval of each point of the grid.
  • the shape inference method is a shape inference method for inferring a three-dimensional shape based on a plurality of camera images captured by the flying body, including: when performing related processing of three-dimensional shape inference, an imaging device provided in the flying body The steps of generating a grid with a plurality of grid points within the imaging range for shooting; the steps of acquiring the first and second camera images captured by the camera device; acquiring the camera device when capturing the first and second camera images The steps of the three-dimensional position and the imaging direction information of the camera device; the step of assuming a plurality of heights of the first lattice point among the plurality of lattice points; The steps of deriving the first assumed projection position of the first grid point projected on the first camera image and the second assumed projection position of the first grid point projected on the second camera image; deriving the first camera image for the assumed multiple heights The similarity between the first assumed projection position and the second assumed projection position in the second captured image; the step of inferring the height of the
  • the steps of multiple heights of the first grid point may include: a step of performing three-dimensional restoration based on the first camera image and the second camera image to obtain point cloud data; a step of generating surface data from the point cloud data; and deriving the first from the surface data The step of initial height of the grid points; and the step of assuming the height of the first grid point in order from the initial height.
  • the first captured image may be more than one existing captured image.
  • the second camera image may be more than one latest camera image.
  • the step of generating point cloud data may include: a step of acquiring first point cloud data generated from the first captured image; and a step of generating second point cloud data from the second captured image.
  • the step of generating surface data may include the step of generating surface data based on the first point cloud data and the second point cloud data.
  • the step of inferring the height of the first lattice point may include: a step of calculating an error score representing the error of the first and second assumed projection positions according to the similarity; and a step of inferring the height of the first lattice point from the error score.
  • the step of calculating the error score may include the step of calculating the error score according to the interval between the first grid point and the adjacent second grid point in the grid.
  • the step of calculating the error score may include the step of calculating the error score according to the smoothness between the first grid point and the second grid point in the grid.
  • the first grid point may be all grid points in the grid.
  • the first grid point may be a partial grid point in the grid.
  • the step of generating a grid may include the step of setting the interval of each point of the grid.
  • the program is used to cause the shape inference device for inferring the three-dimensional shape based on the plurality of captured images captured by the flying body to perform the following steps: Generate a grid with multiple grid points within the shooting range of the device; obtain the first and second camera images captured by the camera device; obtain the three-dimensional position of the camera device when the first and second camera images are captured And the imaging direction information of the camera device; assuming multiple heights of the first grid point among multiple grid points; based on the assumed multiple heights, the first grid point projection is derived from the three-dimensional position of the camera device and the imaging direction of the camera device The first assumed projection position and the first grid point of the first captured image are projected to the second assumed projection position of the second captured image; the first assumed projection position and the second The similarity of the second assumed projection position in the captured image; the height of the first grid point is inferred based on the assumed similarity of multiple heights; and the three-dimensional shape of the imaging range is inferred based on the estimated height of the first grid
  • the recording medium is a computer-readable recording medium recorded with a program for causing the shape inference device to infer a three-dimensional shape based on a plurality of camera images captured by the flying body to perform the following steps:
  • a grid with multiple lattice points is generated within the imaging range of the imaging device provided by the flying object; the first and second camera images captured by the camera device are acquired; The three-dimensional position of the camera device and the imaging direction information of the camera device during the first and second camera images; assuming multiple heights of the first grid point among multiple grid points; assuming multiple heights, according to the three-dimensional height of the camera device
  • FIG. 1 is a schematic diagram showing a first configuration example of the flying system in the first embodiment.
  • FIG. 2 is a schematic diagram showing a second configuration example of the flying system in the first embodiment.
  • FIG. 3 is a diagram showing an example of a specific appearance of an unmanned aircraft.
  • FIG. 4 is a block diagram showing an example of the hardware configuration of an unmanned aircraft.
  • FIG. 5 is a block diagram showing an example of the hardware configuration of the terminal.
  • FIG. 6 is a flowchart showing the procedure of the terrain estimation operation.
  • FIG. 7 is a diagram showing an example of generation of an initial grid.
  • FIG. 8 is a diagram showing an example of generation of a sparse point cloud.
  • FIG. 9 is a diagram showing an example of generation of a surface surrounded by sparse point clouds.
  • FIG. 10 is a diagram showing an example of generation of grid initial values.
  • FIG. 11 is a diagram illustrating a search example of grid height.
  • FIG. 12 is a diagram illustrating an example of estimation of a grid shape.
  • PS1, PS2, PS3, PS4 assumed grid
  • PS1a, PS1b, PS2a, PS2b assumed projection position
  • an unmanned aircraft (UAV: Unmanned Aerial) is used as an example of a flying body.
  • Unmanned aircraft includes aircraft moving in the air.
  • unmanned aerial vehicles are marked as "UAV".
  • the shape inference system takes the flight system as an example.
  • the shape inference device mainly takes an unmanned aircraft as an example, and may also be a terminal.
  • the terminal may include a smart phone, a tablet terminal PC (Personal Computer) or other devices.
  • the shape estimation method specifies the operation in the shape estimation device.
  • a program (for example, a program that causes the shape estimation device to execute various processes) is recorded in the recording medium.
  • FIG. 1 is a diagram showing a first configuration example of the flying system 10 in the first embodiment.
  • the flight system 10 includes an unmanned aircraft 100 and a terminal 80.
  • the unmanned aircraft 100 and the terminal 80 may communicate with each other through wired communication or wireless communication (for example, wireless LAN (Local Area Network)).
  • wireless LAN Local Area Network
  • FIG. 1 the case where the terminal 80 is a portable terminal (for example, a smartphone, a tablet terminal) is illustrated.
  • the flight system may include an unmanned aircraft, a transmitter (transmitter), and a portable terminal.
  • the transmitter When the transmitter is included, the user can use the left and right joysticks arranged in front of the transmitter to control the flight of the unmanned aircraft.
  • the unmanned aerial vehicle, the transmitter, and the portable terminal can communicate with each other through wired communication or wireless communication.
  • FIG. 2 is a schematic diagram showing a second configuration example of the flying system 10 in the first embodiment.
  • the case where the terminal 80 is a PC is illustrated.
  • the function of the terminal 80 may be the same.
  • FIG. 3 is a diagram showing an example of a specific appearance of the unmanned aerial vehicle 100.
  • a perspective view of the unmanned aircraft 100 when flying in the moving direction STV0 is shown.
  • the unmanned aerial vehicle 100 is an example of a mobile body.
  • the roll axis (refer to the x axis) is set in a direction parallel to the ground and along the moving direction STV0.
  • set the pitch axis in the direction parallel to the ground and perpendicular to the roll axis (refer to the y-axis)
  • set the yaw axis in the direction perpendicular to the ground and perpendicular to the roll axis and pitch axis (Refer to the z axis).
  • the unmanned aerial vehicle 100 includes a UAV main body 102, a universal joint 200, an imaging unit 220, and a plurality of imaging units 230.
  • the UAV body 102 includes a plurality of rotors (propellers).
  • the UAV main body 102 makes the unmanned aircraft 100 fly by controlling the rotation of a plurality of rotors.
  • the UAV main body 102 uses, for example, four rotors to fly the unmanned aerial vehicle 100.
  • the number of rotors is not limited to four.
  • the unmanned aerial vehicle 100 may be a fixed-wing aircraft without a rotor.
  • the imaging unit 220 is an imaging camera that captures an object (for example, a scene above an aerial photography target, a landscape such as a mountain or river, or a building on the ground) included in an expected imaging range.
  • an object for example, a scene above an aerial photography target, a landscape such as a mountain or river, or a building on the ground.
  • the plurality of imaging units 230 are sensing cameras that capture the surroundings of the unmanned aircraft 100 in order to control the flight of the unmanned aircraft 100.
  • the two imaging units 230 may be installed on the front of the nose of the unmanned aircraft 100.
  • the other two imaging units 230 may be installed on the bottom surface of the unmanned aerial vehicle 100.
  • the two imaging units 230 on the front side may be paired and function as a so-called stereo camera.
  • the two imaging units 230 on the bottom surface side may also be paired to function as a stereo camera.
  • the three-dimensional space data around the unmanned aerial vehicle 100 can be generated from the images captured by the plurality of imaging units 230.
  • the number of imaging units 230 included in the unmanned aerial vehicle 100 is not limited to four.
  • the unmanned aerial vehicle 100 only needs to include at least one camera 230.
  • the unmanned aircraft 100 may include at least one camera 230 on the nose, tail, side, bottom, and top of the unmanned aircraft 100, respectively.
  • the angle of view that can be set in the imaging unit 230 can be larger than the angle of view that can be set in the imaging unit 220.
  • the imaging unit 230 may have a single focus lens or a fisheye lens.
  • FIG. 4 is a block diagram showing one example of the hardware configuration of the unmanned aerial vehicle 100.
  • Unmanned aerial vehicle 100 includes UAV control unit 110, communication interface 150, memory 160, memory 170, gimbal 200, rotor mechanism 210, imaging unit 220, imaging unit 230, GPS receiver 240, inertial measurement device (IMU: Inertial Measurement 250), magnetic compass 260, barometric altimeter 270, ultrasonic sensor 280 and laser measuring instrument 290.
  • IMU Inertial Measurement 250
  • magnetic compass 260 magnetic compass 260
  • barometric altimeter 270 ultrasonic sensor 280
  • laser measuring instrument 290 laser measuring instrument 290.
  • the UAV control unit 110 is composed of, for example, a CPU (Central Processing Unit: central processing unit), an MPU (Micro Processing Unit: a microprocessor), or a DSP (Digital Signal Processor: digital signal processor).
  • the UAV control unit 110 performs signal processing for overall control of the operation of each part of the unmanned aircraft 100, data input/output processing with other parts, data calculation processing, and data storage processing.
  • the UAV control unit 110 controls the flight of the unmanned aircraft 100 according to a program stored in the memory 160.
  • the UAV control unit 110 can control the flight.
  • the UAV control unit 110 can take aerial images.
  • the UAV control unit 110 acquires position information indicating the position of the unmanned aircraft 100.
  • the UAV control unit 110 may obtain position information indicating the latitude, longitude, and altitude where the unmanned aircraft 100 is located from the GPS receiver 240.
  • the UAV control unit 110 may obtain latitude and longitude information indicating the latitude and longitude of the unmanned aircraft 100 from the GPS receiver 240 and obtain altitude information indicating the altitude of the unmanned aircraft 100 from the barometric altimeter 270 as position information .
  • the UAV control unit 110 may acquire the distance between the ultrasonic radiation point and the ultrasonic reflection point generated by the ultrasonic sensor 280 as height information.
  • the UAV control unit 110 may obtain orientation information indicating the orientation of the unmanned aircraft 100 from the magnetic compass 260.
  • the orientation information can be expressed by, for example, an orientation corresponding to the orientation of the nose of the unmanned aircraft 100.
  • the UAV control unit 110 may acquire position information indicating the position where the unmanned aircraft 100 should exist when the imaging unit 220 shoots the imaging range to be photographed.
  • the UAV control unit 110 may acquire position information indicating the position where the unmanned aircraft 100 should exist from the memory 160.
  • the UAV control unit 110 may acquire position information indicating the position where the unmanned aircraft 100 should exist from other devices via the communication interface 150.
  • the UAV control unit 110 may refer to the three-dimensional map database to determine the position where the unmanned aircraft 100 can exist, and acquire the position as position information indicating the position where the unmanned aircraft 100 should exist.
  • the UAV control unit 110 can acquire imaging range information indicating the imaging ranges of the imaging unit 220 and the imaging unit 230.
  • the UAV control unit 110 may acquire the angle of view information indicating the angles of view of the imaging unit 220 and the imaging unit 230 from the imaging unit 220 and the imaging unit 230 as a parameter for specifying the imaging range.
  • the UAV control unit 110 may acquire information indicating the imaging directions of the imaging unit 220 and the imaging unit 230 as a parameter for specifying the imaging range.
  • the UAV control unit 110 may acquire posture information indicating the posture state of the imaging unit 220 from the universal joint 200 as, for example, information indicating the imaging direction of the imaging unit 220.
  • the posture information of the imaging unit 220 may indicate the angle at which the universal joint 200 rotates from the reference rotation angle of the pitch axis and the yaw axis.
  • the UAV control unit 110 may acquire position information indicating the position of the unmanned aircraft 100 as a parameter for determining the imaging range.
  • the UAV control unit 110 may delimit the imaging range representing the geographic range captured by the imaging unit 220 according to the viewing angles and imaging directions of the imaging unit 220 and the imaging unit 230 and the location of the unmanned aircraft 100, and generate imaging range information, Get camera range information.
  • the UAV control unit 110 can obtain the imaging range information from the memory 160.
  • the UAV control unit 110 can acquire the imaging range information via the communication interface 150.
  • the UAV control unit 110 controls the universal joint 200, the rotor mechanism 210, the imaging unit 220, and the imaging unit 230.
  • the UAV control unit 110 can control the imaging range of the imaging unit 220 by changing the imaging direction or angle of view of the imaging unit 220.
  • the UAV control unit 110 can control the imaging range of the imaging unit 220 supported by the universal joint 200 by controlling the rotation mechanism of the universal joint 200.
  • the imaging range refers to the geographic range captured by the imaging unit 220 or the imaging unit 230.
  • the camera range is defined by latitude, longitude and altitude.
  • the imaging range may be the range of three-dimensional spatial data defined by latitude, longitude, and altitude.
  • the imaging range may be the range of two-dimensional spatial data defined by latitude and longitude.
  • the imaging range can be determined according to the angle of view and the imaging direction of the imaging unit 220 or the imaging unit 230, and the location where the unmanned aircraft 100 is located.
  • the imaging direction of the imaging unit 220 and the imaging unit 230 can be defined by the azimuth and depression angle of the imaging unit 220 and the imaging unit 230 where the front face of the imaging lens is provided.
  • the imaging direction of the imaging unit 220 can be determined according to the orientation of the nose of the drone 100 and the attitude of the imaging unit 220 relative to the universal joint 200.
  • the imaging direction of the imaging unit 230 may be a direction determined according to the nose orientation of the unmanned aircraft 100 and the position where the imaging unit 230 is provided.
  • the UAV control unit 110 may determine the surrounding environment of the unmanned aircraft 100 by analyzing multiple images captured by the multiple imaging units 230.
  • the UAV control unit 110 may control the flight according to the surrounding environment of the unmanned aircraft 100, for example, avoiding obstacles.
  • the UAV control unit 110 may acquire three-dimensional information (three-dimensional information) indicating the three-dimensional shape (three-dimensional shape) of an object existing around the unmanned aircraft 100.
  • the object may be part of a landscape such as buildings, roads, vehicles, trees, etc., for example.
  • the three-dimensional information is, for example, three-dimensional spatial data.
  • the UAV control unit 110 may generate three-dimensional information representing the three-dimensional shape of an object existing around the unmanned aircraft 100 from each image acquired by the plurality of imaging units 230, thereby acquiring three-dimensional information.
  • the UAV control unit 110 may obtain the three-dimensional information indicating the three-dimensional shape of the object existing around the unmanned aircraft 100 by referring to the three-dimensional map database stored in the memory 160 or the memory 170.
  • the UAV control unit 110 may acquire three-dimensional information related to the three-dimensional shape of objects existing around the unmanned aircraft 100 by referring to the three-dimensional map database managed by a server existing on the network.
  • the UAV control unit 110 controls the flight of the unmanned aircraft 100 by controlling the rotor mechanism 210. That is, the UAV control unit 110 controls the rotor mechanism 210 to control the position including the latitude, longitude, and altitude of the unmanned aircraft 100.
  • the UAV control unit 110 may control the imaging range of the imaging unit 220 by controlling the flight of the unmanned aircraft 100.
  • the UAV control section 110 may control the angle of view of the imaging section 220 by controlling the zoom lens included in the imaging section 220.
  • the UAV control unit 110 may use the digital zoom function of the imaging unit 220 to control the angle of view of the imaging unit 220 through digital zoom.
  • the UAV control unit 110 can move the unmanned aircraft 100 to a specific position on a specific date and time to make the camera unit 220 in a desired position Shooting in the desired environment.
  • the UAV control unit 110 may move the unmanned aircraft 100 to a specific position on a specific date and time to enable the imaging unit 220 to Under the desired environment, shoot the desired shooting range.
  • the communication interface 150 communicates with the terminal 80.
  • the communication interface 150 can perform wireless communication by any wireless communication method.
  • the communication interface 150 can perform wired communication by any wired communication method.
  • the communication interface 150 may transmit an aerial image and additional information (metadata) related to the aerial image to the terminal 80.
  • the memory 160 stores the UAV control unit 110 to the universal joint 200, the rotor mechanism 210, the imaging unit 220, the imaging unit 230, the GPS receiver 240, the inertial measurement device 250, the magnetic compass 260, the barometric altimeter 270, the ultrasonic sensor 280 and the laser measuring instrument 290 Programs required for control, etc.
  • the memory 160 may be a computer-readable recording medium, and may include SRAM (Static Random Access Memory: Static Random Access Memory), DRAM (Dynamic Random Access Memory: Dynamic Random Access Memory), EPROM (Erasable Programmable Read Only Only Memory: erasable At least one of flash memory such as programmable read-only memory), EEPROM (Electrically Erasable Programmable Read-Only Memory: electrically erasable programmable read-only memory), and USB (Universal Serial Bus) memory.
  • SRAM Static Random Access Memory: Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • EPROM Erasable Programmable Read Only Only Only Memory: erasable At least one of flash memory such as programmable read-only memory
  • EEPROM Electrically Erasable Programmable Read-Only Memory: electrically erasable programmable read-only memory
  • USB Universal Serial Bus
  • the memory 170 may include at least one of an HDD (Hard Disk: Drive), an SSD (Solid State Drive), an SD card, a USB memory, and other memory.
  • the memory 170 can store various information and various data.
  • the memory 170 can be detached from the unmanned aerial vehicle 100.
  • the memory 170 may record aerial images.
  • the memory 160 or the memory 170 may store information on the aerial photography position and aerial photography path generated by the terminal 80 or the unmanned aerial vehicle 100.
  • the UAV control unit 110 may set the aerial photography position and aerial photography path information as one of the aerial photography parameters related to the aerial photography scheduled by the unmanned aircraft 100 or the flight parameters related to the flight scheduled by the unmanned aircraft 100.
  • the setting information may be stored in the memory 160 or the memory 170.
  • the gimbal 200 can rotatably support the imaging unit 220 about the yaw axis, the pitch axis, and the roll axis.
  • the gimbal 200 can change the imaging direction of the imaging unit 220 by rotating the imaging unit 220 around at least one of the yaw axis, the pitch axis, and the roll axis.
  • the rotor mechanism 210 has a plurality of rotors and a plurality of drive motors that rotate the plurality of rotors.
  • the rotor mechanism 210 is controlled by the UAV control unit 110 to rotate, thereby flying the unmanned aircraft 100.
  • the number of rotors 211 may be four, for example, or other numbers.
  • the unmanned aerial vehicle 100 may be a fixed-wing aircraft without a rotor.
  • the imaging unit 220 captures an object in a desired imaging range and generates data of a captured image.
  • the image data (for example, aerial image) obtained by the imaging unit 220 may be stored in the memory or memory 170 of the imaging unit 220.
  • the imaging unit 220 is an example of an imaging unit.
  • the imaging unit 230 captures the surroundings of the drone 100 and generates data of captured images.
  • the image data of the imaging unit 230 can be stored in the memory 170.
  • the GPS receiver 240 receives a plurality of signals transmitted from a plurality of navigation satellites (that is, GPS satellites) indicating time and the position (coordinates) of each GPS satellite.
  • the GPS receiver 240 calculates the position of the GPS receiver 240 (that is, the position of the unmanned aircraft 100) based on the received multiple signals.
  • the GPS receiver 240 outputs the position information of the unmanned aircraft 100 to the UAV control unit 110.
  • the UAV control unit 110 may calculate the position information of the GPS receiver 240 instead of the GPS receiver 240. In this case, the information indicating the time and the position of each GPS satellite included in the multiple signals received by the GPS receiver 240 is input to the UAV control unit 110.
  • the inertial measurement device 250 detects the posture of the unmanned aircraft 100 and outputs the detection result to the UAV control unit 110.
  • the inertial measurement device 250 can detect the acceleration in the front-rear, left-right, and up-and-down three-axis directions of the unmanned aircraft 100 and the angular velocity in the three-axis directions of the pitch axis, roll axis, and yaw axis as the attitude of the unmanned aircraft 100.
  • the magnetic compass 260 detects the orientation of the nose of the unmanned aircraft 100 and outputs the detection result to the UAV control unit 110.
  • the barometric altimeter 270 detects the flying altitude of the unmanned aircraft 100 and outputs the detection result to the UAV control unit 110.
  • the ultrasonic sensor 280 emits ultrasonic waves, detects ultrasonic waves reflected on the ground and objects, and outputs the detection results to the UAV control unit 110.
  • the detection result may show the distance from the unmanned aircraft 100 to the ground, that is, the height.
  • the detection result may show the distance from the unmanned aircraft 100 to the object (subject).
  • the laser measuring instrument 290 irradiates the object with laser light, receives the reflected light reflected by the object, and measures the distance between the unmanned aircraft 100 and the object (subject) by the reflected light.
  • a time-of-flight method may be used.
  • FIG. 5 is a block diagram showing an example of the hardware configuration of the terminal 80.
  • the terminal 80 includes a terminal control unit 81, an operation unit 83, a communication unit 85, a memory 87, a display unit 88, and a memory 89.
  • the terminal 80 may be held by a user who wishes to control the flight of the unmanned aircraft 100.
  • the terminal control unit 81 is composed of, for example, a CPU, MPU, or DSP.
  • the terminal control unit 81 performs signal processing for overall control of the operation of each part of the terminal 80, data input/output processing with other parts, data calculation processing, and data storage processing.
  • the terminal control unit 81 can acquire data and information from the unmanned aircraft 100 via the communication unit 85.
  • the terminal control section 81 can acquire data and information (for example, various parameters) input via the operation section 83.
  • the terminal control unit 81 can acquire data and information stored in the memory 87.
  • the terminal control unit 81 may transmit data and information (eg, position, speed, and flight path information) to the unmanned aircraft 100 via the communication unit 85.
  • the terminal control unit 81 may send data and information to the display unit 88, and display the display information based on the data and information on the display unit 88.
  • the operation unit 83 receives and acquires data and information input by the user of the terminal 80.
  • the operation unit 83 may include input devices such as buttons, keys, touch screens, and microphones.
  • the case where the operation unit 83 and the display unit 88 are constituted by a touch panel is mainly exemplified.
  • the operation section 83 can receive a touch operation, a click operation, a drag operation, and the like.
  • the operation unit 83 can receive information on various parameters.
  • the information input by the operation unit 83 may be transmitted to the unmanned aerial vehicle 100.
  • the communication unit 85 performs wireless communication with the unmanned aircraft 100 through various wireless communication methods.
  • the wireless communication method of this wireless communication may include, for example, wireless LAN, Bluetooth (registered trademark), or communication via a public wireless network.
  • the communication unit 85 can perform wired communication by any wired communication method.
  • the memory 87 may include, for example, a ROM that stores programs that specify the operation of the terminal 80, setting value data, and a RAM that temporarily stores various information and data used when the terminal control unit 81 performs processing.
  • the memory 87 may include memory other than ROM and RAM.
  • the memory 87 may be provided inside the terminal 80.
  • the memory 87 may be configured to be detachable from the terminal 80.
  • the program may include an application program.
  • the display unit 88 is configured by, for example, an LCD (Liquid Crystal Display), and displays various information and data output from the terminal control unit 81.
  • the display unit 88 can display various data and information related to the execution of the application program.
  • the memory 89 stores and stores various data and information.
  • the memory 89 may be an HDD, SSD, SD card, USB memory, or the like.
  • the memory 89 may be provided inside the terminal 80.
  • the memory 89 may be detachably provided on the terminal 80.
  • the memory 89 may store aerial images and additional information acquired from the unmanned aerial vehicle 100. The additional information can be stored in the memory 87.
  • the unmanned aircraft 100 or the terminal 80 of the flying system 10 performs processing for inferring (terrain inference) the three-dimensional shape of the ground based on a plurality of camera images captured by the unmanned aircraft 100.
  • the terrain here may include a large-area shape of an object (for example, ground, building, or other object) that is reflected in the imaging range when the imaging unit 220 shoots the ground.
  • FIG. 6 is a flowchart showing an example of the terrain estimation processing procedure. As an example, this process can be performed by the UAV control unit 110 of the unmanned aircraft 100 executing a program stored in the memory 160.
  • the terminal 80 can perform an operation for assisting terrain estimation processing. For example, the terminal 80 may accept user operations or display information to the user.
  • the UAV control unit 110 acquires the flight range and various parameters (S1). In this case, the user can input the flight range and parameters to the terminal 80.
  • the terminal control unit 81 may accept user input via the operation unit 83 and acquire the input flight range and parameters.
  • the terminal control section 81 may transmit the flight range and parameters to the terminal 80 via the communication section 85.
  • the terminal control section 81 can acquire map information from an external server via the communication section 85. For example, when the flight range is set to a rectangular range, the user can obtain the flight range information by inputting the positions (latitude, longitude) of the four corners of the rectangle in the map information. In addition, when the flight range is set to a circular range, the user can obtain the information of the flight range by inputting the radius of the circle centered on the flight position. In addition, the user can obtain the flight range information according to the map information by inputting information such as the area and a specific place name (for example, Tokyo). In addition, the terminal control unit 81 may acquire the flight range stored in the memory 87 and the memory 89 from the memory 87 and the memory 89. The flight range may be the predetermined flight range of the unmanned aircraft 100.
  • the parameters may be imaging parameters related to the imaging performed by the imaging unit 220 and flight parameters related to the flight of the unmanned aircraft 100.
  • Camera parameters can include camera position, camera date and time, distance to the subject, camera angle of view, unmanned aircraft 100 posture during shooting, camera direction, camera conditions, camera parameters (shutter speed, exposure value, camera mode, etc.) ).
  • Flight parameters may include flight position (three-dimensional position or two-dimensional position), flight altitude, flight speed, flight acceleration, flight path, flight date and time, and so on.
  • the terminal control unit 81 can acquire various parameters stored in the memory 87 and the memory 89 from the memory 87 and the memory 89.
  • the terminal control section 81 may transmit the acquired flight range and various parameters to the unmanned aircraft 100 via the communication section 85.
  • the UAV control unit 110 can acquire the flight range and various parameters from the terminal 80 via the communication interface 150.
  • the UAV control unit 110 itself may acquire the flight range and various parameters from an external server via the communication interface 150, or the flight range and various parameters from the memory 160, or from various sensors (such as GPS) in the unmanned aerial vehicle 100
  • the receiver 240, the inertial measurement device 250) acquires the flight range and various parameters, or derives them.
  • the UAV control unit 110 can determine the flight path and the camera position according to the flight range and various parameters.
  • the UAV control unit 110 generates an initial grid (grid gd in the initial state) as a grid pattern (S2).
  • the grid gd virtually represents the topography of the imaging range that is the object of shape estimation. Therefore, the grid gd can be set to the same range as the imaging range or the range included in the imaging range. Then, if the height of each point in the imaging range in the actual space is assumed and inferred, the height of the grid point in the grid gd in the virtual space can be assumed and inferred.
  • the grid gd may be grid-shaped, triangular, other polygonal, or other shapes, including the grid point gp which is the vertex of the grid gd.
  • the heights of the grid points gp can all be 0.
  • the initial state height of the grid point gp may be any height value set by the user via the operation unit 83 of the terminal 80.
  • the UAV control unit 110 can acquire the height value from the terminal 80 via the communication interface 150.
  • the interval (grid interval) of each grid point gp may be a predetermined value, or may be set to an arbitrary value by the UAV control unit 110.
  • the grid interval can be 1m, 2m, etc.
  • the terminal control unit 81 of the terminal 80 may specify the grid interval via the operation unit 83, and then transmit the grid interval information to the unmanned aerial vehicle 100 via the communication unit 85.
  • the UAV control unit 110 can acquire and set grid interval information via the communication interface 150.
  • the UAV control unit 110 can acquire the rough shape information.
  • the UAV control unit 110 may sample the approximate shape at the grid point gp, obtain the height at each grid point gp, and set it as the height of the grid point gp.
  • the UAV control unit 110 may link with a map server on a network connected via the communication interface 150 to acquire terrain information of the imaging range as rough shape information.
  • the terrain information may include position information (latitude, longitude, altitude) of each grid point gp in the imaging range.
  • the terrain information may include information on ground shapes such as buildings, mountains, forests, and towers, and information on objects.
  • the UAV control unit 110 controls the flight based on the determined flight path, causes the imaging unit 220 to capture (for example, aerial photography) images, and saves the captured images in the memory 170 (S3).
  • the shooting may be performed at each imaging position along the flight path, and a plurality of captured images may be stored in the memory 170.
  • the UAV control unit 110 may store additional information related to the captured image in the memory 170.
  • the additional information may include the same information as the aforementioned various parameters (eg, flight parameters, camera parameters). In this way, the UAV control unit 110 can acquire information such as the imaging position of the imaging unit 220 (that is, the unmanned aerial vehicle 100 ), the posture at the time of imaging, and the imaging direction when capturing the captured image.
  • the UAV control unit 110 acquires a plurality of captured images from the imaging unit 220 or the memory 170.
  • the UAV control unit 110 may extract a plurality of feature points included in each captured image.
  • the feature point may be any point on the captured image.
  • the UAV control unit 110 may perform matching processing to make the same feature points correspond to each other, and generate corresponding feature points, that is, corresponding points.
  • the UAV control part 110 may generate a sparse point cloud including a plurality of corresponding points.
  • the UAV control unit 110 may generate a sparse point cloud based on sfm, for example.
  • the number of sparse point clouds can be, for example, about several hundred points per image.
  • the data of the points contained in the sparse point cloud may include data representing the three-dimensional position. As described above, the UAV control unit 110 can generate a sparse point cloud from a plurality of captured images (S5).
  • the sparse point cloud may be inconsistent with the grid point gp of the grid gd.
  • the position of the sparse point cloud on the two-dimensional plane can be inconsistent with the position of the grid point gp on the two-dimensional plane (the position without considering the height of the grid), because there is no Strictly specify the grid height.
  • the UAV control unit 110 may store the generated feature points, corresponding points, and sparse point cloud data in the memory 160 or the memory 170. In this way, the UAV control unit 110 only needs to generate the sparse point cloud based on the newly acquired camera image during the next generation of the sparse point cloud.
  • the sparse point cloud before this time can be obtained from the memory 160 or the memory 170. For example, for places that were not photographed in the previous shooting, the shooting will be performed after the next time to generate new feature points, corresponding points, and sparse point cloud data.
  • the UAV control unit 110 may add the sparse point cloud based on the new captured image to the sparse point cloud of the existing captured image.
  • the unmanned aerial vehicle 100 can reduce the processing load for generating sparse point cloud data, shorten the processing time, and improve real-time performance. Therefore, the UAV control unit 110 may repeatedly execute the following processing sequentially: capture an image, generate a sparse point cloud regarding the captured image, and then capture a new image, generate a sparse point cloud regarding the new image.
  • the UAV control unit 110 generates a surface with multiple points (for example, 3 points) contained in the sparse point cloud as vertices based on rasterization techniques such as Delaunay triangulation technology and PSR (Poisson Surface Reconstruction). (Face) sf.
  • the UAV control unit 110 generates the shape of the surface sf including a plurality of points included in the sparse point cloud based on the sparse point cloud.
  • the surface sf may correspond to the shape of the inferred surface of the shape estimation.
  • the UAV control unit 110 corrects the grid point gp (the grid height of each grid point gp) so that the surface sf formed by the sparse point cloud includes the grid point gp in the grid gd, and sets the grid initial value (S6).
  • the initial value of the grid at the grid point gp (the initial value of the grid point height of the grid) is not limited to expressing the height corresponding to the terrain. It is possible to search for the height corresponding to the terrain while changing the height of the grid point gp with the initial value of the grid as the starting point.
  • the UAV control unit 110 changes the grid height along the search axis for searching the grid height (vertical to the axis representing the grid surface of the ground) with the grid initial value as the starting point, and assumes the grid height. With this, the UAV control unit 110 generates an assumed grid point PS having an assumed grid height.
  • the grid height corresponds to, for example, the height of the estimated object and the terrain corresponding to the grid points.
  • the UAV control unit 110 may derive (for example, calculate) the similarity of images (pixels) for each assumed projection position at which the same assumed grid point PS is projected in a plurality of captured images.
  • the UAV control unit 110 estimates the best (considered to be close to the ground) lattice point ps among the plurality of assumed lattice points PS on the search axis A ⁇ 1 based on the similarity of the pixels.
  • the UAV control unit 110 may infer the assumed lattice point PS whose similarity is greater than or equal to the threshold th1 (for example, the largest) as the lattice point gp.
  • the UAV control unit 110 may infer lattice points based on the similarity within a predetermined pixel range including a plurality of adjacent pixels, instead of inferring lattice points (grid height) based on the similarity of a single pixel.
  • the search method using the UAV control unit 110 to search the grid point gp height along the search axis A ⁇ 1 may be a global search method or an optimal method (for example, a binary search method, an LM search method).
  • the UAV control unit 110 may use NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), SAD (Sum of Absolute) Difference, etc. to calculate the similarity of each assumed projection position in each captured image.
  • the UAV control unit 110 may also estimate the optimal grid point based on other indexes gp. For example, when the similarity is high, the UAV control unit 110 may assume an error score with a low score, and infer the best lattice point based on the error score. In this case, the UAV control unit 110 may estimate the optimal lattice point such that the error score is below the threshold th2 (for example, the smallest value).
  • the UAV control unit 110 may perform a search for the grid point gp height for each grid point gp in the grid gd corresponding to the imaging range.
  • each grid point gp here may be all grid points gp in the grid gd, or may be partial grid points gp.
  • the UAV control unit 110 estimates the height of the grid point gp as described above.
  • the UAV control unit 110 estimates the grid shape by estimating the height of each grid point gp (S7).
  • the grid shape is the shape of the grid gd reflecting the estimation result of the actual height of each grid point gp, and is equivalent to the shape of the ground in the imaging range.
  • the UAV control unit 110 determines whether the shooting (for example, aerial photography) of the imaging unit 220 included in the unmanned aircraft 100 is completed (S8). For example, when the unmanned aerial vehicle 100 flies along a predetermined flight path and the image capturing is completed at a predetermined camera position, the UAV control unit 110 may determine that the aerial photography is completed. When the user instructs the end of the aerial photography via the operation unit 83 of the terminal 80, and the UAV control unit 110 acquires the instruction information via the communication unit 85, it can determine that the aerial photography is completed. If the aerial photography is not completed, the UAV control unit 110 will return to S3 to continue the aerial photography. When the aerial photography is completed, the UAV control unit 110 ends the processing of FIG. 6.
  • the unmanned aerial vehicle 100 repeatedly performs ground direction imaging and grid shape estimation. Therefore, as the number of iterations increases, the number of sparse point clouds generated also increases. Therefore, the terrain shape obtained by inferring the grid shape will be closer to the actual terrain shape.
  • the UAV control unit 110 may add the sparse point cloud extracted from the newly added captured image to the sparse point cloud extracted from the existing captured image to generate a new sparse point cloud. With this, the unmanned aerial vehicle 100 can suppress an increase in the amount of calculation when acquiring a captured image, and can realize the estimation of the grid shape, that is, the terrain estimation, which is highly real-time.
  • the UAV control unit 110 may transmit the grid shape information estimated in S7 to the terminal 80 via the communication interface 150.
  • the terminal 80 acquires the estimated grid shape information via the communication unit 85 and displays it via the display unit 88. In this way, the user can confirm the inferred grid shape, that is, the inferred ground shape within the flight range.
  • the grid shape may be estimated from the captured image captured by the imaging unit 230.
  • the terminal 80 may execute at least a part of the terrain estimation processing in FIG. 6.
  • the terminal control unit 81 may acquire the captured image from the unmanned aerial vehicle 100 in S3, generate a sparse point cloud (S5), generate a grid initial value (S6), infer a grid shape (S7), and the like.
  • the terminal control unit 81 may acquire information on whether the aerial photography in S9 is completed from the unmanned aircraft 100.
  • the unmanned aerial vehicle 100 and the terminal 80 may share the terrain estimation processing.
  • FIG. 7 is a diagram showing an example of generation of an initial grid.
  • the shape of the initial grid can be determined according to the imaging range.
  • the shape of the initial grid is a planar rectangle.
  • a quadrilateral with four adjacent grid points gp as vertices is combined to generate a grid gd.
  • the adjacent grid points gp may also be used to generate other shapes (for example, triangles), and the other shapes may be combined to generate a grid gd. That is to say, the unit grid with each grid point gp in the grid gd as the vertex may be a triangle, a quadrangle, or other shapes.
  • the UAV control unit 110 may generate a sparse point cloud based on the feature points included in the captured image.
  • four sparse point clouds (points P1 to P4) are generated.
  • the height of the sparse point P2 is the lowest, and the heights of the sparse points P3 and P4 are higher.
  • FIG. 9 is a diagram showing an example of generation of surfaces sf1 to sf3 surrounded by sparse point clouds (points P1 to P4).
  • the surface sf (sf1, sf2, sf3, ...) representing the shape of a sparse point cloud can be generated by the above rasterization technique.
  • the surface sf1 is a surface surrounded by sparse points P1, P2, and P4.
  • the surface sf2 is a surface surrounded by sparse points P1, P2, and P3.
  • the surface sf3 is a surface surrounded by sparse points P2, P3, and P4.
  • the entire surface represented by the three surfaces sf1 to sf3 has a shape in which the center (sparse point P2) is depressed.
  • FIG. 10 is a diagram showing an example of generation of the grid initial value gpa.
  • the grid point gp having this corrected height becomes the grid initial value gpa at this cycle.
  • the same correction can be performed on all grid points gp to generate the grid initial value gpa for each grid point.
  • the initial values of the grids after correcting the height of each grid point are the initial grid values gpa1, gpa2, gpa3, gpa4, and gpa5.
  • the grid point gp whose height is corrected has a shape that swells upward.
  • FIG. 11 is a diagram illustrating a search example of the height (grid height) of the grid point gp. Shown is an example of grid height search using the captured images GM1, GM2.
  • the unmanned aerial vehicle 100 in flight captures a plurality of captured images GM1 and GM2, and therefore, the imaging positions and imaging directions are different from each other.
  • the actual ground surface SG is assumed to pass near the assumed grid point PS2 of the grid height h2.
  • the UAV control unit 110 searches for the optimal grid height while changing the grid height along the search axis a ⁇ 1, using the grid initial value gpa as a starting point.
  • the grid initial value gpa is the assumed grid point PS1 having the highest grid height among the assumed grid heights, and then the grid height is sequentially lowered for searching.
  • the height of the grid initial value gpa may be set to the assumed grid point PS6 having the lowest grid height among the assumed grid heights, and then the grid height may be sequentially increased for searching.
  • an assumed grid point (for example, assumed grid point PS3) having a grid height in the middle portion instead of the end in the search axis a ⁇ 1 may be set as the height of the grid initial value.
  • the UAV control unit 110 assumes a grid height h1.
  • the position sg1a (pixel) on the ground is drawn at the assumed projection position PS1a of the assumed grid point PS1 of the projection grid height h1.
  • the position sg1b (pixel) on the ground is plotted. Therefore, in the two captured images GM1 and GM2, at the assumed projection positions PS1a and PS1b, instead of drawing the same assumed grid point PS1 assumed to be the ground, different positions sg1a and sg1b of the actual ground SG are drawn. By this, the similarity (similarity of pixels) of the images drawn at the assumed projection positions PS1a, PS1b becomes low.
  • the UAV control unit 110 assumes a grid height h2.
  • the position sg2a (pixel) on the ground is drawn at the assumed projection position PS2a of the assumed grid point PS2 of the projection grid height h2.
  • the position sg2b (pixel) on the ground is drawn at the assumed projection position PS2b of the assumed grid point PS2 of the projection grid height h2.
  • the positions sg2a and sg2b of the actual ground SG are drawn at the assumed projection positions PS2a and PS2b, and the positions sg2a and sg2b of the actual ground SG are close to the same assumed grid point PS2 assumed to be the ground.
  • the similarity (similarity of pixels) of the images drawn at the assumed projection positions PS2a and PS2b when the grid height h2 is assumed is higher than the case where the grid height h1 is assumed.
  • the UAV control unit 110 can calculate the similarity of the images drawn at the assumed projection positions (PS1a and PS1b, or PS2a and PS2b) among the plurality of captured images GM1, GM2, and infer from the calculated similarity
  • the grid height corresponding to the similarity above the threshold th1 (for example, the largest) among the derived multiple similarities may be inferred as the height of the grid point gp.
  • both the captured images GM1 and GM2 may be images captured by the imaging unit 220 this time instead of the existing captured images stored in the memory 170.
  • the captured image GM1 may be an existing captured image stored in the memory 170
  • the captured image GM2 may be the latest captured image captured by the imaging unit 220 this time.
  • FIG. 12 is a diagram illustrating an example of estimation of a grid shape.
  • the UAV control unit 110 estimates that the assumed grid point PS2 after moving upward relative to the grid initial value gpa is the best (closest to accurate) grid point gp based on the similarity of the pixels.
  • the UAV control unit 110 may calculate the similarity of multiple images captured by the imaging unit 220, and infer the grid height with the highest similarity as the height of the optimal grid point gPP.
  • the grid height can be adjusted within the search ranges za to ze, respectively.
  • the UAV control unit 110 can derive the grid initial value from the sparse point cloud.
  • the UAV control unit 110 may start from the initial grid value, assume multiple grid heights, and optimize the grid height based on the pixel similarity of the assumed projection position projected by the assumed grid point PS having the assumed grid height.
  • the UAV control unit 110 may reproject the grid point gp onto different captured images, and calculate the similarity of each location (presumed projection position) of the reprojection.
  • the UAV control unit 110 can recognize the imaging range included in the captured image based on the imaging position and imaging direction. With this, the UAV control section 110 can determine the assumed projection position in the captured image.
  • the UAV control unit 110 can estimate the actual grid height (true value) based on the similarity.
  • the UAV control unit 110 When generating a sparse point cloud, the UAV control unit 110 does not need to use all stored captured images to generate a sparse point cloud.
  • the UAV control unit 110 may store the previously generated sparse point cloud information in advance, and then generate the sparse point cloud for the new captured image.
  • the UAV control unit 110 may add a new sparse point cloud to the stored sparse point cloud and use it as data for generating the surface sf.
  • the unmanned aerial vehicle 100 can simplify the generation processing of the sparse point cloud with a large processing load and a relatively long calculation time, reduce the processing load of the UAV control unit 110, and shorten the processing time. Therefore, the unmanned aerial vehicle 100 can improve the real-time performance when estimating the three-dimensional shape of the ground.
  • the unmanned aerial vehicle 100 estimates the three-dimensional shape of the terrain based on a plurality of camera images captured by the unmanned aerial vehicle 100 (a flying body).
  • the unmanned aerial vehicle 100 includes a UAV control unit 110 (an example of a processing unit) for performing processing related to inferring a three-dimensional shape of terrain.
  • the UAV control unit 110 may generate a grid gd having a plurality of grid points gp within the imaging range of the imaging unit 220 (imaging device) included in the unmanned aerial vehicle 100 for imaging.
  • the UAV control unit 110 can acquire the captured image GM1 (an example of the first captured image) and the captured image GM2 (an example of the second captured image) captured by the imaging unit 220.
  • the UAV control unit 110 can acquire information on the three-dimensional position of the imaging unit 220 and the imaging direction of the imaging unit 220 (corresponding to the posture of the imaging unit 220) when the captured images GM1 and GM2 are captured.
  • the UAV control section 110 may assume a plurality of heights of the assumed lattice points PS (one example of the first lattice point) among the plurality of lattice points gp. It is assumed that the height of the grid point PS corresponds to the height of the estimated object and the terrain corresponding to the grid point, for example.
  • the UAV control unit 110 may derive (for example, calculate) the assumed projection position PS1a (first) of the assumed grid point PS projected on the captured image GM1 based on the three-dimensional position of the imaging unit 220 and the imaging direction of the imaging unit 220 (first (An example of an assumed projection position) and an assumed projection position PS1b projected on the captured image GM2 (an example of a second assumed projection position).
  • the UAV control section 110 may derive the similarity between the assumed projection position PS1a in the captured image GM1 and the assumed projection position PS1b in the captured image GM2.
  • the UAV control unit 110 estimates the height of the grid point gp based on the assumed similarity of a plurality of heights.
  • the UAV control unit 110 may estimate the three-dimensional shape of the imaging range based on the estimated height of the grid point gp.
  • the unmanned aerial vehicle 100 can calculate the similarity for each assumed projection position in each captured image while changing the altitude to optimize the similarity (for example, derive the maximum similarity), thereby inferring the height of the grid point gp, Simplify calculations when inferring three-dimensional shapes. Therefore, the unmanned aerial vehicle 100 does not need to sequentially generate prime points (for example, sfm), generate dense points (for example, MVS: Multi-View Stereo), generate grids, generate textures, etc. in order as in the past.
  • the unmanned aerial vehicle 100 can reduce the processing load of the three-dimensional shape estimation processing and perform the three-dimensional shape estimation online.
  • the unmanned aerial vehicle 100 can improve real-time performance, so that the result of the three-dimensional shape inference processing can be obtained on the spot.
  • each pixel (pixel range) projected at the same position (assuming a grid point PS) among the multiple captured images is the most similar. Therefore, these pixels are more likely to represent the same object.
  • the unmanned aerial vehicle 100 can infer the height of the assumed grid point PS in this case as the height of the actual spatial position corresponding to the image position in the captured image where the pixel is located.
  • the UAV control unit 110 may generate sparse point cloud data (an example of point cloud data) based on the captured images GM1 and GM2, for example, according to the sfm (Structure from Motion) method.
  • the UAV control section 110 may generate a surface sf (an example of surface data) based on sparse point cloud data.
  • the UAV control unit 110 may derive the initial grid height h1 (the grid initial value gpa) of the assumed grid point PS (an example of the initial grid height of the first grid point) from the plane sf.
  • the UAV control unit 110 may assume the height of the assumed grid point PS in order from the grid height h1.
  • the unmanned aerial vehicle 100 can set the grid height h1 of the grid point gp (assumed grid point PS) based on the surface sf based on the sparse point cloud data, and the search time for deriving the grid point gp height can be shortened.
  • the unmanned aerial vehicle 100 can easily search for the true value.
  • the first captured image may be one or more existing captured images (for example, captured images taken before the previous time).
  • the second captured image may be one or more latest captured images (for example, the captured image captured this time).
  • the UAV control unit 110 may acquire the first point cloud data generated from the first captured image, generate the second point cloud data from the second captured image, and generate the surface sf based on the first point cloud data and the second point cloud data.
  • the unmanned aerial vehicle 100 does not need to calculate the existing point cloud data by using the existing image (for example, the first camera image), but only needs to generate the point cloud data based on the latest image (for example, the second camera image) Operation. Therefore, the unmanned aerial vehicle 100 can quickly infer the three-dimensional shape using multiple camera images, improving real-time performance.
  • the grid points gp may be all grid points in the grid gd.
  • the unmanned aerial vehicle 100 can infer the three-dimensional shape of the entire imaging range, for example, with high real-time and quickly.
  • the grid point gp may be a partial grid point in the grid gd.
  • the unmanned aerial vehicle 100 can infer the three-dimensional shape of the specific range where the object in the imaging range is located in a high real-time and quickly, and reduce the calculation cost.
  • the UAV control unit 110 may set the interval between the grid points gp of the grid gd.
  • the terminal control unit 81 may acquire operation information for setting the interval of each grid point gp via the operation unit 83 and then transmit the operation information via the communication unit 85.
  • the UAV control unit 110 may acquire operation information via the communication interface 150, and then set the interval of each grid point gp according to the operation information.
  • the unmanned aerial vehicle 100 can set the fineness of the grid gd (how much of the grid point gp), and realize the fineness of the three-dimensional shape estimation desired by the user.
  • the grid height is estimated based on the similarity of pixels (pixel range) of the assumed projection positions of the plurality of heights in the plurality of captured images.
  • the influence of the non-search target grid on estimating the height of the search target grid is not considered.
  • the shape of the grid is estimated in consideration of the neighboring relationship of the grid is shown.
  • the grid shape is estimated using the UAV control unit 110 as the dominant factor is described, but the grid shape may be estimated using the terminal control unit 81 as the dominant factor.
  • the flight system of the second embodiment has substantially the same configuration as the first embodiment.
  • the same symbols are used, and the descriptions thereof are omitted.
  • the UAV control unit 110 optimizes the height of each grid point gp based on a plurality of captured images.
  • the UAV control unit 110 may calculate the error score E(X) based on the similarity derived in the first embodiment.
  • the UAV control unit 110 can infer the height of each grid point gp based on the calculation result of the error score E(X).
  • the error score E(X) corresponds to the position error in the actual space indicated by each assumed projection position in each captured image.
  • the greater the position error in the actual space represented by each assumed projection position the lower the pixel similarity of the assumed projection position in each captured image.
  • the smaller the position error in the actual space represented by each assumed projection position the higher the pixel similarity of the assumed projection position in each captured image.
  • the error score E(x) can be expressed by formula (1), for example.
  • each grid point gp is a grid point gp that sequentially changes the height x s (for example, h1, h2, ...), and generates a hypothetical grid point PS.
  • the UAV control unit 110 may calculate the error score E(x) by adding the cost of the first item on the right and the degree of influence of the second item on the right.
  • the first term of equation (1) represents the cost of a single search target grid point gp (also called grid point s).
  • the second term indicates the degree of influence of the grid point gp2 (also called grid point t) adjacent to the search target grid point gp on the grid point gp.
  • Set V is the set of grid points gp.
  • the set E is a set of edges connecting the lattice point s and the lattice point t in the adjacent relationship in the grid gd.
  • x s represents the height of the grid point s.
  • x t represents the height of the grid point t.
  • w s represents a function according to the distance from the current imaging position (ie, the position of the unmanned aerial vehicle 100) to the actual spatial position corresponding to the grid point s.
  • W st represents a function according to the distance between lattice point s and lattice point t. The longer the distance, w s, W st the respective smaller value, w s, the value of W st where the term (the first term or second term) becomes small, effect on the error rates E (X) becomes small.
  • ⁇ i (x s ) represents the pixel position of the grid point s (equivalent to the assumed projection position) in the captured image captured by the imaging unit 220 of the unmanned aircraft 100 at the i position.
  • ⁇ i (x s ) represents the pixel position (equivalent to the assumed projection position) of the grid point s in the captured image captured by the imaging unit 220 of the unmanned aircraft 100 at the j position.
  • NCC is an example of a function representing the similarity between pixels, representing the similarity of the image at the pixel position ⁇ i (x s ) and the image at the pixel position ⁇ j (x s ).
  • X st represents the cost of the adjacent relationship, and may include, for example, smoothness information between adjacent lattice points.
  • X st (x s , x t ) includes the cost between two points, the grid point s and the grid point t in an adjacent relationship, and can be expressed by equation (3). In the case of formula (3), it represents the distance (norm) between lattice points s and t.
  • the smoothness information can be stored in the memory 160, and can be obtained from an external server using the UAV control unit 110.
  • the smoothness information may include information indicating low smoothness; if the terrain is plain or water surface (sea surface, river surface), it may include information indicating high smoothness.
  • the X that is, the argmin function shown in equation (4) is a function that passes as the independent variable x the smallest error score E(X) as a function.
  • the function value of the argmin function is a function for minimizing the reprojection error of the search target grid point s on each captured image.
  • MRF optimization methods such as tree weighted message passing algorithm (TRW-S) can be used.
  • the UAV control unit 110 derives the height x s of the grid point s whose error score E(x) is smaller than the threshold rh3 (for example, the smallest value) based on equation (4). In this way, the UAV control unit 110 can optimize the height of each grid point gp based on the plurality of captured images captured by the imaging unit 220.
  • the UAV control unit 110 in addition to the search target grid point s, the UAV control unit 110 also estimates the grid shape by considering its relationship with the adjacent grid point t. With this, the unmanned aerial vehicle 100 can generate a ground shape with higher accuracy by combining grid conditions around the search target even when image noise such as obstacles exists.
  • the relationship between the search target grid point s and the surrounding grid points t is also considered. Therefore, for example, even if there are obstacles mapped on the captured image, no one Piloting the aircraft 100 can also suppress mistaken inference of obstacles as part of the terrain.
  • the UAV control section 110 may determine the similarity between the assumed projection position PS1a and the assumed projection position PS1b (for example ), an error score E(X) representing the pixel value error of the assumed projection position PS1a and the assumed projection position PS1b is calculated.
  • the UAV control unit 110 may estimate the height of the grid point gp (grid point s) based on the error score E(X).
  • the unmanned aerial vehicle 100 can simply use the error score to infer the grid points based on the similarity of the assumed projection positions PS1a and PS2b at the same position (for example, the assumed grid point PS1) projected in the captured images GM1 and GM2. height.
  • the UAV control section 110 may determine the interval between the search target grid point s (an example of the first grid) and the adjacent grid point t (for example, an example of the second grid) in the grid gd (for example, X st ), the error score E(x) is calculated.
  • the unmanned aerial vehicle 100 can adjust the degree of influence on the adjustment of the three-dimensional restoration shape in accordance with the interval (distance) between the search target grid point s and the adjacent grid point t. For example, the unmanned aerial vehicle 100 can make the interval between grid points s and t longer, the less the effect on the three-dimensional restoration shape adjustment. The reason for this is that, if the interval between grid points gp becomes larger, the fluctuation of the ground shape, etc., will reduce the relationship between adjacent grid points s and t.
  • the second term of the error score E(x) (the last half term on the right side of equation (1)) can be set to a value of 0 to eliminate adjacent grid points t Effect on grid point s.
  • the UAV control unit 110 may calculate the error score E(x) based on the smoothness between the lattice point s and the lattice point t.
  • the UAV control unit 110 may correct the above similarity in combination with the smoothness to calculate the error score E(x). For example, when the terrain between adjacent grid points s and t is flat terrain, the unmanned aerial vehicle 100 can increase the corrected similarity (reducing the error score E(X)); when the adjacent grid points s and t When the terrain in between is rich and undulating terrain, the unmanned aerial vehicle 100 can reduce the corrected similarity (increasing the error score E(X)). As a result, the flatter the adjacent grid points s and t, the easier the error score E(X) becomes smaller, and the unmanned aircraft 100 can easily infer the grid height of the search target with a small error.

Abstract

一种在推断地面三维形状时可以提高实时性的形状推断装置。处理部生成具有多个格点的栅格；获取摄像装置拍摄的第一摄像图像及第二摄像图像；假定多个格点中第一格点的多个高度；就假定的多个高度，根据摄像装置的三维位置及摄像装置的摄像方向，导出第一格点投影于第一摄像图像的第一假定投影位置和第一格点投影于第二摄像图像的第二假定投影位置；就假定的多个高度，导出第一假定投影位置和第二假定投影位置的相似度；根据相似度，推断第一格点的高度；根据第一格点的高度，推断摄像范围的三维形状。

Description

形状推断装置、形状推断方法、程序以及记录介质 技术领域
本公开涉及一种根据多个摄像图像来推断三维形状的形状推断装置、形状推断方法、程序以及记录介质。
背景技术
已知现有一种一边按照预先设定的固定路径行进一边进行拍摄的平台(无人机)。该平台从地面基地接收摄像指示并对摄像对象进行拍摄。该平台对摄像对象进行拍摄时，一边按照固定路径飞行，一边根据平台和摄像对象的位置关系来倾斜平台的摄像设备进行拍摄(参照专利文献1)。
【专利文献1】日本特开2010-61216号公报
发明内容
【发明所要解决的技术问题】
使用专利文献1的无人机所拍摄的多个图像，可以推断地面的三维形状。但是，三维形状推断的计算较为复杂，计算成本较高，只能离线处理。例如，在现有的三维形状推断处理中，需要依次生成素点(例如sfm：Structure from Motion)、生成密点(例如mvs：Multi-View Stereo)、生成栅格、生成纹理等，用于推断三维形状的处理负荷较大。因此，无人机很难一边拍摄图像一边实时地推断地面的三维形状。
【用于解决问题的技术手段】
在本发明的一个方面中，形状推断装置是根据飞行体所拍摄的多个摄像图像来推断三维形状的形状推断装置，可以包括进行三维形状推断的相关处理的处理部，处理部在飞行体所具备的摄像装置进行拍摄的摄像范围内生成具有多个格点的栅格；获取摄像装置所拍摄的第一摄像图像及第二摄像图像；获取拍摄第一摄像图像及第二摄像图像时摄像装置的三维位置及摄像装置的摄像方向信息；假定多个格点中第一格点的多个高度；就假定的多个高度，根据摄像装置的三维位置及摄像装置的摄像方向，导出第一格点投影于第一摄像图像的第一假定投影位置和第一格点投影于第二摄像图像的第二假定投影位置；就假定的多个高度，导出第一摄像图像中的第一假定投影位置和第二摄像图像中的第二假定投影位置的相似度；根据假定的多个高度的相似度，推断第一格点的高度；以及根据所推断的第一格点的高度，推断摄像范围的三维形状。
处理部可以根据第一摄像图像及第二摄像图像进行三维复原，获取点云数据，根据点云数据生成面数据，再根据面数据导出第一格点的初始高度，从初始高度开始依次假定第一格点的高度。
第一摄像图像可以是一个以上现有的摄像图像。第二摄像图像可以是一个以上最新的摄像图像。处理部可以获取根据第一摄像图像而生成的第一点云数据，根据第二摄像图像生成第二点云数据，根据第一点云数据和第二点云数据生成面数据。
处理部可以根据相似度计算出表示第一假定投影位置和第二假定投影位置的误差的误差评分，根据误差评分推断第一格点的高度。
处理部可以根据栅格中第一格点与相邻的第二格点之间的间隔，计算出误差评分。
处理部可以根据第一格点与栅格中第二格点之间的平滑度，计算出误差评分。
第一格点可以是栅格中的所有格点。
第一格点可以是栅格中的部分格点。
处理部可以设定栅格各点的间隔。
在本发明的一个方面中，形状推断方法是根据飞行体所拍摄的多个摄像图像来推断三维形状的形状推断方法，包括：进行三维形状推断的相关处理时，在飞行体所具备的摄像装置进行拍摄的摄像范围内生成具有多个格点的栅格的步骤；获取摄像装置所拍摄的第一摄像图像及第二摄像图像的步骤；获取拍摄第一摄像图像及第二摄像图像时摄像装置的三维位置及摄像装置的摄像方向信息的步骤；假定多个格点中第一格点的多个高度的步骤；就假定的多个高度，根据摄像装置的三维位置及摄像装置的摄像方向，导出第一格点投影于第一摄像图像的第一假定投影位置和第一格点投影于第二摄像图像的第二假定投影位置的步骤；就假定的多个高度，导出第一摄像图像中的第一假定投影位置和第二摄像图像中的第二假定投影位置的相似度的步骤；根据假定的多个高度的相似度，推断第一格点的高度的步骤；以及根据所推断的第一格点的高度，推断摄像范围的三维形状的步骤。
假定第一格点多个高度的步骤可以包括：根据第一摄像图像及第二摄像图像进行三维复原，获取点云数据的步骤；根据点云数据生成面数据的步骤；根据面数据导出第一格点的初始高度的步骤；以及从初始高度开始依次假定第一格点的高度的步骤。
第一摄像图像可以是一个以上现有的摄像图像。第二摄像图像可以是一个以上最新的摄像图像。生成点云数据的步骤可以包括：获取根据第一摄像图像而生成的第一点云数据的步骤；以及根据第二摄像图像生成第二点云数据的步骤。生成面数据的步骤可以包括：根据第一点云数据和第二点云数据，生成面数据的步骤。
推断第一格点高度的步骤可以包括：根据相似度计算出表示第一假定投影位置和第二假定投影位置的误差的误差评分的步骤；以及根据误差评分推断第一格点高度的步骤。
计算误差评分的步骤可以包括：根据栅格中第一格点与相邻的第二格点之间的间隔，计算出误差评分的步骤。
计算误差评分的步骤可以包括：根据第一格点与栅格中第二格点之间的平滑度，计算出误差评分的步骤。
第一格点可以是栅格中的所有格点。
第一格点可以是栅格中的部分格点。
生成栅格的步骤可以包括设定栅格各点的间隔的步骤。
在本发明的一个方面中，程序用于使根据飞行体所拍摄的多个摄像图像来推断三维形状的形状推断装置执行如下步骤：进行三维形状推断的相关处理时，在飞行体所具备的摄像装置进行拍摄的摄像范围内生成具有多个格点的栅格；获取摄像装置所拍摄的第一摄像图像及第二摄像图像；获取拍摄第一摄像图像及第二摄像图像时摄像装置的三维位置及摄像装置的摄像方向信息；假定多个格点中第一格点的多个高度；就假定的多个高度，根据摄像装置的三维位置及摄像装置的摄像方向，导出第一格点投影于第一摄像图像的第一假定投影位置和第一格点投影于第二摄像图像的第二假定投影位置；就假定的多个高度，导出第一摄像图像中的第一假定投影位置和第二摄像图像中的第二假定投影位置的相似度；根据假定的多个高度的相似度，推断第一格点的高度；以及根据所推断的第一格点的高度，推断摄像范围的三维形状。
在本发明的一个方面中，记录介质是记录有程序的计算机可读记录介质，所述程序用于使根据飞行体所拍摄的多个摄像图像来推断三维形状的形状推断装置执行如下步骤：进行三维形状推断的相关处理时，在飞行体所具备的摄像装置进行拍摄的摄像范围内生成具有多个格点的栅格；获取摄像装置所拍摄的第一摄像图像及第二摄像图像；获取拍摄第一摄像图像及第二摄像图像时摄像装置的三维位置及摄像装置的摄像方向信息；假定多个格点中第一格点的多个高度；就假定的多个高度，根据摄像装置的三维位置及摄像装置的摄像方向，导出第一格点投影于第一摄像图像的第一假定投影位置和第一格点投影于第二摄像图像的第二假定投影位置；就假定的多个高度，导出第一摄像图像中的第一假定投影位置和第二摄像图像中的第二假定投影位置的相似度；根据假定的多个高度的相似度，推断第一格点的高度；以及根据所推断的第一格点的高度，推断摄像范围的三维形状。
另外，上述发明的内容中并未穷举本公开的全部特征。此外，这些特征组的子组合也可以构成发明。
附图说明
图1是示出第一实施方式中的飞行系统第一构成示例的示意图。
图2是示出第一实施方式中的飞行系统第二构成示例的示意图。
图3是示出无人驾驶航空器的具体外观的一个示例的图。
图4是示出无人驾驶航空器的硬件配置的一个示例的框图。
图5是示出终端的硬件配置的一个示例的框图。
图6是示出地形推断动作步骤的流程图。
图7是示出初始栅格的生成示例的图。
图8是示出稀疏点云的生成示例的图。
图9是示出由稀疏点云围成的面的生成示例的图。
图10是示出栅格初始值的生成示例的图。
图11是说明栅格高度的搜索示例的图。
图12是说明栅格形状的推断示例的图。
【符号说明】
10 飞行系统
80 终端
81 终端控制部
83 操作部
85 通信部
87 内存
88 显示部
89 存储器
100 无人驾驶航空器
110 UAV控制部
150 通信接口
160 内存
170 存储器
200 万向节
210 旋翼机构
220、230 摄像部
240 GPS接收器
250 惯性测量装置
260 磁罗盘
270 气压高度计
280 超声波传感器
290 激光测量仪
GM1、GM2 摄像图像
gd 栅格
gp格点
P1～P4 稀疏点
PS1、PS2、PS3、PS4 假定格点
PS1a、PS1b、PS2a、PS2b 假定投影位置
sf1～sf3 面
za～ze 搜索范围
具体实施方式
以下，通过本发明的实施方式来对本公开进行说明，但是以下实施方式并非限制权利要求书所涉及的发明。实施方式中说明的特征的所有组合未必是发明的解决方案所必须的。
权利要求书、说明书、说明书附图以及说明书摘要中包含作为著作权保护对象的事项。任何人只要如专利局的文档或者记录所表示的那样进行这些文件的复制，著作权人则不会提出异议。但是，在除此以外的情况下，保留一切的著作权。
在以下实施方式中，飞行体以无人驾驶航空器(UAV：Unmanned Aerial Vehicle)为例。无人驾驶航空器包括在空中移动的航空器。在本说明书的附图中，无人驾驶航空器标记为“UAV”。另外，形状推断系统以飞行系统为例。形状推断装置主要以无人驾驶航空器为例，也可以是终端。终端可以包括智能手机、平板终端PC(Personal Computer)或其他装置。形状推断方法对形状推断装置中的动作进行了规定。此外，记录介质中记录有程序(例如使形状推断装置执行各种处理的程序)。
(第一实施方式)
图1是示出第一实施方式中的飞行系统10的第一构成示例的图。飞行系统10包括无人驾驶航空器100及终端80。无人驾驶航空器100和终端80之间可以通过有线通信或无线通信(例如，无线LAN(Local Area Network：局域网))彼此通信。在图1中，例示了终端80是便携式终端(例如，智能手机、平板终端)的情况。
另外，飞行系统可以包括无人驾驶航空器、发送器(发射器)和便携式终端。当包括发送器时，用户能够使用配置在发送器前面的左、右控制杆来控制无人驾驶航空器的飞行。另外，在这种情况下，无人驾驶航空器、发送器以及便携式终端可以通过有线通信或无线通信彼此通信。
图2是示出第一实施方式中的飞行系统10的第二构成示例的示意图。在图2中，例示了终端80是PC的情况。在图1和图2的任意一个中，终端80具有的功能可以相同。
图3是示出无人驾驶航空器100的具体外观的一个示例的图。在图3中，示出了无人驾驶航空器100在移动方向STV0飞行时的立体图。无人驾驶航空器100为移动体的一个示例。
如图3所示，在与地面平行且沿着移动方向STV0的方向上设定滚转轴(参照x轴)。在此情况下，在与地面平行且与滚转轴垂直的方向上设定俯仰轴(参照y轴)，进而，在与地面垂直且与滚转轴及俯仰轴垂直的方向上设定偏航轴(参照z轴)。
无人驾驶航空器100包括UAV主体102、万向节200、摄像部220以及多个摄像部230。
UAV主体102包括多个旋翼(螺旋桨)。UAV主体102通过控制多个旋翼的旋转而使无人驾驶航空器100飞行。UAV主体102使用例如四个旋翼使无人驾驶航空器100飞行。旋翼的数量并不限于四个。此外，无人驾驶航空器100可以是没有旋翼的固定翼机。
摄像部220是对包含在预期摄像范围内的被摄体(例如，作为航拍对象的上空的景象、山川或河流等的景色、地上的建筑物)进行拍摄的拍摄用相机。
多个摄像部230是为了控制无人驾驶航空器100的飞行而对无人驾驶航空器100的周围进行拍摄的传感用相机。两个摄像部230可以设置于无人驾驶航空器100的机头、即正面。并且，其他两个摄像部230可以设置于无人驾驶航空器100的底面。正面侧的两个摄像部230可以成对，起到所谓的立体相机的作用。底面侧的两个摄像部230也可以成对，起到立体相机的作用。可以根据由多个摄像部230拍摄到的图像来生成无人驾驶航空器100周围的三维空间数据。另外，无人驾驶航空器100所包括的摄像部230的数量不限于四个。无人驾驶航空器100只要包括至少一个摄像部230即可。无人驾驶航空器100可以在无人驾驶航空器100的机头、机尾、侧面、底面及顶面分别包括至少一个摄像部230。摄像部230中可设定的视角可大于摄像部220中可设定的视角。摄像部230可以具有单焦点镜头或鱼眼镜头。
图4是示出无人驾驶航空器100的硬件配置的一个示例的框图。无人驾驶航空器100包括UAV控制部110、通信接口150、内存160、存储器170、万向节200、旋翼机构210、摄像部220、摄像部230、GPS接收器240、惯性测量装置(IMU：Inertial Measurement Unit)250、磁罗盘260、气压高度计270、超声波传感器280及激光测量仪290。
UAV控制部110例如由CPU(Central Processing Unit：中央处理器)、MPU(Micro Processing Unit：微处理器)或DSP(Digital Signal Processor：数字信号处理器)构成。UAV控制部110执行用于总体控制无人驾驶航空器100的各部分的动作的信号处理、与其它各部分之间的数据的输入输出处理、数据的运算处理以及数据的存储处理。
UAV控制部110按照存储于内存160中的程序来控制无人驾驶航空器100的飞行。UAV控制部110可以控制飞行。UAV控制部110可以航拍图像。
UAV控制部110获取表示无人驾驶航空器100的位置的位置信息。UAV控制部110可以从GPS接收器240获取表示无人驾驶航空器100所在的纬度、经度以及高度的位置信息。UAV控制部110可以分别从GPS接收器240获取表示无人驾驶航空器100所在的纬度以及经度的纬度经度信息，并从气压高度计270获取表示无人驾驶航空器100所在的高度的高度信息，作为位置信息。UAV控制部110可以获取超声波传感器280产生的超声波放射点与超声波反射点之间的距离，作为高度信息。
UAV控制部110可以从磁罗盘260获取表示无人驾驶航空器100的朝向的朝向信息。朝向信息可以用例如与无人驾驶航空器100的机头的朝向相对应的方位来表示。
UAV控制部110可以获取表示在摄像部220对应该拍摄的摄像范围进行拍摄时无人驾驶航空器100所应该存在的位置的位置信息。UAV控制部110可以从内存160获取表示无人驾驶航空器100所应该存在的位置的位置信息。UAV控制部110可以经由通信接口150从其他装置获取表示无人驾驶航空器100所应该存在的位置的位置信息。UAV控制部110可以参照三维地图数据库来确定无人驾驶航空器100所能够存在的位置，并获取该位置作为表示无人驾驶航空器100所应该存在的位置的位置信息。
UAV控制部110可以获取表示摄像部220和摄像部230各自的摄像范围的摄像范围信息。UAV控制部110可以从摄像部220和摄像部230获取表示摄像部220和摄像部230的视角的视角信息作为用于确定摄像范围的参数。UAV控制部110可以获取表示摄像部220和摄像部230的摄像方向的信息，作为用于确定摄像范围的参数。UAV控制部110可以从万向节200获取表示摄像部220的姿势状态的姿势信息，作为例如表示摄像部220的摄像方向的信息。摄像部220的姿势信息可以表示万向节200的从俯仰轴和偏航轴的基准旋转角度旋转的角度。
UAV控制部110可以获取表示无人驾驶航空器100所在位置的位置信息，作为用于确定摄像范围的参数。UAV控制部110可以根据摄像部220和摄像部230的视角和摄像方向以及无人驾驶航空器100所在的位置，来划定表示摄像部220拍摄的地理范围的摄像范围，并生成摄像范围信息，从而获取摄像范围信息。
UAV控制部110可以从内存160获取摄像范围信息。UAV控制部110可以经由通信接口150获取摄像范围信息。
UAV控制部110控制万向节200、旋翼机构210、摄像部220以及摄像部230。UAV控制部110可以通过变更摄像部220的摄像方向或视角来控制摄像部220的摄像范围。UAV控制部110可以通过控制万向节200的旋转机构来控制由万向节200所支持的摄像部220的摄像范围。
摄像范围是指由摄像部220或摄像部230拍摄的地理范围。摄像范围由纬度、经度和高度定义。摄像范围可以是由纬度、经度和高度定义的三维空间数据的范围。摄像范围可以是由纬度和经度定义的二维空间数据的范围。摄像范围可以根据摄像部220或摄像部230的视角和摄像方向、以及无人驾驶航空器100所在的位置而确定。摄像部220和摄像部230的摄像方向可以由摄像部220和摄像部230的设置有摄像镜头的正面朝向的方位和俯角来定义。摄像部220的摄像方向可以根据无人驾驶飞机100的机头方位以及相对于万向节200的摄像部220的姿态状态而确定的方向。摄像部230的摄像方向可以是根据无人驾驶航空器100的机头方位和设置有摄像部230的位置而确定的方向。
UAV控制部110可以通过分析由多个摄像部230拍摄的多个图像，来确定无人驾驶航空器100的周围环境。UAV控制部110可以根据无人驾驶航空器100的周围环境，例如避开障碍物来控制飞行。
UAV控制部110可以获取表示存在于无人驾驶航空器100周围的对象的立体形状(三维形状)的立体信息(三维信息)。对象例如可以是建筑物、道路、车辆、树木等风景的一部分。立体信息例如是三维空间数据。UAV控制部110可以根据由多个摄像部230获取的各个图像，生成表示存在于无人驾驶航空器100周围的对象的立体形状的立体信息，从而获取立体信息。UAV控制部110可以通过参照存储在内存160或存储器170中的三维地图数据库，来获取表示存在于无人驾驶航空器100周围的对象的立体形状的立体信息。UAV控制部110可以通过参照由网络上存在的服务器所管理的 三维地图数据库，来获取与存在于无人驾驶航空器100的周围的对象的立体形状相关的立体信息。
UAV控制部110通过控制旋翼机构210来控制无人驾驶航空器100的飞行。即，UAV控制部110通过控制旋翼机构210来对包括无人驾驶航空器100的纬度、经度以及高度的位置进行控制。UAV控制部110可以通过控制无人驾驶航空器100的飞行来控制摄像部220的摄像范围。UAV控制部110可以通过控制摄像部220所包括的变焦镜头来控制摄像部220的视角。UAV控制部110可以利用摄像部220的数字变焦功能，通过数字变焦来控制摄像部220的视角。
当摄像部220固定于无人驾驶航空器100并且不能移动摄像部220时，UAV控制部110可以通过使无人驾驶航空器100在特定的日期和时间移动到特定的位置，来使摄像部220在期望的环境下拍摄期望的摄像范围。或者，即使摄像部220不具有变焦功能并且不能变更摄像部220的视角，UAV控制部110也可以通过使无人驾驶航空器100在特定的日期和时间移动到特定的位置，来使摄像部220在期望的环境下拍摄期望的摄像范围。
通信接口150与终端80进行通信。通信接口150可以通过任意的无线通信方法进行无线通信。通信接口150可以通过任意的有线通信方法进行有线通信。通信接口150可以将航拍图像、与航拍图像相关的附加信息(元数据)发送到终端80。
内存160存储UAV控制部110对万向节200、旋翼机构210、摄像部220、摄像部230、GPS接收器240、惯性测量装置250、磁罗盘260、气压高度计270、超声波传感器280和激光测量仪290进行控制所需的程序等。内存160可以是计算机可读记录介质，可以包括SRAM(Static Random Access Memory：静态随机存取存储器)、DRAM(Dynamic Random Access Memory：动态随机存取存储器)、EPROM(Erasable Programmable Read Only Memory：可擦除可编程只读存储器)、EEPROM(Electrically Erasable Programmable Read-Only Memory：电可擦除可编程只读存储器)、以及USB(Universal Serial Bus：通用串行总线)存储器等闪存中的至少一个。内存160可以从无人驾驶航空器100上拆卸下来。内存160可以作为作业用内存进行工作。
存储器170可以包括HDD(Hard Disk Drive：硬盘驱动器)、SSD(Solid State Drive：固态硬盘)、SD卡、USB存储器、其他存储器中的至少一个。存储器170可以保存各种信息、各种数据。存储器170可以从无人驾驶航空器100上拆卸下来。存储器170可以记录航拍图像。
内存160或存储器170可以保存由终端80或无人驾驶航空器100所生成的航拍位置、航拍路径的信息。可以通过UAV控制部110设定航拍位置、航拍路径的信息，作为由无人驾驶航空器100预定的航拍所涉及的航拍参数或者由无人驾驶航空器100预定的飞行所涉及的飞行参数中的一个。该设定信息可以保存在内存160或存储器170中。
万向节200可以以偏航轴、俯仰轴以及滚转轴为中心可旋转地支持摄像部220。万向节200可以通过使摄像部220以偏航轴、俯仰轴以及滚转轴中的至少一个为中心旋转，来变更摄像部220的摄像方向。
旋翼机构210具有多个旋翼和使多个旋翼旋转的多个驱动电机。旋翼机构210通过由UAV控制部110控制旋转，从而使无人驾驶航空器100飞行。旋翼211的数量例如可以是四个，也可以是其他数量。此外，无人驾驶航空器100可以是没有旋翼的固定翼机。
摄像部220对期望的摄像范围中的被摄体进行拍摄并生成摄像图像的数据。通过摄像部220的拍摄而得到的图像数据(例如航拍图像)可以存储于摄像部220具有的内存或存储器170中。摄像部220是摄像部的一个示例。
摄像部230对无人驾驶航空器100的周围进行拍摄并生成摄像图像的数据。摄像部230的图像数据可以存储于存储器170中。
GPS接收器240接收从多个导航卫星(即GPS卫星)发送的、表示时间以及各GPS卫星的位置(坐标)的多个信号。GPS接收器240根据接收到的多个信号，计算出GPS接收器240的位置(即无人驾驶航空器100的位置)。GPS接收器240将无人驾驶航空器100的位置信息输出到UAV控制部110。另外，可以由UAV控制部110代替GPS接收器240来进行GPS接收器240的位置信息的计算。在此情况下，GPS接收器240所接收到的多个信号中所包含的表示时间以及各GPS卫星的位置的信息被输入到UAV控制部110中。
惯性测量装置250检测无人驾驶航空器100的姿势，并将检测结果输出到UAV控制部110。惯性测量装置250可以检测无人驾驶航空器100的前后、左右、以及上下三轴方向的加速度以及俯仰轴、滚转轴和偏航轴三轴方向的角速度，作为无人驾驶航空器100的姿势。
磁罗盘260检测无人驾驶航空器100的机头的方位，并将检测结果输出到UAV控制部110。
气压高度计270检测无人驾驶航空器100的飞行高度，并将检测结果输出到UAV控制部110。
超声波传感器280发射超声波，检测地面、物体反射的超声波，并将检测结果输出到UAV控制部110。检测结果可以示出从无人驾驶航空器100到地面的距离，即高度。检测结果可以示出从无人驾驶航空器100到物体(被摄体)的距离。
激光测量仪290对物体照射激光，接收物体反射的反射光，并通过反射光来测量无人驾驶航空器100与物体(被摄体)之间的距离。作为根据激光的距离测量方法的一个示例，可以为飞行时间法。
图5是示出终端80的硬件配置的一个示例的框图。终端80包括终端控制部81、操作部83、通信部85、内存87、显示部88以及存储器89。终端80可以由希望控制无人驾驶航空器100的飞行的用户所持有。
终端控制部81例如采用CPU、MPU或DSP构成。终端控制部81进行用于整体控制终端80的各部分动作的信号处理、与其他各部分之间的数据的输入输出处理、数据的运算处理以及数据的存储处理。
终端控制部81可以经由通信部85获取来自无人驾驶航空器100的数据、信息。终端控制部81可以获取经由操作部83输入的数据、信息(例如，各种参数)。终端控制部81可以获取保存在内存87中的数据、信息。终端控制部81可以经由通信部85向无人驾驶航空器100发送数据、信息(例如，位置、速度、飞行路径的信息)。终端控制部81可以将数据、信息发送至显示部88，将根据此数据、信息的显示信息显示在显示部88上。
操作部83接收并获取由终端80的用户输入的数据、信息。操作部83也可以包括按钮、按键、触控显示屏、话筒等输入装置。此处，主要例示了操作部83与显示部88由触控面板构成的情况。在这种情况下，操作部83可以接收触摸操作、点击操作、拖动操作等。操作部83可以接受各种参数的信息。操作部83输入的信息可以发送到无人驾驶航空器100。
通信部85通过各种无线通信方式与无人驾驶航空器100之间进行无线通信。此无线通信的无线通信方式可以包括例如无线LAN、Bluetooth(注册商标)，或经由公共无线网络进行的通信。通信部85可以通过任意的有线通信方式进行有线通信。
内存87例如可以具有：存储有规定终端80的动作的程序、设定值数据的ROM、暂时保存终端控制部81进行处理时所使用的各种信息、数据的RAM。内存87可包括ROM及RAM以外的内存。内存87可以设置于终端80内部。内存87可设置成可从终端80上拆卸下来。程序可以包括应用程序。
显示部88例如采用LCD(Liquid Crystal Display，液晶显示器)构成，显示从终端控制部81输出的各种信息、数据。显示部88可以显示应用程序的执行所涉及的各种数据、信息。
存储器89存储并保存各种数据、信息。存储器89可以是HDD、SSD、SD卡、USB存储器等。存储器89可以设置在终端80内部。存储器89可以可拆卸地设置在终端80上。存储器89可以保存从无人驾驶航空器100获取的航拍图像及其附加信息。附加信息可以保存在内存87中。
接着，对飞行系统10的动作进行说明。飞行系统10的无人驾驶航空器100或终端80根据无人驾驶航空器100所拍摄的多个摄像图像，进行用于推断(地形推断)地面三维形状的处理。另外，此处的地形可以包括摄像部220对地面方向拍摄时映入摄像范围的物体(例如，地面、建筑物、其他对象)的大面积形状。
图6是示出地形推断处理步骤的一个示例的流程图。作为一个示例，可以通过无人驾驶航空器100的UAV控制部110执行存储于内存160中的程序来进行该处理。并且，终端80可以进行辅助地形推断处理的动作。例如，终端80可以受理用户操作，或者向用户显示信息。
UAV控制部110获取飞行范围和各种参数(S1)。这种情况下，用户可以对终端80输入飞行范围和参数。终端控制部81可以经由操作部83受理用户输入，获取所输入的飞行范围和参数。终端控制部81可以经由通信部85向终端80发送飞行范围和参数。
终端控制部81可以经由通信部85从外部服务器获取地图信息。例如当飞行范围被设定为矩形范围时，用户可以通过输入矩形的四个角在地图信息中的位置(纬度、经度)来获得飞行范围的信息。此外，当飞行范围被设定为圆形范围时，用户可以通过输入以飞行位置为中心的圆的半径来获得飞行范围的信息。此外，用户可以通过输入区域、特定地名(例如东京)等信息，根据地图信息获得飞行范围的信息。此外，终端控制部81可以从内存87、存储器89获取保存在内存87、存储器89中的飞行范围。飞行范围可以是无人驾驶航空器100的预定飞行范围。
参数可以是摄像部220进行拍摄所涉及的摄像参数及无人驾驶航空器100进行飞行所涉及的飞行参数。摄像参数可以包括摄像位置、摄像日期和时间、到被摄体的距离、摄像视角、拍摄时无人驾驶航空器100的姿势、摄像方向、摄像条件、相机参数(快门速度、曝光值、摄像模式等)。飞行参数可以包括飞行位置(三维位置或二维位置)、飞行高度、飞行速度、飞行加速度、飞行路径、飞行日期和时间等。终端控制部81可以从内存87、存储器89获取保存在内存87、存储器89中的各种参数。终端控制部81可以经由通信部85将所获取的飞行范围和各种参数发送到无人驾驶航空器100。
在无人驾驶航空器100中，UAV控制部110可以经由通信接口150从终端80获取飞行范围和各种参数。另外，UAV控制部110自身也可以经由通信接口150从外部 服务器获取飞行范围和各种参数，或者从内存160获取飞行范围和各种参数，或者从无人驾驶航空器100内的各传感器(例如GPS接收器240、惯性测量装置250)获取飞行范围和各种参数，或者将其导出。UAV控制部110可以根据飞行范围和各种参数来确定飞行路径及摄像位置。
UAV控制部110生成初始栅格(初始状态的栅格gd)作为栅格图案(S2)。
栅格gd虚拟地表现出作为形状推断对象的摄像范围的地形。因此，可以将栅格gd设定为与摄像范围相同的范围，或者包含于摄像范围中的范围。继而，如果对实际空间中摄像范围中的各点高度进行假定及推断，则可以对虚拟空间中栅格gd中的格点高度进行假定及推断。栅格gd可以为格子状、三角形、其他多边形、其他形状，包含栅格gd的顶点即格点gp。
初始状态下各格点gp的高度可以全部为0值。另外，关于格点gp的初始状态高度，也可以为用户经由终端80的操作部83而设定的任意高度值。这种情况下，UAV控制部110可以经由通信接口150从终端80获取高度值。
各格点gp的间隔(栅格间隔)可以为预定值，也可以通过UAV控制部110设定为任意值。栅格间隔可以为1m、2m等。例如，终端80的终端控制部81可以经由操作部83指定栅格间隔，再经由通信部85将栅格间隔信息发送给无人驾驶航空器100。UAV控制部110可以经由通信接口150获取、设定栅格间隔信息。
此外，如果准备了摄像范围的大致形状信息，UAV控制部110可以获取大致形状信息。UAV控制部110可以在格点gp采样大致形状，获得其在各格点gp的高度，并设定为格点gp的高度。例如，UAV控制部110可以与经由通信接口150而连接的网络上的地图服务器联动，获取摄像范围的地形信息作为大致形状信息。地形信息可以包括摄像范围内各格点gp的位置信息(纬度、经度、高度)。此外，地形信息可以包括建筑物、山、森林、铁塔等地面形状的信息、物体的信息。
UAV控制部110依据所确定的飞行路径控制飞行，使摄像部220拍摄(例如航拍)图像，并将摄像图像保存于存储器170中(S3)。可以在沿着飞行路径中的各摄像位置进行该拍摄，并将多个摄像图像保存于存储器170中。此外，UAV控制部110可以将摄像图像的相关附加信息保存于存储器170中。该附加信息可以包括和上述各种参数(例如飞行参数、摄像参数)相同的信息。借此，UAV控制部110可以获取拍摄摄像图像时摄像部220(即无人驾驶航空器100)的摄像位置、拍摄时的姿势、摄像方向等信息。
UAV控制部110从摄像部220或存储器170获取多个摄像图像。UAV控制部110可以提取各摄像图像中所包含的多个特征点。特征点可以是摄像图像上的任意位置点。对于各摄像图像中所包含的多个特征点，UAV控制部110可以实施匹配处理，使相同特征点彼此相对应，生成对应的特征点即对应点。UAV控制部110可以生成包括多个对应点的稀疏点云。UAV控制部110例如可以依据sfm生成稀疏点云。稀疏点云的数量例如可以为每一个图像数百点左右。稀疏点云中所包含的点的数据可以包括表示三维位置的数据。如上所述，UAV控制部110可以根据多个摄像图像生成稀疏点云(S5)。
另外，稀疏点云可以和栅格gd的格点gp不一致。稀疏点云在二维平面上的位置(不考虑实际空间中的高度的位置)可以和格点gp在二维平面上的位置(不考虑格点高度的位置)不一致，因为初始栅格中没有严格地规定栅格高度。
UAV控制部110可以将所生成的特征点、对应点、稀疏点云数据存储于内存160或存储器170中。借此，UAV控制部110在下一次之后的稀疏点云的生成中只需根据新获取的摄像图像来生成稀疏点云即可，本次之前的稀疏点云可以从内存160或存储 器170中获取。例如，对于本次之前的拍摄中未拍摄的场所，将于下一次之后拍摄，生成新的特征点、对应点、稀疏点云数据。此外，即便是本次之前的拍摄中已拍摄的场所，但本次之前的拍摄中的摄像方向、摄像角度不同，同一场所的信息量不足时，也可以生成新的特征点、对应点、稀疏点云数据。也就是说，UAV控制部110可以将根据新摄像图像的稀疏点云追加到现有摄像图像的稀疏点云中。这种情况下，无人驾驶航空器100可以减轻用于生成稀疏点云数据的处理负荷，缩短处理时间，提高实时性。因此，UAV控制部110可以逐次反复地实施如下处理：拍摄图像，生成关于所拍摄的图像的稀疏点云，然后拍摄新图像，生成关于新图像的稀疏点云。
UAV控制部110根据Delaunay三角剖分技术、PSR(Poisson Surface Reconstruction：泊松表面重建)等栅格化技术，生成以稀疏点云中所包含的多个点(例如3个点)为顶点的surface(面)sf。也就是说，UAV控制部110根据稀疏点云，生成包含稀疏点云中所包含的多个点的面sf的形状。此外，面sf可以对应于形状推断的推断对象表面的形状。UAV控制部110以由稀疏点云所形成的面sf中包含栅格gd中的格点gp的方式对格点gp(各格点gp的栅格高度)进行补正，并设为栅格初始值(S6)。另外，格点gp中的栅格初始值(栅格的格点高度的初始值)并不仅限于表示对应于地形的高度。可以以栅格初始值为起点，一边变更该格点gp的高度一边搜索对应于地形的高度。
UAV控制部110以栅格初始值为起点，沿着用于搜索栅格高度的搜索轴(垂直于表示地面的栅格面的轴)变更栅格高度，同时假定栅格高度。借此，UAV控制部110生成具有假定的栅格高度的假定格点PS。栅格高度例如对应于与格点相对应的推断对象及地形高度。UAV控制部110可以对多个摄像图像中投影同一假定格点PS的各个假定投影位置，导出(例如计算)图像(像素)的相似度。UAV控制部110根据该像素的相似度，推断搜索轴A×1上的多个假定格点PS中的最佳(被认为是接近地面)格点ps。UAV控制部110可以将相似度在阈值th1以上(例如最大)的假定格点PS推断为格点gp。另外，UAV控制部110也可以根据包含相邻的多个像素在内的预定像素范围内的相似度来推断格点，而不是根据单一像素的相似度来推断格点(栅格高度)。
利用UAV控制部110沿搜索轴A×1搜索格点gp高度的搜索方式可以是全域搜索法，也可以是最优法(例如二分搜索法、LM搜索法)。UAV控制部110可以使用NCC(Normalized Cross-Correlation)、SSD(Sum of Squared Difference)、SAD(Sum of Absolute Difference)等，计算出各摄像图像中各假定投影位置的相似度。
另外，将各摄像图像中各假定投影位置的像素相似度设为用于推断格点gp(栅格高度)的指标仅为一个示例，UAV控制部110也可以根据其他指标来推断最佳格点gp。例如，在上述相似度较高时，UAV控制部110可以设想为评分较低的误差评分，根据误差评分来推断最佳格点。这种情况下，UAV控制部110可以按照例如误差评分在阈值th2以下(例如最小)的方式推断最佳格点。
UAV控制部110可以对与摄像范围相对应的栅格gd中的各格点gp实施格点gp高度的搜索。另外，此处各格点gp可以是栅格gd中的所有格点gp，也可以是部分格点gp。
UAV控制部110按照上述方式推断格点gp的高度。UAV控制部110通过推断各格点gp的高度来推断栅格形状(S7)。栅格形状是反映了各格点gp实际高度的推断结果的栅格gd的形状，相当于摄像范围内的地面形状。
UAV控制部110判断无人驾驶航空器100所包括的摄像部220的拍摄(例如航拍)是否完成(S8)。例如，当无人驾驶航空器100沿预定的飞行路径飞行，在预定的摄像位置结束了图像拍摄后，UAV控制部110可以判断为航拍完成。用户经由终端80的 操作部83指示航拍结束，UAV控制部110经由通信部85获取该指示信息时，即可判断为航拍完成。如果航拍没有完成，UAV控制部110将返回S3，继续进行航拍。航拍完成时，UAV控制部110结束图6的处理。
无人驾驶航空器100反复地进行地面方向的拍摄和栅格形状的推断。因此，随着反复次数的增加，生成的稀疏点云数量也随之增加。因此，推断栅格形状而得到的地形形状会愈发接近实际地形形状。这种情况下，UAV控制部110可以将从新追加的摄像图像提取的稀疏点云追加到从现有摄像图像提取的稀疏点云中，生成新的稀疏点云。借此，无人驾驶航空器100可以抑制获取摄像图像时计算量的增加，可以实现实时性较高的栅格形状的推断即地形推断。
另外，UAV控制部110可以经由通信接口150将S7中推断的栅格形状信息发送给终端80。终端80经由通信部85获取所推断的栅格形状信息，并经由显示部88对其进行显示。借此，用户可以对所推断的栅格形状、即飞行范围内推断的地面形状进行确认。
另外，除了摄像部220所拍摄的摄像图像外，也可以根据摄像部230所拍摄的摄像图像来推断栅格形状。
另外，也可以由终端80执行图6的地形推断处理中的至少一部分处理。例如，终端控制部81可以在S3中从无人驾驶航空器100获取摄像图像，生成稀疏点云(S5)，生成栅格初始值(S6)，推断栅格形状(S7)等。此外，终端控制部81也可以从无人驾驶航空器100获取S9中航拍是否完成的信息。另外，无人驾驶航空器100和终端80也可以分摊实施地形推断处理。
接着，对图6的各步骤中的处理进行补充。
图7是示出初始栅格的生成示例的图。可以根据摄像范围来确定初始栅格的形状。在图7中，初始栅格的形状为平面矩形。在图7中，对以相邻的4个格点gp为顶点的四边形进行组合，生成栅格gd。也可以利用相邻的格点gp生成其他形状(例如，三角形)，组合其他形状生成栅格gd。也就是说，以栅格gd中的各格点gp为顶点的单位栅格可以是三角形、四边形或其他形状。
图8是示出稀疏点云的生成示例的图。可以根据摄像图像中所包含的特征点，利用UAV控制部110生成稀疏点云。在图8中，生成4个稀疏点云(点P1～P4)。4个稀疏点云(点P1～P4)中，稀疏点P2的高度最低，稀疏点P3、P4的高度较高。
图9是示出由稀疏点云(点P1～P4)围成的面sf1～sf3的生成示例的图。表示稀疏点云形状的面sf(sf1、sf2、sf3、...)可以通过上述栅格化技术而生成。面sf1是由稀疏点P1、P2、P4围成的面。面sf2是由稀疏点P1、P2、P3围成的面。面sf3是由稀疏点P2、P3、P4围成的面。由3个面sf1～sf3所表现的面整体具有中央(稀疏点P2)凹陷的形状。
图10是示出栅格初始值gpa的生成示例的图。对格点gp的高度进行补正，使得由稀疏点云(点P1～P4)围成的面sf1～sf3中包含格点gp。例如，结合面对初始状态下高度＝0的点进行补正，或者结合根据本次所生成的稀疏点云的面sf，对图6的处理中前一次循环时所推断的格点gp的高度进行补正。具有该经补正后的高度的格点gp变为本次循环时的栅格初始值gpa。可以对所有的格点gp进行同样的补正，对各个格点生成栅格初始值gpa。
在图10中，对各格点的高度进行补正后的栅格初始值为栅格初始值gpa1、gpa2、gpa3、gpa4、gpa5。和各格点gp的高度＝0时的初始栅格相比，高度经补正的格点gp呈向上方隆起的形状。
图11是说明格点gp的高度(栅格高度)的搜索示例的图。所示出的是使用摄像图像GM1、GM2的栅格高度搜索示例。在图11中，飞行中的无人驾驶航空器100会拍摄多个摄像图像GM1、GM2，因此，摄像位置及摄像方向互不相同。另外，在图11中，实际地面SG设想为通过栅格高度h2的假定格点PS2附近。
UAV控制部110以栅格初始值gpa为起点，沿着搜索轴a×1一边变更栅格高度一边搜索最佳栅格高度。在图11中，栅格初始值gpa是具有假定的栅格高度中最高栅格高度的假定格点PS1，然后依次降低栅格高度进行搜索。另外，也可以将栅格初始值gpa的高度设为具有假定的栅格高度中最低栅格高度的假定格点PS6，然后依次提高栅格高度进行搜索。此外，也可以将具有搜索轴a×1中的中间部分而非端部的栅格高度的假定格点(例如，假定格点PS3)设为栅格初始值的高度。
首先，UAV控制部110假定为栅格高度h1。这种情况下，在摄像图像GM1中，在投影栅格高度h1的假定格点PS1的假定投影位置PS1a，绘制地面上的位置sg1a(像素)。在摄像图像GM2中，在投影栅格高度h1的假定格点PS1的假定投影位置PS1b，绘制地面上的位置sg1b(像素)。因此，在两个摄像图像GM1、GM2中，在假定投影位置PS1a、PS1b上，并非绘制假定为地面的相同假定格点PS1，而是绘制不同的实际地面SG的位置sg1a、sg1b。借此，在假定投影位置PS1a、PS1b所绘制的图像的相似度(像素的相似度)变低。
接着，UAV控制部110假定为栅格高度h2。这种情况下，在摄像图像GM1中，在投影栅格高度h2的假定格点PS2的假定投影位置PS2a绘制地面上的位置sg2a(像素)。在摄像图像GM1中，在投影栅格高度h2的假定格点PS2的假定投影位置PS2b绘制地面上的位置sg2b(像素)。因此，在两个摄像图像GM1、GM2中，在假定投影位置PS2a、PS2b绘制实际地面SG的位置sg2a、sg2b，实际地面SG的位置sg2a、sg2b接近假定为地面的相同假定格点PS2。借此，在假定为栅格高度h2时的假定投影位置PS2a、PS2b所绘制的图像的相似度(像素的相似度)要高于假定为栅格高度h1的情况。
借此，UAV控制部110可以计算出在多个摄像图像GM1、GM2中的假定投影位置(PS1a和PS1b、或者PS2a和PS2b)所绘制的图像的相似度，并根据计算出的相似度，推断最佳(推断为实际地面)栅格高度的格点。例如，可以将与导出的多个相似度中阈值th1以上(例如最大)的相似度所对应的栅格高度推断为格点gp的高度。
另外，摄像图像GM1、GM2两者可以是摄像部220本次所拍摄的图像，而不是存储于存储器170中的现有摄像图像。另外，摄像图像GM1可以是存储于存储器170中的现有摄像图像，摄像图像GM2可以是摄像部220本次所拍摄的最新摄像图像。此外，用于推断栅格高度的摄像图像可以是三张以上。可以是多张现有的摄像图像。也可以是多张最新的摄像图像。
图12是说明栅格形状的推断示例的图。在图12中，UAV控制部110根据上述像素的相似度，推断相对于栅格初始值gpa向上方移动后的假定格点PS2是最佳(最接近准确的)格点gp。UAV控制部110可以计算出摄像部220所拍摄的多张图像的相似度，将相似度最高的栅格高度推断为最优化的格点gPP的高度。在图12中，可以分别在搜索范围za～ze内调整栅格高度。
如此一来，在第一实施方式的飞行系统10中，UAV控制部110可以根据稀疏点云导出栅格初始值。UAV控制部110可以以栅格初始值为起点，假定多个栅格高度，根据具有假定的栅格高度的假定格点PS所投影的假定投影位置的像素相似度，使栅格高度最优化。也就是说，UAV控制部110可以将格点gp再次投影到不同的摄像图 像上，计算再次投影的各场所(假定投影位置)的相似度。UAV控制部110可以根据摄像位置和摄像方向，识别摄像图像中所包含的摄像范围。借此，UAV控制部110可以确定摄像图像中的假定投影位置。UAV控制部110可以根据上述相似度，推断实际的栅格高度(真实值)。
UAV控制部110在生成稀疏点云时不需要使用存储的所有摄像图像来生成稀疏点云。UAV控制部110可以预先存储之前所生成的稀疏点云信息，再针对新的摄像图像生成稀疏点云。UAV控制部110可以对存储的稀疏点云追加新的稀疏点云，用作用于生成面sf的数据。借此，无人驾驶航空器100可以简化处理负荷较大、运算时间相对较长的稀疏点云的生成处理，减轻UAV控制部110的处理负荷，缩短处理时间。因此，无人驾驶航空器100可以提高推断地面三维形状时的实时性。
如此一来，无人驾驶航空器100(形状推断装置的一个示例)根据无人驾驶航空器100(飞行体)所拍摄的多个摄像图像，推断地形的三维形状。无人驾驶航空器100包括UAV控制部110(处理部的一个示例)，用于进行推断地形三维形状的相关处理。UAV控制部110可以在无人驾驶航空器100所包括的摄像部220(摄像装置)进行拍摄的摄像范围内生成具有多个格点gp的栅格gd。UAV控制部110可以获取摄像部220所拍摄的摄像图像GM1(第一摄像图像的一个示例)及摄像图像GM2(第二摄像图像的一个示例)。UAV控制部110可以获取拍摄摄像图像GM1、GM2时摄像部220的三维位置及摄像部220的摄像方向(相当于摄像部220的姿势)信息。UAV控制部110可以假定多个格点gp中的假定格点PS(第一格点的一个示例)的多个高度。假定格点PS的高度例如对应于与格点相对应的推断对象及地形高度。对于假定的多个高度，UAV控制部110可以根据摄像部220的三维位置及摄像部220的摄像方向，导出(例如计算出)假定格点PS投影于摄像图像GM1的假定投影位置PS1a(第一假定投影位置的一个示例)和投影于摄像图像GM2的假定投影位置PS1b(第二假定投影位置的一个示例)。对于假定的多个高度，UAV控制部110可以导出摄像图像GM1中的假定投影位置PS1a和摄像图像GM2中的假定投影位置PS1b的相似度。UAV控制部110根据假定的多个高度的相似度，推断格点gp的高度。UAV控制部110可以根据所推断的格点gp的高度，推断摄像范围的三维形状。
借此，无人驾驶航空器100可以一边变更高度一边对各摄像图像中的各假定投影位置计算出相似度，使相似度最优化(例如，导出最大相似度)，从而推断格点gp的高度，简化推断三维形状时的计算。因此，无人驾驶航空器100不需要像以往那样依次生成素点(例如sfm)、生成密点(例如MVS：Multi-View Stereo)、生成栅格、生成纹理等全部处理。无人驾驶航空器100可以减轻三维形状推断处理的处理负荷，在线实施三维形状的推断。并且，无人驾驶航空器100可以提高实时性，从而能够现场获得三维形状推断处理的结果。
例如，相似度较大时，多个摄像图像中投影同一位置(假定格点PS)的各个像素(像素范围)最为相似，因此，这些像素表示同一对象的可能性较高。无人驾驶航空器100可以将这种情况下的假定格点PS的高度推断为对应于该像素所在摄像图像中的图像位置的实际空间位置的高度。
此外，UAV控制部110可以根据摄像图像GM1、GM2，例如依据sfm(Structure from Motion)方法生成稀疏点云数据(点云数据的一个示例)。UAV控制部110可以根据稀疏点云数据生成面sf(面数据的一个示例)。UAV控制部110可以根据面sf导出假定格点PS的初始栅格高度h1(栅格初始值gpa)(第一格点的初始高度的一个示例)。UAV控制部110可以从栅格高度h1开始依次对假定格点PS的高度进行假定。
借此，无人驾驶航空器100可以根据基于稀疏点云数据的面sf，设定格点gp(假定格点PS)的栅格高度h1，可以缩短用于导出格点gp高度的搜索时间。此外，即便格点gp的初始高度和真实值(实际高度)之间偏差较大，无人驾驶航空器100仍易搜索出真实值。
此外，第一摄像图像可以是一个以上现有的摄像图像(例如，前一次之前所拍摄的摄像图像)。第二摄像图像可以是一个以上最新的摄像图像(例如，本次所拍摄的摄像图像)。UAV控制部110可以获取根据第一摄像图像而生成的第一点云数据，根据第二摄像图像生成第二点云数据，并根据第一点云数据和第二点云数据生成面sf。
借此，无人驾驶航空器100通过使用现有的图像(例如第一摄像图像)，就不需要计算现有的点云数据，只需开展根据最新图像(例如第二摄像图像)生成点云数据的运算即可。因此，无人驾驶航空器100可以使用多个摄像图像快速地推断三维形状，提高实时性。
此外，格点gp可以是栅格gd中的所有格点。借此，无人驾驶航空器100可以高实时性、快速地推断例如整个摄像范围的三维形状。
此外，格点gp可以是栅格gd中的部分格点。借此，无人驾驶航空器100可以高实时性、快速地推断例如摄像范围中物体所在的特定范围的三维形状，降低计算成本。
此外，UAV控制部110可以设定栅格gd的各格点gp的间隔。例如，终端控制部81可以经由操作部83获取用于设定各格点gp的间隔的操作信息，再经由通信部85发送该操作信息。UAV控制部110可以经由通信接口150获取操作信息，再根据操作信息设定各格点gp的间隔。
借此，无人驾驶航空器100可以设定栅格gd的细度(格点gp的多少)，实现用户期望的三维形状推断的精细性。
(第二实施方式)
在第二实施方式中，设想为在栅格高度的搜索对象中假定多个高度，根据多个高度在多个摄像图像中的假定投影位置的像素(像素范围)相似度，推断栅格高度。另一方面，在第一实施方式中，没有考虑到非搜索对象栅格对于推断搜索对象栅格的高度的影响。在第二实施方式中，示出了考虑到栅格的相邻关系，推断栅格形状的情况。另外，在本实施方式中，列举了以UAV控制部110为主导来推断栅格形状的情况，但也可以用终端控制部81为主导来推断栅格形状。
第二实施方式的飞行系统具有和第一实施方式基本相同的配置。对于与第一实施方式相同的结构要素，通过使用相同的符号，省略其说明。
UAV控制部110根据多个摄像图像，使各格点gp的高度最优化。这种情况下，UAV控制部110可以根据第一实施方式中导出的相似度，计算出误差评分E(X)。UAV控制部110可以根据误差评分E(X)的计算结果，推断各格点gp的高度。将假定格点PS投影于多个摄像图像上时，误差评分E(X)相当于各摄像图像中各假定投影位置所表示的实际空间中的位置误差。各假定投影位置所表示的实际空间中的位置误差越大，各摄像图像中假定投影位置的像素相似度就越低。各假定投影位置所表示的实际空间中的位置误差越小，各摄像图像中假定投影位置的像素相似度就越高。误差评分E(x)例如可以用式(1)来表示。
【式1】
Figure PCTCN2019123314-appb-000001
在误差评分E(x)的计算中，UAV控制部110对各格点gp假定为依次变更高度x s(例如h1、h2、...)的格点gp，生成假定格点PS。UAV控制部110可以通过将右边第一项的成本和右边第二项的影响度相加来计算出误差评分E(x)。式(1)的第一项表示单个搜索对象格点gp(也称为格点s)的成本。第二项表示与搜索对象格点gp相邻的格点gp2(也称为格点t)对格点gp的影响度。集合V是格点gp的集合。集合E是对栅格gd中处于相邻关系的格点s和格点t进行连结的边的集合。
式(1)中，x s表示格点s的高度。x t表示格点t的高度。w s表示根据从当前摄像位置(即，无人驾驶航空器100的位置)到对应于格点s的实际空间位置的距离的函数。W st表示根据格点s和格点t之间的距离的函数。距离越长，w s、W st各自的值就越小，w s、W st所在项(第一项或第二项)的值变小，对误差评分E(X)的影响变小。
此外，
Figure PCTCN2019123314-appb-000002
表示多个摄像图像间假定格点PS的再次投影位置(假定投影位置)的相似度，用式(2)来表示。
【式2】
φ s(x s)＝NCC(π i(x s)，π j(x s))......(2)
π i(x s)表示在i位置由无人驾驶航空器100的摄像部220所拍摄的摄像图像中格点s的像素位置(相当于假定投影位置)。π i(x s)表示在j位置由无人驾驶航空器100的摄像部220所拍摄的摄像图像中格点s的像素位置(相当于假定投影位置)。NCC是表示像素间相似度的函数的一个示例，表示像素位置π i(x s)的图像和像素位置π j(x s)的图像的相似度。像素间的再次投影位置的相似度较高时，
Figure PCTCN2019123314-appb-000003
值较小，也就是说，误差评分E(X)变小。另一方面，像素间的再次投影位置的相似度较低时，
Figure PCTCN2019123314-appb-000004
值较大，也就是说，误差评分E(X)变大。
X st表示相邻关系的成本，例如可以包括相邻的格点间的平滑度信息。X st(x s、x t)包括格点s和处于相邻关系的格点t这2点间的成本，可以用式(3)来表示。式(3)的情况下，表示格点s、t之间的距离(范数)。
【式3】
χ st(x s，x t)＝|x s-x t| 2......(3)
平滑度信息可以保存于内存160中，可以利用UAV控制部110从外部服务器获取。例如为地形起伏剧烈的土地(山地等)时，平滑度信息可以包括表示平滑度较低的信息；如果地形为平野或水面(海面、河面)，可以包括表示平滑度较高的信息。
式(4)中所示X即argmin函数是以作为函数的误差评分E(X)最小的x为自变量进行传递的函数。argmin函数的函数值是用于使搜索对象格点s投影于各摄像图像的再次投影误差最小的函数。另外，作为argmin函数的最优化解法(即，栅格形状的最优化解法)，可以采用树重加权的消息传递算法(TRW-S)等MRF最优化方法。
【式4】
X＝argmin X E(X)......(4)
UAV控制部110根据式(4)导出误差评分E(x)小于阈值rh3(例如最小)的格点s的高度x s。借此，UAV控制部110可以根据摄像部220所拍摄的多个摄像图像，使各格点gp的高度最优化。
例如，摄像图像中映射有障碍物时，在各摄像图像的假定格点PS的投影中，障碍物可能会映射到假定投影位置，这时候，假定投影位置的像素值会发生变化。因此，可能会误将障碍物推断为地形的一部分。对此，在本实施方式中，除了搜索对象格点s外，UAV控制部110还会考虑其与相邻的格点t之间的关系来推断栅格形状。借此，无人驾驶航空器100在有障碍物等图像噪声存在时，也能通过结合搜索对象周边的格点情况来更高精度地生成地面形状。
如上所述，在本实施方式中，和第一实施方式相比，还考虑了搜索对象格点s与周围格点t之间的关系，因此，例如即便摄像图像上映射有障碍物，无人驾驶航空器100也能抑制误将障碍物推断为地形的一部分。
如上所述，UAV控制部110可以根据假定投影位置PS1a和假定投影位置PS1b的相似度(例如
Figure PCTCN2019123314-appb-000005
)，计算出表示假定投影位置PS1a和假定投影位置PS1b的像素值误差的误差评分E(X)。UAV控制部110可以根据误差评分E(X)，推断格点gp(格点s)的高度。
借此，无人驾驶航空器100可以根据各摄像图像GM1、GM2中投影同一位置(例如，假定格点PS1)的各假定投影位置PS1a、PS2b的相似度，使用误差评分来简单地推断格点的高度。
此外，UAV控制部110可以根据栅格gd中搜索对象格点s(第一栅格的一个示例)与相邻的格点t(例如，第二栅格的一个示例)之间的间隔(例如X st)，计算出误差评分E(x)。
借此，无人驾驶航空器100可以结合搜索对象格点s与相邻的格点t之间的间隔(距离)，调整对三维复原形状调整的影响度。例如，无人驾驶航空器100可以使得格点s、t的间隔越长，对三维复原形状调整的影响越小。其原因在于，如果格点gp的间隔变大，地面形状的起伏等会使相邻的格点s、t之间的关系变小。另外，距离相隔一定程度以上(例如阈值th4以上)时，可以将误差评分E(x)的第二项(式(1)右边的后半项)设为0值，消除相邻的格点t对格点s的影响。
此外，UAV控制部110可以根据格点s与格点t之间的平滑度，计算出误差评分E(x)。
相邻的格点s、t之间的关系会因格点s、t之间的地形平滑与否而不同。UAV控制部110可以结合平滑度对上述相似度进行补正，计算出误差评分E(x)。例如当相邻格点s、t之间的地形为平坦的地形时，无人驾驶航空器100可以提高补正后的相似度(降低误差评分E(X))；当相邻格点s、t之间的地形为富有起伏的地形时，无人驾驶航空器100可以降低补正后的相似度(增大误差评分E(X))。借此，相邻的格点s、t之间越平坦，误差评分E(X)越容易变小，无人驾驶航空器100容易以较小的误差推断搜索对象的栅格高度。
以上使用实施方式对本公开进行了说明，但是本公开的技术范围并不限于上述实施方式所记载的范围。对本领域普通技术人员来说，显然可对上述实施方式加以各种变更或改良。从权利要求书的记载即可明白，加以了这样的变更或改良的方式都可包含在本公开的技术范围之内。
权利要求书、说明书以及说明书附图中所示的装置、系统、程序和方法中的动作、顺序、步骤、以及阶段等各项处理的执行顺序，只要没有特别明示“在...之前”、“事 先”等，且只要前面处理的输出并不用在后面的处理中，即可以以任意顺序实现。关于权利要求书、说明书以及附图中的操作流程，为方便起见而使用“首先”、“接着”等进行了说明，但并不意味着必须按照这样的顺序实施。

Claims (20)

  1. 一种形状推断装置，其是根据飞行体所拍摄的多个摄像图像来推断三维形状的形状推断装置，其特征在于，
    包括用于进行所述三维形状推断相关处理的处理部，
    所述处理部在所述飞行体所具备的摄像装置进行拍摄的摄像范围内生成具有多个格点的栅格；
    获取所述摄像装置所拍摄的第一摄像图像及第二摄像图像；
    获取所述拍摄第一摄像图像及第二摄像图像时所述摄像装置的三维位置及所述摄像装置的摄像方向信息；
    假定所述多个格点中第一格点的多个高度；
    就假定的多个高度，根据所述摄像装置的三维位置及所述摄像装置的摄像方向，导出所述第一格点投影于所述第一摄像图像的第一假定投影位置和所述第一格点投影于所述第二摄像图像的第二假定投影位置；
    就假定的多个高度，导出所述第一摄像图像中的所述第一假定投影位置和所述第二摄像图像中的所述第二假定投影位置的相似度；
    根据假定的多个高度的所述相似度，推断所述第一格点的高度；以及
    根据所推断的所述第一格点的高度，推断所述摄像范围的三维形状。
  2. 如权利要求1所述的形状推断装置，其特征在于，
    所述处理部根据所述第一摄像图像及所述第二摄像图像，生成稀疏点云数据；
    根据所述稀疏点云数据生成面数据；
    根据所述面数据导出所述第一格点的初始高度；
    从所述初始高度开始依次假定所述第一格点的高度。
  3. 如权利要求2所述的形状推断装置，其特征在于，
    所述第一摄像图像是一个以上现有的摄像图像，
    所述第二摄像图像是一个以上最新的摄像图像，
    所述处理部获取根据所述第一摄像图像而生成的第一点云数据；
    根据所述第二摄像图像，生成第二点云数据；
    根据所述第一点云数据和所述第二点云数据，生成所述面数据。
  4. 如权利要求1至3中任一项所述的形状推断装置，其特征在于，
    所述处理部根据所述相似度，计算出表示所述第一假定投影位置和所述第二假定投影位置的误差的误差评分；
    根据所述误差评分推断所述第一格点的高度。
  5. 如权利要求4所述的形状推断装置，其特征在于，
    所述处理部根据所述栅格中所述第一格点与相邻的第二格点之间的间隔，计算出所述误差评分。
  6. 如权利要求4或5所述的形状推断装置，其特征在于，
    所述处理部根据所述第一格点与所述栅格中第二格点之间的平滑度，计算出所述误差评分。
  7. 如权利要求1至6中任一项所述的形状推断装置，其特征在于，
    所述第一格点是所述栅格中的所有格点。
  8. 如权利要求1至6中任一项所述的形状推断装置，其特征在于，
    所述第一格点是所述栅格中的部分格点。
  9. 如权利要求1至8中任一项所述的形状推断装置，其特征在于，
    所述处理部设定所述栅格各点的间隔。
  10. 一种形状推断方法，其是根据飞行体所拍摄的多个摄像图像来推断三维形状的形状推断方法，其特征在于，包括以下步骤：
    在所述飞行体所具备的摄像装置进行拍摄的摄像范围内生成具有多个格点的栅格；
    获取所述摄像装置所拍摄的第一摄像图像及第二摄像图像；
    获取所述拍摄第一摄像图像及第二摄像图像时所述摄像装置的三维位置及所述摄像装置的摄像方向信息；
    假定所述多个格点中第一格点的多个高度；
    就假定的多个高度，根据所述摄像装置的三维位置及所述摄像装置的摄像方向，导出所述第一格点投影于所述第一摄像图像的第一假定投影位置和所述第一格点投影于所述第二摄像图像的第二假定投影位置；
    就假定的多个高度，导出所述第一摄像图像中的所述第一假定投影位置和所述第二摄像图像中的所述第二假定投影位置的相似度；
    根据假定的多个高度的所述相似度，推断所述第一格点的高度的步骤；以及
    根据所推断的所述第一格点的高度，推断所述摄像范围的三维形状。
  11. 如权利要求10所述的形状推断方法，其特征在于，
    所述假定第一格点的多个高度的步骤包括以下步骤：
    根据所述第一摄像图像及所述第二摄像图像，生成点云数据；
    根据所述点云数据生成面数据；
    根据所述面数据导出所述第一格点的初始高度；以及
    从所述初始高度开始依次假定所述第一格点的高度。
  12. 如权利要求11所述的形状推断方法，其特征在于，
    所述第一摄像图像是一个以上现有的摄像图像，
    所述第二摄像图像是一个以上最新的摄像图像，
    所述生成点云数据的步骤包括：
    获取根据所述第一摄像图像而生成的第一点云数据；以及
    根据所述第二摄像图像，生成第二点云数据，
    所述生成面数据的步骤包括以下步骤：根据所述第一点云数据和所述第二点云数据，生成所述面数据。
  13. 如权利要求10至12中任一项所述的形状推断方法，其特征在于，
    所述推断第一格点的高度的步骤包括以下步骤：
    根据所述相似度，计算出表示所述第一假定投影位置和所述第二假定投影位置的像素值误差的误差评分；以及
    根据所述误差评分推断所述第一格点的高度。
  14. 如权利要求13所述的形状推断方法，其特征在于，
    所述计算误差评分的步骤包括以下步骤：根据所述栅格中所述第一格点与相邻的第二格点之间的间隔，计算出所述误差评分。
  15. 如权利要求13或14所述的形状推断方法，其特征在于，
    所述计算误差评分的步骤包括以下步骤：根据所述第一格点与所述栅格中第二格点之间的平滑度，计算出所述误差评分。
  16. 如权利要求10至15中任一项所述的形状推断方法，其特征在于，
    所述第一格点是所述栅格中的所有格点。
  17. 如权利要求10至15中任一项所述的形状推断方法，其特征在于，
    所述第一格点是所述栅格中的部分格点。
  18. 如权利要求10至17中任一项所述的形状推断方法，其特征在于，
    所述生成栅格的步骤包括设定所述栅格各点的间隔的步骤。
  19. 一种程序，其特征在于，其用于使根据飞行体所拍摄的多个摄像图像来推断三维形状的形状推断装置执行如下步骤：
    在所述飞行体所具备的摄像装置进行拍摄的摄像范围内生成具有多个格点的栅格；
    获取所述摄像装置所拍摄的第一摄像图像及第二摄像图像；
    获取所述拍摄第一摄像图像及第二摄像图像时所述摄像装置的三维位置及所述摄像装置的摄像方向信息；
    假定所述多个格点中第一格点的多个高度；
    就假定的多个高度，根据所述摄像装置的三维位置及所述摄像装置的摄像方向，导出所述第一格点投影于所述第一摄像图像的第一假定投影位置和所述第一格点投影于所述第二摄像图像的第二假定投影位置；
    就假定的多个高度，导出所述第一摄像图像中的所述第一假定投影位置和所述第二摄像图像中的所述第二假定投影位置的相似度；
    根据假定的多个高度的所述相似度，推断所述第一格点的高度；以及
    根据所推断的所述第一格点的高度，推断所述摄像范围的三维形状。
  20. 一种记录介质，其特征在于，其是记录有程序的计算机可读记录介质，所述程序用于使根据飞行体所拍摄的多个摄像图像来推断三维形状的形状推断装置执行如下步骤：
    在所述飞行体所具备的摄像装置进行拍摄的摄像范围内生成具有多个格点的栅格；
    获取所述摄像装置所拍摄的第一摄像图像及第二摄像图像；
    获取所述拍摄第一摄像图像及第二摄像图像时所述摄像装置的三维位置及所述摄像装置的摄像方向信息；
    假定所述多个格点中第一格点的多个高度；
    就假定的多个高度，根据所述摄像装置的三维位置及所述摄像装置的摄像方向，导出所述第一格点投影于所述第一摄像图像的第一假定投影位置和所述第一格点投影于所述第二摄像图像的第二假定投影位置；
    就假定的多个高度，导出所述第一摄像图像中的所述第一假定投影位置和所述第二摄像图像中的所述第二假定投影位置的相似度；
    根据假定的多个高度的所述相似度，推断所述第一格点的高度；以及
    根据所推断的所述第一格点的高度，推断所述摄像范围的三维形状。
PCT/CN2019/123314 2018-12-13 2019-12-05 形状推断装置、形状推断方法、程序以及记录介质 WO2020119572A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201980008243.0A CN111788457A (zh) 2018-12-13 2019-12-05 形状推断装置、形状推断方法、程序以及记录介质

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018233603A JP2020095519A (ja) 2018-12-13 2018-12-13 形状推定装置、形状推定方法、プログラム、及び記録媒体
JP2018-233603 2018-12-13

Publications (1)

Publication Number Publication Date
WO2020119572A1 true WO2020119572A1 (zh) 2020-06-18

Family

ID=71077122

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/123314 WO2020119572A1 (zh) 2018-12-13 2019-12-05 形状推断装置、形状推断方法、程序以及记录介质

Country Status (3)

Country Link
JP (1) JP2020095519A (zh)
CN (1) CN111788457A (zh)
WO (1) WO2020119572A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112598350B (zh) * 2020-12-24 2024-02-20 山西迪奥普科技有限公司 一种基于点云扫描的仓储堆料管理方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011024116A2 (en) * 2009-08-30 2011-03-03 Rafael Advanced Defense Systems Ltd. System and method for virtual range estimation
CN103493104A (zh) * 2011-04-29 2014-01-01 微软公司 从空间手势推断空间对象描述
CN104819706A (zh) * 2014-01-31 2015-08-05 株式会社拓普康 测量系统
CN104951745A (zh) * 2014-03-28 2015-09-30 富士重工业株式会社 车外环境识别装置
CN105928498A (zh) * 2015-02-26 2016-09-07 赫克斯冈技术中心 通过基于模板的uav控制来确定对象数据
DE102015224854A1 (de) * 2015-12-10 2017-06-14 Siemens Aktiengesellschaft Verfahren zur Erstellung einer Tiefenkarte

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001229388A (ja) * 2000-02-18 2001-08-24 Hitachi Ltd 画像データのマッチング方法
JP3618649B2 (ja) * 2000-08-22 2005-02-09 アジア航測株式会社 不定形窓を用いた画像間拡張イメージマッチング方法
JP3661187B2 (ja) * 2002-03-27 2005-06-15 ソニー株式会社 ３次元地形情報生成システム及び方法、並びにコンピュータ・プログラム
JP4624287B2 (ja) * 2006-03-17 2011-02-02 株式会社パスコ 建物形状変化検出方法及び建物形状変化検出システム
DE102010029091B4 (de) * 2009-05-21 2015-08-20 Koh Young Technology Inc. Formmessgerät und -verfahren
CN105823438B (zh) * 2012-05-22 2018-10-19 株式会社高永科技 三维形状测量装置的高度测量方法
WO2018158927A1 (ja) * 2017-03-02 2018-09-07 エスゼット ディージェイアイ テクノロジー カンパニー リミテッド ３次元形状推定方法、飛行体、モバイルプラットフォーム、プログラム及び記録媒体

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011024116A2 (en) * 2009-08-30 2011-03-03 Rafael Advanced Defense Systems Ltd. System and method for virtual range estimation
CN103493104A (zh) * 2011-04-29 2014-01-01 微软公司 从空间手势推断空间对象描述
CN104819706A (zh) * 2014-01-31 2015-08-05 株式会社拓普康 测量系统
CN104951745A (zh) * 2014-03-28 2015-09-30 富士重工业株式会社 车外环境识别装置
CN105928498A (zh) * 2015-02-26 2016-09-07 赫克斯冈技术中心 通过基于模板的uav控制来确定对象数据
DE102015224854A1 (de) * 2015-12-10 2017-06-14 Siemens Aktiengesellschaft Verfahren zur Erstellung einer Tiefenkarte

Also Published As

Publication number Publication date
CN111788457A (zh) 2020-10-16
JP2020095519A (ja) 2020-06-18

Similar Documents

Publication Publication Date Title
JP6765512B2 (ja) 飛行経路生成方法、情報処理装置、飛行経路生成システム、プログラム及び記録媒体
US20210133996A1 (en) Techniques for motion-based automatic image capture
JP6803800B2 (ja) 情報処理装置、空撮経路生成方法、空撮経路生成システム、プログラム、及び記録媒体
JP6962775B2 (ja) 情報処理装置、空撮経路生成方法、プログラム、及び記録媒体
US20210185235A1 (en) Information processing device, imaging control method, program and recording medium
JP2021117047A (ja) 無人飛行体を用いた写真測量方法および無人飛行体を用いた写真測量システム
JP2019028560A (ja) モバイルプラットフォーム、画像合成方法、プログラム、及び記録媒体
WO2020198963A1 (zh) 关于拍摄设备的数据处理方法、装置及图像处理设备
CN111344650B (zh) 信息处理装置、飞行路径生成方法、程序以及记录介质
WO2020119572A1 (zh) 形状推断装置、形状推断方法、程序以及记录介质
WO2018214401A1 (zh) 移动平台、飞行体、支持装置、便携式终端、摄像辅助方法、程序以及记录介质
WO2019189381A1 (ja) 移動体、制御装置、および制御プログラム
JP2020016663A (ja) 検査システム
WO2021115192A1 (zh) 图像处理装置、图像处理方法、程序及记录介质
US20210092306A1 (en) Movable body, image generation method, program, and recording medium
US20210256732A1 (en) Image processing method and unmanned aerial vehicle
WO2019061859A1 (zh) 移动平台、摄像路径生成方法、程序、以及记录介质
WO2020001629A1 (zh) 信息处理装置、飞行路径生成方法、程序以及记录介质
WO2021035746A1 (zh) 图像处理方法、装置和可移动平台
WO2020088397A1 (zh) 位置推定装置、位置推定方法、程序以及记录介质
WO2020108290A1 (zh) 图像生成装置、图像生成方法、程序以及记录介质
WO2023047799A1 (ja) 画像処理装置、画像処理方法及びプログラム
JP2020016664A (ja) 検査システム
Sambolek et al. Determining the Geolocation of a Person Detected in an Image Taken with a Drone

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19896871

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19896871

Country of ref document: EP

Kind code of ref document: A1