JPWO2018193574A1 - Flight path generation method, information processing apparatus, flight path generation system, program, and recording medium - Google Patents

Abstract

A flight path generation method for generating a flight path of a flying object imaging a subject includes a step of obtaining a schematic shape of an object included in the subject, a step of extracting a side surface in the schematic shape, and setting an imaging position corresponding to the side surface. And generating a flight path passing through the imaging position.

Description

  The present disclosure relates to a flight path generation method, an information processing device, a flight path generation system, a program, and a recording medium for generating a flight path of a flying object.

  2. Description of the Related Art There is known a platform (for example, an unmanned aerial vehicle) that carries a photographing device and performs photographing while flying on a preset fixed route (for example, see Patent Document 1). The platform receives commands such as a flight path and a shooting instruction from the ground base, flies according to the instructions, performs shooting, and sends the acquired image to the ground base. When capturing an imaging target, the platform inclines and captures an imaging device of the platform based on the positional relationship between the platform and the imaging target while flying along a set fixed path.

  Conventionally, it is also known to estimate a three-dimensional shape of a subject such as a building based on a captured image such as an aerial photograph taken by an unmanned aerial vehicle (for example, UAV: Unmanned Aerial Vehicle) flying in the air. In order to automate photographing (for example, aerial photography) using an unmanned aerial vehicle, a technique for generating a flight path of the unmanned aerial vehicle in advance is used. Therefore, in order to estimate the three-dimensional shape of a subject such as a building using an unmanned aerial vehicle, the unmanned aerial vehicle was caused to fly according to a previously generated flight path, and the unmanned aerial vehicle was photographed at different imaging positions in the flight path. It is necessary to acquire a plurality of captured images of a subject.

Japanese Patent Application Laid-Open No. 2010-61216

  In the platform described in Patent Literature 1, an image is taken while passing through a fixed path, but the existence of an object (for example, a building) positioned vertically from the fixed path is not sufficiently considered. Therefore, it is difficult to sufficiently obtain a captured image of the side surface of the object or another captured image hidden by a part of the object that can be observed from above. Therefore, the number of captured images for estimating the three-dimensional shape is insufficient, and the estimation accuracy of the three-dimensional shape is reduced.

  When capturing an image of the side surface of an object, it is conceivable that a photographer grasps the imaging device and captures an image of the side surface of the object. In this case, since the user needs to move to the periphery of the object, the convenience of the user is reduced. In addition, since the imaging is manually performed by the user, there is a possibility that a captured image in a desired state (for example, a desired imaging position of the object, a desired imaging size of the object, a desired imaging direction of the object) cannot be sufficiently acquired.

  In the case where the side surface of a specific object is imaged by an unmanned aerial vehicle, it is conceivable that the flight path of the unmanned aerial vehicle is manually determined in advance. When a desired position around an object is designated as an imaging position, it is conceivable that a position (latitude, longitude, altitude) in a three-dimensional space is designated by a user input. In this case, since each imaging position is determined by a user input, user convenience is reduced. In addition, detailed information of the object is required in advance for determining the flight path, so that preparation is troublesome.

  In one aspect, a flight path generation method for generating a flight path of a flying object that images a subject, the method including: obtaining a schematic shape of an object included in the subject; extracting a side surface in the general shape; Setting a corresponding imaging position; and generating a flight path passing through the imaging position.

  The step of setting the imaging position may include, for each extracted side surface, setting an imaging position facing the side surface.

  The step of setting the imaging position may include a step of setting a plurality of imaging positions having a predetermined imaging position interval corresponding to the side surface.

  Generating the flight path may include determining an imaging path passing through the plurality of imaging positions and generating a flight path including the imaging path.

  The flight path generation method further includes a step of generating an imaging plane parallel to the side surface at a predetermined imaging distance, and the step of setting an imaging position has a predetermined imaging position interval in the imaging plane. The method may include a step of setting a plurality of imaging positions.

  The step of setting the imaging position may use, as the predetermined imaging position interval, an imaging position interval in which a part of an image captured at each imaging position overlaps with another.

  The flight path generation method further includes a step of calculating a polyhedron surrounding a schematic shape of the object, and the step of extracting a side surface includes a surface along the vertical direction in the polyhedron, or a surface standing within a predetermined angle range in the vertical direction. May be extracted as an aspect.

  The flight path generation method further includes a step of calculating a polyhedron in which the schematic shape of the object is simplified, and the step of extracting the side surface includes a step of extracting a side surface of the polyhedron along a vertical direction or a predetermined angle range in the vertical direction. The method may include a step of extracting the cut surface as a side surface.

  The step of calculating a polyhedron may include a step of calculating a polyhedron corresponding to a plurality of schematic shapes of the object, and combining a plurality of adjacent polyhedrons.

  Generating a flight path may include generating a flight path that passes through the imaging position on one of the side surfaces, and generating a flight path that passes through the imaging position on the next side surface adjacent to the side surface.

  The flight path generation method further includes a step of acquiring a captured image of the object facing downward, and the step of acquiring the schematic shape includes a step of acquiring three-dimensional shape data of the schematic shape of the object using the captured image. May be included.

  In one aspect, an information processing device that generates a flight path of a flying object that captures an image of a subject includes a processing unit that executes a process related to the flight path, and the processing unit obtains a schematic shape of an object included in the subject. This is an information processing apparatus that extracts a side surface in a schematic shape, sets an imaging position corresponding to the side surface, and generates a flight path passing through the imaging position.

  The processing unit may set an imaging position facing the side surface for each extracted side surface in setting the imaging position.

  In the setting of the imaging position, the processing unit may set a plurality of imaging positions with a predetermined imaging position interval corresponding to the side surface.

  In generating the flight path, the processing unit may determine an imaging path passing through a plurality of imaging positions and generate a flight path including the imaging path.

  The processing unit further generates an imaging plane parallel to the side surface at a predetermined imaging distance and sets a plurality of imaging positions having a predetermined imaging position interval on the imaging plane in setting the imaging position. Good.

  In the setting of the imaging position, the processing unit may use, as the predetermined imaging position interval, an imaging position interval in which a part of an image captured at each imaging position overlaps with another.

  The processing unit may further calculate a polyhedron surrounding the approximate shape of the object, and extract a side along a vertical direction or a surface standing within a predetermined angle range in the vertical direction in the polyhedron in the side surface extraction.

  The processing unit further calculates a polyhedron in which the schematic shape of the object is simplified, and in the extraction of the side surface, extracts a surface along the vertical direction in the polyhedron or a surface standing within a predetermined angle range in the vertical direction as the side surface. Good.

  In the calculation of the polyhedron, the processing unit may calculate a polyhedron corresponding to a plurality of schematic shapes of the object, respectively, and combine a plurality of adjacent polyhedrons.

  In generating the flight path, the processing unit may generate a flight path that passes through the imaging position on one of the side surfaces, and may generate a flight path that passes through the imaging position on the next side surface adjacent to the side surface.

  The processing unit may further obtain a captured image obtained by capturing the object downward, and may obtain three-dimensional shape data of the general shape of the object using the captured image in obtaining the general shape.

  In one aspect, a flight object that captures an image of a subject, and a processing unit that generates a flight path of the flying object, is a flight path generation system, wherein the processing unit acquires a schematic shape of an object included in the subject, A flight path generation system that extracts a side surface in a schematic shape, sets an imaging position corresponding to the side surface, generates a flight path passing through the imaging position, and acquires and sets the flight path of the flying object.

  In one aspect, the program corresponds to a step of obtaining a schematic shape of an object included in the subject, extracting a side surface in the schematic shape, and a computer, the computer generating a flight path of the flying object imaging the subject. A program for executing a step of setting an imaging position and a step of generating a flight path passing through the imaging position.

  In one aspect, a recording medium includes: a computer that generates a flight path of a flying object that images a subject; obtaining a schematic shape of an object included in the subject; extracting a side surface in the schematic shape; A computer-readable recording medium storing a program for executing a step of setting an imaging position to be executed and a step of generating a flight path passing through the imaging position.

  Note that the above summary of the present invention does not list all of the features of the present disclosure. Further, a sub-combination of these feature groups can also be an invention.

FIG. 1 is a schematic diagram illustrating a first configuration example of a flight path generation system according to an embodiment. It is a figure showing an example of the appearance of an unmanned aerial vehicle. It is a figure showing an example of the concrete appearance of an unmanned aerial vehicle. FIG. 2 is a block diagram illustrating an example of a hardware configuration of an unmanned aerial vehicle included in the flight route generation system in FIG. 1. It is a figure showing an example of appearance of a transmitter with which a personal digital assistant was attached. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a transmitter included in the flight route generation system in FIG. 1. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a mobile terminal included in the flight route generation system in FIG. 1. 6 is a flowchart illustrating an example of a processing procedure of a flight route generation method according to the embodiment. It is a figure for explaining an example of input of a flight range. FIG. 3 is a diagram for explaining schematic imaging on a flight path. It is a figure for explaining generation of three-dimensional shape data of an outline shape based on an outline imaging obtained by a flight course. 6 is a flowchart illustrating an example of a processing procedure of a three-dimensional shape estimation method according to the embodiment. FIG. 9 is a diagram for describing a first operation example of flight path generation using a schematic shape of an object in the embodiment. FIG. 4 is a diagram for describing setting of a plurality of imaging positions on an imaging plane. 6 is a flowchart illustrating a processing procedure of a first operation example of generating a flight path using a schematic shape of an object according to the embodiment. FIG. 9 is a diagram for describing a second operation example of generating a flight path using a schematic shape of an object in the embodiment. 9 is a flowchart illustrating a processing procedure of a second operation example of generating a flight path using a schematic shape of an object according to the embodiment. FIG. 11 is a diagram for describing a third operation example of generating a flight path using a schematic shape of an object in the embodiment. 11 is a flowchart illustrating a processing procedure of a third operation example of generating a flight path using a schematic shape of an object according to the embodiment. FIG. 3 is a schematic diagram illustrating a second configuration example of the flight path generation system according to the embodiment. It is a block diagram showing an example of hardware constitutions of an unmanned aerial vehicle concerning a 3rd example of composition of a flight course generating system in an embodiment. It is a block diagram showing an example of hardware constitutions of a transmitter concerning a 4th example of composition of a flight course generating system in an embodiment.

  Hereinafter, the present disclosure will be described through embodiments of the present invention, but the following embodiments do not limit the invention according to the claims. Not all combinations of the features described in the embodiments are necessarily essential to the solution of the invention.

  The claims, the description, the drawings, and the abstract include matters covered by copyright. The copyright owner will not object to any person's reproduction of these documents, as indicated in the JPO file or record. However, in all other cases, all copyrights are reserved.

  A flight path generation system according to the present disclosure is configured to include a flying object as an example of a moving object, and a platform for remotely controlling operation or processing of the flying object.

  An information processing device according to the present disclosure is a computer included in at least one of a platform and a flying object, and executes various processes related to the operation of the flying object.

  Flying vehicles include airborne vehicles (eg, drones, helicopters). The flying object may be an unmanned aerial vehicle (UAV) having an imaging device. The flying object flies along a preset flight path in order to image a subject (for example, a ground shape of a building, a road, a park, or the like within a certain range) in the imaging range, and is set on the flight path. The subject is imaged at a plurality of imaging positions. The subject includes, for example, an object such as a building or a road.

  The platform is a computer, for example, a transmitter for instructing remote control of various processes including movement of the flying object, or a communication terminal connected to the transmitter or the flying object so that information and data can be input and output. is there. The communication terminal may be, for example, a mobile terminal, a PC (Personal Computer), or the like. Note that the flying object itself may be included as a platform.

  In the flight route generation method according to the present disclosure, various processes (steps) in the information processing device (platform, flying object) or the flight route generation system are defined.

  A program according to the present disclosure is a program for causing an information processing device (platform, flying object) or a flight route generation system to execute various processes (steps).

  A recording medium according to the present disclosure has recorded thereon a program (that is, a program for causing an information processing device (platform, flying object) or a flight path generation system to execute various processes (steps)).

  In the following embodiments, an unmanned aerial vehicle (UAV) will be exemplified as a vehicle. In the drawings attached to this specification, the unmanned aerial vehicle is referred to as “UAV”. In the present embodiment, the unmanned aerial vehicle sets a flight path including an imaging position where the side surface of the object can be imaged.

[Flying path generation system, first configuration example]
FIG. 1 is a schematic diagram illustrating a first configuration example of a flight route generation system 10 according to the embodiment. The flight path generation system 10 includes an unmanned aerial vehicle 100, a transmitter 50, and a mobile terminal 80. The unmanned aerial vehicle 100, the transmitter 50, and the mobile terminal 80 can communicate with each other using wired communication or wireless communication (for example, wireless LAN (Local Area Network) or Bluetooth (registered trademark)). The transmitter 50 is used, for example, while being held by both hands of a person who uses the transmitter 50 (hereinafter, referred to as a “user”).

  FIG. 2 is a diagram illustrating an example of the appearance of the unmanned aerial vehicle 100. FIG. 3 is a diagram illustrating an example of a specific appearance of the unmanned aerial vehicle 100. A side view when the unmanned aerial vehicle 100 flies in the movement direction STV0 is shown in FIG. 2, and a perspective view when the unmanned aerial vehicle 100 flies in the movement direction STV0 is shown in FIG. The unmanned aerial vehicle 100 is an example of a moving object that includes imaging devices 220 and 230 as an example of an imaging unit and moves. The moving body is a concept including, in addition to the unmanned aerial vehicle 100, another aircraft moving in the air, a vehicle moving on the ground, a ship moving on water, and the like. Here, as shown in FIGS. 2 and 3, it is assumed that a roll axis (see the x-axis in FIGS. 2 and 3) is defined in a direction parallel to the ground and along the movement direction STV0. In this case, a pitch axis (see the y-axis in FIGS. 2 and 3) is defined in a direction parallel to the ground and perpendicular to the roll axis, and further a direction perpendicular to the ground and perpendicular to the roll axis and the pitch axis. The yaw axis (the z-axis in FIGS. 2 and 3) is determined.

  The unmanned aerial vehicle 100 is configured to include a UAV main body 102, a gimbal 200, an imaging device 220, and a plurality of imaging devices 230. The unmanned aerial vehicle 100 moves based on a remote control instruction transmitted from a transmitter 50 as an example of a platform. The movement of the unmanned aerial vehicle 100 means a flight, and includes at least ascending, descending, turning left, turning right, moving left horizontally, and moving right horizontal.

  The UAV main body 102 includes a plurality of rotors (propellers). The UAV body 102 moves the unmanned aerial vehicle 100 by controlling the rotation of a plurality of rotors. The UAV main body 102 moves the unmanned aerial vehicle 100 using, for example, four rotors. The number of rotors is not limited to four. Further, the unmanned aerial vehicle 100 may be a fixed wing aircraft having no rotary wing.

  The imaging device 220 is an imaging camera for imaging a subject (for example, a building on the ground) included in a desired imaging range. The subject may include, for example, an object such as a building, as well as a state of the sky as an aerial photograph target of the unmanned aerial vehicle 100 and scenery such as a mountain or a river.

  The plurality of imaging devices 230 are sensing cameras that image the periphery of the unmanned aerial vehicle 100 in order to control the movement of the unmanned aerial vehicle 100. Two imaging devices 230 may be provided in front of the unmanned aerial vehicle 100, which is the nose. Further, two other imaging devices 230 may be provided on the bottom surface of the unmanned aerial vehicle 100. The two imaging devices 230 on the front side may be paired and function as a so-called stereo camera. The two imaging devices 230 on the bottom side may also be paired and function as a stereo camera. Three-dimensional spatial data around the unmanned aerial vehicle 100 may be generated based on images captured by the plurality of imaging devices 230. Note that the number of imaging devices 230 included in the unmanned aerial vehicle 100 is not limited to four. The unmanned aerial vehicle 100 may include at least one imaging device 230. The unmanned aerial vehicle 100 may include at least one imaging device 230 on each of the nose, the nose, the side surface, the bottom surface, and the ceiling surface of the unmanned aerial vehicle 100. The angle of view that can be set by the imaging device 230 may be wider than the angle of view that can be set by the imaging device 220. The imaging device 230 may include a single focus lens or a fisheye lens.

[Configuration example of unmanned aerial vehicle]
Next, a configuration example of the unmanned aerial vehicle 100 will be described.

  FIG. 4 is a block diagram illustrating an example of a hardware configuration of the unmanned aerial vehicle 100 included in the flight route generation system 10 of FIG. The unmanned aerial vehicle 100 includes a processing unit 110, a communication interface 150, a memory 160, a storage 170, a battery 190, a gimbal 200, a rotary wing mechanism 210, an imaging device 220, an imaging device 230, and a GPS reception device. The apparatus includes an instrument 240, an inertial measurement unit (IMU: Inertial Measurement Unit) 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, and a laser measuring device 290. The communication interface 150 is an example of a communication unit.

  The processing unit 110 is configured using a processor, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). The processing unit 110 performs signal processing for integrally controlling the operation of each unit of the unmanned aerial vehicle 100, data input / output processing with other units, data calculation processing, and data storage processing.

  The processing unit 110 controls the flight of the unmanned aerial vehicle 100 according to a program stored in the memory 160. The processing unit 110 controls the movement (i.e., flight) of the unmanned aerial vehicle 100 according to a command received from the remote transmitter 50 via the communication interface 150. Memory 160 may be removable from unmanned aerial vehicle 100.

  The processing unit 110 acquires image data of a subject captured by the imaging device 220 and the imaging device 230 (hereinafter, may be referred to as “captured image”).

  The processing unit 110 controls the gimbal 200, the rotary wing mechanism 210, the imaging device 220, and the imaging device 230. The processing unit 110 controls the imaging range of the imaging device 220 by changing the imaging direction or the angle of view of the imaging device 220. The processing unit 110 controls an imaging range of the imaging device 220 supported by the gimbal 200 by controlling a rotation mechanism of the gimbal 200.

  In the present specification, the imaging range refers to a geographical area captured by the imaging device 220 or the imaging device 230. The imaging range is defined by latitude, longitude, and altitude. The imaging range may be a range in three-dimensional spatial data defined by latitude, longitude, and altitude. The imaging range is specified based on the angle of view and imaging direction of the imaging device 220 or the imaging device 230, and the position where the unmanned aerial vehicle 100 exists. The imaging direction of the imaging device 220 and the imaging device 230 is defined by the azimuth of the front of the imaging device 220 and the imaging device 230 provided with the imaging lens and the depression angle. The imaging direction of the imaging device 220 is a direction specified from the azimuth of the nose of the unmanned aerial vehicle 100 and the state of the attitude of the imaging device 220 with respect to the gimbal 200. The imaging direction of the imaging device 230 is a direction specified from the azimuth of the nose of the unmanned aerial vehicle 100 and the position where the imaging device 230 is provided.

  The processing unit 110 controls the flight of the unmanned aerial vehicle 100 by controlling the rotary wing mechanism 210. That is, the processing unit 110 controls the position of the unmanned aerial vehicle 100 including the latitude, longitude, and altitude by controlling the rotary wing mechanism 210. The processing unit 110 may control the imaging range of the imaging device 220 and the imaging device 230 by controlling the flight of the unmanned aerial vehicle 100. The processing unit 110 may control the angle of view of the imaging device 220 by controlling the zoom lens included in the imaging device 220. The processing unit 110 may use the digital zoom function of the imaging device 220 to control the angle of view of the imaging device 220 by digital zoom. The processing unit 110 captures an image of a subject in a horizontal direction, a predetermined angle direction, or a vertical direction by the imaging device 220 or the imaging device 230 at an imaging position (waypoint (Waypoint) described later) existing in the middle of the set flight path. Let it. The direction of the predetermined angle is a direction of the predetermined angle suitable for the information processing apparatus (unmanned aerial vehicle or platform) to estimate the three-dimensional shape of the subject.

  When the imaging device 220 is fixed to the unmanned aerial vehicle 100 and the imaging device 220 cannot be moved, the processing unit 110 moves the unmanned aerial vehicle 100 to a specific position at a specific date and time to obtain a desired image under a desired environment. The imaging range can be imaged by the imaging device 220. Alternatively, even when the image capturing apparatus 220 does not have a zoom function and the angle of view of the image capturing apparatus 220 cannot be changed, the processing unit 110 moves the unmanned aerial vehicle 100 to a specific position on the specified date and time. Under the above environment, a desired imaging range can be imaged by the imaging device 220.

  The communication interface 150 communicates with the transmitter 50. The communication interface 150 receives various commands to the processing unit 110 from the remote transmitter 50.

  The memory 160 is an example of a storage unit. In the memory 160, the processing unit 110 includes a gimbal 200, a rotary wing mechanism 210, an imaging device 220, an imaging device 230, a GPS receiver 240, an inertial measurement device 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, and laser measurement. And a program necessary for controlling the device 290. The memory 160 stores the images captured by the imaging devices 220 and 230. The memory 160 may be a computer-readable recording medium, such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), and It may include at least one of a flash memory such as a USB memory. The memory 160 may be provided inside the UAV body 102. It may be provided detachably from the UAV body 102.

  The storage 170 is an example of a storage unit. The storage 170 accumulates and holds various data and information. The storage 170 may be a hard disk drive (HDD), a solid state drive (SSD), a memory card, a USB memory, or the like. The storage 170 may be provided inside the UAV body 102. The storage 170 may be provided detachably from the UAV body 102.

  The battery 190 has a function as a drive source for each unit of the unmanned aerial vehicle 100, and supplies necessary power to each unit of the unmanned aerial vehicle 100.

  The gimbal 200 rotatably supports the imaging device 220 about at least one axis. The gimbal 200 may rotatably support the imaging device 220 about a yaw axis, a pitch axis, and a roll axis. The gimbal 200 may change the imaging direction of the imaging device 220 by rotating the imaging device 220 about at least one of the yaw axis, the pitch axis, and the roll axis.

  The rotary wing mechanism 210 includes a plurality of rotary blades and a plurality of drive motors for rotating the plurality of rotary blades.

  The imaging device 220 captures an object in a desired imaging range and generates data of a captured image. Image data obtained by imaging by the imaging device 220 is stored in the memory of the imaging device 220 or the memory 160.

  The imaging device 230 captures an image around the unmanned aerial vehicle 100 and generates data of a captured image. Image data of the imaging device 230 is stored in the memory 160.

  The GPS receiver 240 receives a plurality of signals indicating times transmitted from a plurality of navigation satellites (that is, GPS satellites) and a position (coordinate) of each GPS satellite. The GPS receiver 240 calculates the position of the GPS receiver 240 (that is, the position of the unmanned aerial vehicle 100) based on the received signals. The GPS receiver 240 outputs the position information of the unmanned aerial vehicle 100 to the processing unit 110. The calculation of the position information of the GPS receiver 240 may be performed by the processing unit 110 instead of the GPS receiver 240. In this case, the processing unit 110 receives input of the time included in the plurality of signals received by the GPS receiver 240 and information indicating the position of each GPS satellite.

  Inertial measurement device 250 detects the attitude of unmanned aerial vehicle 100 and outputs the detection result to processing section 110. The inertial measurement device 250 calculates, as the attitude of the unmanned aerial vehicle 100, acceleration in the three axial directions of the unmanned aerial vehicle 100 in front and rear, left and right, and up and down, and angular velocities in three axial directions of a pitch axis, a roll axis, and a yaw axis. To detect.

  The magnetic compass 260 detects the azimuth of the nose of the unmanned aerial vehicle 100 and outputs the detection result to the processing unit 110.

  The barometric altimeter 270 detects the altitude at which the unmanned aerial vehicle 100 flies, and outputs the detection result to the processing unit 110.

  The ultrasonic sensor 280 emits ultrasonic waves, detects ultrasonic waves reflected by the ground or an object, and outputs a detection result to the processing unit 110. The detection result may indicate, for example, the distance (ie, altitude) from the unmanned aerial vehicle 100 to the ground. The detection result may indicate, for example, the distance from the unmanned aerial vehicle 100 to the object.

  The laser measuring device 290 irradiates the object with laser light, receives the light reflected by the object, and measures the distance between the unmanned aerial vehicle 100 and the object by the reflected light. The distance measurement result is input to the processing unit 110. As a method of measuring the distance by the laser beam, for example, a time-of-flight method may be used.

  Next, an example of the function of the processing unit 110 of the unmanned aerial vehicle 100 will be described.

  The processing unit 110 may specify the environment around the unmanned aerial vehicle 100 by analyzing a plurality of images captured by the plurality of imaging devices 230. The processing unit 110 controls the flight while avoiding an obstacle, for example, based on the environment around the unmanned aerial vehicle 100. The processing unit 110 may generate three-dimensional space data around the unmanned aerial vehicle 100 based on a plurality of images captured by the plurality of imaging devices 230, and control flight based on the three-dimensional space data.

  The processing unit 110 acquires date and time information indicating the current date and time. The processing unit 110 may acquire date and time information indicating the current date and time from the GPS receiver 240. The processing unit 110 may obtain date and time information indicating the current date and time from a timer (not shown) mounted on the unmanned aerial vehicle 100.

  The processing unit 110 acquires position information indicating the position of the unmanned aerial vehicle 100. The processing unit 110 may acquire, from the GPS receiver 240, position information indicating the latitude, longitude, and altitude at which the unmanned aerial vehicle 100 exists. The processing unit 110 transmits, from the GPS receiver 240, latitude and longitude information indicating the latitude and longitude at which the unmanned aerial vehicle 100 is present, and altitude information indicating the altitude at which the unmanned aerial vehicle 100 is present from the barometric altimeter 270 or the ultrasonic sensor 280, respectively. It may be obtained as position information.

  The processing unit 110 may obtain direction information indicating the direction of the unmanned aerial vehicle 100 from the magnetic compass 260. The direction information may indicate a direction corresponding to the direction of the nose of the unmanned aerial vehicle 100, for example.

  The processing unit 110 may acquire position information indicating a position where the unmanned aerial vehicle 100 should exist when the imaging device 220 captures an imaging range to be imaged. The processing unit 110 may acquire position information indicating a position where the unmanned aerial vehicle 100 should exist from the memory 160. The processing unit 110 may acquire position information indicating a position where the unmanned aerial vehicle 100 should exist from another device such as the transmitter 50 via the communication interface 150. The processing unit 110 refers to the three-dimensional map database, specifies a position where the unmanned aerial vehicle 100 can exist, and determines the position where the unmanned aerial vehicle 100 exists in order to image the imaging range to be imaged. You may acquire as position information which shows a position.

  The processing unit 110 may acquire imaging information indicating an imaging range of each of the imaging device 220 and the imaging device 230. The processing unit 110 may acquire angle-of-view information indicating the angle of view of the imaging device 220 and the imaging device 230 from the imaging device 220 and the imaging device 230 as a parameter for specifying the imaging range. The processing unit 110 may acquire information indicating the imaging direction of the imaging device 220 and the imaging device 230 as a parameter for specifying the imaging range. The processing unit 110 may acquire posture information indicating the posture state of the imaging device 220 from the gimbal 200, for example, as information indicating the imaging direction of the imaging device 220. The information indicating the posture state of the imaging device 220 may be indicated by, for example, a rotation angle from a reference rotation angle of the pitch axis and the yaw axis of the gimbal 200. The processing unit 110 may acquire information indicating the direction of the unmanned aerial vehicle 100, for example, as information indicating the imaging direction of the imaging device 220. The processing unit 110 may acquire position information indicating a position where the unmanned aerial vehicle 100 exists as a parameter for specifying the imaging range. The processing unit 110 defines an imaging range indicating a geographical range to be imaged by the imaging device 220 based on the angle of view and the imaging direction of the imaging device 220 and the imaging device 230, and the position where the unmanned aerial vehicle 100 exists. The imaging information may be acquired by generating imaging information indicating the imaging range.

  The processing unit 110 may acquire imaging information indicating an imaging range in which the imaging device 220 should image. The processing unit 110 may acquire imaging information to be imaged by the imaging device 220 from the memory 160. The processing unit 110 may acquire imaging information to be imaged by the imaging device 220 from another device such as the transmitter 50 via the communication interface 150.

  The processing unit 110 acquires three-dimensional information (three-dimensional information) indicating a three-dimensional shape (three-dimensional shape) of an object existing around the unmanned aerial vehicle 100. The object is, for example, a part of a landscape such as a building, a road, a car, and a tree. The three-dimensional information is, for example, three-dimensional space data. The processing unit 110 may acquire the stereoscopic information by generating stereoscopic information indicating the stereoscopic shape of the object existing around the unmanned aerial vehicle 100 from each image obtained from the plurality of imaging devices 230. The processing unit 110 may acquire the three-dimensional information indicating the three-dimensional shape of the object existing around the unmanned aerial vehicle 100 by referring to the three-dimensional map database stored in the memory 160. The processing unit 110 may acquire three-dimensional information on the three-dimensional shape of an object existing around the unmanned aerial vehicle 100 by referring to a three-dimensional map database managed by a server existing on the network.

  Next, a configuration example of the transmitter 50 and the portable terminal 80 will be described.

  FIG. 5 is a diagram illustrating an example of the appearance of the transmitter 50 on which the mobile terminal 80 is mounted. FIG. 5 illustrates a smartphone as an example of the mobile terminal 80. The mobile terminal 80 may be, for example, a smartphone, a tablet terminal, or the like. It is assumed that the directions of up, down, front, rear, left and right with respect to the transmitter 50 follow the directions of the arrows shown in FIG. The transmitter 50 is used, for example, while being held by both hands of the user who uses the transmitter 50.

  The transmitter 50 has, for example, a substantially rectangular parallelepiped (in other words, substantially box-shaped) resin housing 50B having a substantially square bottom surface and a height shorter than one side of the bottom surface. A left control rod 53L and a right control rod 53R are arranged protrudingly at substantially the center of the housing surface of the transmitter 50.

  The left control rod 53L and the right control rod 53R are used in operations for remotely controlling the movement of the unmanned aerial vehicle 100 by the user (for example, moving the unmanned aerial vehicle 100 back and forth, left and right, up and down, and change direction). Is done. FIG. 5 shows the positions of the left control rod 53L and the right control rod 53R in the initial state where no external force is applied from both hands of the user. After the external force applied by the user is released, the left control rod 53L and the right control rod 53R automatically return to predetermined positions (for example, initial positions shown in FIG. 5).

  The power button B1 of the transmitter 50 is arranged on the near side (in other words, on the user side) of the left control rod 53L. When the user presses the power button B1 once, for example, the remaining capacity of the battery built in the transmitter 50 is displayed on the remaining battery level display section L2. When the power button B1 is pressed again by the user, for example, the power of the transmitter 50 is turned on, and power is supplied to each unit (see FIG. 6) of the transmitter 50 to be usable.

  An RTH (Return To Home) button B2 is arranged on the near side (in other words, on the user side) of the right control rod 53R. When the RTH button B2 is pressed by the user, the transmitter 50 transmits a signal for automatically returning the unmanned aerial vehicle 100 to a predetermined position. Thus, the transmitter 50 can automatically return the unmanned aerial vehicle 100 to a predetermined position (for example, a takeoff position stored in the unmanned aerial vehicle 100). The RTH button B2 is used, for example, when the user loses sight of the unmanned aerial vehicle 100 during aerial photographing with the unmanned aerial vehicle 100 outdoors, or when the user becomes unable to operate due to radio wave interference or unexpected trouble. Available to

  A remote status display section L1 and a remaining battery level display section L2 are arranged in front of the power button B1 and the RTH button B2 (in other words, on the user side). The remote status display unit L1 is configured using, for example, an LED (Light Emission Diode), and displays a wireless connection state between the transmitter 50 and the unmanned aerial vehicle 100. The battery remaining amount display unit L2 is configured using, for example, an LED, and displays the remaining amount of the battery built in the transmitter 50.

  Two antennas AN1 and AN2 are arranged to protrude from the rear side surface of the housing 50B of the transmitter 50 on the rear side of the left control rod 53L and the right control rod 53R. The antennas AN1 and AN2 transmit a signal for controlling the movement of the unmanned aerial vehicle 100 to the unmanned aerial vehicle 100 based on a user's operation of the left control rod 53L and the right control rod 53R. The antennas AN1 and AN2 can cover a transmission and reception range of, for example, 2 km. In addition, the antennas AN1 and AN2 are connected to the image pickup devices 220 and 230 of the unmanned aerial vehicle 100 wirelessly connected to the transmitter 50, or the various data acquired by the unmanned aerial vehicle 100 from the unmanned aerial vehicle 100. When transmitted, these images or various data can be received.

  In FIG. 5, the transmitter 50 does not include a display unit, but may include a display unit.

  The mobile terminal 80 may be mounted on the holder HLD. The holder HLD may be joined and attached to the transmitter 50. Thereby, the portable terminal 80 is mounted on the transmitter 50 via the holder HLD. The portable terminal 80 and the transmitter 50 may be connected via a wired cable (for example, a USB cable). The mobile terminal 80 and the transmitter 50 may be connected by wireless communication (for example, Bluetooth (registered trademark)). The mobile terminal 80 may not be mounted on the transmitter 50, and the mobile terminal 80 and the transmitter 50 may be independently provided.

[Configuration example of transmitter]
FIG. 6 is a block diagram illustrating an example of a hardware configuration of the transmitter 50 included in the flight route generation system 10 of FIG. The transmitter 50 includes a left control rod 53L, a right control rod 53R, a processing unit 61, a wireless communication unit 63, an interface unit 65, a memory 67, a battery 69, a power button B1, and an RTH button B2. , An operation section set OPS, a remote status display section L1, and a remaining battery level display section L2. The transmitter 50 is an example of an operation terminal for remotely controlling the unmanned aerial vehicle 100.

  The processing unit 61 is configured using a processor (for example, a CPU, an MPU, or a DSP). The processing unit 61 performs signal processing for integrally controlling the operation of each unit of the transmitter 50, data input / output processing with other units, data arithmetic processing, and data storage processing.

  The processing unit 61 acquires the data of the captured image captured by the imaging device 220 of the unmanned aerial vehicle 100 via the wireless communication unit 63, stores the data in the memory 67, and outputs the data to the mobile terminal 80 via the interface unit 65. May be. In other words, the processing unit 61 may cause the mobile terminal 80 to display an image captured by the imaging device 220 of the unmanned aerial vehicle 100. Thus, the captured image captured by the imaging device 220 of the unmanned aerial vehicle 100 can be displayed on the mobile terminal 80.

  The processing unit 61 may generate a signal for controlling the movement of the unmanned aerial vehicle 100 specified by the operation of the left control rod 53L and the right control rod 53R by the user. The processing unit 61 may transmit the generated signal to the unmanned aerial vehicle 100 via the wireless communication unit 63 and the antennas AN1 and AN2 to remotely control the unmanned aerial vehicle 100. Accordingly, the transmitter 50 can remotely control the movement of the unmanned aerial vehicle 100. The processing unit 61 may acquire the map information of the map database stored in the external server or the like via the wireless communication unit 63.

  The left control rod 53L is used for an operation for remotely controlling the movement of the unmanned aerial vehicle 100 by, for example, the left hand of the user. The right control rod 53R is used for an operation for remotely controlling the movement of the unmanned aerial vehicle 100 by, for example, the right hand of the user. The movement of the unmanned aerial vehicle 100 includes, for example, movement in a forward direction, movement in a backward direction, movement in a left direction, movement in a right direction, movement in a rising direction, movement in a descending direction, and movement in the left direction. , Or a movement of rotating the unmanned aerial vehicle 100 rightward, or a combination thereof, and so on.

  The battery 69 has a function as a drive source of each unit of the transmitter 50, and supplies necessary power to each unit of the transmitter 50.

  When the power button B1 is pressed once, the processing section 61 displays the remaining capacity of the battery 69 incorporated in the transmitter 50 on the remaining battery level display section L2. Thus, the user can easily check the remaining capacity of the battery built in the transmitter 50. The processing unit 61 may display the remaining capacity of the battery incorporated in the unmanned aerial vehicle 100 on the battery remaining amount display unit L2. When the power button B1 is pressed twice, the processing unit 61 instructs the battery 69 incorporated in the transmitter 50 to supply power to each unit in the transmitter 50. Thus, the user turns on the power of the transmitter 50 and can easily start using the transmitter 50.

  When the RTH button B2 is pressed, the processing unit 61 generates a signal for automatically returning the unmanned aerial vehicle 100 to a predetermined position (for example, a takeoff position of the unmanned aerial vehicle 100), and the wireless communication unit 63 and the antenna AN1. , AN2 to the unmanned aerial vehicle 100. Thus, the user can automatically return (return) the unmanned aerial vehicle 100 to a predetermined position by a simple operation on the transmitter 50.

  The operation unit set OPS is configured using a plurality of operation units (for example, operation units OP1,..., Operation unit OPn) (n: an integer of 2 or more). The operation unit set OPS supports remote control of the unmanned aerial vehicle 100 by the operation units other than the left control rod 53L, the right control rod 53R, the power button B1, and the RTH button B2 shown in FIG. Various operation units). The various operation units referred to herein include, for example, a button for instructing imaging of a still image using the imaging device 220 of the unmanned aerial vehicle 100, and starting and ending recording of a moving image using the imaging device 220 of the unmanned aerial vehicle 100 Button, a dial for adjusting the tilt of the gimbal 200 (see FIG. 4) of the unmanned aerial vehicle 100 in the tilt direction, a button for switching the flight mode of the unmanned aerial vehicle 100, and setting of the imaging device 220 of the unmanned aerial vehicle 100. Dial is applicable.

  Since the remote status display section L1 and the remaining battery level display section L2 have been described with reference to FIG. 5, the description is omitted here.

  The wireless communication unit 63 is connected to two antennas AN1 and AN2. The wireless communication unit 63 transmits and receives information and data to and from the unmanned aerial vehicle 100 via the two antennas AN1 and AN2 using a predetermined wireless communication method (for example, WiFi (registered trademark)).

  The interface unit 65 inputs and outputs information and data between the transmitter 50 and the mobile terminal 80. The interface unit 65 may be a USB port (not shown) provided in the transmitter 50, for example. The interface unit 65 may be an interface other than the USB port.

  The memory 67 is an example of a storage unit. The memory 67 temporarily stores, for example, a ROM (Read Only Memory) in which a program for defining the operation of the processing unit 61 and data of set values are stored, and various information and data used in the processing of the processing unit 61. (Random Access Memory). The programs and the data of the setting values stored in the ROM of the memory 64 may be copied to a predetermined recording medium (for example, a CD-ROM or a DVD-ROM). In the RAM of the memory 64, for example, data of a captured image captured by the imaging devices 220 and 230 of the unmanned aerial vehicle 100 may be stored.

[Configuration example of mobile terminal]
FIG. 7 is a block diagram illustrating an example of a hardware configuration of the mobile terminal 80 included in the flight route generation system 10 of FIG. The mobile terminal 80 may include a processing unit 81, an interface unit 82, an operation unit 83, a wireless communication unit 85, a memory 87, a display unit 88, a storage 89, and a battery 99. The mobile terminal 80 has a function as an example of an information processing device, and the processing unit 81 of the mobile terminal 80 is an example of a processing unit of the information processing device.

  The processing unit 81 is configured using a processor (for example, a CPU, an MPU, or a DSP). The processing unit 81 performs signal processing for integrally controlling the operation of each unit of the mobile terminal 80, data input / output processing with other units, data arithmetic processing, and data storage processing.

  The processing unit 81 may acquire data and information from the unmanned aerial vehicle 100 via the wireless communication unit 85. The processing unit 81 may obtain data and information from the transmitter 50 or another device via the interface unit 82. The processing unit 81 may acquire data or information input via the operation unit 83. The processing unit 81 may acquire data and information stored in the memory 87. The processing unit 81 may send the data and information to the display unit 88 and cause the display unit 88 to display display information based on the data and information. The processing unit 81 may send data and information to the storage 89 and store the data and information. The processing unit 81 may acquire data and information stored in the storage 89.

  The processing unit 81 may execute an application for instructing control of the unmanned aerial vehicle 100. The processing unit 81 may generate various data used in the application.

  The interface unit 82 inputs and outputs information and data between the portable terminal 80 and the transmitter 50 or another device. The interface unit 82 may be, for example, a USB connector (not shown) provided on the mobile terminal 80. The interface unit 65 may be an interface other than the USB connector.

  The operation unit 83 receives data and information input by the operator of the mobile terminal 80. The operation unit 83 may include buttons, keys, a touch panel, a microphone, and the like. Here, it is mainly illustrated that the operation unit 83 and the display unit 88 are configured by a touch panel. In this case, the operation unit 83 can receive a touch operation, a tap operation, a drag operation, and the like.

  The wireless communication unit 85 communicates with the unmanned aerial vehicle 100 by various wireless communication systems. The wireless communication method may include, for example, wireless LAN, Bluetooth (registered trademark), short-range wireless communication, or communication via a public wireless line. The wireless communication unit 85 may communicate with another device to transmit and receive data and information.

  The memory 87 is an example of a storage unit. The memory 87 has, for example, a ROM in which a program that regulates the operation of the portable terminal 80 and data of set values are stored, and a RAM that temporarily stores various information and data used in the processing of the processing unit 81. May be. The memory 87 may include a memory other than the ROM and the RAM. The memory 87 may be provided inside the mobile terminal 80. The memory 87 may be provided detachably from the mobile terminal 80. The program may include an application program.

  The display unit 88 is configured using, for example, an LCD (Liquid Crystal Display) or an organic EL (ElectroLuminescence) display, and displays various information and data output from the processing unit 81. The display unit 88 may display data of a captured image captured by the imaging devices 220 and 230 of the unmanned aerial vehicle 100.

  The storage 89 is an example of a storage unit. The storage 89 accumulates and holds various data and information. The storage 89 may be a flash memory, a solid state drive (SSD), a memory card, a USB memory, or the like. The storage 89 may be provided detachably from the main body of the mobile terminal 80.

  The battery 99 has a function as a drive source of each unit of the mobile terminal 80 and supplies necessary power to each unit of the mobile terminal 80.

  Next, an example of a function of the processing unit 81 of the mobile terminal 80 will be described.

  The processing unit 81 as an example of a processing unit of the information processing device includes a flight path processing unit 811 that performs processing related to generation of a flight path of the unmanned aerial vehicle 100. The processing unit 81 includes a shape data processing unit 812 that performs processing relating to estimation and generation of three-dimensional shape data of a subject.

  The flight path processing unit 811 generates a flight path of the unmanned aerial vehicle 100 that images a subject. The flight path processing unit 811 may acquire an input parameter. The flight path processing unit 811 may acquire the input parameters input by the transmitter 50 by receiving the input parameters via the interface unit 82 or the wireless communication unit 85. Further, the flight path processing unit 811 may acquire at least a part of the information included in the input parameter from another device instead of acquiring from the transmitter 50. The flight path processing unit 811 may acquire at least a part of the information included in the input parameters from a server or the like existing on the network. The acquired input parameters may be stored in the memory 87. The processing unit 81 of the mobile terminal 80 may refer to the memory 87 as appropriate (for example, when generating a flight path and when generating three-dimensional shape data).

  The input parameters may include information on the approximate shape of the object, information on the flight range, information on the flight altitude, information on the imaging distance, and information on the imaging position interval. The input parameter may include information on the set resolution. Note that the set resolution is the resolution of the captured images captured by the imaging devices 220 and 230 of the unmanned aerial vehicle 100 (that is, a resolution for obtaining an appropriate captured image so that the three-dimensional shape of the subject can be estimated with high accuracy). Resolution), and may be stored in the memory 160 of the unmanned aerial vehicle 100 or the memory 67 of the transmitter 50.

  The input parameters include, in addition to the above-described parameters, information on an imaging position (that is, a waypoint) in the flight path of the unmanned aerial vehicle 100 and various parameters for generating a flight path passing through the imaging position. Good. The imaging position is a position in a three-dimensional space.

  In addition, the input parameter may include, for example, information on the overlap ratio of the imaging range when the unmanned aerial vehicle 100 images the subject at the imaging position. Further, the input parameter may include information on the interval between the imaging positions in the flight path. The imaging position interval is an interval (distance) between two adjacent imaging positions among a plurality of imaging positions (waypoints) arranged on the flight path. In addition, the input parameter may include information on the angle of view of the imaging device 220 or 230 of the unmanned aerial vehicle 100.

  Further, the flight path processing unit 811 may receive and acquire the identification information of the subject. The flight path processing unit 811 communicates with an external server via the interface unit 82 or the wireless communication unit 85 based on the identified subject identification information, and outputs information on the shape of the subject corresponding to the subject identification information and the subject. The size information may be received and acquired.

  The overlap ratio of the imaging ranges indicates a rate at which two imaging ranges overlap when the imaging device 220 or the imaging device 230 of the unmanned aerial vehicle 100 captures images at imaging positions adjacent in the horizontal direction or the vertical direction. The overlap ratio of the imaging ranges is at least one of information on the overlap ratio of the imaging ranges in the horizontal direction (also referred to as horizontal overlap ratio) and information on the overlap ratio of the imaging ranges in the vertical direction (also referred to as vertical overlap ratio). May include. The horizontal overlap rate and the vertical overlap rate may be the same or different. When the horizontal overlap ratio and the vertical overlap ratio have different values, both the information of the horizontal overlap ratio and the information of the vertical overlap ratio may be included in the input parameter. When the horizontal overlap rate and the vertical overlap rate have the same value, information of one overlap rate having the same value may be included in the input parameter.

  The image capturing position interval is a spatial image capturing interval, and is a distance between adjacent image capturing positions among a plurality of image capturing positions at which the unmanned aerial vehicle 100 should capture images on the flight path. The imaging position interval may include at least one of a horizontal imaging position interval (also referred to as a horizontal imaging interval) and a vertical imaging position interval (also referred to as a vertical imaging interval). The flight path processing unit 811 may calculate and acquire the imaging position interval including the horizontal imaging interval and the vertical imaging interval, or may acquire from the input parameters.

  That is, the flight path processing unit 811 may arrange an imaging position (waypoint) at which the imaging device 220 or 230 captures an image on the flight path. The intervals between the imaging positions (intervals between the imaging positions) may be arranged at equal intervals, for example. The imaging positions are arranged so that the imaging ranges of the captured images at adjacent imaging positions partially overlap. This is to enable estimation of a three-dimensional shape using a plurality of captured images. Since the imaging device 220 or 230 has a predetermined angle of view, shortening the interval between imaging positions partially overlaps both imaging ranges.

  The flight path processing unit 811 may calculate the imaging position interval based on, for example, the altitude at which the imaging position is arranged (imaging altitude) and the resolution of the imaging device 220 or 230. As the imaging altitude is higher or the imaging distance is longer, the overlap ratio of the imaging range is higher, so that the imaging position interval can be made longer (sparser). The lower the imaging altitude or the shorter the imaging distance, the smaller the overlap ratio of the imaging range. Therefore, the interval between the imaging positions is shortened (dense). The flight path processing unit 811 may further calculate an imaging position interval based on the angle of view of the imaging device 220 or 230. The flight path processing unit 811 may calculate the imaging position interval by another known method.

  The flight path processing unit 811 may acquire information on the angle of view of the imaging device 220 or the angle of view of the imaging device 230 from the imaging device 220 or the imaging device 230. The angle of view of the imaging device 220 or the angle of view of the imaging device 230 may be the same or different between the horizontal direction and the vertical direction. The angle of view of the imaging device 220 or the angle of view of the imaging device 230 in the horizontal direction is also referred to as a horizontal angle of view. The angle of view of the imaging device 220 or the angle of view of the imaging device 230 in the vertical direction is also referred to as the vertical angle of view. When the horizontal angle of view and the vertical angle of view are the same, the flight path processing unit 811 may acquire information of one angle of view having the same value.

  The flight path processing unit 811 determines the imaging position (waypoint) of the subject by the unmanned aerial vehicle 100 based on the flight range and the imaging position interval. The imaging positions of the unmanned aerial vehicle 100 may be arranged at equal intervals in the horizontal direction, and the distance between the last imaging position and the first imaging position may be shorter than the imaging position interval. This interval is the horizontal imaging interval. The imaging positions of the unmanned aerial vehicle 100 may be arranged at equal intervals in the vertical direction, and the distance between the last imaging position and the first imaging position may be shorter than the imaging position interval. This interval is the vertical imaging interval.

  The flight path processing unit 811 generates a flight path passing through the determined imaging position for each imaging plane corresponding to the side surface of the object. The flight path processing unit 811 sequentially passes through the adjacent imaging positions in the flight course of one imaging plane, passes through all the imaging positions in this flight course, and then determines the flight path that enters the flight course of the next imaging plane. May be generated. Similarly, in the flight course of the next imaging plane, the flight path processing unit 811 sequentially passes through the adjacent imaging positions, passes through all the imaging positions in this flight course, and then proceeds to the flight course of the next imaging plane. An approaching flight path may be generated. The flight path may be formed so that the altitude decreases as the flight path starts from the sky side. On the other hand, the flight path may be formed so that the altitude increases as the flight path proceeds starting from the ground side.

  The processing unit 110 of the unmanned aerial vehicle 100 may control the flight of the unmanned aerial vehicle 100 according to the generated flight path. The processing unit 110 may cause the imaging device 220 or the imaging device 230 to image a subject at an imaging position existing in the middle of the flight path. Therefore, the imaging device 220 or the imaging device 230 may image the side surface of the subject at the imaging position in the flight path. The image captured by the imaging device 220 or the imaging device 230 may be stored in the memory 160 or the storage 170 of the unmanned aerial vehicle 100 or the memory 87 or the storage 89 of the mobile terminal 80. The processing unit 110 may refer to the memory 160 as appropriate (for example, when setting a flight path).

  The shape data processing unit 812 performs three-dimensional information (three-dimensional shape) indicating a three-dimensional shape (three-dimensional shape) of an object (subject) based on a plurality of captured images captured at different imaging positions by one of the imaging devices 220 and 230. Information, three-dimensional shape data). Therefore, the captured image may be used as one image for restoring the three-dimensional shape data. The captured image for restoring the three-dimensional shape data may be a still image. A known method may be used as a method for generating three-dimensional shape data based on a plurality of captured images. Known methods include, for example, MVS (Multi View Stereo), PMVS (Patch-based MVS), and SfM (Structure from Motion).

  The captured image used for generating the three-dimensional shape data may be a still image. The plurality of captured images used for generating the three-dimensional shape data include two captured images whose imaging ranges partially overlap each other. When the three-dimensional shape data is generated in the same range, the number of captured images used for generating the three-dimensional shape data increases as the ratio of the overlap (that is, the overlap ratio of the imaging ranges) increases. Therefore, the shape data processing unit 812 can improve the accuracy of restoring the three-dimensional shape. On the other hand, when the three-dimensional shape data is generated in the same range, the number of captured images used for generating the three-dimensional shape data decreases as the overlapping rate of the imaging ranges decreases. Therefore, the shape data processing unit 812 can reduce the time for generating three-dimensional shape data. Note that the plurality of captured images need not include two captured images whose imaging ranges partially overlap each other.

  The shape data processing unit 812 may acquire a plurality of captured images including a captured image of a side surface of a subject. Accordingly, the shape data processing unit 812 can collect a large number of image features on the side surface of the subject and can improve the restoration accuracy of the three-dimensional shape around the subject, as compared with a case where a captured image obtained by uniformly capturing the vertical direction from the sky is acquired. Can be improved.

  Next, an operation example of the flight route generation system 10 will be described.

[Flight path generation]
FIG. 8 is a flowchart illustrating an example of a processing procedure of a flight route generation method according to the embodiment. The illustrated example illustrates a process of performing aerial photography of a target area to acquire a schematic shape of an object, and generating a flight path for three-dimensional shape estimation based on the acquired schematic shape. In the present example, it is assumed that the processing unit 81 of the mobile terminal 80 performs the processing independently.

  The flight path processing unit 811 of the processing unit 81 generates a flight path of the unmanned aerial vehicle 100 for object shooting when performing shooting for estimating the three-dimensional shape of the object. The flight path processing unit 811 inputs the flight range of the unmanned aerial vehicle 100 and specifies the range of the imaging target area (S11).

  The processing unit 110 of the unmanned aerial vehicle 100 inputs the information of the designated flight range, flies in the corresponding flight range, and aerially shoots the object in the shooting target area vertically downward at a predetermined imaging position in the vertical direction. (S12). In this case, the processing unit 110 roughly captures an object at a small number of capturing positions (hereinafter, may be referred to as “rough capturing”). The processing unit 110 of the unmanned aerial vehicle 100 acquires a bird's-eye view captured image at each imaging position, and records the captured image in the memory 160.

  The flight path processing unit 811 of the processing unit 81 acquires the captured image obtained by the general imaging (downward aerial photography) vertically below the imaging target area, and stores it in the memory 87 or the storage 89. The flight path processing unit 811 estimates the approximate shape of an object (building, ground, etc.) by a known three-dimensional shape restoration technique using the acquired captured image group, thereby acquiring the approximate shape (S13). The three-dimensional shape data of the schematic shape may include, for example, polygon data.

  In addition, instead of acquiring the outline shape of the object by aerial photography of the target area, a three-dimensional map database held by another device such as the mobile terminal 80 or a server is used, and the map of the three-dimensional map database is used. The three-dimensional shape data of the approximate shape may be obtained based on the three-dimensional information (for example, polygon data) of the building, the road, and the like included in the information.

  The flight path processing unit 811 uses the acquired general shape of the object to generate a detailed imaging flight path for estimating the three-dimensional shape of the object (S14). Some examples of a procedure for generating a flight path using the schematic shape of an object will be described later.

  According to the above operation example, a flight path for estimating the three-dimensional shape of the object can be generated, and detailed imaging of the object can be automated. In addition, the setting of an appropriate flight path for the object can be automated.

[Schematic shape acquisition]
Next, an example of a method of obtaining a schematic shape of an object in the flight path generation processing will be described.

  FIG. 9 is a diagram for describing an input example of the flight range A1.

  For example, the processing unit 81 of the mobile terminal 80 inputs information on the flight range A1 using the operation unit 83. The operation unit 83 may receive a user input of a desired range in which generation of the three-dimensional shape data indicated in the map information M1 is desired as the flight range A1. The information on the flight range A1 is not limited to a desired range, and may be a predetermined flight range. The predetermined flight range may be, for example, one of ranges for periodically generating three-dimensional shape data and measuring the three-dimensional shape.

  FIG. 10 is a diagram for explaining schematic imaging in the flight path FPA.

  In the processing unit 81, the flight path processing unit 811 may set the interval (imaging position interval) between the imaging positions CP to the interval d11 in the flight path FPA. The interval d11 is a sparse interval (eg, several tens of meters apart) at which the size of an object (eg, a building) can be estimated. The interval d11 is set to at least an interval in which the imaging ranges at the adjacent imaging positions CP partially overlap. Imaging at each imaging position CP at the interval d11 of the flight path FPA may be referred to as general imaging. The unmanned aerial vehicle 100 can reduce the imaging time by imaging at sparse intervals compared to imaging at dense intervals. In the vertical direction (the direction toward the ground, the direction of gravity) of the flight path on which the unmanned aerial vehicle 100 flies, the scenery including the building BL and the mountain MT may spread. Therefore, the building BL and the mountain MT exist in the imaging range and are the imaging targets. The schematic shape of the object can be obtained from the captured image obtained by the general imaging.

  FIG. 11 is a diagram for describing generation of three-dimensional shape data of a schematic shape based on the schematic imaging obtained by the flight path FPA.

  In the processing unit 81, the shape data processing unit 812 generates the three-dimensional shape data SD1 of the schematic shape of the object based on the plurality of captured images CI1 obtained at the respective imaging positions CP by the schematic imaging of the flight path FPA. By confirming the three-dimensional shape data SD1 by display or the like, the user can grasp the general shape of the ground existing in the vertical direction of the flight path FPA. The user can confirm the presence of the mountain MT by checking the shape (schematic shape) obtained from the three-dimensional shape data SD1 based on the schematic imaging, but cannot confirm the existence of the building BL. This is because the mountain MT has a gentle contour, and even if the mountain MT is imaged from the sky following the flight path FPA, an image necessary for generating the three-dimensional shape data SD1 is sufficient in the captured image CI1. This also means that the side of the building BL is sufficiently imaged at the imaging position CP of the flight path FPA in which the outline of the building BL is substantially parallel to the vertical direction and the unmanned aerial vehicle 100 travels over the building BL in the horizontal direction. This is because it is difficult to do so. In other words, information necessary for three-dimensional shape estimation cannot be acquired from the captured image taken downward in the vicinity of the building BL.

  Therefore, the flight path processing unit 811 uses the data of the approximate shape of the object to move the side of the object toward the direction parallel to the vertical direction of the object, that is, the horizontal direction (the normal direction of the vertical direction). A flight path and an imaging position are generated and set so that an image is taken from the user. The shape data processing unit 812 generates three-dimensional shape data of the object using a captured image including a captured image on the side of the object captured according to the generated flight path. Thereby, the estimation accuracy of the three-dimensional shape of the object can be improved.

[Three-dimensional shape estimation]
FIG. 12 is a flowchart illustrating an example of a processing procedure of the three-dimensional shape estimation method according to the embodiment. In this example, it is assumed that the processing unit 81 of the mobile terminal 80 as an example of the processing unit of the information processing apparatus mainly performs the processing.

  The flight path processing unit 811 of the processing unit 81 sets a flight path for the unmanned aerial vehicle 100 using the generated flight path (S21). The processing unit 110 of the unmanned aerial vehicle 100 flies in the flight range of the imaging target area according to the set flight path, and aerially photographs the object sideways at a predetermined imaging position (S22). In this case, the processing unit 110 captures the object in detail by partially overlapping the imaging range at every predetermined imaging position interval (hereinafter, may be referred to as “detailed imaging”). The processing unit 110 of the unmanned aerial vehicle 100 acquires a captured image at each imaging position, and records the captured image in the memory 160.

  The shape data processing unit 812 of the processing unit 81 acquires a captured image obtained by detailed imaging (lateral aerial photography) of the imaging target area and stores the acquired image in the memory 87 or the storage 89. The shape data processing unit 812 generates three-dimensional shape data by estimating the three-dimensional shape of an object (building, ground, etc.) from the acquired captured image group by a known three-dimensional shape restoration technique (S23).

  Accordingly, three-dimensional shape data including the shape of the side surface of the object can be generated using the detailed captured image of the object captured from the side. Therefore, it is possible to estimate the detailed shape of the side surface, which has been difficult to restore in the captured image obtained by capturing the object downward, and to improve the accuracy of the three-dimensional shape data of the object.

[First operation example of flight path generation]
FIG. 13 is a diagram illustrating a first operation example of generating a flight path using the schematic shape of an object according to the embodiment. The first operation example is an example in which a polyhedron, such as a cube, surrounding an object is calculated to generate an imaging plane directed to the side of the object.

  The flight path processing unit 811 of the processing unit 81 calculates a polyhedron surrounding the outer shape of the object by using the acquired schematic shape. The polyhedron is a solid that touches the outside or is slightly larger than the general shape of the object. The illustrated example shows an example in which a cube 301 is calculated as an example of a polyhedron. The flight path processing unit 811 extracts at least one side surface 303 from the polyhedron of the cube 301. The side surface 303 may be a surface along the vertical direction of the polyhedron, or a surface standing within a predetermined angle range in the vertical direction. The flight path processing unit 811 calculates a normal line 304 outward of the polyhedron with respect to the extracted side surface 303. The normal line 304 can be calculated by the cross product of two vectors (for example, a vector connecting any one of the vertices) along the surface of the side surface 303. The flight path processing unit 811 calculates an imaging plane 305 having a predetermined imaging distance and being parallel to the side surface 303 using the acquired normal line 304. The photographing plane 305 is located at a predetermined photographing distance from the side surface 303 and is a plane perpendicular to the normal 304. The flight path processing unit 811 sets a plurality of imaging positions (waypoints) 306 having a predetermined imaging position interval in the generated imaging plane 305 within this plane, and determines an imaging path 307 passing through each imaging position 306. As a result, a flight path including the photographing path 307 is generated. The imaging direction at each imaging position 306 is opposite to the direction of the normal line 304 and faces the side surface of the object. When the polyhedron surrounding the object is a cube, a rectangular parallelepiped, or a column, the imaging plane is a vertical plane, and the imaging direction is a horizontal direction perpendicular to the imaging plane.

  FIG. 14 is a diagram for describing the setting of a plurality of imaging positions 306 on the imaging plane 305. The flight path processing unit 811 sets a predetermined shooting distance L in the normal direction with respect to the side surface 303 of the polyhedron, and calculates a shooting plane 305 parallel to the side surface 303 at a position separated from the side surface 303 by the shooting distance L. The flight path processing unit 811 sets a predetermined imaging position interval d on the imaging plane 305, and determines the imaging position 306 for each imaging position interval d. The setting of the photographing distance L and the photographing position interval d may be performed by the following method, for example.

  [1] The user specifies a shooting distance L [m] and an imaging position interval d [m]. The processing unit 81 of the mobile terminal 80 inputs information of the shooting distance L and the imaging position interval d by the operation unit 83 according to the operation input by the user, and stores the information in the memory 87. Thereby, the imaging position for detailed imaging can be set based on the imaging distance and the imaging position interval specified by the user.

[2] The user specifies the shooting distance L [m] and the overlap ratio r _side [%] of the imaging range, and calculates the imaging position interval d [m] from the shooting distance L and the overlap ratio r _side . The imaging position interval d can be calculated by the following equation (1) using the imaging distance L, the overlap ratio r _side , and the angle of view FOV (Field of View) of the imaging device.

The processing unit 81 of the mobile terminal 80 inputs information of the shooting distance L and the overlap ratio r _side by the operation unit 83 according to the operation input by the user, and stores the information in the memory 87. The processing unit 81 acquires information on the angle of view FOV of the imaging device 220 from the unmanned aerial vehicle 100 by the interface unit 82 or the wireless communication unit 85, and stores the information in the memory 87. The processing unit 81 calculates the imaging position interval d using the above equation (1). Thus, the imaging position interval can be calculated based on the overlapping ratio of the imaging distance and the imaging range specified by the user, and the imaging position for detailed imaging can be set.

[3] The user specifies the resolution r [m / pixel] of the captured image and the overlap ratio r _side [%] of the imaging range, and calculates the imaging position interval d [m] from the resolution r and the overlap ratio r _side . Further, the photographing distance L [m] is calculated from the photographing position interval d. The imaging position interval d can be calculated by the following equation (2) using the resolution r, the width w of the captured image, and the overlap ratio r _side .

The shooting distance L can be calculated by the following equation (3) using the imaging position interval d, the overlap ratio r _side , and the angle of view FOV of the imaging device.

The processing unit 81 of the mobile terminal 80 inputs the information of the resolution r and the overlap ratio r _side by the operation unit 83 according to the operation input by the user, and stores the information in the memory 87. The processing unit 81 calculates the imaging position interval d using the above equation (2). The processing unit 81 acquires information on the angle of view FOV of the imaging device 220 from the unmanned aerial vehicle 100 by the interface unit 82 or the wireless communication unit 85, and stores the information in the memory 87. The processing unit 81 calculates the shooting distance L by the above equation (3). Thus, the imaging position interval can be calculated based on the resolution of the captured image specified by the user and the overlap ratio of the imaging range, and the imaging position for detailed imaging can be set.

  The flight path processing unit 811 arranges a plurality of imaging positions 306 at equal intervals on the imaging plane 305 based on the set imaging position intervals d and sets the imaging path 307 passing through these imaging positions 306. decide. The imaging position at the end of the imaging plane 305, such as the imaging position of the start point and the end point on the imaging plane 305, may be set to a range of 1 / 2d or less from the side edge of the imaging plane 305.

  FIG. 15 is a flowchart illustrating a processing procedure of a first operation example of generating a flight path using the schematic shape of an object according to the embodiment. The flight path processing unit 811 of the processing unit 81 calculates a polyhedron (cube 301) surrounding the outer shape of the object using the acquired schematic shape (S31). The flight path processing unit 811 sequentially extracts at least one (four in the cube) side surfaces 303 in the polyhedron of the cube 301 (S32). The flight path processing unit 811 calculates a normal line 304 outward of the polyhedron with respect to the extracted one side surface 303 (S33). Using the acquired normal line 304, the flight path processing unit 811 calculates an imaging plane 305 parallel to the side surface 303 at a position separated by a predetermined imaging distance L (S34).

  The flight path processing unit 811 sets a plurality of imaging positions (waypoints) 306 having a predetermined imaging position interval d on one calculated imaging plane 305, and performs imaging in a direction from each of these imaging positions toward the object. An imaging route 307 is generated (S35). The flight path processing unit 811 determines whether the generation of the imaging paths 307 of all the extracted side surfaces 303 has been completed for the object (S36). If the generation of the imaging paths for all the side surfaces 303 has not been completed, the flight path processing unit 811 returns to the processing in step S32, extracts the next side surface 303, and repeats the same processing until the generation of the imaging paths 307 (S32). ~ S35). In step S36, when the generation of the imaging paths of all the side surfaces 303 is completed, the flight path processing unit 811 combines the imaging paths 307 of the respective imaging planes 305 to generate a flight path (S37).

  The processing unit 110 of the unmanned aerial vehicle 100 communicates with the mobile terminal 80 via the communication interface 150, acquires information on the flight route generated by the flight route processing unit 811 and sets the flight route of the unmanned aerial vehicle 100. The processing unit 110 flies around the object according to the set flight path, and images the object with the imaging devices 220 and 230 at each of a plurality of imaging positions (waypoints). The processing unit 110 sequentially captures images at the respective imaging positions 306 for each of the imaging planes 305 by a flight path obtained by combining the imaging paths 307 of the imaging planes 305. That is, when the imaging of the object included in the subject is completed at each imaging position 306 on the imaging plane 305 corresponding to one side surface 303 of the roughly-shaped polyhedron (cube 301), the processing unit 110 proceeds to the next side surface. Is moved to an imaging plane corresponding to the image plane, for example, an imaging plane on a side surface adjacent to the current side surface, and imaging is performed at each imaging position on this imaging plane. As described above, the unmanned aerial vehicle 100 acquires the captured image of the side captured toward the side surface of the object at the imaging positions of all the imaging planes set in the flight path.

  The processing unit 81 of the mobile terminal 80 communicates with the unmanned aerial vehicle 100 via the interface unit 82 or the wireless communication unit 85, and acquires a captured image captured by the unmanned aerial vehicle 100. The shape data processing unit 812 of the processing unit 81 generates three-dimensional shape data of the object (building, ground, etc.) using the acquired captured image of the side of the object, thereby obtaining details including the shape of the side surface of the object. A three-dimensional shape can be estimated.

  In addition, the captured image may include a lower captured image obtained by performing detailed imaging of the object vertically downward at an imaging position interval for detailed imaging together with the laterally captured image. In this case, the flight path processing unit 811 sets an imaging path including a plurality of imaging positions on the upper surface of the object, similarly to the side surface, and generates a flight path.

  By the above operation example, a polyhedron surrounding the schematic shape of the object is calculated, and a surface along the vertical direction in the polyhedron, or a surface standing within a predetermined angle range in the vertical direction is extracted as a side surface, so that the surface is directed toward the side of the object. Thus, the side surface of the approximate shape that can be imaged can be extracted. For this reason, it is possible to set an imaging position at which detailed imaging can be performed when the object is viewed from the side, using the schematic shape of the object.

[Second operation example of flight path generation]
FIG. 16 is a diagram illustrating a second operation example of generating a flight path using the schematic shape of an object according to the embodiment. The second operation example is an example in which a mesh indicating a schematic shape of an object is simplified to generate an imaging plane facing the side of the object.

  The flight path processing unit 811 of the processing unit 81 uses the acquired general shape to simplify the mesh indicating the general shape of the object. As a method for simplifying the mesh, a known method may be used. Known methods include, for example, a vertex clustering method and an incremental decimation method. In the mesh simplification process, smoothing is performed to simplify polygonal data to simplify a complicated shape and reduce the number of polygons representing one surface. In the illustrated example, the flight path processing unit 811 performs a simplification process on the schematic shape 311 to calculate a simplified polyhedron 312. The flight path processing unit 811 extracts at least one side surface 313 from the simplified polyhedron 312 and calculates a normal 314 to the extracted side surface 313 outward of the polyhedron. In this case, the direction of the normal to each plane of the polyhedron 312 is determined, and if the absolute value of the normalized normal component Nz is smaller than 0.1 (| Nz | <0.1), the plane is set to the side surface. 313 is extracted. The side surface 313 may be a surface along a vertical direction in the polyhedron, or a surface standing within a predetermined angle range in the vertical direction. The flight path processing unit 811 calculates an imaging plane 315 having a predetermined imaging distance L and parallel to the side surface 313 using the acquired normal 314. The flight path processing unit 811 sets a plurality of imaging positions (waypoints) 316 having a predetermined imaging position interval d on the calculated imaging plane 315, determines an imaging path 317 passing through each imaging position 316, and performs this imaging. A flight path including the path 317 is generated. In this case, the photographing plane is a steep plane within a predetermined range with respect to the vertical direction, and the photographing direction is a direction that is substantially horizontal toward the side, that is, a direction that faces the side surface of the object.

  FIG. 17 is a flowchart illustrating a processing procedure of a second operation example of the flight path generation using the schematic shape of the object according to the embodiment. The flight path processing unit 811 of the processing unit 81 uses the acquired schematic shape to simplify the mesh of the schematic shape of the object, and calculates a polyhedron 312 in which the schematic shape 311 is simplified (S41). The flight path processing unit 811 extracts at least one side surface 313 from the polyhedron 312 (S42). The flight path processing unit 811 calculates a normal 314 outward of the polyhedron with respect to the extracted one side surface 313 (S43). Using the calculated normal 314, the flight path processing unit 811 calculates an imaging plane 315 parallel to the side surface 313 at a position separated by a predetermined imaging distance L (S44).

  The flight path processing unit 811 sets a plurality of imaging positions (waypoints) 316 having a predetermined imaging position interval d in one generated imaging plane 315, and performs imaging in a direction from each of these imaging positions toward the object. An imaging route 317 is generated (S45). The flight path processing unit 811 determines whether the generation of the imaging paths 317 of all the extracted side surfaces 313 has been completed for the object (S46). If the generation of the imaging paths for all the side surfaces 313 has not been completed, the flight path processing unit 811 returns to the processing in step S42, extracts the next side surface 313 adjacent to the current side surface, and generates the imaging path 317. The same processing is repeated until (S42 to S45). In step S46, when the generation of the photographing paths of all the side surfaces 313 is completed, the flight path processing unit 811 combines the photographing paths 317 of the respective photographing planes 315 to generate a flight path (S47).

  By the above operation example, a polyhedron in which the schematic shape of the object is simplified is calculated, and a surface along the vertical direction in the polyhedron, or a surface standing within a predetermined angle range in the vertical direction is extracted as a side surface, so that the side of the object is extracted. , A side surface of a schematic shape that can be imaged can be extracted. For this reason, it is possible to set an imaging position at which detailed imaging can be performed when the object is viewed from the side, using the schematic shape of the object.

[Third operation example of flight path generation]
FIG. 18 is a diagram illustrating a third operation example of generating a flight path using the schematic shape of an object according to the embodiment. The third operation example is an example in which a plurality of polyhedrons such as cubes surrounding an object are combined to generate an imaging plane directed to the side of the object.

  The flight path processing unit 811 of the processing unit 81 calculates a plurality of polyhedrons surrounding the outline shape of each object, for a plurality of objects such as buildings, using the acquired outline shape. In the illustrated example, an example is shown in which cubic or rectangular parallelepiped polyhedrons 321A, 321B, and 321C are calculated as an example of a plurality of polyhedrons. The flight path processing unit 811 combines the plurality of polyhedrons 321A, 321B, and 321C, and calculates the combined polyhedron 322. By coupling the adjacent polyhedrons, collision of the unmanned aerial vehicle 100 with the object at the time of side detailed imaging is avoided. The flight path processing unit 811 extracts at least one side surface 323 of the combined polyhedron 322. The side surface 323 may be a surface along the vertical direction of the polyhedron, or a surface standing within a predetermined angle range in the vertical direction. The flight path processing unit 811 calculates the normal 324 outward of the polyhedron with respect to the extracted side surface 323. Using the calculated normal 324, the flight path processing unit 811 calculates an imaging plane 325 having a predetermined imaging distance L and being parallel to the side surface 323. The flight path processing unit 811 sets a plurality of imaging positions (waypoints) 326 having a predetermined imaging position interval d within the calculated imaging plane 325, determines an imaging path 327 passing through each imaging position 326, A flight path including the photographing path 327 is generated. The imaging direction at each imaging position 326 is opposite to the direction of the normal 324 and faces the side surface of the object. When the polyhedron surrounding the object is a cube, a rectangular parallelepiped, or a column, the imaging plane is a vertical plane, and the imaging direction is a horizontal direction perpendicular to the imaging plane.

  FIG. 19 is a flowchart illustrating a processing procedure of a third operation example of generating a flight path using the schematic shape of an object according to the embodiment. The flight path processing unit 811 of the processing unit 81 calculates a plurality of polyhedrons 321A, 321B, and 321C surrounding the outer shape of each of the plurality of objects by using the acquired schematic shape (S51). The flight path processing unit 811 combines the polyhedrons 321A, 321B, and 321C and calculates the combined polyhedron 322 (S52). The flight path processing unit 811 sequentially extracts at least one side surface 323 from the combined polyhedron 322 (S32). The flight path processing unit 811 extracts the side surface 323 of the polyhedron (S32), calculates the normal 324 (S33), calculates the imaging plane 325 (S34), determines the imaging position 326, and generates the imaging path 327 (S35). Each process of the determination (S36) of the completion of generation of all shooting paths is the same as that of the first operation example shown in FIG. 15, and the description is omitted here.

  In step S36, when the generation of the imaging routes of all the side surfaces 323 is not completed, the flight path processing unit 811 returns to the process of step S32, extracts the next side surface 323 adjacent to the current side surface, and performs imaging. The same processing is repeated until the path 327 is generated (S32 to S35). When the generation of the imaging paths of all the side surfaces 323 is completed, the flight path processing unit 811 combines the imaging paths 327 of the respective imaging planes 325 in the combined polyhedron 322 to generate a flight path (S57).

  The above-described second operation example and third operation example are combined to simplify the schematic shape of the object and combine a plurality of polyhedrons, to extract the side surface, set the imaging plane and the imaging position, and set the imaging path. By making the determination, a flight path may be generated.

  According to the above operation example, a polyhedron corresponding to a plurality of general shapes of the object is calculated, and a plurality of adjacent polyhedrons are combined, and a surface along the vertical direction of the combined polyhedron or a vertical angle range is set. By extracting the cut surface as a side surface, it is possible to extract a roughly shaped side surface that can be imaged facing the side of the object. For this reason, it is possible to set an imaging position at which detailed imaging can be performed when the object is viewed from the side, using the schematic shape of the object. By combining a plurality of polyhedrons and extracting a side surface, it is possible to prevent the flying object from colliding with the object during detailed imaging of the side when the object exists in close proximity.

  In the above configuration example, the mobile terminal 80 functioning as an information processing device sets an imaging position at which detailed imaging can be performed when the object is viewed from the side using the schematic shape of the object, and sets a flight path passing through the imaging position. it can.

  According to the present embodiment, it is possible to set an imaging position at which a detailed imaging can be performed when the object is viewed from the side, using the schematic shape of the object. By extracting the side surface of the schematic shape, it is possible to set the imaging position for detailed side imaging corresponding to the side surface of the object. By generating a flight path that passes through the set imaging position, a flight path for detailed imaging including the side of the object can be set. By setting an imaging position facing the side surface for each extracted side surface, detailed imaging in the horizontal direction when the object is viewed from the side can be performed.

  According to the present embodiment, by setting a plurality of imaging positions having a predetermined imaging position interval corresponding to the extracted side surface, a captured image having an appropriate resolution and an appropriate overlap rate can be obtained. By determining an imaging route that passes through a plurality of set imaging positions and generating a flight route including the imaging route, a flight route for detailed imaging including the side of the object can be set. By generating a shooting plane parallel to the extracted side surface at a predetermined shooting distance, an imaging position facing the side surface of the object can be easily determined. By calculating the normal to the side surface, a shooting plane parallel to the side surface at a predetermined shooting distance can be easily generated. By using, as the predetermined image capturing position interval, an image capturing position interval at which a part of the image captured at each image capturing position overlaps with another, an image capturing position at which a captured image having an appropriate overlap rate can be obtained can be set. By generating a flight path passing through the imaging position on one side and generating a flight path passing through the imaging position on the next side adjacent to this side, corresponding to a plurality of sides of the object, for each side You can fly in order and take images efficiently. By acquiring a captured image of the object facing downward and acquiring three-dimensional shape data of the schematic shape of the object using the captured image, it is possible to acquire a schematic shape of a desired object.

[Flying path generation system, second configuration example]
FIG. 20 is a schematic diagram illustrating a second configuration example of the flight path generation system 10A according to the embodiment. The flight path generation system 10A includes an unmanned aerial vehicle 100 and a PC (Personal Computer) 70. The unmanned aerial vehicle 100 and the PC 70 can communicate with each other using wired communication or wireless communication (for example, wireless LAN or Bluetooth (registered trademark)). The PC 70 may be a computer such as a desktop PC, a notebook PC, and a tablet terminal. The PC 70 may be a computer including a server and a client terminal connected by a network. The PC 70 is an example of an information processing device.

  The PC 70 may include a processor (for example, a CPU, an MPU or a DSP) as an example of a processing unit, a memory as an example of a storage unit, a communication interface, a display, an input device, and a storage. The PC 70 as an example of the information processing apparatus has the same functions as the processing unit 81, the flight path processing unit 811, and the shape data processing unit 812 included in the mobile terminal 80 illustrated in FIG.

  In the above configuration example, the PC 70 functioning as an information processing device can set an imaging position at which detailed imaging of the object can be viewed from the side using the schematic shape of the object, and can set a flight path passing through the imaging position.

[Flying path generation system, third configuration example]
FIG. 21 is a block diagram illustrating an example of a hardware configuration of an unmanned aerial vehicle 100A according to a third configuration example of the flight path generation system 10B in the embodiment. The unmanned aerial vehicle 100A of the flight path generation system 10B includes a processing unit 110A instead of the processing unit 110 as compared with the unmanned aerial vehicle 100 shown in FIG. The unmanned aerial vehicle 100A has a function as an example of an information processing device, and the processing unit 110A of the unmanned aerial vehicle 100A is an example of a processing unit of the information processing device. In the unmanned aerial vehicle 100A of FIG. 21, the same components as those of the unmanned aerial vehicle 100 of FIG. 4 are denoted by the same reference numerals, and description thereof will be omitted or simplified.

  The processing unit 110A as an example of the processing unit of the information processing device includes a flight path processing unit 111 and a shape data processing unit 112. The flight path processing unit 111 has the same function as the flight path processing unit 811 included in the mobile terminal 80 illustrated in FIG. The shape data processing unit 112 has the same function as the shape data processing unit 812 included in the mobile terminal 80 illustrated in FIG.

  In the above configuration example, the processing unit 110A of the unmanned aerial vehicle 100A functioning as an information processing device sets an imaging position at which detailed imaging of the object viewed from the side is set using the schematic shape of the object, and sets the imaging position. You can set the flight route that passes.

[Flight path generation system, fourth configuration example]
FIG. 22 is a block diagram illustrating an example of a hardware configuration of a transmitter 50A according to a fourth configuration example of the flight path generation system 10C in the embodiment. The transmitter 50A includes a processing unit 61A instead of the processing unit 61 as compared with the transmitter 50 shown in FIG. The transmitter 50A has a function as an example of an information processing device, and the processing unit 61A of the transmitter 50A is an example of a processing unit of the information processing device. In the transmitter 50A of FIG. 22, the same components as those of the transmitter 50 of FIG. 6 are denoted by the same reference numerals, and description thereof will be omitted or simplified.

  The processing unit 61A as an example of the processing unit of the information processing device includes a flight path processing unit 611 and a shape data processing unit 612. The flight path processing unit 611 has the same function as the flight path processing unit 811 included in the mobile terminal 80 illustrated in FIG. The shape data processing unit 612 has the same function as the shape data processing unit 812 included in the mobile terminal 80 illustrated in FIG.

  According to the above configuration example, the processing unit 61A of the transmitter 50A functioning as an information processing device sets an imaging position at which detailed imaging of an object can be viewed from the side using the schematic shape of the object, and passes through the imaging position. You can set the flight path.

  In the above embodiment, the generated flight path is set to the flying object, and the flying object flies over the imaging target area in accordance with the flight path, and performs imaging including side detailed imaging of the object, and acquires the acquired imaging. The image may be used for generating three-dimensional shape data of the object existing in the shooting target area. The captured image obtained by the side detailed imaging may be used for inspection of the side surface of the object.

  In the above embodiment, the example in which the information processing apparatus that executes the steps in the flight route generation method is provided in any of the portable terminal 80, the unmanned aerial vehicle 100A, and the transmitter 50A has been described. The apparatus may include a device and perform steps in a flight path generation method.

  As described above, the present disclosure has been described using the embodiments, but the technical scope of the invention according to the present disclosure is not limited to the scope described in the above-described embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the embodiments described above. It is apparent from the description of the claims that embodiments with such changes or improvements can be included in the technical scope of the present invention.

  The execution order of each processing such as operation, procedure, step, and step in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before”, “before”. And the like, and can be realized in any order as long as the output of the previous process is not used in the subsequent process. Even if the operation flow in the claims, the specification, and the drawings is described using “first”, “next”, and the like for convenience, it means that it is essential to implement in this order is not.

10, 10B, 10C Flight route generation system 50, 50A Transmitter 61, 61A Processing unit 63 Wireless communication unit 65 Interface unit 67 Memory 70 PC
80 Mobile terminal 81 Processing unit 82 Interface unit 83 Operation unit 85 Wireless communication unit 87 Memory 88 Display unit 89 Storage 100, 100A Unmanned aerial vehicle (UAV)
110, 110A processing unit 111 flight path processing unit 112 shape data processing unit 150 communication interface 160 memory 170 storage 200 gimbal 210 rotating wing mechanism 220, 230 imaging device 301 cubes 303, 313, 323 side surfaces 304, 314, 324 normal 305, 315, 325 Image plane 306, 316, 326 Image position (waypoint)
307, 317, 327 Imaging path 311 Schematic shape 312, 321A, 321B, 321C Polyhedron 322 Combined polyhedron 611 Flight path processing unit 612 Shape data processing unit 811 Flight path processing unit 812 Shape data processing unit OPS Operation unit set

Claims (25)

A flight path generation method for generating a flight path of a flying object that images a subject,
Obtaining a schematic shape of an object included in the subject;
Extracting a side surface in the general shape;
Setting an imaging position corresponding to the side surface;
Generating a flight path passing through the imaging position.
Flight path generation method. The step of setting the imaging position,
Setting a shooting position facing the side surface for each of the extracted side surfaces,
The flight path generation method according to claim 1. The step of setting the imaging position,
In accordance with the side surface, including a step of setting a plurality of imaging positions having a predetermined imaging position interval,
The flight path generation method according to claim 1. Generating the flight path includes:
Determining a shooting path passing through the plurality of imaging positions, and generating a flight path including the shooting path;
The flight path generation method according to claim 3. Generating a shooting plane parallel to the side surface at a predetermined shooting distance, further comprising:
The step of setting the imaging position,
Setting a plurality of imaging positions having a predetermined imaging position interval in the imaging plane,
The flight path generation method according to claim 1. The step of setting the imaging position,
As the predetermined imaging position interval, an imaging position interval in which a part of the captured image captured at each imaging position overlaps with another,
The flight path generation method according to claim 3. Calculating a polyhedron surrounding the general shape of the object,
Extracting the side surface,
A step of extracting a surface along the vertical direction in the polyhedron, or a surface standing within a predetermined angle range in the vertical direction as a side surface,
The flight path generation method according to claim 1. Calculating a polyhedron in which the schematic shape of the object is simplified,
Extracting the side surface,
A step of extracting a surface along the vertical direction in the polyhedron, or a surface standing within a predetermined angle range in the vertical direction as a side surface,
The flight path generation method according to claim 1. Calculating the polyhedron,
Calculating a polyhedron corresponding to a plurality of schematic shapes of the object, and combining a plurality of adjacent polyhedrons,
The flight path generation method according to claim 7. Generating the flight path includes:
Generating a flight path passing the imaging position on one of the side surfaces, and generating a flight path passing the imaging position on the next side surface adjacent to the side surface,
The flight path generation method according to claim 1. Obtaining a captured image obtained by capturing the object downward, further comprising:
The step of obtaining the general shape,
Obtaining a three-dimensional shape data of a schematic shape of the object using the captured image,
The flight path generation method according to claim 1. An information processing apparatus that generates a flight path of a flying object that images a subject,
A processing unit that performs processing related to the flight path,
The processing unit includes:
Obtain a schematic shape of an object included in the subject,
Extract the side surface in the general shape,
Setting an imaging position corresponding to the side,
Generating a flight path passing through the imaging position;
Information processing device. The processing unit includes:
In setting the imaging position,
For each of the extracted side surfaces, set an imaging position facing the side surface,
The information processing apparatus according to claim 12. The processing unit includes:
In setting the imaging position,
Corresponding to the side surface, setting a plurality of imaging positions having a predetermined imaging position interval,
An information processing apparatus according to claim 12. The processing unit includes:
In generating the flight path,
Determine a shooting path passing through the plurality of imaging positions, and generate a flight path including the shooting path,
The information processing device according to claim 14. The processing unit further includes:
Generate a shooting plane parallel to the side surface with a predetermined shooting distance,
In setting the imaging position,
Setting a plurality of imaging positions having a predetermined imaging position interval on the imaging plane,
The information processing apparatus according to claim 12. The processing unit includes:
In setting the imaging position,
As the predetermined imaging position interval, an imaging position interval in which a part of the captured image captured at each imaging position overlaps with another,
The information processing device according to claim 14. The processing unit further includes:
Calculating a polyhedron surrounding the general shape of the object,
In the extraction of the side surface,
A surface along the vertical direction in the polyhedron, or a surface standing within a predetermined vertical angle range is extracted as a side surface,
The information processing apparatus according to claim 12. The processing unit further includes:
Calculating a polyhedron that simplifies the schematic shape of the object,
In the extraction of the side surface,
A surface along the vertical direction in the polyhedron, or a surface standing within a predetermined vertical angle range is extracted as a side surface,
The information processing apparatus according to claim 12. The processing unit includes:
In calculating the polyhedron,
Calculating respective polyhedrons corresponding to a plurality of schematic shapes of the object, and combining a plurality of adjacent polyhedrons;
The information processing device according to claim 18. The processing unit includes:
In generating the flight path,
Generate a flight path passing the imaging position on one of the side surfaces, and generate a flight path passing the imaging position on the next side surface adjacent to the side surface,
The information processing apparatus according to claim 12. The processing unit further includes:
Obtain a captured image of the object taken downward,
In obtaining the general shape,
Acquiring three-dimensional shape data of a schematic shape of the object using the captured image;
The information processing apparatus according to claim 12. A flying object that captures an image of a subject, and a processing unit that generates a flight route of the flying object,
The processing unit includes:
Obtain a schematic shape of an object included in the subject,
Extract the side surface in the general shape,
Setting an imaging position corresponding to the side,
Generating a flight path passing through the imaging position,
The flying object is
Obtaining and setting the flight path,
Flight path generation system. A computer that generates the flight path of the flying object that images the subject,
Obtaining a schematic shape of an object included in the subject;
Extracting a side surface in the general shape;
Setting an imaging position corresponding to the side surface;
Generating a flight path passing through the imaging position; and
program. A computer that generates the flight path of the flying object that images the subject,
Obtaining a schematic shape of an object included in the subject;
Extracting a side surface in the general shape;
Setting an imaging position corresponding to the side surface;
Generating a flight path passing through the imaging position, and a computer-readable recording program for executing
recoding media.

Images