JPWO2018167893A1 - Shape generation method, image acquisition method, mobile platform, flying object, program, and recording medium - Google Patents

Abstract

Information about a plurality of imaging positions of a flying object having a plurality of imaging devices is obtained, and an imaging device to be used for imaging for each of the plurality of imaging positions is selected from among the plurality of imaging devices, and selected at each imaging position. An image is taken by an imaging device, the shape of the subject is restored for each imaging device based on the captured image, and these shapes are combined. When selecting an imaging device, at least one imaging device is selected from the plurality of imaging devices based on a ratio of a portion having a predetermined light amount or less in an imaging region at the imaging position. Thus, the shape of a subject having a sharp difference in brightness can be obtained with high accuracy without stopping the flying object.

Description

  The present disclosure relates to a shape generation method for generating a shape of a subject based on an image captured by a flying object, an image acquisition method, a mobile platform, a flying object, a program, and a recording medium.

  2. Description of the Related Art A platform (e.g., an unmanned aerial vehicle) that mounts an imaging device such as a camera and performs imaging while flying on a fixed route set in advance is known (for example, see Patent Document 1). The platform receives commands such as a flight path and a shooting instruction from the ground base, flies according to the instructions, performs shooting, and sends the acquired image to the ground base. When capturing an imaging target, the platform inclines and captures an imaging device of the platform based on the positional relationship between the platform and the imaging target while flying along a set fixed path.

  Conventionally, it is also known to form a ground shape in a certain imaging range based on an image such as an aerial photograph taken by an unmanned aerial vehicle (for example, UAV: Unmanned Aerial Vehicle) flying in the air. In order to automate photographing (for example, aerial photography) using an unmanned aerial vehicle, a technique for generating a flight path of the unmanned aerial vehicle in advance is used. In order to form a three-dimensional shape on the ground with a fixed imaging range using an unmanned aerial vehicle, the unmanned aerial vehicle was flown according to a previously generated flight path, and the unmanned aerial vehicle was photographed at different imaging positions in the flight path It is necessary to acquire a plurality of captured images.

Japanese Patent Application Laid-Open No. 2010-61216

  When forming a three-dimensional shape on the ground, after capturing images at different imaging positions with an unmanned aerial vehicle, the ground shape in the imaging range is restored using techniques such as SFM (Structure from Motion) based on these captured images Is common. In SFM, feature points must be detected from a captured image. For this purpose, it is preferable that the captured image has an appropriate exposure, and an automatic exposure mode (AE) in which the amount of light in the imaging environment is measured and the exposure is automatically set is often used. However, an image pickup device mounted on an unmanned aerial vehicle has a limit in a light amount range (hereinafter, referred to as “dynamic range”) that can be photographed by one exposure, and when the brightness of a subject exceeds the upper limit of the dynamic range. Since overexposure occurs and underexposure occurs below the lower limit, it is difficult to simultaneously obtain shapes in a dark area and a bright area when the contrast is large. Therefore, when imaging a building with a strong difference in brightness or a shadow of a building in an environment of direct sunlight, a scene in a strong backlighting environment, or the like, a part of the captured image may be overexposed or underexposed. However, there has been a problem that it is difficult to detect a feature point of the portion, and there is a shortage of information that can be obtained, and the shape cannot be restored normally.

  As a means for avoiding overexposure and underexposure due to the limit of the dynamic range, high dynamic range synthesis (HDR) is known. In HDR, a plurality of images having different exposure settings are photographed, and the images are combined to generate an image (high dynamic range image) having a wide dynamic range with less whiteout and blackout. However, HDR needs to acquire a plurality of images from the same imaging position, and is not suitable for capturing while moving as in imaging by a flying object. In addition, when an image is captured by the same camera, there is a limit to the exposure setting when acquiring an image serving as an HDR material.

  In one aspect, the shape generation method includes a step of acquiring information on a plurality of imaging positions of a flying object having a plurality of imaging devices, and an imaging device used for imaging each of the plurality of imaging positions from among the plurality of imaging devices. Selecting, imaging at the imaging device selected at each imaging position, restoring the shape of the subject based on the captured image for each imaging device, and restoring the shape for each imaging device Synthesizing the image pickup device, and the step of selecting an image pickup device includes, for each image pickup position, at least one of a plurality of image pickup devices Selecting one imaging device.

  The step of selecting at least one imaging device includes selecting the first imaging device when the ratio is equal to or less than the first threshold, and selecting the first imaging device when the ratio exceeds a second threshold that is larger than the first threshold. Selecting a second imaging device for which an exposure parameter higher than that of the first imaging device is set, and if the ratio is greater than the first threshold and less than or equal to the second threshold, the first imaging device and the second imaging device It may include selecting both of the devices.

  In one aspect, a method for generating a shape on a mobile platform includes acquiring information on a plurality of imaging positions of a flying object having a plurality of imaging devices, and performing imaging for each of the plurality of imaging positions from among the plurality of imaging devices. Selecting an imaging device to be used, transmitting the information on the plurality of imaging positions, and information on the imaging device selected for each imaging position to the flying object, and selecting the imaging device. And selecting at least one image pickup device from the plurality of image pickup devices based on a ratio of a portion of the image pickup area at or below the predetermined light amount at each image pickup position.

  The step of selecting at least one imaging device includes selecting the first imaging device when the ratio is equal to or less than the first threshold, and selecting the first imaging device when the ratio exceeds a second threshold that is larger than the first threshold. Selecting a second imaging device for which an exposure parameter higher than that of the first imaging device is set, and if the ratio is greater than the first threshold and less than or equal to the second threshold, the first imaging device and the second imaging device It may include selecting both of the devices.

  Obtaining an image captured by the imaging device selected at each imaging position from the flying object; for each imaging device, restoring the shape of the subject based on the captured image; and for each imaging device, Synthesizing the modified shape.

  The step of selecting an imaging device may select a plurality of imaging devices at at least one of the plurality of imaging positions.

  The step of selecting an imaging device includes the steps of acquiring information on an illumination angle of a light source, acquiring information on an obstruction of a light source at each imaging position, and information on an illumination angle of a light source and a light source at each imaging position. Estimating a ratio occupied by a portion having a predetermined light amount or less in the imaging area at the imaging position based on the information on the obstacle.

  The step of obtaining information on the irradiation angle of the light source includes the steps of obtaining time information and geographic information of the imaging position, and estimating information on the irradiation angle of the light source using the time information and the geographic information. Good.

  In one aspect, an image acquisition method in a flying object for shape generation includes: acquiring information on a plurality of imaging positions of a flying object having a plurality of imaging devices; Acquiring information about an imaging device used for imaging for each of the imaging positions, and capturing an image using an imaging device corresponding to the information about the imaging device used for the acquired imaging, for each of the plurality of imaging positions; In the information on the imaging device used for imaging, based on the ratio of the portion of the imaging area at each imaging position less than the predetermined amount of light, according to the selection of at least one imaging device from a plurality of imaging devices This is the generated information.

  The step of obtaining information on the imaging device used for imaging may include a step of receiving information on the imaging device used for imaging from the mobile platform for each of the plurality of imaging positions.

  The step of acquiring information on the imaging device used for imaging includes, for each imaging position, a step of measuring the amount of light in an imaging region at the imaging position at the time of imaging, and a step of measuring a predetermined amount of light in the imaging region at the imaging position based on the amount of light. The method may include a step of estimating a ratio occupied by the following portions, and a step of selecting at least one imaging device from a plurality of imaging devices based on the ratio and generating information on the imaging device used for imaging.

  Acquiring information on the imaging device used for imaging, for each imaging position, acquiring information on an illumination angle of the light source, acquiring information on an obstruction of the light source at the imaging position, and illuminating angle of the light source; Estimating a ratio occupied by a portion equal to or less than a predetermined amount of light in the imaging region at the imaging position based on the information regarding the light source block at the imaging position and at least one of the plurality of imaging devices based on the ratio. Selecting one of the imaging devices and generating information about the imaging device to be used for imaging.

  The step of obtaining information on the irradiation angle of the light source includes the steps of obtaining time information and geographic information of the imaging position, and estimating information on the irradiation angle of the light source using the time information and the geographic information. Good.

  The information on the imaging device used for imaging is such that when the proportion of the portion of the imaging area at each imaging position that is equal to or less than the predetermined amount of light is less than or equal to the first threshold, the first imaging device is selected, and If the second imaging device having a higher exposure parameter than that of the first imaging device is selected, the second imaging device having a higher exposure parameter than the first imaging device is selected, and the ratio exceeds the first threshold and is equal to or less than the second threshold. In some cases, both the first imaging device and the second imaging device may be selected and generated.

  The information on the imaging device used for imaging may be generated by selecting a plurality of imaging devices at at least one of the plurality of imaging positions.

  The step of imaging using the imaging device includes, during the movement of the flying object, for each of the plurality of imaging positions, imaging using the imaging device specified by the information on the imaging device used for the acquired imaging. Good.

  The method may further include a step of transmitting images captured at a plurality of imaging positions to an information processing apparatus that performs shape generation.

  In one aspect, a mobile platform includes a storage unit, a communication unit that communicates with a flying object, and a processing unit, wherein the processing unit includes information about a plurality of imaging positions of the flying object having a plurality of imaging devices. And selecting an imaging device to be used for imaging for each of the plurality of imaging positions from the plurality of imaging devices, and obtaining information on the plurality of imaging positions and information on the imaging device selected for each of the imaging positions. The transmission to the flying object by the communication unit and the selection of the imaging device are performed by at least one of the plurality of imaging devices based on the ratio of the portion of the imaging area at the imaging position that is equal to or less than the predetermined light amount. This is realized by selecting one imaging device.

  The processing unit selects the first imaging device when the ratio is equal to or less than the first threshold, and selects a higher exposure than the first imaging device when the ratio exceeds the second threshold that is larger than the first threshold. Select the second imaging device for which the parameter is set, and if the ratio exceeds the first threshold and is equal to or less than the second threshold, select both the first imaging device and the second imaging device. Good.

  The processing unit further obtains an image captured by the imaging device selected at each imaging position from the flying object, restores the shape of the subject based on the captured image for each imaging device, and The restored shape may be combined.

  The processing unit may select a plurality of imaging devices at at least one of the plurality of imaging positions.

  The processing unit obtains information about the irradiation angle of the light source, acquires information about the light blocking object at each imaging position, and based on the information about the light source irradiation angle and information about the light blocking object at each imaging position. Thus, the ratio occupied by a portion having a predetermined light amount or less in the imaging region at the imaging position may be estimated.

  The processing unit may acquire the time information and the geographic information of the imaging position, and may estimate information about the irradiation angle of the light source using the time information and the geographic information.

  In one aspect, an air vehicle includes a storage unit, a communication unit that communicates with a mobile platform, a processing unit, and a plurality of imaging devices, wherein the processing unit includes information regarding a plurality of imaging positions of the air vehicle. Acquiring information about the imaging device used for imaging for each of the plurality of imaging positions selected from among the plurality of imaging devices, and acquiring the information about the imaging device used for the acquired imaging for each of the plurality of imaging positions. An image is captured using an imaging device corresponding to the information, and information on the imaging device used for imaging is based on a ratio of a portion of a predetermined amount of light or less in an imaging region at each imaging position. This is information generated according to the selection of the imaging device.

  The processing unit may receive, from the mobile platform, information on an imaging device used for imaging for each of the plurality of imaging positions.

  The image processing apparatus further includes an optical sensor, and the processing unit measures, for each imaging position, a light amount of the imaging region at the imaging position at the time of imaging, and based on the light amount, a part of the imaging region at the imaging position at a predetermined light amount or less is determined. The occupation ratio may be estimated, and based on the ratio, at least one of the plurality of imaging devices may be selected to generate information regarding the imaging device used for imaging.

  The processing unit acquires, for each imaging position, information about the irradiation angle of the light source, acquires information about the light blocking object at the imaging position, information about the irradiation angle of the light source, and information about the light blocking object at the imaging position. Based on the ratio, the ratio of the portion occupied by the predetermined amount of light or less in the imaging region at the imaging position is estimated, and based on the ratio, information on the imaging device to be used for imaging by selecting at least one imaging device from the plurality of imaging devices. May be generated.

  The processing unit may acquire the time information and the geographic information of the imaging position, and may estimate information about the irradiation angle of the light source using the time information and the geographic information.

  The information on the imaging device used for imaging is such that when the proportion of the portion of the imaging area at each imaging position that is equal to or less than the predetermined amount of light is less than or equal to the first threshold, the first imaging device is selected, and If the second imaging device having a higher exposure parameter than that of the first imaging device is selected, the second imaging device having a higher exposure parameter than the first imaging device is selected, and the ratio exceeds the first threshold and is equal to or less than the second threshold. In some cases, both the first imaging device and the second imaging device may be selected and generated.

  The information on the imaging device used for imaging may be generated by selecting a plurality of imaging devices at at least one of the plurality of imaging positions.

  The processing unit may capture an image of each of the plurality of imaging positions using the imaging device specified by the information on the imaging device used for the acquired imaging while the flying object is moving.

  The processing unit may transmit images captured at a plurality of imaging positions to an information processing device that performs shape generation.

  In one embodiment, a step of acquiring information on a plurality of imaging positions of a flying object having a plurality of imaging devices on a mobile platform that is a computer, and from among the plurality of imaging devices, using each of the plurality of imaging positions for imaging. Selecting an imaging device, transmitting information to the flying object with information on the plurality of imaging positions and information on the imaging device selected for each imaging position, and selecting an imaging device. And selecting at least one image pickup device from among the plurality of image pickup devices based on the ratio of the portion of the image pickup area at or below the predetermined light amount at the image pickup position.

  In one embodiment, a step of acquiring information on a plurality of imaging positions of a flying object having a plurality of imaging devices on a mobile platform that is a computer, and from among the plurality of imaging devices, using each of the plurality of imaging positions for imaging. The step of selecting an imaging device, the step of transmitting information on a plurality of imaging positions, and the step of transmitting information on the imaging device selected for each imaging position to the flying object, and the step of selecting an imaging device, Selecting at least one imaging device from a plurality of imaging devices based on a ratio of a portion having a predetermined light amount or less in an imaging region at the imaging position with respect to each imaging position. Medium.

  In one aspect, for a flying object having a plurality of imaging devices, which is a computer, acquiring information on a plurality of imaging positions of the flying object, and for each of the plurality of imaging positions selected from the plurality of imaging devices. Acquiring information on an imaging device used for imaging, and, for each of a plurality of imaging positions, capturing an image using an imaging device corresponding to the information on the acquired imaging device used for imaging. The information on the imaging device to be used is a program which is information generated according to selection of at least one imaging device from a plurality of imaging devices based on a ratio of a portion of a predetermined amount of light or less in an imaging region at each imaging position. It is.

  In one aspect, for a flying object having a plurality of imaging devices, which is a computer, acquiring information on a plurality of imaging positions of the flying object, and for each of the plurality of imaging positions selected from the plurality of imaging devices. Acquiring information on an imaging device used for imaging, and, for each of a plurality of imaging positions, capturing an image using an imaging device corresponding to the information on the acquired imaging device used for imaging. The information on the imaging device to be used is a program which is information generated according to selection of at least one imaging device from a plurality of imaging devices based on a ratio of a portion of a predetermined amount of light or less in an imaging region at each imaging position. Is a storage medium for storing.

  Note that the above summary of the present invention does not list all of the features of the present disclosure. Further, a sub-combination of these feature groups can also be an invention.

It is a figure showing the example of composition of the shape generation system of each embodiment. It is a figure showing an example of the appearance of an unmanned aerial vehicle. It is a figure showing an example of the concrete appearance of an unmanned aerial vehicle. FIG. 2 is a block diagram illustrating an example of a hardware configuration of an unmanned aerial vehicle constituting the shape generation system of FIG. 1. It is a figure showing an example of the appearance of a transmitter. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a transmitter included in the shape generation system in FIG. 1. FIG. 3 is a diagram illustrating an example of an imaging range according to each embodiment. FIG. 3 is a sequence diagram illustrating processing in the shape generation system according to the first embodiment. FIG. 4 is a diagram schematically illustrating a flight path and a plurality of imaging positions in an imaging range. It is a flowchart which shows an example of the step which selects an imaging device. FIG. 4 is a diagram schematically illustrating a state where an imaging device is selected at a plurality of imaging positions. FIG. 4 is a diagram schematically illustrating restoring and forming a shape for each imaging device. It is a sequence diagram showing processing in a shape generation method system concerning a 2nd embodiment.

  Hereinafter, the present disclosure will be described through embodiments of the present invention, but the following embodiments do not limit the invention according to the claims. Not all combinations of the features described in the embodiments are necessarily essential to the solution of the invention.

  The claims, specification, drawings, and abstract include subject matter protected by copyright. The copyright owner will not object to any person's reproduction of these documents, as indicated in the JPO file or record. However, in all other cases, all copyrights are reserved.

  The shape generation method according to the present disclosure includes an unmanned aerial vehicle (UAV) as an example of a moving object, a mobile platform for remotely controlling the operation or processing of the unmanned aerial vehicle, and combining and combining images. Various processes (steps) in a shape generation system including a computer (PC) that performs the process are defined.

  The method for acquiring an image of a flying object for generating a shape according to the present disclosure is one in which various processes (steps) for an unmanned aerial vehicle are specified in the method for generating a shape according to the present disclosure.

  The shape generation method in the mobile platform according to the present disclosure defines various processes (steps) in the transmitter in the shape generation method according to the present disclosure.

  An air vehicle according to the present disclosure includes an aircraft (eg, a drone, a helicopter) that moves in the air. The flying object may be an unmanned flying object having a plurality of imaging devices, and is set in advance to image a subject (for example, a ground shape of a building, a road, a park, or the like within a certain range) in an imaging range. It flies along the flight path, and images at a plurality of imaging positions (waypoints described later) set on the flight path.

  The mobile platform according to the present disclosure is a computer, a transmitter for instructing remote control of various processes including, for example, movement of an unmanned aerial vehicle, and a terminal connected to the transmitter so that information and data can be input and output. An information processing apparatus such as a PC or a tablet connected to a device or an unmanned aerial vehicle so that information and data can be input and output. Note that the unmanned aerial vehicle itself may be included as a mobile platform.

  A program according to the present disclosure is a program for causing an unmanned aerial vehicle or a mobile platform to execute various processes (steps).

  The recording medium according to the present disclosure stores a program (that is, a program for causing an unmanned aerial vehicle or a mobile platform to execute various processes (steps)).

  In each embodiment according to the present disclosure, the unmanned aerial vehicle 100 flies along a flight path set in advance in an imaging range.

  The flight path may be set by any conventional method, and may be, for example, a path that flies the shortest distance within the range set by an existing algorithm, a path that flies in the shortest time, or a path that saves the most power.

  The flight path includes information on a plurality of imaging positions (that is, waypoints). The unmanned aerial vehicle 100 sequentially moves along the set route and captures images at waypoints.

  In addition, it is preferable that each waypoint is set at intervals at which the imaging areas of adjacent waypoints overlap. This makes it possible to detect the feature points of the entire imaging range, and to avoid a shortage of information for forming a complete shape in the imaging range when the shape is restored from the captured image by SFM technology or the like. it can.

  FIG. 1 is a diagram illustrating a configuration example of a shape generation system according to each embodiment. 1 includes at least an unmanned aerial vehicle 100 and a transmitter 50. Preferably, the information processing apparatus further includes an information processing device (for example, a PC, a tablet, or the like) (not shown) for combining and combining captured images. The unmanned aerial vehicle 100 and the transmitter 50 can communicate information and data with each other using wired communication or wireless communication (for example, a wireless LAN (Local Area Network) or Bluetooth (registered trademark)). In FIG. 1, illustration of a state where the terminal device is attached to the housing of the transmitter 50 is omitted. The transmitter 50 as an example of the operation terminal is used while being held by both hands of a person who uses the transmitter 50 (hereinafter, referred to as “user”), for example.

  FIG. 2 is a diagram illustrating an example of the appearance of the unmanned aerial vehicle 100. FIG. 3 is a diagram illustrating an example of a specific appearance of the unmanned aerial vehicle 100. A side view when the unmanned aerial vehicle 100 flies in the movement direction STV0 is shown in FIG. 2, and a perspective view when the unmanned aerial vehicle 100 flies in the movement direction STV0 is shown in FIG. The unmanned aerial vehicle 100 is an example of a moving object that includes two imaging devices 220-1 and 220-2 and moves. The moving body is a concept including, in addition to the unmanned aerial vehicle 100, another aircraft moving in the air, a vehicle moving on the ground, a ship moving on water, and the like. Here, as shown in FIGS. 2 and 3, it is assumed that a roll axis (see the x-axis in FIGS. 2 and 3) is defined in a direction parallel to the ground and along the movement direction STV0. In this case, a pitch axis (see the y-axis in FIGS. 2 and 3) is defined in a direction parallel to the ground and perpendicular to the roll axis, and further a direction perpendicular to the ground and perpendicular to the roll axis and the pitch axis. The yaw axis (the z-axis in FIGS. 2 and 3) is determined.

  The unmanned aerial vehicle 100 is configured to include the UAV main body 102, the gimbal 200, a plurality of imaging devices 220-1, 220-2, and a plurality of obstacle sensors 230. The unmanned aerial vehicle 100 can move based on a remote control instruction transmitted from the transmitter 50 as an example of the mobile platform according to the present disclosure. The movement of the unmanned aerial vehicle 100 means a flight, and includes at least ascending, descending, turning left, turning right, moving left horizontally, and moving right horizontal.

  The UAV main body 102 includes a plurality of rotors. The UAV body 102 moves the unmanned aerial vehicle 100 by controlling the rotation of a plurality of rotors. The UAV main body 102 moves the unmanned aerial vehicle 100 using, for example, four rotors. The number of rotors is not limited to four. Further, the unmanned aerial vehicle 100 may be a fixed wing aircraft having no rotary wing.

  The imaging device 220-1 and the imaging device 220-2 are cameras that capture an image of a subject (for example, a ground shape such as a building, a road, or a park described above) included in a desired imaging range. Hereinafter, an example in which the two imaging devices 220-1 and 220-2 are attached to one gimbal 200 will be described. However, actually, each of the imaging devices 220-1 and 220-2 may be attached to a different gimbal 200 and separately controlled. . Further, the number of imaging devices is not limited to two, but may be larger.

  The plurality of obstacle sensors 230 can detect obstacles around the unmanned aerial vehicle.

  Next, a configuration example of the unmanned aerial vehicle 100 will be described.

  FIG. 4 is a block diagram illustrating an example of a hardware configuration of the unmanned aerial vehicle 100 included in the shape generation system 10 of FIG. The unmanned aerial vehicle 100 includes a UAV processing unit 110, a communication interface 150, a memory 160, a gimbal 200, a rotary wing mechanism 210, an imaging device 220-1, an imaging device 220-2, an obstacle sensor 230, The configuration includes a GPS receiver 240, a battery 250, an optical sensor 260, and a timer 270.

  The UAV processing unit 110 is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). The UAV processing unit 110 performs signal processing for controlling the operation of each unit of the unmanned aerial vehicle 100 in an integrated manner, data input / output processing with other units, data arithmetic processing, and data storage processing.

  The UAV processing unit 110 controls the flight of the unmanned aerial vehicle 100 according to a program stored in the memory 160. The UAV processing unit 110 controls the movement (that is, flight) of the unmanned aerial vehicle 100 according to a command received from the remote transmitter 50 via the communication interface 150. Memory 160 may be removable from unmanned aerial vehicle 100.

  The communication interface 150 communicates with the transmitter 50 (see FIG. 4). The communication interface 150 receives various commands to the UAV processing unit 110 from the remote transmitter 50.

  In the memory 160, the UAV processing unit 110 controls the gimbal 200, the rotary wing mechanism 210, the imaging device 220-1, the imaging device 220-2, the obstacle sensor 230, the GPS receiver 240, the battery 250, the optical sensor 260, and the timer 270. Stores programs and the like necessary to perform The memory 160 may be a computer-readable recording medium, such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), and It may include at least one of a flash memory such as a USB memory. The memory 160 may be provided inside the UAV body 102. It may be provided detachably from the UAV body 102.

  The gimbal 200 rotatably supports the imaging device 220-1 and the imaging device 220-2 about at least one axis. The gimbal 200 may rotatably support the imaging device 220 about a yaw axis, a pitch axis, and a roll axis. The gimbal 200 may change the imaging direction of the imaging device 220 by rotating the imaging device 220-1 and the imaging device 220-2 around at least one of the yaw axis, the pitch axis, and the roll axis. Further, as described above, each of the imaging device 220-1 and the imaging device 220-2 may be provided with one gimbal.

  The rotary wing mechanism 210 includes a plurality of rotary blades and a plurality of drive motors for rotating the plurality of rotary blades.

  The imaging device 220-1 and the imaging device 220-2 capture an image of a subject in a desired imaging range and generate data of a captured image. It is preferable that the two imaging devices 220-1 and 220-2 have the same angle of view and different exposure parameters. Image data obtained by imaging by the imaging device 220-1 and the imaging device 220-2 is stored in the memory or the memory 160 of each of the imaging device 220-1 and the imaging device 220-2. Preferably, the image data is stored for each imaging device.

  The obstacle sensor 230 is, for example, an infrared sensor, an imaging device, an ultrasonic sensor, or the like, detects an obstacle around the unmanned aerial vehicle 100, and outputs the information to the UAV processing unit 110.

  The GPS receiver 240 receives a plurality of signals indicating times transmitted from a plurality of navigation satellites (that is, GPS satellites) and a position (coordinate) of each GPS satellite. The GPS receiver 240 calculates the position of the GPS receiver 240 (that is, the position of the unmanned aerial vehicle 100) based on the received signals. The GPS receiver 240 outputs the position information of the unmanned aerial vehicle 100 to the UAV processing unit 110. The calculation of the position information of the GPS receiver 240 may be performed by the UAV processing unit 110 instead of the GPS receiver 240. In this case, to the UAV processing unit 110, information indicating the time included in the plurality of signals received by the GPS receiver 240 and the position of each GPS satellite is input.

  The battery 250 has a function as a drive source for each unit of the unmanned aerial vehicle 100 and supplies necessary power to each unit of the unmanned aerial vehicle 100.

  The optical sensor 260 detects the amount of light in the imaging area, and outputs a detection result to the UAV processing unit 110.

  The timer 270 manages the time information and outputs the time information to the UAV processing unit 110.

  Next, a configuration example of the transmitter 50 will be described.

  FIG. 5 is a perspective view showing an example of the appearance of the transmitter 50. It is assumed that the directions of up, down, front, rear, left and right with respect to the transmitter 50 follow the directions of arrows shown in FIG. 5, respectively. The transmitter 50 is used, for example, while being held by both hands of the user who uses the transmitter 50.

  The transmitter 50 has, for example, a substantially rectangular parallelepiped (in other words, substantially box-shaped) resin housing 50B having a substantially square bottom surface and a height shorter than one side of the bottom surface. The specific configuration of the transmitter 50 will be described later with reference to FIG. A left control rod 53L and a right control rod 53R are arranged protrudingly at substantially the center of the housing surface of the transmitter 50.

  The left control rod 53L and the right control rod 53R are used in operations for remotely controlling the movement of the unmanned aerial vehicle 100 by the user (for example, moving the unmanned aerial vehicle 100 back and forth, left and right, up and down, and change direction). Is done. FIG. 5 shows the positions of the left control rod 53L and the right control rod 53R in the initial state where no external force is applied from both hands of the user. After the external force applied by the user is released, the left control rod 53L and the right control rod 53R automatically return to predetermined positions (for example, initial positions shown in FIG. 5).

  The power button B1 of the transmitter 50 is arranged on the near side (in other words, on the user side) of the left control rod 53L. When the power button B1 is pressed once by the user, for example, the remaining capacity of a battery (not shown) built in the transmitter 50 is displayed on the battery remaining amount display section L2. When the power button B1 is pressed again by the user, for example, the power of the transmitter 50 is turned on, and power is supplied to each unit (see FIG. 6) of the transmitter 50 to be usable.

  An RTH (Return To Home) button B2 is arranged on the near side (in other words, on the user side) of the right control rod 53R. When the RTH button B2 is pressed by the user, the transmitter 50 transmits a signal for automatically returning the unmanned aerial vehicle 100 to a predetermined position. Thus, the transmitter 50 can automatically return the unmanned aerial vehicle 100 to a predetermined position (for example, a takeoff position stored in the unmanned aerial vehicle 100). The RTH button B2 is used, for example, when the user loses sight of the unmanned aerial vehicle 100 during aerial photographing with the unmanned aerial vehicle 100 outdoors, or when the user becomes unable to operate due to radio wave interference or unexpected trouble. Available to

  A remote status display section L1 and a remaining battery level display section L2 are arranged in front of the power button B1 and the RTH button B2 (in other words, on the user side). The remote status display unit L1 is configured using, for example, an LED (Light Emission Diode), and displays a wireless connection state between the transmitter 50 and the unmanned aerial vehicle 100. The battery remaining amount display unit L2 is configured using, for example, an LED, and displays the remaining amount of capacity of a battery (not shown) built in the transmitter 50.

  Two antennas AN1 and AN2 are arranged to protrude from the rear side surface of the housing 50B of the transmitter 50 on the rear side of the left control rod 53L and the right control rod 53R. The antennas AN1 and AN2 output the signal generated by the transmitter processing unit 61 (that is, the signal for controlling the movement of the unmanned aerial vehicle 100) based on the operation of the left control rod 53L and the right control rod 53R by the user. Transmit to the flying object 100. The antennas AN1 and AN2 can cover a transmission and reception range of, for example, 2 km. In addition, the antennas AN1 and AN2 are used to transmit images captured by the imaging devices 220-1 and 220-2 of the unmanned aerial vehicle 100 wirelessly connected to the transmitter 50 or various data acquired by the unmanned aerial vehicle 100 for unmanned flight. When transmitted from the body 100, these images or various data can be received.

  The touch panel display TPD1 is configured using, for example, an LCD (Crystal Liquid Display) or an organic EL (Electroluminescence). The shape, size, and arrangement position of the touch panel display TPD1 are arbitrary, and are not limited to the illustrated example in FIG.

  FIG. 6 is a block diagram illustrating an example of a hardware configuration of the transmitter 50 included in the shape generation system 10 of FIG. The transmitter 50 includes a left control rod 53L, a right control rod 53R, a transmitter processing unit 61, a wireless communication unit 63, a memory 64, a power button B1, an RTH button B2, an operation unit set OPS, The configuration includes a remote status display unit L1, a battery remaining amount display unit L2, and a touch panel display TPD1. The transmitter 50 is an example of an operation terminal for remotely controlling the unmanned aerial vehicle 100.

  The left control rod 53L is used for an operation for remotely controlling the movement of the unmanned aerial vehicle 100 by, for example, the left hand of the user. The right control rod 53R is used for an operation for remotely controlling the movement of the unmanned aerial vehicle 100 by, for example, the right hand of the user. The movement of the unmanned aerial vehicle 100 includes, for example, movement in a forward direction, movement in a backward direction, movement in a left direction, movement in a right direction, movement in a rising direction, movement in a descending direction, and movement in the left direction. , Or a movement of rotating the unmanned aerial vehicle 100 rightward, or a combination thereof, and so on.

  When the power button B1 is pressed once, a signal indicating that the power button B1 has been pressed once is input to the transmitter processing unit 61. According to this signal, the transmitter processing section 61 displays the remaining capacity of the battery (not shown) built in the transmitter 50 on the remaining battery level display section L2. Thus, the user can easily check the remaining capacity of the battery built in the transmitter 50. When the power button B1 is pressed twice, a signal indicating that the power button B1 has been pressed twice is passed to the transmitter processing unit 61. According to this signal, the transmitter processing unit 61 instructs a battery (not shown) built in the transmitter 50 to supply power to each unit in the transmitter 50. Thus, the user turns on the power of the transmitter 50 and can easily start using the transmitter 50.

  When the RTH button B2 is pressed, a signal indicating that the RTH button B2 has been pressed is input to the transmitter processing unit 61. According to this signal, the transmitter processing unit 61 generates a signal for automatically returning the unmanned aerial vehicle 100 to a predetermined position (for example, the take-off position of the unmanned aerial vehicle 100), and controls the wireless communication unit 63 and the antennas AN1 and AN2. To the unmanned aerial vehicle 100 via the Thus, the user can automatically return (return) the unmanned aerial vehicle 100 to a predetermined position by a simple operation on the transmitter 50.

  The operation unit set OPS is configured using a plurality of operation units (for example, operation units OP1,..., Operation unit OPn) (n: an integer of 2 or more). The operation unit set OPS supports remote control of the unmanned aerial vehicle 100 by the operation units other than the left control rod 53L, the right control rod 53R, the power button B1, and the RTH button B2 shown in FIG. Various operation units). The various operation units referred to herein include, for example, a button for instructing imaging of a still image using the imaging device 220 of the unmanned aerial vehicle 100, and starting and ending recording of a moving image using the imaging device 220 of the unmanned aerial vehicle 100 Button, a dial for adjusting the tilt of the gimbal 200 (see FIG. 4) of the unmanned aerial vehicle 100 in the tilt direction, a button for switching the flight mode of the unmanned aerial vehicle 100, and setting of the imaging device 220 of the unmanned aerial vehicle 100. Dial is applicable.

  The operation unit set OPS has a parameter operation unit OPA for inputting information of input parameters for generating a waypoint of the unmanned aerial vehicle 100. The parameter operation unit OPA may be formed by a stick, a button, a key, a touch panel, and the like. The parameter operation unit OPA may be formed by the left control rod 53L and the right control rod 53R. The timing of inputting each parameter included in the input parameters by the parameter operation unit OPA may be the same or different.

  Since the remote status display section L1 and the remaining battery level display section L2 have been described with reference to FIG. 5, the description is omitted here.

  The transmitter processing unit 61 is configured using a processor (for example, a CPU, an MPU, or a DSP). The transmitter processing unit 61 performs signal processing for integrally controlling the operation of each unit of the transmitter 50, data input / output processing with other units, data arithmetic processing, and data storage processing.

  The wireless communication unit 63 is connected to two antennas AN1 and AN2. The wireless communication unit 63 transmits and receives information and data to and from the unmanned aerial vehicle 100 via the two antennas AN1 and AN2 using a predetermined wireless communication method (for example, WiFi (registered trademark)). The wireless communication unit 63 transmits the information of the input parameters from the transmitter processing unit 61 to the unmanned aerial vehicle 100.

  The memory 64 stores, for example, a ROM (Read Only Memory) in which a program that defines the operation of the transmitter processing unit 61 and data of setting values are stored, and various information and data used in processing of the transmitter processing unit 61. And a RAM (Random Access Memory) for temporarily storing. The programs and the data of the setting values stored in the ROM of the memory 64 may be copied to a predetermined recording medium (for example, a CD-ROM or a DVD-ROM). The RAM of the memory 64 stores, for example, data of an aerial image captured by the imaging device 220 of the unmanned aerial vehicle 100.

  The touch panel display TPD1 may display various data processed by the transmitter processing unit 61. Touch panel display TPD1 displays input parameter information that has been input. Therefore, the user of the transmitter 50 can confirm the contents of the input parameters by referring to the touch panel display TPD1.

  Note that the transmitter 50 may be connected to a terminal device by wire or wirelessly instead of including the touch panel display TPD1. The information of the input parameter may be displayed on the terminal device, similarly to the touch panel display TPD1. The terminal device may be a smartphone, a tablet terminal, a PC (Personal Computer), or the like. Further, the terminal device inputs at least one of the input parameters, sends the input parameters to the transmitter 50 by wire communication or wireless communication, and the wireless communication unit 63 of the transmitter 50 transmits the input parameters to the unmanned aerial vehicle 100. Good.

  In addition, the operation of the unmanned aerial vehicle 100 may be one that flies along a preset flight path and captures an image, as described later, other than a case based on an operation by the transmitter 50.

  Hereinafter, embodiments of processing in the shape generation system according to the present disclosure will be described with reference to the drawings. FIG. 7 is a diagram illustrating an example of an imaging range according to each embodiment. In order to easily explain the features of the shape generation system according to the present disclosure, in each of the following embodiments, when an unmanned aerial vehicle images a rectangular ground shape (imaging range A) as shown in FIG. The case where a circular shade formed by blocking sunlight due to a mountain or the like exists to the left from the center of the imaging range A is taken as an example, but is not limited to this. The present disclosure can be applied to any other situations, for example, in a case where illumination is blocked by furniture, a fixture, or the like in imaging of an indoor environment. Of course, the imaging range may be an irregular and complicated range other than a rectangle, and the number of shades included in the imaging range and the shape of each shade may be various.

(1st Embodiment)
FIG. 8 is a sequence diagram illustrating processing in the shape generation system according to the first embodiment. The shape generation system according to the present embodiment includes a transmitter 50, an unmanned aerial vehicle 100 including an imaging device 220-1 (imaging device 1) and an imaging device 220-2 (imaging device 2), and an information processing device (for example, a PC, Tablet).

  First, the transmitter 50 acquires information on the imaging device 1 and the imaging position (waypoint) (step S11). The information on the waypoint is coordinate information on a map on which the unmanned aerial vehicle 100 performs an imaging operation determined based on a flight route set in advance in the imaging range A.

  The flight path is set by the transmitter 50 or a terminal device (smartphone, tablet, or the like) connected to the transmitter 50 or another terminal device by a conventional method. Route, the route that flies in the shortest time, or the route that saves the most power.

  The information on the waypoint may include, for example, longitude, latitude, and altitude information. At this time, the transmitter 50 may acquire the information about the waypoint recorded in the built-in memory, or may acquire the information about the waypoint from the outside via the wireless communication unit 63.

  FIG. 9 is a diagram schematically illustrating a flight path and a plurality of imaging positions in the imaging range A. As shown in FIG. 9, 24 waypoints are provided on the flight path in the imaging range A. That is, the unmanned aerial vehicle 100 performs the first imaging at the waypoint 1 in the imaging processing (step S13) described later, then flies in the direction of the arrow, and performs the second imaging when passing the waypoint 2. After that, when flying in the direction of the arrow again and passing waypoint 3, the third imaging is performed. In this way, after the flight and the imaging are repeated and the 24th imaging is performed at the waypoint 24, the aerial imaging is ended.

  The waypoints are preferably set at intervals at which captured images at adjacent waypoints overlap. Thus, when the information processing apparatus surely restores the shape of the imaging range A from the plurality of acquired images, it is possible to avoid a shortage of information for forming a complete shape in the imaging range A. .

  Next, an imaging device used at each waypoint is selected (step S12). In the present embodiment, the unmanned aerial vehicle 100 includes two imaging devices (imaging device 1 and imaging device 2) having different exposure parameters, and performs imaging using only the imaging device 1 for each waypoint, or Either imaging using only the imaging device 2 or imaging using the imaging device 1 and the imaging device 2 at the same time is selected.

  FIG. 10 is a flowchart illustrating an example of the step of selecting an imaging device (step S12). As shown in FIG. 10, first, the ratio of the waypoint occupied by the shaded portion in the imaging area is obtained (step S121). In the present embodiment, the imaging range A includes a bright portion that is exposed to direct sunlight and a dark portion that is a shadow formed by blocking the sun's light by a building or the like, and has an environment with a large difference in brightness. Therefore, a portion equal to or less than the predetermined light amount is defined as a "shadow portion".

  The transmitter 50 can estimate the ratio occupied by the shaded portion based on information on the irradiation angle of the light source and the obstacle of the light source.

  In the present embodiment, the irradiation angle of the light source indicates the irradiation angle of the sun. The transmitter 50 can estimate the sun irradiation angle from the time information and the position information of the waypoint, for example.

  In this case, the transmitter 50 may acquire the time information from, for example, a built-in timer, or may acquire the time information from outside via GPS, the Internet, or the like. The waypoint position information may be longitude, latitude, and altitude information included in the information on the waypoint acquired in step S11.

  As the information on the light source obstruction, for example, the transmitter 50 may obtain a schematic shape (for example, an existing elevation map) of the imaging region with reference to a three-dimensional map database on the Internet.

  After acquiring the ratio of the shadow portion in the waypoint, the transmitter 50 selects an imaging device to be used based on the ratio.

  Specifically, when the proportion occupied by the shadow portion is 30% or less (first threshold value) (“YES” in step S122), the transmitter 50 performs imaging with the exposure parameter having a relatively low exposure amount. The device 1 is selected (Step S123). Thereby, it can be expected that an image in which the sunlit portion of the imaging region is properly exposed is obtained.

  On the other hand, if the proportion occupied by the shadow portion exceeds 70% (the second threshold value) (“NO” in step S124), the transmitter 50 selects the imaging device 2 having an exposure parameter with a relatively high exposure amount. (Step S125). Thereby, it can be expected that an image in which the shaded portion of the imaging region is properly exposed is obtained.

  Further, if the ratio of the shadow portion exceeds 30% (first threshold value) and is 70% (second threshold value) or less (“NO” in step S124), the transmitter 50 sets the imaging device. 1 and the imaging device 2 are both selected (step S126). Thereby, it can be expected that both an image in which the shaded portion is properly exposed and an image in which the sunlit portion is properly exposed are obtained.

  Note that the above 30% and 70% are merely examples of the first threshold value and the second threshold value, respectively. Actually, other numerical values may be set as necessary.

  Preferably, the first threshold and the second threshold are set so that the transmitter 50 selects both the imaging device 1 and the imaging device 2 in at least one waypoint. As a result, at least a part of the shape restored by the characteristic points of the image captured by the imaging device 1 described later and the shape restored by the characteristic points of the image captured by the imaging device 2 overlap, and a plurality of captured images Not only can accurately identify the positional relationship between the images, but also it is ensured that the information necessary for restoration by the information processing apparatus is not insufficient.

  FIG. 11 is a diagram schematically illustrating a state where an imaging device is selected at each waypoint. In FIG. 11, when the imaging device 1 is selected, it is indicated by “○”, and when the imaging device 2 is selected, it is indicated by “□”.

  As shown in FIG. 11, the transmitter 50 uses the image capturing apparatus 1 because the waypoints 1 to 7, 11 to 14, and 18 to 24 are sunlit portions and the proportion occupied by the shaded portions is 30% or less. select. Since the waypoints 8, 10, 15, and 17 are on the boundary between the shade and the sun, and the shaded portion exceeds 30% and is 70% or less, both the imaging device 1 and the imaging device 2 are selected. The waypoints 9 and 16 are in the shade, and the ratio of the shaded portion exceeds 70%, so that the imaging device 2 is selected.

  After the imaging device to be used for each waypoint is selected (step S12), returning to FIG. 8, the transmitter 50 generates information on the imaging device for each waypoint based on the selected result, and The information about the imaging device used at each waypoint and the information about the imaging device used at each waypoint are transmitted to the unmanned aerial vehicle 100.

  The information on the waypoints and the information on the imaging device used in each waypoint may be transmitted by the transmitter 50 to the unmanned aerial vehicle 100 by a wireless or wired communication method. Further, the transmitter 50 may record such information on a storage medium such as a memory card and transmit the information by any other method such as inserting the storage medium into an unmanned aerial vehicle.

  The unmanned aerial vehicle 100 flies along the flight path after receiving information on the waypoints and information on the imaging device used at each waypoint, and performs imaging using the imaging device selected at each waypoint. (Step S13).

  Preferably, the unmanned aerial vehicle 100 captures an image during its movement (that is, without stopping when the waypoint is reached). Thereby, the unmanned aerial vehicle 100 can not only shorten the aerial photographing time, but also save electric power for stopping and restarting the unmanned aerial vehicle 100.

  It is preferable that the unmanned aerial vehicle 100 stores the image captured in step S13 in a memory built in the imaging device or a memory built in the unmanned aerial vehicle for each imaging device.

  When the aerial photography ends, the unmanned aerial vehicle 100 transmits the captured image to the information processing device.

  These images may be transmitted by the unmanned aerial vehicle 100 to the information processing device by a wireless or wired communication method, or the unmanned aerial vehicle 100 may record the image on a storage medium such as a memory card and store the storage medium in the information processing device. May be transmitted by any other method, such as insertion into the

  The information processing apparatus that has received the image restores the shape of the subject by a conventional method such as SFM (Structure from Motion) for each imaging apparatus (step S14).

  FIG. 12 is a diagram schematically illustrating restoring and forming a shape for each imaging device. As shown in FIG. 12, when the shape of the image captured by the imaging device 1 is restored, the sunlit portion and the vicinity of the boundary between the shade and the sun are normally restored, and the information of the deep shaded portion is missing (the left side). (See shape B). This is because when the ratio of shadow portions such as the waypoints 9 and 16 exceeds 70%, the image is not captured by the image capturing apparatus 1 and the information is insufficient. On the other hand, feature points are efficiently detected in portions other than B and can be restored normally.

  On the other hand, when the shape of the image captured by the imaging device 2 is restored, the shaded portion and the vicinity of the boundary between the shade and the sun are normally restored, and information on the sunlit portion is missing (see B in the shape on the right). This is because the image capturing apparatus 2 is selected only when the shaded portion such as the waypoints 8 to 10 and the waypoints 15 to 17 exceeds 30%. On the other hand, in portions other than B, the feature points are efficiently detected and can be restored normally.

  Finally, the information processing device combines the two restored shapes (step S15). As a result, the portions B for which information is insufficient are complemented with each other, and the shape of the entire imaging range A is generated.

  At this time, since there is a waypoint in which both the imaging device 1 and the imaging device 2 are selected, the shape restored based on the image captured by the imaging device 1 and the waypoint based on the image captured by the imaging device 2 With the restored shape, they overlap near the boundary between the shade and the sun. Thus, the synthesized shape can prevent the occurrence of a missing portion such as B in FIG. 12 without causing information shortage, and a highly accurate shape is guaranteed.

  Note that the unmanned aerial vehicle 100 may transmit the captured image to the transmitter 50 as shown by the broken line in FIG. In this case, the transmitter 50 restores the shape of the subject for each imaging device (step S14 ') and combines the restored shape (step S15'). A detailed description of the restoration / synthesis at this time is the same as that of each process (steps S14 and S15) when the above-described image is transmitted to the information processing apparatus, and thus will not be described.

(Second embodiment)
In the first embodiment, the transmitter 50 selects an imaging device to be used for imaging for each waypoint. The second embodiment is different from the first embodiment in that the unmanned aerial vehicle 100 selects the imaging device. For the sake of convenience, the description of the parts that overlap with the first embodiment will be omitted.

  FIG. 13 is a sequence diagram illustrating processing in the shape generation system according to the second embodiment.

  First, the transmitter 50 acquires information on the imaging device 1 and the imaging position (waypoint) (Step S21).

  Thereafter, the transmitter 50 transmits information on the waypoint to the unmanned aerial vehicle 100. The information on the waypoints may be transmitted directly to the unmanned aerial vehicle 100 by the transmitter 50 by a wireless or wired communication method, or may be recorded on a storage medium such as a memory card in the transmitter 50, and the storage medium may be unmanned. Transmission may be by any other method, such as insertion into a vehicle.

  After receiving the information on the waypoint, the unmanned aerial vehicle 100 acquires the ratio occupied by the shadow portion in the waypoint, and selects an imaging device to be used based on the ratio (step S22).

  At this time, as a first acquisition method of the proportion occupied by the shaded portion in the waypoint, the unmanned aerial vehicle measures the proportion occupied by the shaded portion in the waypoint by using the information regarding the irradiation angle of the light source and the light source blockage. Can be estimated based on

  In the present embodiment, the irradiation angle of the light source refers to the irradiation angle of the sun. For example, the unmanned aerial vehicle 100 can be estimated from the time information and the position information of the waypoint.

  In this case, the unmanned aerial vehicle 100 may acquire the time information from the timer 270, or may acquire the time information from outside via GPS, the Internet, or the like. The waypoint location information may be longitude, latitude, and altitude information included in the information about the waypoint acquired in step S21.

  The information relating to the obstacle of the light source may be obtained, for example, by the unmanned aerial vehicle 100 referring to a three-dimensional map database on the Internet to obtain a schematic shape (for example, an existing altitude map) of the imaging region.

  As a second method of obtaining the ratio of the shaded portion occupying the waypoint, the unmanned aerial vehicle 100 flies along the flight path and uses the optical sensor 260 such as a light meter to capture the image area of the waypoint at the time of imaging. Measure the light intensity.

  At this time, the unmanned aerial vehicle 100 may continuously measure the amount of light in the imaging region during the flight, and estimate the proportion of the shadow portion in the waypoint by the time of imaging. Further, the light quantity may be measured at the timing of arriving at each waypoint (the timing just before arriving may be used).

  The unmanned aerial vehicle 100 selects an imaging device to be used based on the ratio of the shaded portion in the waypoint (Step S22), and then captures an image using the imaging device selected in each waypoint (Step S23).

  The image captured in step S23 is stored in the memory built in the imaging device or the memory 160 built in the unmanned aerial vehicle 100 for each imaging device.

  When the aerial photography ends, the unmanned aerial vehicle 100 transmits the image stored in the imaging device to the information processing device.

  These images may be transmitted by the unmanned aerial vehicle 100 to the information processing device by a wireless or wired communication method, or may be recorded on a storage medium such as a memory card by the unmanned aerial vehicle 100 and the storage medium may be transmitted to the information processing device. May be transmitted by any other method, such as insertion into the

  The information processing apparatus that has received the image restores the shape of the subject by a conventional method such as SFM (Structure from Motion) for each imaging apparatus (Step S24), and combines the two restored shapes last (Step S24). S25). As a result, the portions B for which information is insufficient are complemented with each other, and the shape of the entire imaging range A is generated.

  In the above, the shape generation system including the transmitter 50, the unmanned aerial vehicle 100, and the information processing device has been described, but the configuration is not actually limited to these configurations. The number of imaging devices provided in the unmanned aerial vehicle 100 is not limited to two, and may be provided more.

  In each of the above embodiments, the process executed by the transmitter 50 may be executed by any other type of mobile platform or information processing device. Further, these processes may be executed by the unmanned aerial vehicle 100 itself.

  The processing executed by the unmanned aerial vehicle 100 in each of the above embodiments may be executed by a moving object having any other imaging function.

  In each of the above embodiments, the process executed by the information processing device may be executed by another information processing device such as a smartphone or a tablet, or may be executed by the unmanned aerial vehicle 100 itself.

  The processing in the shape generation system according to each embodiment constitutes the shape generation method of the present disclosure as a whole, and the processing executed by the transmitter 50 constitutes the shape generation method in the mobile platform according to the present disclosure. The processing to be executed constitutes an image acquisition method in a flying object for shape generation according to the present disclosure.

  Among the processes (steps) in the shape generation system, the process (step) performed by the transmitter is performed in the transmitter processing unit 61 of the transmitter 50, and the process (step) performed by the drone is performed by the UAV of the unmanned aerial vehicle 100. The processing may be executed in the processing unit 110.

  At this time, the transmitter processing unit 61 may cause the transmitter 50, which is a computer, to execute a program for executing a process (step) executed by the transmitter in the shape generation method. This program may be stored in the memory 64 or another storage medium.

  Similarly, the UAV processing unit 110 may execute a program that causes the unmanned aerial vehicle 100, which is a computer, to execute a process (step) executed by the drone in the shape generation method. This program may be stored in the memory 160 or another storage medium.

  According to the shape generation method, the image acquisition method, the mobile platform, the flying object, the program, and the recording medium according to the present disclosure, it is possible to generate a high-precision shape even in an environment with a large difference in brightness, regardless of the dynamic range of the imaging device. Become.

  According to the shape generation method, the image acquisition method, the mobile platform, the flying object, the program, and the recording medium according to the present disclosure, the flying object can be imaged while moving without stopping on the flight path, The photographing time can be reduced and power consumption can be reduced.

  Further, according to the shape generation method, the image acquisition method, the mobile platform, the flying object, the program, and the recording medium according to the present disclosure, the shape can be formed with high efficiency even when the number of waypoints is large. For example, when there are two imaging devices as in each of the above-described embodiments, it is only necessary to perform the restoration operation twice and then perform the combining operation once. There is no need to combine images each time.

  As described above, the present disclosure has been described using the embodiments, but the technical scope of the present invention according to the present disclosure is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above-described embodiment. It is apparent from the description of the claims that embodiments with such changes or improvements can be included in the technical scope of the present invention.

  The execution order of each processing such as operation, procedure, step, and step in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before”, “before”. And the like, and can be realized in any order as long as the output of the previous process is not used in the subsequent process. Even if the operation flow in the claims, the specification, and the drawings is described using “first”, “next”, and the like for convenience, it means that it is essential to implement in this order is not.

Reference Signs List 10 shape generation system 50 transmitter 61 transmitter processing unit 63 wireless communication unit 64 memory 100 unmanned aerial vehicle 102 UAV main body 110 UAV processing unit 150 communication interface 160 memory 220-1 imaging device 220-2 imaging device

Claims (36)

Obtaining information on a plurality of imaging positions of a flying object having a plurality of imaging devices;
From the plurality of imaging devices, selecting an imaging device to be used for imaging for each of the plurality of imaging positions,
Imaging by the selected imaging device at each imaging position;
For each of the imaging devices, restoring the shape of the subject based on the captured image,
Synthesizing the restored shape for each imaging device,
The step of selecting the imaging device,
Based on a ratio of a portion occupied by a predetermined amount of light or less in an imaging region at the imaging position, including selecting at least one imaging device from the plurality of imaging devices.
Shape generation method. Selecting the at least one imaging device,
If the ratio is equal to or less than a first threshold, selecting a first imaging device;
When the ratio exceeds a second threshold that is larger than the first threshold, selecting a second imaging device for which an exposure parameter higher than the first imaging device is set;
If the ratio is greater than the first threshold and less than or equal to the second threshold, selecting both the first imaging device and the second imaging device.
The shape generation method according to claim 1. Obtaining information on a plurality of imaging positions of a flying object having a plurality of imaging devices;
Selecting, from the plurality of imaging devices, an imaging device to be used for imaging for each of the plurality of imaging positions; information on the plurality of imaging positions; and information on the selected imaging device for each imaging position. Transmitting to the flying vehicle;
In a shape generation method on a mobile platform having
The step of selecting the imaging device,
Based on a ratio of a portion occupied by a predetermined amount of light or less in an imaging region at the imaging position, including selecting at least one imaging device from the plurality of imaging devices.
Shape generation method. Selecting the at least one imaging device,
If the ratio is equal to or less than a first threshold, selecting a first imaging device;
When the ratio exceeds a second threshold that is larger than the first threshold, selecting a second imaging device for which an exposure parameter higher than the first imaging device is set;
If the ratio is greater than the first threshold and less than or equal to the second threshold, selecting both the first imaging device and the second imaging device.
The shape generation method according to claim 3. Obtaining an image taken by the selected imaging device at each imaging position from the flying object;
For each of the imaging devices, restoring the shape of the subject based on the captured image,
Synthesizing the restored shape for each of the imaging devices, further comprising:
The shape generation method according to claim 3. The step of selecting the imaging device,
At least one of the plurality of imaging positions selects a plurality of imaging devices,
The shape generation method according to claim 3. The step of selecting the imaging device,
Obtaining information on the irradiation angle of the light source;
Obtaining information on the obstruction of the light source at each of the imaging positions;
Estimating, based on information about the irradiation angle of the light source and information about the obstruction of the light source at each of the imaging positions, a ratio occupied by a portion having a predetermined light amount or less in an imaging region at the imaging position. ,
The shape generation method according to any one of claims 3 to 6. The step of acquiring information on the irradiation angle of the light source,
Obtaining time information and geographic information of the imaging position;
Using the time information and the geographic information, estimating information about the irradiation angle of the light source,
The shape generation method according to claim 7. Obtaining information on a plurality of imaging positions of a flying object having a plurality of imaging devices;
Obtaining information on an imaging device used for imaging for each of the plurality of imaging positions selected from the plurality of imaging devices;
For each of the plurality of imaging positions, imaging using an imaging device corresponding to the information on the acquired imaging device used for the imaging,
In the image acquisition method in a flying object for shape generation having
The information on the imaging device used for the imaging is generated based on a ratio of a portion of a predetermined amount of light or less in an imaging region at each imaging position, in accordance with selection of at least one imaging device from the plurality of imaging devices. Information,
Image acquisition method. The step of acquiring information on the imaging device used for the imaging,
For each of the plurality of imaging positions, including receiving information about an imaging device used for the imaging from a mobile platform,
The image acquisition method according to claim 9. The step of acquiring information on the imaging device used for the imaging,
For each of the imaging positions,
Measuring the amount of light in the imaging area at the imaging position during imaging,
Estimating, based on the light amount, a ratio occupied by a portion having a predetermined light amount or less in an imaging region at the imaging position;
Selecting at least one imaging device from the plurality of imaging devices based on the ratio, and generating information on the imaging device used for the imaging,
The image acquisition method according to claim 9. The step of acquiring information on the imaging device used for the imaging,
For each of the imaging positions,
Obtaining information on the irradiation angle of the light source;
Obtaining information on the obstacle of the light source at the imaging position;
Based on the information on the irradiation angle of the light source and the information on the obstruction of the light source at the imaging position, estimating a ratio occupied by a portion having a predetermined light amount or less in the imaging region at the imaging position,
Based on the ratio, selecting at least one imaging device from the plurality of imaging devices to generate information on the imaging device used for the imaging,
The image acquisition method according to claim 9. The step of acquiring information on the irradiation angle of the light source,
Obtaining time information and geographic information of the imaging position;
Using the time information and the geographic information, estimating information about the irradiation angle of the light source,
The image acquisition method according to claim 12. Information on the imaging device used for the imaging,
If the ratio occupied by a portion having a predetermined light amount or less in the imaging area at each imaging position is equal to or less than the first threshold, the first imaging device is selected, and the second ratio is higher than the first threshold. Is greater than the first imaging device, a second imaging device set with a higher exposure parameter than the first imaging device is selected, and if the ratio is greater than the first threshold and less than or equal to the second threshold, Both the first imaging device and the second imaging device are selected and generated,
The image acquisition method according to any one of claims 9 to 13. Information on the imaging device used for the imaging,
At least one of the plurality of imaging positions is generated by selecting a plurality of imaging devices,
The image acquisition method according to any one of claims 9 to 14. The step of imaging using the imaging device,
During the movement of the flying object, for each of the plurality of imaging positions, including imaging using an imaging device identified by information on the imaging device used for the acquired imaging,
The image acquisition method according to claim 9. The method further includes transmitting images captured at the plurality of image capturing positions to an information processing device that performs the shape generation.
The image acquisition method according to any one of claims 9 to 16. In a mobile platform having a communication unit that performs communication with an air vehicle and a processing unit,
The processing unit includes:
Obtain information on a plurality of imaging positions of a flying object having a plurality of imaging devices,
From among the plurality of imaging devices, an imaging device used for imaging for each of the plurality of imaging positions is selected,
Information on the plurality of imaging positions, and information on the selected imaging device for each imaging position is transmitted to the flying object by the communication unit,
Selecting the imaging device,
For each imaging position,
Achieved by selecting at least one imaging device from the plurality of imaging devices, based on a ratio occupied by a portion having a predetermined light amount or less in an imaging region at the imaging position.
Mobile platform. The processing unit includes:
If the ratio is equal to or less than a first threshold, the first imaging device is selected. If the ratio exceeds a second threshold that is larger than the first threshold, an exposure parameter higher than the first imaging device is selected. Is selected, and if the ratio is greater than the first threshold and less than or equal to the second threshold, both the first imaging device and the second imaging device are selected. Select the
The mobile platform according to claim 18. The processing unit further includes:
Acquiring an image captured by the selected imaging device at each imaging position from the flying object,
For each imaging device, restore the shape of the subject based on the captured image,
Synthesizing the shape restored for each of the imaging devices,
A mobile platform according to claim 18 or claim 19. The processing unit includes:
At least one of the plurality of imaging positions selects a plurality of imaging devices,
A mobile platform according to any one of claims 18 to 20. The processing unit includes:
Get information about the illumination angle of the light source,
Obtaining information on the obstacle of the light source at each of the imaging positions,
Based on the information on the irradiation angle of the light source and the information on the obstruction of the light source at each of the imaging positions, estimate a ratio occupied by a portion having a predetermined light amount or less in an imaging region at the imaging position.
A mobile platform according to any one of claims 18 to 21. The processing unit includes:
Obtaining time information and geographic information of the imaging position,
Using the time information and the geographic information, estimating information on the irradiation angle of the light source,
A mobile platform according to claim 22. A storage unit, a communication unit that performs communication with the mobile platform, a processing unit, and a plurality of imaging devices, in a flying object having,
The processing unit includes:
Acquiring information on a plurality of imaging positions of the flying object,
From among the plurality of imaging devices, information on the imaging device used for imaging for each of the plurality of imaging positions is obtained,
For each of the plurality of imaging positions, an image is taken using an imaging device corresponding to the acquired information on the imaging device used for the imaging,
The information on the imaging device used for the imaging is generated based on a ratio of a portion of a predetermined amount of light or less in an imaging region at each imaging position, in accordance with selection of at least one imaging device from the plurality of imaging devices. Information,
Flying object. The processing unit includes:
For each of the plurality of imaging positions, receiving information on the imaging device used for the imaging from a mobile platform,
An air vehicle according to claim 24. Further comprising an optical sensor,
The processing unit includes:
For each of the imaging positions,
Measure the amount of light in the imaging area at the imaging position at the time of imaging,
Based on the light amount, estimate the proportion of a portion of the imaging area at the imaging position that is equal to or less than a predetermined light amount,
Based on the ratio, select at least one imaging device from the plurality of imaging devices, generate information about the imaging device used for the imaging,
An air vehicle according to claim 24. The processing unit includes:
For each of the imaging positions,
Get information about the illumination angle of the light source,
Obtaining information about the obstacle of the light source at the imaging position,
Based on the information on the irradiation angle of the light source and the information on the obstacle of the light source at the imaging position, estimate the proportion occupied by a portion having a predetermined light amount or less in the imaging region at the imaging position,
Based on the ratio, select at least one imaging device from the plurality of imaging devices to generate information about the imaging device used for the imaging,
An air vehicle according to claim 24. The processing unit includes:
Obtaining time information and geographic information of the imaging position,
Using the time information and the geographic information, estimating information on the irradiation angle of the light source,
A flying object according to claim 27. Information on the imaging device used for the imaging,
If the ratio occupied by a portion having a predetermined light amount or less in the imaging area at each imaging position is equal to or less than the first threshold, the first imaging device is selected, and the second ratio is higher than the first threshold. Is greater than the first imaging device, a second imaging device set with a higher exposure parameter than the first imaging device is selected, and if the ratio is greater than the first threshold and less than or equal to the second threshold, Both the first imaging device and the second imaging device are selected and generated,
A flying object according to any one of claims 24 to 28. Information on the imaging device used for the imaging,
At least one of the plurality of imaging positions is generated by selecting a plurality of imaging devices,
A flying object according to any one of claims 24 to 29. The processing unit includes:
During the movement of the flying object, for each of the plurality of imaging positions, an image is captured using an imaging device specified by information on the imaging device used for the acquired imaging,
The flying object according to any one of claims 24 to 30. The processing unit includes:
Transmitting an image captured at the plurality of imaging positions to an information processing apparatus;
The flying object according to any one of claims 24 to 31. For mobile platforms that are computers,
Obtaining information on a plurality of imaging positions of a flying object having a plurality of imaging devices;
Selecting, from the plurality of imaging devices, an imaging device to be used for imaging for each of the plurality of imaging positions; information on the plurality of imaging positions; and information on the selected imaging device for each imaging position. Transmitting to the flying vehicle, and
The step of selecting the imaging device,
For each imaging position,
Based on a ratio of a portion occupied by a predetermined amount of light or less in an imaging region at the imaging position, the method includes selecting at least one imaging device from the plurality of imaging devices.
program. For mobile platforms that are computers,
Obtaining information on a plurality of imaging positions of a flying object having a plurality of imaging devices;
From the plurality of imaging devices, selecting an imaging device to be used for imaging for each of the plurality of imaging positions,
Transmitting the information on the plurality of imaging positions and the information on the selected imaging device for each imaging position to the flying object,
The step of selecting the imaging device,
For each imaging position,
Based on a ratio occupied by a portion equal to or less than a predetermined amount of light in an imaging region at the imaging position, including a step of selecting at least one imaging device from the plurality of imaging devices, storing a program,
Storage medium. A flying object having a plurality of imaging devices, which is a computer,
Obtaining information on a plurality of imaging positions of the flying object;
Obtaining information on an imaging device used for imaging for each of the plurality of imaging positions selected from the plurality of imaging devices;
For each of the plurality of imaging positions, imaging using an imaging device corresponding to the acquired information on the imaging device used for the imaging, and
The information on the imaging device used for the imaging is generated based on a ratio of a portion of a predetermined amount of light or less in an imaging region at each imaging position, in accordance with selection of at least one imaging device from the plurality of imaging devices. Information,
program. A flying object having a plurality of imaging devices, which is a computer,
Obtaining information on a plurality of imaging positions of the flying object;
Obtaining information on an imaging device used for imaging for each of the plurality of imaging positions selected from the plurality of imaging devices;
For each of the plurality of imaging positions, imaging using an imaging device corresponding to the acquired information on the imaging device used for the imaging,
The information on the imaging devices used for the imaging is generated based on a ratio of a portion of a predetermined light amount or less in an imaging region at each imaging position, in accordance with selection of at least one imaging device from the plurality of imaging devices. Storing information, programs,
Storage medium.

Images