JP2017201757A - Image acquisition system, image acquisition method, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for accurately compositing images acquired by a plurality of unmanned aircrafts, and an apparatus using the same.SOLUTION: There is provided an image acquisition system comprising: a first unmanned aircraft that includes a first camera; a second unmanned aircraft that includes a second camera; a commanding device that includes a third camera; and an image processing apparatus that performs composition processing of compositing a first image obtained by photographing an imaging object with the first camera and a second image obtained by photographing the imaging object with the second camera, wherein the commanding device acquires a third image by photographing the first unmanned aircraft and the second unmanned aircraft photographing the imaging object, and the image processing apparatus detects the respective attitudes of the first unmanned aircraft and the second unmanned aircraft on the basis of the third image, and performs processing of correcting the difference between the attitudes in performing the composition processing of compositing the first image and the second image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image acquisition method using an unmanned aerial vehicle (UAV (Unmanned aerial vehicle)).

  An inspection method using an infrared camera is widely used for maintenance of buildings such as building walls and piers. This inspection method makes it possible to inspect a wide area more quickly than conventional hammering sound inspections by visualizing the temperature difference between a normal part and cavities inside the concrete or scratches on the surface (abnormal part). It is a feature.

  On the other hand, UAV has made remarkable progress in recent years with the technological advancement of sensors and batteries, and products and systems that are integrated with various fields have been developed. The building maintenance field mentioned above is no exception.

Japanese Patent No. 5697592 JP2015-119372A

  Patent Document 1 proposes a system in which a camera capable of acquiring an infrared image is mounted on a UAV and the UAV is used for inspection of a structure. In the inspection using the above-described infrared camera, the infrared image and the visible light image are compared to determine the abnormal part. This discrimination is facilitated by superimposing (combining) the infrared image and the visible light image. At this time, by acquiring two types of images at the same time, it is possible to generate a synthesized image with higher accuracy in a short time. Although it is desirable to mount two types of cameras on one UAV for the above-mentioned simultaneous acquisition, it is difficult to realize at present due to restrictions on the amount of UAV mounted and the drive power supply. As a configuration that can be easily realized, two different cameras are mounted on two UAVs, and simultaneous imaging is performed, whereby the above-described two types of images can be simultaneously acquired. However, in this configuration, the positional relationship and orientation of the two cameras are not fixed, and the camera always changes depending on the external environment such as airflow. For this reason, a device for improving the image synthesis accuracy is required.

  In Patent Document 2, images of a structure are acquired by a plurality of cameras, and image alignment is performed based on feature points in the images or reference markers. Image alignment using such feature points is generally performed, and in a situation where the camera is fixed, alignment can be performed with high accuracy. However, in this method, when there is a change in the position of the camera described above, there is a problem in that the accuracy is lowered or the image processing time is increased. Further, if a marker is used, it is necessary to install the marker on the imaging target, which is disadvantageous in terms of work time and cost.

  As another method, there is a method of correcting an image based on the positional relationship and orientation of each other's cameras when the image is acquired. As a method for obtaining the positional relationship and orientation of the camera, it is conceivable to use sensors mounted on the UAV, but the response speed and accuracy of the sensor are problematic. In general UAVs, GPS (Global Positioning System) is often used outdoors, and barometric pressure sensors are often used for altitude. However, the accuracy of GPS is on the order of 1 m, and the pressure sensor is also on the order of 0.1 m, which is insufficient for use in image correction.

  An object of the present invention is to provide a method for accurately synthesizing images acquired by a plurality of unmanned aerial vehicles and an apparatus using the method.

  According to a first aspect of the present invention, there is provided a first unmanned aerial vehicle having a first camera, a second unmanned aerial vehicle having a second camera, a command device having a third camera, and the first camera. An image processing apparatus that performs a synthesis process for synthesizing a first image obtained by imaging an imaging target and a second image obtained by imaging the imaging target with the second camera; In the image acquisition system, the command device acquires the third image by imaging the first unmanned aircraft and the second unmanned aircraft that images the imaging target with the third camera. Then, the image processing device detects the attitudes of the first unmanned aircraft and the second unmanned aircraft based on the third image, and combines the first image and the second image. The difference in posture when processing Providing an image acquisition system which is characterized in that a positive processes.

  According to a second aspect of the present invention, an image is obtained using a first unmanned aerial vehicle having a first camera, a second unmanned aerial vehicle having a second camera, a third camera, and an image processing device. An image acquisition method for acquiring, wherein the first unmanned aircraft acquires a first image by imaging an imaging target with the first camera, and the second unmanned aircraft is configured to acquire the second image. Capturing a second image by imaging the imaging target with the camera, and the third camera imaging the first unmanned aircraft and the second unmanned aircraft that capture the imaging target. A step of acquiring a third image, and the image processing device detects the attitudes of the first unmanned aircraft and the second unmanned aircraft based on the third image, and 1 image and the second To provide an image acquisition method characterized by comprising the steps of: performing processing for correcting the difference in the position when performing composition processing of the image.

  According to a third aspect of the present invention, the first unmanned aerial vehicle acquires the first image data obtained by imaging the imaging target with the first camera; A step of acquiring data of a second image obtained by imaging the imaging object by a camera; and the first unmanned aircraft and the second unmanned aircraft imaging the imaging object by a third camera. Acquiring the data of the third image obtained by imaging, detecting the attitudes of the first unmanned aircraft and the second unmanned aircraft based on the third image, and And a step of correcting the difference in posture when performing the synthesis process of the second image and the second image.

  According to a fourth aspect of the present invention, there is provided a program characterized by causing a processor to execute each step of the image processing method.

  ADVANTAGE OF THE INVENTION According to this invention, the method and apparatus using the same which synthesize | combine the image acquired with several unmanned aircraft accurately can be provided.

Overall view of UAV image acquisition system according to the first embodiment Block diagram of a UAV image acquisition system according to the first embodiment Sequence chart of image acquisition process according to first embodiment Overall view of UAV image acquisition system according to second embodiment Sequence chart of image acquisition process according to second embodiment

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each embodiment exemplifies a UAV image acquisition system in which a plurality of unmanned aerial vehicles (UAVs) each captures an imaging target and generates a composite image from images obtained by each UAV.

[First embodiment]
<Overall configuration>
FIG. 1 is a schematic diagram illustrating a schematic device configuration of a UAV image acquisition system according to the first embodiment of the present invention. Detailed image acquisition and processing sequences will be described later.

  The UAV image acquisition system includes an infrared camera mounted UAV 101 (hereinafter referred to as UAV 101), a visible light camera mounted UAV 102 (hereinafter referred to as UAV 102), a command device 103, an image processing device 104, a computer 105, and a display device 106. The image processing apparatus 104 is incorporated in the computer 105 as a dedicated processing board. As an example, the computer 105 uses a personal computer (PC), and the display device 106 uses a liquid crystal display. The image processing apparatus 104 includes an image processing dedicated processor (FPGA, ASIC, etc.), a memory, and an image processing board having various I / Os. The function of the image processing apparatus 104 described later is required for the processor. This is realized by executing a simple program.

  In this embodiment, the system configuration shown in FIG. 1 is adopted, but the configuration of the present invention is not limited to this. For example, the image processing device 104 may be incorporated in the command device 103, or a notebook PC, a smartphone, a tablet terminal, or the like in which the computer 105 and the display device 106 are integrated may be used. Alternatively, all functions of the command device 103, the image processing device 104, the computer 105, and the display device 106 may be configured by one device. Alternatively, some functions may be executed by a server (such as a cloud server) on the network.

  The UAV 101 acquires image data in the infrared region, and the UAV 102 acquires image data in the visible light region. Image data acquisition timing (timing to release the shutter) is controlled by a trigger signal described later. Image data acquired by the UAV is referred to as “original image data”. The original image data acquired by the UAV 101 is referred to as “infrared original image data”, and the original image data acquired by the UAV 102 is referred to as “visible light original image data”. The infrared original image data and the visible light original image data are subjected to synthesis processing by an image processing device 104 described later, and are displayed on the display device 106 as an image.

  It should be noted that the UAV 101 and UAV 102 can use the same aircraft except that the mounted cameras are different. However, in order to identify the UAV, it is preferable to apply different coatings and markings (unless the shape of the mounted camera is greatly different). The UAV to be used is capable of automatic navigation by GPS or wireless remote control in the same manner as commonly used. Further, it has a function of receiving a trigger signal, which will be described later, and capturing an image, and a function of transmitting the acquired image to the image processing apparatus wirelessly.

<Command device>
The configuration of the command device 103 will be described with reference to FIG. FIG. 2 is a schematic diagram showing a schematic configuration of the command device and the image processing device.

The command device 103 transmits a trigger signal for starting imaging to the UAV 101 and the UAV 102. The command device 103 acquires image data (posture image data) for calculating the postures of the UAV 101 and the UAV 102, receives original image data acquired by the UAV 101 and the UAV 102, and transfers them to the image processing device 104.
The command device 103 includes a command control system 201, an imaging unit 202, a distance measuring unit 203, a pan head unit 204, and a data transmission / reception unit 205.

  The command control system 201 exchanges commands with the imaging unit 202, the distance measurement unit 203, the pan head unit 204, and the data transmission / reception unit 205, which will be described later, and data with the image processing control system 206.

  The imaging unit 202 includes an imaging module and an imaging processing control system. The imaging module has an optical system (lens, diaphragm, etc.), a lens driving mechanism for focusing the optical system, and an imaging sensor. The imaging processing control system is a circuit that controls each part of the imaging module. The imaging processing control system controls the imaging module based on a command from the command control system 201 to capture the UAVs 101 and 102, and controls image processing of the image data (posture image data) in which the UAVs 101 and 102 are captured through the command control system 201. Send to system 206. The posture image data is image data used for the purpose of detecting the posture (position and orientation) of each of the two UAVs 101 and 102. Therefore, any image data in which the two UAVs 101 and 102 are clearly shown to the extent that the identification of the aircraft and the detection of the posture (position and orientation) are possible can be used as the posture image data.

  The distance measuring unit 203 includes a distance measuring sensor that measures the distance to the object, and a distance measurement processing control system that controls the distance measuring sensor. The method of the distance measuring sensor is not limited, but in the present embodiment, a laser distance measuring sensor that measures the distance by TOF (Time of Flight) of laser light is used. The distance measurement processing control system drives a distance measurement sensor based on a command from the command control system 201 to measure the distance between the command device 103 and each UAV, and performs image processing on the acquired distance data through the command control system 201. This is sent to the control system 206.

  The pan head unit 204 includes a pan head driving mechanism in which the imaging unit 202 and the distance measuring unit 203 are installed, and a driving control system thereof. The drive control system drives the drive mechanism based on a command from the command control system 201 to control the azimuth angle and elevation angle of the camera platform, and directs the imaging unit 202 and the distance measurement unit 203 in appropriate directions.

  The data transmission / reception unit 205 transmits / receives data to / from the UAV 101 and the UAV 102 based on a command from the command control system 201. The data transmission / reception unit 205 also transmits information necessary for the UAVs 101 and 102 to capture, such as the position data of the UAV imaging target and the imaging start trigger signal described later. Also, the original image data received from each UAV 101 and 102 is sent to the image processing control system 206 through the command control system 201.

<Image processing device>
The image processing device 104 calculates correction parameters for appropriately combining the original image data from the UAV posture image data and distance data acquired by the command device 103. Further, the image processing apparatus 104 performs a composition process of the original image data using the correction parameter, and generates display data for displaying the image data subjected to the composition process on the display apparatus 106. Further, the image processing device 104 acquires information to be transmitted to the UAV through the command device 103 such as the position data of the imaging target of each UAV from the user through the computer 105.

  As shown in FIG. 2, the image processing apparatus 104 includes an image processing control system 206, a UAV detection unit 207, an attitude calculation unit 208, a correction parameter calculation unit 209, a composition processing unit 211, a development processing unit 212, and a display data generation unit 213. And a user information acquisition unit 215. The UAV detection unit 207 to the correction parameter calculation unit 209 are collectively referred to as a correction parameter calculation unit 210, and the combination processing unit 211 to the display data generation unit 213 are collectively referred to as a composite image generation unit 214.

  The image processing control system 206 controls the correction parameter calculation unit 210, the composite image generation unit 214, and the user information acquisition unit 215, and exchanges data with the command device 103 (command control system 201).

  The correction parameter calculation unit 210 detects the postures of the UAVs 101 and 102 using the posture image data and the distance data acquired by the commanding device 103, and calculates correction parameters used for a synthesis process to be described later. The correction parameter calculation unit 210 includes a UAV detection unit 207, an attitude calculation unit 208, and a correction parameter calculation unit 209. The UAV detection unit 207 detects UAV features (paint color, marker, camera shape) in the posture image data, and identifies the UAV. In the present embodiment, the UAV detection unit 207 determines whether the UAV in the posture image data is the UAV 101 or the UAV 102 based on the feature. The posture calculation unit 208 detects the postures of the UAVs 101 and 102 identified by the UAV detection unit 207 and outputs posture information. The posture can be calculated by a known method using a feature point of a detection target (UAV) (eg, a method using Bag-of-Features expression). The correction parameter calculation unit 209 estimates the angle and distance of the camera mounted on the UAV from the posture information and distance data, and calculates correction parameters for generating a composite image.

The composite image generation unit 214 generates image data to be displayed on the display device 106 using the original image data acquired by each of the UAVs 101 and 102 and the posture image data and distance data acquired by the command device 103. The composite image generation unit 214 includes a composite processing unit 211, a development processing unit 212, and a display data generation unit 213. Based on the correction parameter calculated by the correction parameter calculation unit 209, the composition processing unit 211 performs correction processing on the original image data acquired by the UAVs 101 and 102, and then performs image composition processing to generate composite image data. The correction process is a process for adjusting the position, orientation, size, and the like of the two original image data, and includes, for example, a deformation process such as affine transformation, a rotation process, and a trimming process. The development processing unit 212 converts each pixel value of the composite image data into XYZ chromaticity coordinate values. This converted image data is referred to as XYZ image data. In addition, digital filter processing such as noise removal, JPEG (Joint Photographic Experts if necessary)
(Group) or the like. The display data generation unit 213 converts the XYZ image data generated by the development processing unit 212 into RGB color system image data that can be displayed on the display device 106 using a lookup table.

  The user information acquisition unit 215 acquires user information from the storage device in the computer 105. The user information includes, for example, UAV body setting information (imaging target, flight route / schedule to imaging target, return location, etc.), information on a camera mounted on the UAV (view angle, focal length, etc.), and the like. The machine setting information and camera information are input in advance by the user via a UI or the like, and held in a storage device in the computer 105. The machine setting information acquired by the user information acquisition unit 215 is sent to the UAVs 101 and 102 through the image processing control system 206, the command control system 201, and the data transmission / reception unit 205. Unless otherwise specified, communication between the command device 103 and the UAVs 101 and 102 is performed via the data transmission / reception unit 205.

<UAV control and imaging processing>
Detailed steps for obtaining composite image data in the UAV 101 and UAV 102, the command device 103, and the image processing device 104 according to this embodiment will be described with reference to the sequence chart of FIG.

In step S301, the user information acquisition unit 215 acquires the above-described user information (machine setting information, camera information, etc.). The acquired user information is sent to the command device 103 through the image processing control system 206.

  In step S302, the command device 103 initializes the UAV 101 and UAV 102 (via the data transmission / reception unit 205) based on the machine setting information acquired in step S301. Specifically, information such as an imaging target, an imaging position (a UAV position when imaging the imaging target), a flight route, a flight schedule, a period place, and the like are transmitted to each UAV 101 and 102 and set.

  In step S303, the UAV 101 and UAV 102 are moved to the imaging position based on the setting in step S302. The movement to the imaging position is preferably performed by automatic navigation using GPS or the like. After the movement, each UAV 101, 102 finely adjusts the position and orientation of the own device so that the imaging target is within the angle of view of the mounted camera. When the subject to be imaged falls within the angle of view, each UAV 101, 102 sends a notification of movement completion to the command device 103.

  In step S304, the command device 103 prepares for UAV imaging (posture image data acquisition) and distance measurement (distance data acquisition) in the subsequent stage. Specifically, the command control system 201 controls the pan head unit 204 so that the two UAVs 101 and 102 at the imaging position (in the air) are within the angle of view of the imaging unit 202, and the command device 103 (imaging unit 202 And the direction of the distance measuring unit 203) is changed.

  The determination of whether or not the UAV is within the angle of view may be performed by any method. For example, the calculation may be performed theoretically (geometrically) from the imaging position set in each of the UAVs 101 and 102 in step S302, the installation position of the command device 103, and the angle of view of the imaging unit 202. Alternatively, the existence position of each UAV 101, 102 is calculated from the GPS information of each UAV, and geometrically calculated from the existence position of each UAV 101, 102, the installation position of the command device 103, and the angle of view of the imaging unit 202 Good. Alternatively, object recognition may be performed on the image acquired by the imaging unit 202 to determine whether or not the two UAVs 101 and 102 are included in the image.

  When the UAV imaging preparation (step S304) and the movement of the UAVs 101 and 102 (step S303) are completed, the process proceeds to the next step S305. In step S <b> 305, the command device 103 transmits a trigger signal for instructing imaging to each of the UAVs 101 and 102. Based on this trigger signal, the process of step S306 by each of the UAVs 101 and 102 and the process of step S307 by the command device 103 are executed in synchronization (substantially simultaneously).

  In step S306, each of the UAV 101 and the UAV 102 performs imaging of an imaging target (acquisition of original image data). Imaging is performed by releasing the shutter of the camera simultaneously with receiving the trigger signal. The image acquisition time (time stamp) is recorded in the header of the acquired original image data. In the sequence of the present embodiment, imaging is performed once. However, when the imaging is performed a plurality of times, a unique serial number (unique serial number) is recorded for each UAV in the header of the original image data. The reason why the image acquisition time and serial number are recorded is to perform pairing of original image data acquired simultaneously with different UAVs. After imaging, each UAV 101, 102 transmits the acquired original image data to the command device 103. Thereby, infrared original image data and visible light original image data are acquired about one imaging object.

In step S307, simultaneously with the trigger signal transmission in step S305, the command device 103 performs UAV imaging (acquisition of posture image data) and distance measurement (acquisition of distance data). The posture image data is acquired by the imaging unit 202, and the distance data is acquired by the distance measuring unit 203. Thereby, when UAV101 imaged the imaging target (step S306),
The position of the UAV 101 (three-dimensional position in the air) and the posture (orientation) of the UAV 101 with respect to the imaging target are recorded. Similarly, when the UAV 102 images the imaging target (step S306), the position of the UAV 102 (three-dimensional position in the air) and the posture (orientation) of the UAV 102 with respect to the imaging target are recorded. In order to make correspondence between the original image data, the posture image data, and the distance data, a time stamp or a unique serial number may be recorded in the posture image data and the distance data.

  In step S308, the UAVs 101 and 102 are returned to a predetermined location. The timing of the return may be based on an instruction from the command device 103, or may be determined by each UAV 101, 102 based on the aircraft setting information transmitted to each UAV 101, 102. The movement is preferably performed by automatic navigation using GPS or the like as in step S303.

  In step S 309, the command control system 201 transfers the original image data, attitude image data, and distance data acquired from each UAV 101, 102 to the image processing apparatus 104.

  In step S <b> 310, the correction parameter calculation unit 210 calculates correction parameters used for image composition of the original image data acquired from each of the UAVs 101 and 102. Specifically, the UAV detection unit 207 identifies UAV 101 and UAV 102 from the posture image data, and the posture calculation unit 208 detects the feature points of the UAV 101 and UAV 102 to estimate the posture. Then, the correction parameter calculation unit 209 determines the line-of-sight vector for the imaging target of the camera mounted on the UAV 101 based on the posture information and distance data of the UAV 101 (the camera position and distance relative to the imaging target, the direction of the optical axis of the camera, etc.). Is estimated. Further, the correction parameter calculation unit 209 estimates a line-of-sight vector with respect to an imaging target of the camera mounted on the UAV 102 based on the posture information and distance data of the UAV 102. The difference in the line-of-sight vector between the UAV 101 and the UAV 102 or information corresponding thereto is the correction parameter. The correction parameter is information for correcting a difference in attitude (camera coordinate system difference) between a plurality of UAVs (cameras), or a position / orientation between a plurality of images obtained by UAVs (cameras) having different attitudes. -It can be said that the information is used to correct the size.

  In step S <b> 311, the original image data acquired by the UAV 101 and the original image data acquired by the UAV 102 are combined to generate display data of the combined image data and output it to the display device 106. Specifically, the composition processing unit 211 confirms the time stamp of the original image data, and obtains the original image data acquired by the UAVs 101 and 102 at the same time (substantially simultaneously in consideration of signal, circuit delay, etc.). select. When imaging of the imaging target is performed a plurality of times, the unique serial number is also referred to. After that, the composition processing unit 211 performs image correction such as deformation or trimming on one or both of the original image data so that the line-of-sight vectors of the two original image data are matched based on the correction parameter. Then, the composition processing unit 211 generates a composite image by superimposing the two corrected original image data. The superposition may be performed by a known method such as alpha blend. After the composition processing, the composite image is converted into an image form that can be displayed on the display device 106 by the development processing unit 212 and the display data generation unit 213 and output.

<Advantages of the first embodiment>
As described above, in the present embodiment, different cameras are mounted on a plurality of UAVs, and images are simultaneously acquired by a trigger signal. Furthermore, by observing the posture and distance of the UAV at the time of imaging from the outside, it is possible to efficiently and accurately acquire a composite image of images acquired by different cameras without using a marker or the like.

[Second Embodiment]
<Overall configuration>
FIG. 4 is a schematic diagram illustrating a schematic configuration of a UAV image acquisition system according to the second embodiment of the present invention. The biggest difference between the present embodiment and the first embodiment is that the function of the command device 103 in the first embodiment is installed in the UAV and image acquisition is executed. In addition, in the first embodiment, the original image data is acquired by capturing a still image, whereas in the present embodiment, the moving image is captured by UAV, and the frame is extracted from the moving image data to capture the original image data. It differs in that it is acquired.

  The UAV image acquisition system according to the present embodiment has at least three UAVs: an infrared camera mounted UAV 401 (hereinafter referred to as UAV 401), a visible light camera mounted UAV 402 (hereinafter referred to as UAV 402), and a command UAV 403. The UAV image acquisition system includes an image processing device 404, a computer 405, a display device 406, and a data transmission / reception unit 407.

  The UAV 401 has the same function and configuration as the UAV 101, and the UAV 402 has the same function and configuration as the UAV 102, so that the description thereof is omitted. Since the image processing apparatus 404 has the same configuration as the image processing apparatus 104, the computer 405 has the same configuration as the computer 105, and the display apparatus 406 has the same configuration as the display apparatus 106, description thereof will be omitted.

  The command UAV 403 is a UAV having the same function as the command device 103 in the first embodiment, and is equipped with a camera for acquiring posture image data of the UAV 401 and UAV 402 and a distance measuring device for acquiring distance data. Yes. Further, the command UAV 403 has a function corresponding to the data transmission / reception unit 205 for performing data communication with the image processing apparatus 404 (not shown).

  The data transmission / reception unit 407 is connected to the image processing apparatus 404 and performs data transmission / reception with the UAV 401, UAV 402, and UAV 403. The data transmission / reception unit 407 is substantially the same as the data transmission / reception unit 205, and transmits information necessary for the UAV to perform imaging (position data to be imaged, etc.) and original image data received from the UAV as an image processing device 404. The function to send to.

<UAV control and imaging processing>
Detailed steps for obtaining composite image data in the UAV 401 and UAV 402, the command UAV 403, and the image processing apparatus 404 according to this embodiment will be described with reference to the sequence chart of FIG.

  Since the process of step S501 is the same as that of step S301 of the first embodiment, a description thereof will be omitted. However, in the present embodiment, three-dimensional CAD data (computer-aided design data) of UAV 401 and UAV 402 is acquired from a user or a storage device in computer 405 in addition to the machine setting information and camera information.

  In step S502, the image processing apparatus 404 initializes the UAV 401, the UAV 402, and the command UAV 403 via the data transmission / reception unit 407 based on the machine setting information acquired in step S501. Specifically, information such as an imaging target, an imaging position (a UAV position when capturing an imaging target or a posture image), a flight route, a flight schedule, a period place, and the like is transmitted to each UAV 401, 402, 403 and set. .

  In step S503, the UAV 401, UAV 402, and command UAV 403 are moved to the imaging position based on the setting in step S502. The movement to the imaging position is preferably performed by automatic navigation using GPS or the like. After the movement, the UAV 401 and the UAV 402 finely adjust their position and orientation so that the imaging target is within the angle of view of the mounted camera.

  In step S <b> 504, the command UAV 403 prepares for acquisition of subsequent-stage posture image data and distance data acquisition. Specifically, the position and orientation of the command UAV 403 are finely adjusted so that both UAVs 401 and 402 at the imaging position (in the air) fall within the angle of view of the camera mounted on the command UAV 403. The determination of whether or not the UAV is within the angle of view may be performed by any method such as the method described in the first embodiment.

  In step S505, the UAVs 401 and 402 each prepare for imaging of an imaging target. First, the UAV 401 and the UAV 402 finely adjust the positions so that the imaging target is within the angle of view of the mounted camera while hovering at the imaging position. Thereafter, the UAV 401 and the UAV 402 each transmit a notification of completion of imaging preparation to the command UAV 403 and acquire moving image data until a subsequent trigger signal is received.

  In step S506, the command UAV 403 transmits a trigger signal for instructing the UAVs 401 and 402 to perform imaging. Based on this trigger signal, the process of step S507 by each UAV 401, 402 and the process of step S508 by the command UAV 403 are executed synchronously (substantially simultaneously).

  In step S507, the UAV 401 and the UAV 402 each acquire original image data. The acquisition of the original image data is performed by extracting the frame image at the time when the trigger signal is received from the moving image data that has been continuously captured since step S505.

  In step S508, simultaneously with the trigger signal transmission in step S506, the command UAV 403 performs imaging (acquisition of posture image data) and ranging (acquisition of distance data) of UAV 401 and UAV 402. As a result, the UAV 401 position (three-dimensional position in the air) and the posture (orientation) of the UAV 401 with respect to the imaging target when the UAV 401 images the imaging target (step S507) are recorded. Similarly, the UAV 402 position (a three-dimensional position in the air) and the posture (orientation) of the UAV 402 with respect to the imaging target when the UAV 402 images the imaging target (step S507) are recorded. In order to make correspondence between the original image data, the posture image data, and the distance data, a time stamp or a unique serial number may be recorded in the posture image data and the distance data.

  In step S509, each UAV confirms the aircraft setting information set in step S502, and when the imaging schedule (imaging of all imaging targets) is completed, the UAV returns to a predetermined location. After the feedback, UAV 401 and UAV 402 transmit original image data, and command UAV 403 transmits attitude image data and distance data to image processing device 404.

  In step S510, as in step S310 in the first embodiment, the image processing apparatus 404 calculates correction parameters used for image composition of original image data. In the present embodiment, the image processing apparatus 404 performs identification of each UAV 401 and 402 and calculation of posture information based on the three-dimensional CAD data (three-dimensional shape information) acquired in step S501. For example, UAV identification and posture detection can be performed by obtaining various UAV posture shapes from 3D CAD data and comparing them with the UAV shapes reflected in the posture image data.

  Since the process of step S511 is the same as that of step S311 of 1st embodiment, description is abbreviate | omitted.

<Advantages of Second Embodiment>
As described above, in the present embodiment, the UAV also has a function for controlling imaging by the UAV, so that the composite image can be efficiently and highly accurately even under limited conditions (for example, a place where people cannot enter). Can be acquired.

[Other Embodiments]
In the first and second embodiments, the image processing apparatuses 104 and 404 are incorporated into the computers 105 and 405 as dedicated boards. However, similar functions may be realized by software executed on the computers 105 and 405. .
In the first and second embodiments, the UAV is moved by automatic navigation, but the user may move the UAV by remote control.

  In the first and second embodiments, distance measurement is performed using a laser beam to acquire distance data. However, other known distance measurement methods such as a method using an ultrasonic wave and a method using a distance finder are also available. The distance data may be acquired by a method. In addition, a relative distance may be calculated by image processing based on the size of the UAV and the structure or the like shown in the posture image data, and the distance data may be calculated based on this. In this case, since the distance can be measured by the imaging camera, a dedicated device such as a distance measuring sensor is not necessary.

In the first and second embodiments, the original image data, the posture image data, and the distance data are wirelessly transmitted to the image processing apparatus. However, the original image data, the attitude image data, and the distance data are temporarily stored in a storage device (for example, an SD memory card) installed in the UAV. The user may take out and read after returning.
In the first and second embodiments, the UAV is returned immediately after the end of imaging. However, the above-described imaging sequence may be performed by moving to the next imaging target position.
In the second embodiment, the function of the command UAV 403 may be installed in all the UAVs 401 to 403 so that the UAV that performs the function of the command UAV 403 can be changed according to the situation. In this case, for example, the imaging time can be shortened by setting the aircraft that has arrived at the imaging target last as the command UAV.

In the first and second embodiments, the process of capturing an infrared image and a visible light image with two UAVs and superimposing them is described. However, the number of UAVs, the type of image to be acquired, and the contents of the composition process are as follows. It doesn't matter. For example, imaging of an imaging target (acquisition of original image data) may be performed by three or more UAVs. A plurality of UAVs may acquire the same type of image, or may acquire a type of image other than an infrared image or a visible light image. Further, the composition process may be, for example, image joining (stitching), image difference, or the like in addition to image superposition. The present invention can be preferably applied if correction for aligning a plurality of images taken with different UAVs is necessary.
A configuration obtained by appropriately combining various techniques in the above embodiments also belongs to the category of the present invention.

101: UAV with infrared camera
102: UAV with visible light camera
103: Command device 104: Image processing device

Claims (13)

A first unmanned aerial vehicle having a first camera;
A second unmanned aerial vehicle having a second camera;
A commanding device having a third camera;
A synthesis process for synthesizing the first image obtained by imaging the imaging target with the first camera and the second image obtained by imaging the imaging target with the second camera is performed. An image processing device;
An image acquisition system comprising:
The command device captures the first unmanned aerial vehicle and the second unmanned aerial vehicle that images the imaging target by the third camera, thereby obtaining a third image;
The image processing device detects postures of the first unmanned aircraft and the second unmanned aircraft based on the third image, and performs the composition processing of the first image and the second image. An image acquisition system that performs a process of correcting the difference in posture when performing. A process in which the first unmanned aerial vehicle acquires the first image, a process in which the second unmanned aerial vehicle acquires the second image, and a process in which the command device acquires the third image; The image acquisition system according to claim 1, wherein the two are executed simultaneously. A process in which the first unmanned aerial vehicle acquires the first image, a process in which the second unmanned aerial vehicle acquires the second image, and a process in which the command device acquires the third image; The image acquisition system according to claim 1, wherein the image acquisition system is executed based on a trigger signal generated by the command device. The acquisition of the first image is performed when the first unmanned aircraft releases the shutter of the first camera based on the trigger signal,
The image acquisition system according to claim 3, wherein the acquisition of the second image is performed when the second unmanned aerial vehicle releases the shutter of the second camera based on the trigger signal. The acquisition of the first image is performed by extracting a frame image at a time based on the trigger signal from moving image data captured from the first camera,
The image acquisition system according to claim 3, wherein the acquisition of the second image is performed by extracting a frame image at a time based on the trigger signal from moving image data captured from the second camera. . The image according to any one of claims 1 to 5, wherein the process of correcting the difference in posture includes a process of aligning a position and an orientation of the first image and the second image. Acquisition system. The image acquisition system according to claim 1, wherein the command device is an unmanned aerial vehicle. The image acquisition system according to any one of claims 1 to 7, wherein an image acquisition time or a unique serial number is recorded in the first image and the second image. The image processing device detects feature points of the first unmanned aircraft and the second unmanned aircraft from the third image, respectively, and thereby detects each of the first unmanned aircraft and the second unmanned aircraft. The image acquisition system according to claim 1, wherein the posture is estimated. The image processing device includes the first unmanned aircraft and the second unmanned aircraft based on the third image and three-dimensional shape information of the first unmanned aircraft and the second unmanned aircraft. Each posture is estimated, The image acquisition system of any one of Claims 1-8 characterized by the above-mentioned. A first unmanned aerial vehicle having a first camera;
A second unmanned aerial vehicle having a second camera;
A third camera,
An image processing device;
An image acquisition method for acquiring an image using
The first unmanned aerial vehicle acquiring a first image by imaging an imaging target with the first camera;
The second unmanned aerial vehicle acquiring the second image by imaging the imaging object with the second camera;
The third camera acquiring a third image by imaging the first unmanned aerial vehicle and the second unmanned aerial vehicle that images the imaging target; and
The image processing device detects the attitudes of the first unmanned aircraft and the second unmanned aircraft based on the third image, and performs a composition process of the first image and the second image. Performing a process of correcting the difference in posture,
An image acquisition method comprising: Obtaining data of a first image obtained by a first unmanned aerial vehicle imaging an imaging target with a first camera;
Obtaining data of a second image obtained by a second unmanned aerial vehicle imaging the imaging target with a second camera;
Obtaining data of a third image obtained by imaging the first unmanned aerial vehicle and the second unmanned aerial vehicle that images the imaging target with a third camera;
The attitude difference between the first unmanned aircraft and the second unmanned aircraft is detected based on the third image, and the first image and the second image are combined. Performing a process of correcting
An image processing method comprising: A program causing a processor to execute each step of the image processing method according to claim 12.

Images