WO2018185940A1

WO2018185940A1 - Imaging control device, imaging device, imaging system, mobile body, imaging control method and program

Info

Publication number: WO2018185940A1
Application number: PCT/JP2017/014560
Authority: WO
Inventors: 明邵; 本庄　謙一
Original assignee: エスゼットディージェイアイテクノロジーカンパニーリミテッド
Priority date: 2017-04-07
Filing date: 2017-04-07
Publication date: 2018-10-11
Also published as: JP6503607B2; JPWO2018185940A1

Abstract

This imaging control device may be provided with a derivation unit which derives the spatial frequency band of a captured image. The imaging control device may be provided with a control unit which, on the basis of the spatial frequency band of the image derived by the derivation unit, controls the positional relation between the imaging surface and the lens by selecting one of, or by combining both, first AF processing, in which autofocus processing is performed on the basis of the amount of blur in multiple images captured in states with different positional relations between the imaging surface and the lens, and second AF processing, in which autofocus processing is performed with a phase difference method.

Description

Imaging control apparatus, imaging apparatus, imaging system, moving object, imaging control method, and program

The present invention relates to an imaging control device, an imaging device, an imaging system, a moving body, an imaging control method, and a program.

An autofocus process (AF process) using two parallax images and an AF process using a plurality of images with different amounts of blur captured with different shooting parameters are known.
Patent Document 1 Japanese Patent No. 5932476

Challenges to be solved

According to the AF processing using the blur amount of the image, it is necessary to capture at least two images, and the AF processing time may be long. According to the AF process using two parallax images, the accuracy of the AF process may be reduced when a specific frequency component such as a high frequency component is included in the spatial frequency of the image.

General disclosure

The imaging control apparatus according to an aspect of the present invention may include a derivation unit that derives a spatial frequency band of a captured image. The imaging control device executes autofocus processing based on blur amounts of a plurality of images captured in a state where the positional relationship between the imaging surface and the lens is different based on the spatial frequency band of the image derived by the deriving unit. A control unit is provided that controls the positional relationship between the imaging surface and the lens by selecting one of the first AF process and the second AF process for performing autofocus processing by a phase difference method, or by combining both. Good.

The control unit selects one of the first AF process and the second AF process based on the spatial frequency band of the image derived by the derivation unit and the spatial frequency band predetermined for the second AF process. Alternatively, the positional relationship between the imaging surface and the lens may be controlled by combining both.

The control unit selects one of the first AF process and the second AF process based on the ratio of the spatial frequency band included in the predetermined spatial frequency band in the spatial frequency band of the image derived by the derivation unit. Or a combination of both may control the positional relationship between the imaging surface and the lens.

The control unit may control the positional relationship between the imaging surface and the lens by selecting the second AF process when the ratio is equal to or greater than the threshold and selecting the first AF process when the ratio is smaller than the threshold.

The control unit selects the second AF process when the ratio is equal to or greater than the first threshold, and combines the first AF process and the second AF process when the ratio is smaller than the first threshold and equal to or greater than the second threshold, When the threshold value is smaller than 2, the first AF process may be selected to control the positional relationship between the imaging surface and the lens.

The control unit compares the spatial frequency band of the image derived by the deriving unit with a predetermined spatial frequency band that can be reproduced by a plurality of pixels that generate the phase difference detection signal, and thereby performs the first AF process and the first AF process. The positional relationship between the imaging surface and the lens may be controlled by selecting either one of the 2AF processing or combining both.

The second AF process may be executed based on a phase difference detection signal output from an image sensor in which a plurality of pixels and a plurality of other pixels that generate color component signals are arranged in a predetermined arrangement pattern. .

An imaging device according to one embodiment of the present invention may include the imaging control device. The imaging device may include an image sensor having an imaging surface. The imaging device may include a lens.

An imaging system according to an aspect of the present invention may include the imaging device. The imaging system may include a support mechanism that supports the imaging device.

The moving body according to one embodiment of the present invention may move by mounting the imaging system.

The imaging control method according to an aspect of the present invention may include a step of deriving a spatial frequency band of a captured image. The imaging control method includes a first AF process that performs an autofocus process based on blur amounts of a plurality of images that are captured in a state where the positional relationship between the imaging surface and the lens is different based on the spatial frequency band of the derived image. And a step of controlling the positional relationship between the imaging surface and the lens by selecting either one of the second AF processing for executing the autofocus processing by the phase difference method, or combining both.

The program according to an aspect of the present invention may cause a computer to execute a step of deriving a spatial frequency band of a captured image. The program includes: a first AF process that performs an autofocus process based on blur amounts of a plurality of images captured in a state where the positional relationship between the imaging surface and the lens is different based on the derived spatial frequency band of the image; The step of controlling the positional relationship between the imaging surface and the lens may be executed by the computer by selecting either one of the second AF processes that execute the autofocus process by the phase difference method, or by combining both.

According to one aspect of the present invention, it is possible to suppress an increase in AF processing time and a decrease in AF processing accuracy in a balanced manner.

The above summary of the invention does not enumerate all the features of the present invention. A sub-combination of these feature groups can also be an invention.

It is a figure which shows an example of the external appearance of an unmanned aircraft and a remote control device. It is a figure which shows an example of the functional block of an unmanned aerial vehicle. It is a figure which shows an example of the curve which shows the relationship between a blurring amount and a lens position. It is a figure which shows an example of the procedure which calculates the distance to an object based on the amount of blurs. It is a figure for demonstrating the relationship between the position of an object, the position of a lens, and a focal distance. It is a figure which shows an example of the pixel arrangement | sequence of an image sensor. It is a figure for demonstrating the ratio of the spatial frequency band of an image. It is a figure for demonstrating the ratio of the spatial frequency band of an image. It is a flowchart which shows an example of the procedure of AF process. It is a foot chart which shows an example of the procedure of AF process. It is a figure which shows an example of a hardware constitutions.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the claimed invention. Moreover, not all the combinations of features described in the embodiments are essential for the solution means of the invention. It will be apparent to those skilled in the art that various modifications or improvements can be made to the following embodiments. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

The claims, the description, the drawings, and the abstract include matters that are subject to copyright protection. The copyright owner will not object to any number of copies of these documents as they appear in the JPO file or record. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where a block is either (1) a stage in a process in which an operation is performed or (2) an apparatus responsible for performing the operation. May represent a “part”. Certain stages and “units” may be implemented by programmable circuits and / or processors. Dedicated circuitry may include digital and / or analog hardware circuitry. Integrated circuits (ICs) and / or discrete circuits may be included. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. The memory element or the like may be included.

The computer readable medium may include any tangible device capable of storing instructions to be executed by a suitable device. As a result, a computer readable medium having instructions stored thereon comprises a product that includes instructions that can be executed to create a means for performing the operations specified in the flowcharts or block diagrams. Examples of computer readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer readable media include floppy disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), Electrically erasable programmable read only memory (EEPROM), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated A circuit card or the like may be included.

The computer readable instructions may include either source code or object code written in any combination of one or more programming languages. The source code or object code includes a conventional procedural programming language. Conventional procedural programming languages include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or Smalltalk, JAVA, C ++, etc. It may be an object-oriented programming language and a “C” programming language or a similar programming language. Computer readable instructions may be directed to a general purpose computer, special purpose computer, or other programmable data processing device processor or programmable circuit locally or in a wide area network (WAN) such as a local area network (LAN), the Internet, etc. ). The processor or programmable circuit may execute computer readable instructions to create a means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

FIG. 1 shows an example of the external appearance of an unmanned aerial vehicle (UAV) 10 and a remote control device 300. The UAV 10 includes a UAV main body 20, a gimbal 50, a plurality of imaging devices 60, and an imaging device 100. The gimbal 50 and the imaging device 100 are an example of an imaging system. The UAV 10 is an example of a moving body propelled by a propulsion unit. The moving body is a concept including a flying body such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, etc. in addition to the UAV.

The UAV main body 20 includes a plurality of rotor blades. The plurality of rotor blades is an example of a propulsion unit. The UAV main body 20 causes the UAV 10 to fly by controlling the rotation of a plurality of rotor blades. For example, the UAV main body 20 causes the UAV 10 to fly using four rotary wings. The number of rotor blades is not limited to four. The UAV 10 may be a fixed wing machine that does not have a rotating wing.

The imaging apparatus 100 is an imaging camera that images a subject included in a desired imaging range. The gimbal 50 supports the imaging device 100 in a rotatable manner. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 supports the imaging device 100 so as to be rotatable about the pitch axis using an actuator. The gimbal 50 further supports the imaging device 100 using an actuator so as to be rotatable about the roll axis and the yaw axis. The gimbal 50 may change the posture of the imaging device 100 by rotating the imaging device 100 about at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are sensing cameras that image the surroundings of the UAV 10 in order to control the flight of the UAV 10. Two imaging devices 60 may be provided in the front which is the nose of UAV10. Two other imaging devices 60 may be provided on the bottom surface of the UAV 10. The two imaging devices 60 on the front side may be paired and function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired and function as a stereo camera. Based on images picked up by a plurality of image pickup devices 60, three-dimensional spatial data around the UAV 10 may be generated. The number of imaging devices 60 included in the UAV 10 is not limited to four. The UAV 10 only needs to include at least one imaging device 60. The UAV 10 may include at least one imaging device 60 on each of the nose, the tail, the side surface, the bottom surface, and the ceiling surface of the UAV 10. The angle of view that can be set by the imaging device 60 may be wider than the angle of view that can be set by the imaging device 100. The imaging device 60 may have a single focus lens or a fisheye lens.

The remote operation device 300 communicates with the UAV 10 to remotely operate the UAV 10. The remote operation device 300 may communicate with the UAV 10 wirelessly. The remote control device 300 transmits to the UAV 10 instruction information indicating various commands related to movement of the UAV 10 such as ascending, descending, accelerating, decelerating, moving forward, moving backward, and rotating. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at an altitude indicated by the instruction information received from the remote operation device 300. The instruction information may include an ascending command that raises the UAV 10. The UAV 10 rises while accepting the ascent command. Even if the UAV 10 receives the ascending command, the UAV 10 may limit the ascent when the altitude of the UAV 10 has reached the upper limit altitude.

FIG. 2 shows an example of functional blocks of the UAV10. The UAV 10 includes a UAV control unit 30, a memory 32, a communication interface 34, a propulsion unit 40, a GPS receiver 41, an inertial measurement device 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a gimbal 50, and the imaging device 100. .

The communication interface 34 communicates with other devices such as the remote operation device 300. The communication interface 34 may receive instruction information including various commands for the UAV control unit 30 from the remote operation device 300. The memory 32 includes a propulsion unit 40, a GPS receiver 41, an inertial measurement device (IMU) 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a gimbal 50, an imaging device 60, and the imaging device 100. Stores programs and the like necessary for controlling The memory 32 may be a computer-readable recording medium and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 32 may be provided inside the UAV main body 20. It may be provided so as to be removable from the UAV main body 20.

The UAV control unit 30 controls the flight and imaging of the UAV 10 according to a program stored in the memory 32. The UAV control unit 30 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like. The UAV control unit 30 controls the flight and imaging of the UAV 10 according to a command received from the remote control device 300 via the communication interface 34. The propulsion unit 40 propels the UAV 10. The propulsion unit 40 includes a plurality of rotating blades and a plurality of drive motors that rotate the plurality of rotating blades. The propulsion unit 40 causes the UAV 10 to fly by rotating a plurality of rotor blades via a plurality of drive motors in accordance with a command from the UAV control unit 30.

The GPS receiver 41 receives a plurality of signals indicating times transmitted from a plurality of GPS satellites. The GPS receiver 41 calculates the position of the GPS receiver 41, that is, the position of the UAV 10 based on the received signals. The IMU 42 detects the posture of the UAV 10. The IMU 42 detects, as the posture of the UAV 10, acceleration in the three axial directions of the front, rear, left, and right of the UAV 10, and angular velocity in the three axial directions of pitch, roll, and yaw. The magnetic compass 43 detects the heading of the UAV 10. The barometric altimeter 44 detects the altitude at which the UAV 10 flies. The barometric altimeter 44 detects the atmospheric pressure around the UAV 10, converts the detected atmospheric pressure into an altitude, and detects the altitude. The temperature sensor 45 detects the temperature around the UAV 10.

The imaging apparatus 100 includes an imaging unit 102 and a lens unit 200. The lens unit 200 is an example of a lens device. The imaging unit 102 includes an image sensor 120, an imaging control unit 110, and a memory 130. The image sensor 120 may be configured by a CCD or a CMOS. The image sensor 120 outputs image data of an optical image formed through the plurality of lenses 210 to the imaging control unit 110. The imaging control unit 110 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like. The imaging control unit 110 may control the imaging device 100 in accordance with an operation command for the imaging device 100 from the UAV control unit 30. The memory 130 may be a computer-readable recording medium and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 130 stores a program and the like necessary for the imaging control unit 110 to control the image sensor 120 and the like. The memory 130 may be provided inside the housing of the imaging device 100. The memory 130 may be provided so as to be removable from the housing of the imaging apparatus 100.

The lens unit 200 includes a plurality of lenses 210, a lens moving mechanism 212, and a lens control unit 220. The plurality of lenses 210 may function as a zoom lens, a varifocal lens, and a focus lens. At least some or all of the plurality of lenses 210 are arranged to be movable along the optical axis. The lens unit 200 may be an interchangeable lens that is detachably attached to the imaging unit 102. The lens moving mechanism 212 moves at least some or all of the plurality of lenses 210 along the optical axis. The lens control unit 220 drives the lens moving mechanism 212 in accordance with a lens control command from the imaging unit 102 to move one or a plurality of lenses 210 along the optical axis direction. The lens control command is, for example, a zoom control command and a focus control command.

The imaging apparatus 100 configured in this manner performs an autofocus process (AF process) and images a desired subject.

The imaging apparatus 100 determines the distance from the lens to the subject (subject distance) in order to execute the AF process. As a method for determining the subject distance, there is a method of determining based on the blur amounts of a plurality of images captured in a state where the positional relationship between the lens and the imaging surface is different. Here, this method is referred to as a blur detection auto focus (BDAF) method.

For example, the blur amount (Cost) of the image can be expressed by the following equation (1) using a Gaussian function. In Expression (1), x indicates a pixel position in the horizontal direction. σ represents a standard deviation value.

FIG. 3 shows an example of a curve represented by Equation (1). By focusing the focus lens on the lens position corresponding to the minimum point 502 of the curve 500, the object included in the image I can be focused.

FIG. 4 is a flowchart illustrating an example of a distance calculation procedure of the BDAF method. First, the imaging apparatus 100, in a state in which the lens and the image plane is in the first positional relationship, for storing the image I ₁ of the first sheet in the memory 130 by photographing. Then, by moving the imaging surface of the focusing lens or the image sensor 120 in the optical axis direction, the lens and the imaging surface in the state in the second positional relationship, captured image I ₂ of the second sheet by the imaging device 100 And stored in the memory 130 (S101). For example, like so-called hill-climbing AF, the focus lens or the imaging surface of the image sensor 120 is moved in the optical axis direction so as not to exceed the focal point. The moving amount of the imaging surface of the focus lens or the image sensor 120 may be 10 μm, for example.

Then, the imaging device 100 divides the image _{I 1} into a plurality of regions (S102). Calculates a feature amount for each pixel in the image I2, may divide the image I ₁ into a plurality of regions of pixel groups having a feature amount that is similar as a single region. A pixel group in a range set in the AF processing frame in the image I ₁ may be divided into a plurality of regions. Imaging device 100 divides the image I ₂ into a plurality of regions corresponding to a plurality of areas of the image I _1. The imaging apparatus 100 includes an object included in each of the plurality of regions based on the amount of blur of each of the plurality of regions of the image I ₁ and the amount of blur of each of the plurality of regions of the image I _2. Is calculated (S103).

The distance calculation procedure will be further described with reference to FIG. The distance from the lens L (principal point) to the object 510 (object surface) is A, the distance from the lens L (principal point) to the position (image plane) where the object 510 forms an image on the imaging surface is B, and the focal length is F. And In this case, the relationship between the distance A, the separation B, and the focal length F can be expressed by the following formula (2) from the lens formula.

The focal length F is specified by the lens position. Therefore, if the distance B at which the object 510 forms an image on the imaging surface can be specified, the distance A from the lens L to the object 510 can be specified using Expression (2).

As shown in FIG. 5, the distance B is specified by calculating the position at which the object 510 forms an image from the blur size (the circles of confusion 512 and 514) of the object 510 projected on the imaging surface. A can be specified. That is, the imaging position can be specified in consideration of the fact that the size of blur (the amount of blur) is proportional to the imaging surface and the imaging position.

Here, the distance from the image I ₁ close to the imaging surface to the lens L and D _1. The distance from the image I ₂ far from the image plane to the lens L is D ₂ . Each image is blurred. The point spread function at this time is PSF, and the images at D ₁ and D ₂ are I _d1 and I _d2 , respectively. In this case, for example, the image I ₁ can be expressed by the following equation (3) by a convolution operation.

Further, the Fourier transform function of the image data _{I d1} and _{I d2} as f, the point spread image _{I d1} and _{I d2} function PSF ₁ and the optical transfer function of the PSF ₂ Fourier transform (Optical Transfer Function) of the OTF ₁ and OTF ₂ , the ratio is as shown in the following equation (4).

The value C shown in the equation (4) corresponds to the amount of change in the blur amounts of the images I _d1 and I _d2 , that is, the value C corresponds to the difference between the blur amount of the image I _{d1 and} the blur amount of the image I _d2n. .

According to the BDAF method as described above, since it is necessary to acquire at least two images, AF processing may take time.

A phase difference AF processing method that performs AF processing using two parallax images is known as a method that can shorten the AF processing time relatively. The phase difference AF processing method includes an image plane phase difference AF (PDAF) method. According to the PDAF method, for example, an image sensor in which a plurality of pixels are arranged as shown in FIG. 6 is used. The image sensor includes a pixel column 600 including a Bayer array pixel block that outputs a color component detection signal, and a pixel column 602 and a pixel column 604 including pixel blocks including pixels that output a phase difference detection signal. The pixel P1 that outputs the phase difference detection signal included in the pixel column 602 and the pixel P2 that outputs the phase difference detection signal included in the pixel column 604 have different light incident directions. Therefore, an image having a different phase is obtained from the image obtained from the pixel P1 and the image obtained from the pixel P2. Then, the distance to the subject can be derived based on the amount and direction of deviation between the image included in the image obtained from the pixel P1 and the image included in the image obtained from the pixel P2.

In many cases, the PDAF method can execute AF processing faster than the BDAF method. However, the pixels for phase difference detection are relatively spaced apart. Therefore, the image quality of the image obtained from the image P1 and the pixel P2 is relatively low. For this reason, if a high frequency component is included in the spatial frequency band of the image captured by the image sensor, the accuracy of the AF processing may decrease.

Therefore, according to the imaging apparatus 100 according to the present embodiment, AF processing of the BDAF method and AF processing of the PADF method are selected based on the spatial frequency band of the image, or a combination of both, Execute the process.

The imaging control unit 110 includes a derivation unit 112 and a focusing control unit 114. The deriving unit 112 derives the spatial frequency band of the image captured by the image sensor 120. The deriving unit 112 may derive the spatial frequency band of the image by performing Fourier transform on the image and decomposing the image for each spatial frequency component.

Based on the spatial frequency band of the image derived by the deriving unit 112, the focusing control unit 114 determines the blur amounts of a plurality of images captured in a state where the positional relationship between the imaging surface of the image sensor 120 and the lens 210 is different. The imaging surface of the image sensor 120 and the lens 210 are selected by selecting either one of BDAF processing that executes autofocus processing based on the PDAF processing that performs autofocus processing by the phase difference method, or a combination of both. Control the positional relationship of The focus control unit 114 may control the position of the focus lens by selecting one of BDAF processing and PDAF processing based on the spatial frequency band of the image, or by combining both. The focus control unit 114 is an example of a control unit. The BDAF process is an example of a first AF process. The PFAF process is an example of a second AF process.

The focus control unit 114 selects either BDAF processing or PDAF processing based on the spatial frequency band of the image derived by the deriving unit 112 and the spatial frequency band predetermined for the PDAF processing. Alternatively, the positional relationship between the imaging surface of the image sensor 120 and the lens 210 may be controlled by combining both. The focusing control unit 114 performs either one of the BDAF process and the PDAF process based on the ratio of the spatial frequency band included in the predetermined spatial frequency band in the spatial frequency band of the image derived by the deriving unit 112. The position relationship between the imaging surface of the image sensor 120 and the lens 210 may be controlled by selecting or combining both.

The focus control unit 114 selects the PDAF process when the ratio is greater than or equal to the threshold value, and selects the BDAF process when the ratio is smaller than the threshold value, thereby determining the positional relationship between the imaging surface of the image sensor 120 and the lens 210. You may control. The focus control unit 114 selects the PDAF process when the ratio is greater than or equal to the first threshold, and combines the BDAF process and the PDAF process when the ratio is smaller than the first threshold and greater than or equal to the second threshold, If it is smaller than two thresholds, the positional relationship between the imaging surface of the image sensor 120 and the lens 210 may be controlled by selecting BDAF processing. For example, the focus control unit 114 weights the distance to the subject specified by the BDAF process and the distance to the subject specified by the PADF process, and based on the weighted distance, the position of the focus lens May be controlled. The focus control unit 114 may change the weighting of the BDAF process and the PDAF process depending on the ratio.

The focusing control unit 114 compares the spatial frequency band of the image derived by the deriving unit 112 with a predetermined spatial frequency band that can be reproduced by a plurality of pixels that generate the phase difference detection signal, thereby obtaining the BDAF. The positional relationship between the imaging surface of the image sensor 120 and the lens 210 may be controlled by selecting one of the processing and the PDAF processing or combining both.

The focus control unit 114 is output from the image sensor 120 in which a plurality of pixels that generate a phase difference detection signal and a plurality of other pixels that generate a color component signal are arranged in a predetermined arrangement pattern. The PDAF process may be executed based on the phase difference detection signal. The plurality of pixels that generate the phase difference detection signal are, for example, the pixel P1 and the pixel P2 illustrated in FIG. The plurality of other pixels that generate the color component signal are, for example, the pixel R, the pixel G, and the pixel B illustrated in FIG. The predetermined spatial frequency band may be a spatial frequency band that can be reproduced by the pixel P1 and the pixel P2, for example.

7A and 7B, the spatial frequency band that can be reproduced by the pixel R, the pixel G, and the pixel B is, for example, a region 700. On the other hand, the spatial frequency band that can be reproduced by the pixel P1 and the pixel P2 is, for example, a region 702. Region 700 and region 702 each contain a low frequency component. The frequency component included in the region 702 has a lower proportion of high frequency components than the frequency component included in the region 700. Therefore, when the image picked up by the image sensor 120 includes many high-frequency components, the spatial frequency components that can be reproduced by the pixels P1 and P2 are small, and the focus control unit 114 cannot execute the AF processing with high accuracy by the PDAF processing. There is a case.

On the other hand, the BDAF process is executed based on the color component signal. That is, the focus control unit 114 can perform BDAF processing with high accuracy on an image including a spatial frequency band in the range of the region 700.

7A and 7B show the relationship between the spatial frequency and the gain of the spatial frequency. The larger the gain, the more spatial frequency components corresponding to the gain are included in the image. 7A and 7B, a region 710 is a region corresponding to the spatial frequency band of the image derived by the deriving unit 112. The region 712 is a region corresponding to the spatial frequency band of the image included in the region 702 in the spatial frequency band of the image. The focusing control unit 114 controls the position of the focus lens by selecting one of BDAF processing and PDAF processing or combining both based on the ratio of the area of the region 712 to the area of the region 710. It's okay.

For example, as shown in FIG. 7A, the focus control unit 114 may select the PDAF process and control the position of the focus lens if the ratio of the region 712 is equal to or greater than a predetermined threshold. On the other hand, as shown in FIG. 7B, the focus control unit 114 may select the BDAF process and control the position of the focus lens when the ratio of the area 712 is smaller than a predetermined threshold.

If the maximum spatial frequency component included in the spatial frequency band of the image derived by the deriving unit 112 is equal to or less than a predetermined spatial frequency component, the focusing control unit 114 selects the PDAF process and selects the maximum spatial frequency component. If the frequency component is larger than a predetermined spatial frequency component, BDAF processing may be selected to control the position of the focus lens. The predetermined spatial frequency component may be determined based on, for example, the maximum spatial frequency component in the spatial frequency band that can be reproduced by the pixel P1 and the pixel P2.

FIG. 8 is a flowchart showing an example of the procedure of AF processing. First, the deriving unit 112 acquires an image captured by the image sensor 120. The deriving unit 112 derives a spatial frequency band of the image. Furthermore, the deriving unit 112 derives a ratio B of the spatial frequency band of the image that occupies a predetermined spatial frequency band for the PDAF processing (S200).

The focus control unit 114 determines whether or not the derived ratio B is equal to or greater than a predetermined first threshold Th1 (S202). If the ratio B is equal to or greater than the first threshold Th1, the focus control unit 114 selects the PDAF process and controls the positional relationship between the imaging surface of the image sensor 120 and the lens 210 (S204). On the other hand, if the ratio B is smaller than the first threshold Th1, the focusing control unit 114 selects the BDAF process and controls the positional relationship between the imaging surface of the image sensor 120 and the lens 210 (S206).

Thus, based on the spatial frequency component included in the image used for the AF process, the focus control unit 114 switches between the PDAF process and the BDAF process. As a result, when the high-frequency component is included in the image, the focusing control unit 114 can prevent the AF processing accuracy from being lowered by executing the PDAF processing. Further, when the high frequency component is not included in the image, it is possible to prevent the AF process from taking a long time by the focusing control unit 114 executing the BDAF process. Therefore, according to the imaging device 100 according to the present embodiment, it is possible to suppress an increase in AF processing time and a decrease in AF processing accuracy with a good balance. You may distinguish a low frequency component and a high frequency component on the 1st threshold value Th as a boundary.

FIG. 9 is a flowchart showing another example of the AF processing procedure. First, the deriving unit 112 derives a spatial frequency band of the image. Further, the deriving unit 112 derives a ratio B of the spatial frequency band of the image that occupies a predetermined spatial frequency band for the PDAF processing (S300).

The focus control unit 114 determines whether or not the derived ratio B is equal to or greater than a predetermined first threshold Th1 (S302). If the ratio B is equal to or greater than the first threshold Th1, the focus control unit 114 selects the PDAF process and controls the positional relationship between the imaging surface of the image sensor 120 and the lens 210 (S304). On the other hand, if the ratio B is smaller than the first threshold value Th1, the focusing control unit 114 determines whether the ratio B is smaller than the first threshold value Th1 and is equal to or larger than a predetermined second threshold value Th2 (S306).

If the ratio B is smaller than the first threshold Th1 and equal to or greater than a predetermined second threshold Th2, the focus control unit 114 combines the PDAF process and the BDAF process to combine the imaging surface of the image sensor 120 and the lens 210. (S308). For example, the focus control unit 114 may control the position of the focus lens based on the position of the focus lens specified by the PDAF process and the position of the focus lens specified by the BDAF process. The focusing control unit 114 weights each focus lens position specified by the PDAF process and the BDAF process according to the ratio B, and controls the position of the focus lens based on each weighted distance. Good. The focus control unit 114 may weight each distance so that the weight of the distance specified by the PDAF process is larger than the weight of the distance specified by the BDAF process as the ratio B is larger. .

If the ratio B is smaller than the second threshold, the focus control unit 114 selects the BDAF process and controls the positional relationship between the imaging surface of the image sensor 120 and the lens 210 (S310).

As described above, based on the spatial frequency component included in the image, the focus control unit 114 executes the AF process by switching the PDAF process and the BDAF process, or combining the PDAF process and the BDAF process. As a result, when the high-frequency component is included in the image, it is possible to prevent the focus control unit 114 from executing the PDAF process and reducing the accuracy of the AF process. In addition, when the image does not contain many high-frequency components, it is possible to prevent the focusing control unit 114 from executing the BDAF process and taking a long time for the AF process. When the ratio B is around 50%, for example, instead of selecting only one of the PDAF process and the BDAF process, the AF process is executed by combining the PDAF process and the BDAF process. Thereby, when the ratio B is, for example, around 50%, the accuracy of the AF process can be improved as compared with the case where the AF process is executed by only one of the PDAF process and the BDAF process. Therefore, according to the imaging device 100 according to the present embodiment, it is possible to suppress an increase in AF processing time and a decrease in AF processing accuracy with a good balance.

FIG. 10 illustrates an example of a computer 1200 in which aspects of the present invention may be embodied in whole or in part. A program installed in the computer 1200 can cause the computer 1200 to function as an operation associated with the apparatus according to the embodiment of the present invention or as one or more “units” of the apparatus. Alternatively, the program can cause the computer 1200 to execute the operation or the one or more “units”. The program can cause the computer 1200 to execute a process according to an embodiment of the present invention or a stage of the process. Such a program may be executed by CPU 1212 to cause computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222 and an input / output unit, which are connected to the host controller 1210 via the input / output controller 1220. Computer 1200 also includes ROM 1230. The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

The communication interface 1222 communicates with other electronic devices via a network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores therein a boot program executed by the computer 1200 at the time of activation and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, a USB memory, or an IC card or a network. The program is installed in the RAM 1214 or the ROM 1230 that is also an example of a computer-readable recording medium, and is executed by the CPU 1212. Information processing described in these programs is read by the computer 1200 to bring about cooperation between the programs and the various types of hardware resources. An apparatus or method may be configured by implementing information operations or processing in accordance with the use of computer 1200.

For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214 and performs communication processing on the communication interface 1222 based on the processing described in the communication program. You may order. The communication interface 1222 reads transmission data stored in a RAM 1214 or a transmission buffer area provided in a recording medium such as a USB memory under the control of the CPU 1212 and transmits the read transmission data to a network, or The reception data received from the network is written into a reception buffer area provided on the recording medium.

In addition, the CPU 1212 allows the RAM 1214 to read all or necessary portions of a file or database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. Good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in the recording medium and subjected to information processing. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval that are described throughout the present disclosure for data read from the RAM 1214 and specified by the instruction sequence of the program. Various types of processing may be performed, including / replacement, etc., and the result is written back to RAM 1214. In addition, the CPU 1212 may search for information in files, databases, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. The entry that matches the condition is searched from the plurality of entries, the attribute value of the second attribute stored in the entry is read, and thereby the first attribute that satisfies the predetermined condition is associated. The attribute value of the obtained second attribute may be acquired.

The program or software module described above may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, whereby the program is transferred to the computer 1200 via the network. provide.

The execution order of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior”. It should be noted that they can be implemented in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for the sake of convenience, it means that it is essential to carry out in this order. is not.

10 UAV
20 UAV body 30 UAV control unit 32 Memory 34 Communication interface 40 Propulsion unit 41 GPS receiver 42 Inertial measurement device 43 Magnetic compass 44 Barometric altimeter 45 Temperature sensor 50 Gimbal 60 Imaging device 100 Imaging device 102 Imaging unit 110 Imaging control unit 112 Deriving unit 114 Focus control unit 120 Image sensor 130 Memory 200 Lens unit 210 Lens 212 Lens moving mechanism 220 Lens control unit 300 Remote operation device 1200 Computer 1210 Host controller 1212 CPU
1214 RAM
1220 Input / output controller 1222 Communication interface 1230 ROM

Claims

A deriving unit for deriving a spatial frequency band of the captured image;
Based on the spatial frequency band of the image derived by the deriving unit, a first AF that performs autofocus processing based on blur amounts of a plurality of images captured in a state where the positional relationship between the imaging surface and the lens is different A control unit that controls the positional relationship between the imaging surface and the lens by selecting either one of the processing and the second AF processing for performing autofocus processing by a phase difference method, or by combining both Imaging control device.
The control unit performs the first AF process and the second AF process based on the spatial frequency band of the image derived by the derivation unit and a spatial frequency band predetermined for the second AF process. The imaging control apparatus according to claim 1, wherein the positional relationship between the imaging surface and the lens is controlled by selecting either one or combining both.
The control unit may perform the first AF process and the second AF process based on a ratio of a spatial frequency band included in the predetermined spatial frequency band in the spatial frequency band of the image derived by the derivation unit. The imaging control apparatus according to claim 2, wherein the positional relationship between the imaging surface and the lens is controlled by selecting one of the two or a combination of both.
The control unit selects the second AF process when the ratio is greater than or equal to a threshold, and selects the first AF process when the ratio is smaller than the threshold, thereby positioning the imaging surface and the lens. The imaging control device according to claim 3, wherein the imaging control device controls the relationship.
The control unit selects the second AF process when the ratio is equal to or greater than a first threshold, and when the ratio is smaller than the first threshold and equal to or greater than a second threshold, the first AF process and the second AF process are selected. The imaging control apparatus according to claim 3, wherein a positional relationship between the imaging surface and the lens is controlled by selecting the first AF process when combined with 2AF processing and smaller than the second threshold.
The control unit compares the spatial frequency band of the image derived by the deriving unit with a predetermined spatial frequency band that can be reproduced by a plurality of pixels that generate a phase difference detection signal. The imaging control apparatus according to claim 1, wherein the positional relationship between the imaging surface and the lens is controlled by selecting one of the first AF process and the second AF process or combining both.
The second AF process is executed based on the phase difference detection signal output from an image sensor in which the plurality of pixels and a plurality of other pixels that generate color component signals are arranged in a predetermined arrangement pattern. The imaging control device according to claim 6.
The imaging control device according to any one of claims 1 to 7,
An imaging apparatus comprising: an image sensor having the imaging surface; and the lens.
An imaging device according to claim 8,
An imaging system comprising: a support mechanism that supports the imaging device.
A moving body that is mounted with the imaging system according to claim 9 and moves.
Deriving a spatial frequency band of the captured image;
First AF processing for performing autofocus processing based on blur amounts of a plurality of images captured in a state where the positional relationship between the imaging surface and the lens is different based on the spatial frequency band of the derived image; An imaging control method comprising: controlling a positional relationship between the imaging surface and the lens by selecting any one of the second AF processing for executing the autofocus processing by a phase difference method, or combining both.
Deriving a spatial frequency band of the captured image;
First AF processing for performing autofocus processing based on blur amounts of a plurality of images captured in a state where the positional relationship between the imaging surface and the lens is different based on the spatial frequency band of the derived image; A step of controlling a positional relationship between the imaging surface and the lens by selecting any one of the second AF processing for performing the autofocus processing by the phase difference method, or by combining both. program.