JP6503607B2 - Imaging control apparatus, imaging apparatus, imaging system, moving object, imaging control method, and program - Google Patents

Description

  The present invention relates to an imaging control apparatus, an imaging apparatus, an imaging system, a moving object, an imaging control method, and a program.

Known are auto-focusing processing (AF processing) using two parallax images, and AF processing using a plurality of images with different amounts of blur captured with different imaging parameters, as in Patent Document 1.
Patent Document 1 Patent No. 5932476

Problem to be solved

  According to the AF process using the blur amount of the image, it is necessary to capture at least two images, and the time of the AF process may be long. According to the AF process using two parallax images, the accuracy of the AF process may decrease when the spatial frequency of the image includes a specific frequency component such as a high frequency component.

General disclosure

  The imaging control apparatus according to an aspect of the present invention may include a deriving unit that derives a spatial frequency band of a captured image. The imaging control apparatus executes autofocus processing based on the blur amount of a plurality of images captured in a state in which the positional relationship between the imaging surface and the lens is different based on the spatial frequency band of the image derived by the deriving unit. A control unit is provided that controls the positional relationship between the imaging surface and the lens by selecting either one of the first AF processing and the second AF processing that performs autofocus processing in the phase difference method, or combining both. Good.

  The control unit selects one of the first AF processing and the second AF processing based on the spatial frequency band of the image derived by the deriving unit and the spatial frequency band predetermined for the second AF processing. Alternatively, the positional relationship between the imaging surface and the lens may be controlled by combining both.

  The control unit selects one of the first AF processing and the second AF processing based on the ratio of the spatial frequency band included in the predetermined spatial frequency band among the spatial frequency bands of the image derived by the deriving unit. The positional relationship between the imaging surface and the lens may be controlled by combining or both.

  The control unit may control the positional relationship between the imaging surface and the lens by selecting the second AF process when the ratio is equal to or higher than the threshold and selecting the first AF process when the ratio is smaller than the threshold.

  The control unit selects the second AF process if the ratio is equal to or greater than the first threshold, and combines the first AF process and the second AF process if the ratio is smaller than the first threshold and equal to or greater than the second threshold. If it is smaller than two thresholds, the positional relationship between the imaging surface and the lens may be controlled by selecting the first AF processing.

  The control unit compares the spatial frequency band of the image derived by the derivation unit with a predetermined spatial frequency band reproducible by the plurality of pixels generating the phase difference detection signal, thereby performing the first AF processing and the first AF processing. The positional relationship between the imaging surface and the lens may be controlled by selecting either or both of the 2AF processing.

  The second AF processing may be performed based on a phase difference detection signal output from an image sensor in which a plurality of pixels and a plurality of other pixels generating color component signals are arranged in a predetermined array pattern. .

  An imaging device according to an aspect of the present invention may include the imaging control device. The imaging device may include an image sensor having an imaging surface. The imaging device may comprise a lens.

  An imaging system according to an aspect of the present invention may include the imaging device. The imaging system may include a support mechanism that supports the imaging device.

  The mobile unit according to an aspect of the present invention may move with the imaging system mounted thereon.

  The imaging control method according to an aspect of the present invention may include the step of deriving a spatial frequency band of a captured image. The imaging control method performs the first AF processing based on the blur amount of a plurality of images captured in a state in which the positional relationship between the imaging surface and the lens is different based on the derived spatial frequency band of the image. And a step of controlling the positional relationship between the imaging surface and the lens by selecting one or the other of the second AF processing for performing the autofocus processing in the phase difference method, or combining both.

  A program according to an aspect of the present invention may cause a computer to execute a step of deriving a spatial frequency band of a captured image. The program executes a first AF process of executing an autofocus process based on the blur amount of a plurality of images captured in a state in which the positional relationship between the imaging surface and the lens is different based on the derived spatial frequency band of the image; The computer may be made to execute the step of controlling the positional relationship between the imaging surface and the lens by selecting either one of the second AF processing to execute the autofocus processing in the phase difference method or combining the two.

  According to one aspect of the present invention, it is possible to suppress an increase in the time of AF processing and a decrease in the accuracy of AF processing in a well-balanced manner.

  The above summary of the invention does not enumerate all of the features of the present invention. A subcombination of these feature groups can also be an invention.

It is a figure which shows an example of the external appearance of a unmanned aerial vehicle and a remote control. It is a figure which shows an example of the functional block of a unmanned aerial vehicle. It is a figure which shows an example of the curve which shows the relationship between blur amount and a lens position. It is a figure which shows an example of the procedure which calculates the distance to an object based on blur amount. It is a figure for demonstrating the relationship between the position of an object, the position of a lens, and a focal distance. It is a figure which shows an example of the pixel array of an image sensor. It is a figure for demonstrating the ratio of the spatial frequency band of an image. It is a figure for demonstrating the ratio of the spatial frequency band of an image. 5 is a flowchart illustrating an example of a procedure of AF processing. 6 is a fut chart showing an example of the procedure of AF processing. It is a figure showing an example of hardware constitutions.

  Hereinafter, the present invention will be described through the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Further, not all combinations of features described in the embodiments are essential to the solution of the invention. It will be apparent to those skilled in the art that various changes or modifications can be added to the following embodiments. It is also apparent from the scope of the claims that the embodiments added with such alterations or improvements can be included in the technical scope of the present invention.

  The claims, the description, the drawings, and the abstract contain matters which are subject to copyright protection. The copyright holder will not object to any copy of these documents as they appear in the Patent Office file or record. However, in all other cases, all copyrights are reserved.

  Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the blocks are responsible for (1) process steps or (2) operations being performed. May represent a "part" of The particular stages and "parts" may be implemented by programmable circuits and / or processors. Dedicated circuitry may include digital and / or analog hardware circuitry. Integrated circuits (ICs) and / or discrete circuits may be included. Programmable circuits may include reconfigurable hardware circuits. Reconfigurable hardware circuits include logic AND, logic OR, logic XOR, logic NAND, logic NOR, and other logic operations, flip flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc. Memory elements, etc. may be included.

  Computer readable media may include any tangible device capable of storing instructions for execution by a suitable device. As a result, a computer readable medium having instructions stored thereon will comprise an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer readable media include floppy disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), Electrically Erasable Programmable Read Only Memory (EEPROM), Static Random Access Memory (SRAM), Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD), Blu-Ray (RTM) Disc, Memory Stick, Integrated A circuit card or the like may be included.

  Computer readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes a conventional procedural programming language. Conventional procedural programming languages include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or like Smalltalk, JAVA, C ++, etc. It may be an object oriented programming language, and a "C" programming language or similar programming language. Computer readable instructions may be local or to a wide area network (WAN) such as a local area network (LAN), the Internet, etc., relative to a processor or programmable circuitry of a general purpose computer, special purpose computer, or other programmable data processing device. May be provided via The processor or programmable circuitry may execute computer readable instructions to create a means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

  FIG. 1 shows an example of the appearance of an unmanned aerial vehicle (UAV) 10 and a remote control device 300. The UAV 10 includes a UAV main body 20, a gimbal 50, a plurality of imaging devices 60, and an imaging device 100. The gimbal 50 and the imaging device 100 are examples of an imaging system. The UAV 10 is an example of a mobile unit propelled by the propulsion unit. In addition to the UAV, the mobile object is a concept including an aircraft such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like.

  The UAV body 20 comprises a plurality of rotors. The plurality of rotors are an example of the propulsion unit. The UAV body 20 causes the UAV 10 to fly by controlling the rotation of a plurality of rotors. The UAV body 20 causes the UAV 10 to fly using, for example, four rotors. The number of rotors is not limited to four. The UAV 10 may also be a fixed wing aircraft that does not have a rotor.

  The imaging device 100 is a camera for imaging which captures an object included in a desired imaging range. The gimbal 50 rotatably supports the imaging device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the imaging device 100 on the pitch axis using an actuator. The gimbal 50 rotatably supports the imaging device 100 about each of the roll axis and the yaw axis using an actuator. The gimbal 50 may change the attitude of the imaging device 100 by rotating the imaging device 100 about at least one of the yaw axis, the pitch axis, and the roll axis.

  The plurality of imaging devices 60 are cameras for sensing that capture the periphery of the UAV 10 to control the flight of the UAV 10. Two imaging devices 60 may be provided at the front, which is the nose of the UAV 10. Two further imaging devices 60 may be provided on the bottom of the UAV 10. The two front imaging devices 60 may be paired to function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of imaging devices 60. The number of imaging devices 60 provided in the UAV 10 is not limited to four. The UAV 10 may include at least one imaging device 60. The UAV 10 may include at least one imaging device 60 on each of the nose, aft, sides, bottom, and ceiling of the UAV 10. The angle of view that can be set by the imaging device 60 may be wider than the angle of view that can be set by the imaging device 100. The imaging device 60 may have a single focus lens or a fisheye lens.

  The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may communicate with the UAV 10 wirelessly. The remote control device 300 transmits instruction information indicating various commands related to the movement of the UAV 10 such as rising, falling, acceleration, deceleration, forward, reverse, and rotation to the UAV 10. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at the altitude indicated by the instruction information received from the remote control device 300. The instruction information may include a lift instruction to raise the UAV 10. The UAV 10 goes up while accepting the up command. Even if the UAV 10 receives an ascent instruction, if the UAV 10's altitude reaches the upper limit altitude, the UAV 10 may limit the ascent.

  FIG. 2 shows an example of a functional block of the UAV 10. The UAV 10 includes a UAV control unit 30, a memory 32, a communication interface 34, a propulsion unit 40, a GPS receiver 41, an inertia measurement device 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a gimbal 50, and an imaging device 100. .

  The communication interface 34 communicates with other devices such as the remote control device 300. The communication interface 34 may receive instruction information including various instructions for the UAV control unit 30 from the remote control device 300. The memory 32 includes a propulsion unit 40, a GPS receiver 41, an inertial measurement unit (IMU) 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a gimbal 50, an imaging device 60, and an imaging device 100. Stores programs etc. required to control the The memory 32 may be a computer readable recording medium, and may include at least one of a SRAM, a DRAM, an EPROM, an EEPROM, and a flash memory such as a USB memory. The memory 32 may be provided inside the UAV body 20. It may be provided to be removable from the UAV main body 20.

  The UAV control unit 30 controls the flight and imaging of the UAV 10 in accordance with a program stored in the memory 32. The UAV control unit 30 may be configured by a microprocessor such as a CPU or an MPU or a microcontroller such as an MCU. The UAV control unit 30 controls the flight and imaging of the UAV 10 in accordance with an instruction received from the remote control device 300 via the communication interface 34. The promotion unit 40 promotes the UAV 10. The propulsion unit 40 has a plurality of rotors and a plurality of drive motors for rotating the plurality of rotors. The propulsion unit 40 rotates the plurality of rotary blades via the plurality of drive motors in accordance with the instruction from the UAV control unit 30 to fly the UAV 10.

  The GPS receiver 41 receives a plurality of signals indicating times transmitted from a plurality of GPS satellites. The GPS receiver 41 calculates the position of the GPS receiver 41, that is, the position of the UAV 10 based on the received plurality of signals. The IMU 42 detects the attitude of the UAV 10. The IMU 42 detects, as the posture of the UAV 10, accelerations in the front, rear, left, right, and top three axial directions of the UAV 10, and angular velocities in three axial directions of pitch, roll, and yaw. The magnetic compass 43 detects the heading of the UAV 10. The barometric altimeter 44 detects the altitude at which the UAV 10 flies. The barometric pressure altimeter 44 detects the barometric pressure around the UAV 10, converts the detected barometric pressure into a height, and detects the height. The temperature sensor 45 detects the temperature around the UAV 10.

  The imaging device 100 includes an imaging unit 102 and a lens unit 200. The lens unit 200 is an example of a lens device. The imaging unit 102 includes an image sensor 120, an imaging control unit 110, and a memory 130. The image sensor 120 may be configured by a CCD or a CMOS. The image sensor 120 outputs image data of an optical image formed through the plurality of lenses 210 to the imaging control unit 110. The imaging control unit 110 may be configured by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, or the like. The imaging control unit 110 may control the imaging device 100 according to an operation command of the imaging device 100 from the UAV control unit 30. The memory 130 may be a computer readable recording medium, and may include at least one of SRAM, DRAM, EPROM, EEPROM, and flash memory such as USB memory. The memory 130 stores programs and the like necessary for the imaging control unit 110 to control the image sensor 120 and the like. The memory 130 may be provided inside the housing of the imaging device 100. The memory 130 may be provided to be removable from the housing of the imaging device 100.

  The lens unit 200 includes a plurality of lenses 210, a lens moving mechanism 212, and a lens control unit 220. The plurality of lenses 210 may function as a zoom lens, a varifocal lens, and a focus lens. At least a part or all of the plurality of lenses 210 is disposed movably along the optical axis. The lens unit 200 may be an interchangeable lens provided detachably to the imaging unit 102. The lens moving mechanism 212 moves at least a part or all of the plurality of lenses 210 along the optical axis. The lens control unit 220 drives the lens moving mechanism 212 according to the lens control command from the imaging unit 102 to move the one or more lenses 210 along the optical axis direction. The lens control command is, for example, a zoom control command and a focus control command.

  The imaging apparatus 100 configured in this manner performs autofocus processing (AF processing) to image a desired subject.

  The imaging apparatus 100 determines the distance from the lens to the subject (subject distance) in order to execute the AF process. As a method of determining the subject distance, there is a method of determining based on the blur amount of a plurality of images captured in a state in which the positional relationship between the lens and the imaging surface is different. Here, this method is referred to as a Bokeh Detection Auto Focus (BDAF) method.

For example, the blur amount (Cost) of the image can be expressed by the following equation (1) using a Gaussian function. In equation (1), x indicates the pixel position in the horizontal direction. σ indicates a standard deviation value.

  FIG. 3 shows an example of the curve represented by equation (1). By focusing the focusing lens on the lens position corresponding to the minimum point 502 of the curve 500, an object included in the image I can be focused.

FIG. 4 is a flowchart showing an example of the distance calculation procedure of the BDAF method. First, the imaging apparatus 100, in a state in which the lens and the image plane is in the first positional relationship, for storing the image I ₁ of the first sheet in the memory 130 by photographing. Then, by moving the imaging surface of the focusing lens or the image sensor 120 in the optical axis direction, the lens and the imaging surface in the state in the second positional relationship, captured image I ₂ of the second sheet by the imaging device 100 Then, it is stored in the memory 130 (S101). For example, as in the case of so-called mountain climbing AF, the imaging lens of the focus lens or the image sensor 120 is moved in the optical axis direction so as not to exceed the in-focus point. The amount of movement of the focusing lens or the imaging surface of the image sensor 120 may be, for example, 10 μm.

Then, the imaging device 100 divides the image _{I 1} into a plurality of regions (S102). Calculates a feature amount for each pixel in the image I2, may divide the image I ₁ into a plurality of regions of pixel groups having a feature amount that is similar as a single region. The pixel groups in the range that is set in the AF processing frame of the image I ₁ may be divided into a plurality of regions. Imaging device 100 divides the image I ₂ into a plurality of regions corresponding to a plurality of areas of the image I _1. Object image pickup apparatus 100 includes a respective blur amount of the plurality of regions of the image I _1, based on the respective blur amount of the plurality of regions of the image I _2, respectively included in the plurality of regions for each of a plurality of regions The distance up to is calculated (S103).

The procedure of calculating the distance will be further described with reference to FIG. The distance from the lens L (principal point) to the object 510 (object plane) is A, the distance from the lens L (principal point) to the position (image plane) where the object 510 forms an image on the imaging plane is B, and the focal distance is F I assume. In this case, the relationship between the distance A, the distance B, and the focal length F can be expressed by the following formula (2) from the lens formula.

  The focal length F is specified by the lens position. Therefore, if the distance B at which the object 510 forms an image on the imaging plane can be specified, the distance A from the lens L to the object 510 can be specified using Equation (2).

  As shown in FIG. 5, the distance B is specified by calculating the position where the object 510 forms an image from the size of the blur of the object 510 projected on the imaging surface (circles of confusion 512 and 514), and further the distance A can be identified. That is, the imaging position can be specified in consideration of the fact that the size of the blurring (the amount of blurring) is proportional to the imaging plane and the imaging position.

Here, the distance from the image I ₁ close to the imaging plane to the lens L is D ₁ . The distance from the far image I ₂ from the image plane to the lens L and D _2. Each image is blurred. The images at this point spread function (Point Spread Function) PSF, D ₁ and D ₂ are I _{d 1} and I _{d 2} respectively. In this case, for example, the image I ₁ can be expressed by the following equation (3) by a convolution operation.

Further, the Fourier transform function of the image data _{I d1} and _{I d2} as f, the point spread image _{I d1} and _{I d2} function PSF ₁ and the optical transfer function of the PSF ₂ Fourier transform (Optical Transfer Function) of the OTF ₁ and OTF _The ratio is taken as ₂ as in the following equation (4).

The value shown in Equation (4) C, each amount of blur change amount of the image _{I d1} and _{I d2,} i.e., the value C is equivalent to the difference between the blurring amount and blur of the image _{I d2n} image _{I d1} .

  According to the above-described BDAF method, it is necessary to acquire at least two images, and therefore it may take time for the AF processing.

  As a method capable of relatively shortening the time of AF processing, a phase difference AF processing method of performing AF processing using two parallax images is known. As the phase difference AF processing method, there is an image plane phase difference AF (PDAF) method. According to the PDAF method, for example, an image sensor in which a plurality of pixels are arranged as shown in FIG. 6 is used. The image sensor includes a pixel array 600 including a pixel block of a Bayer array that outputs color component detection signals, and a pixel array 602 and a pixel array 604 including pixel blocks including pixels that output phase difference detection signals. The direction in which light is incident is different between the pixel P1 that outputs the phase difference detection signal included in the pixel column 602 and the pixel P2 that outputs the phase difference detection signal included in the pixel column 604. Therefore, an image having a different phase is obtained from the image obtained from the pixel P1 and the image obtained from the pixel P2. Then, the distance to the subject can be derived based on the amount and direction of deviation between the image included in the image obtained from the pixel P1 and the image included in the image obtained from the pixel P2.

  The PDAF method can often execute AF processing faster than the BDAF method. However, the pixels for phase difference detection are relatively spaced apart. Therefore, the image quality of the image obtained from the image P1 and the pixel P2 is relatively low. Therefore, if a high frequency component is included in the spatial frequency band of the image captured by the image sensor, the accuracy of the AF processing may decrease.

  Therefore, according to the imaging apparatus 100 according to the present embodiment, the AF process of the BDAF method and the AF process of the PADF method are selected by selecting one on the basis of the spatial frequency band of the image, or by combining both. Execute the process

  The imaging control unit 110 includes a derivation unit 112 and a focusing control unit 114. The derivation unit 112 derives the spatial frequency band of the image captured by the image sensor 120. The deriving unit 112 may derive the spatial frequency band of the image by performing Fourier transform on the image to decompose the image into spatial frequency components.

  The focusing control unit 114 sets the blur amount of a plurality of images captured in a state in which the positional relationship between the imaging surface of the image sensor 120 and the lens 210 is different based on the spatial frequency band of the image derived by the derivation unit 112. The imaging surface of the image sensor 120 and the lens 210 can be selected by selecting either or both of BDAF processing that performs autofocus processing and PDAF processing that performs autofocus processing using a phase difference method. Control the positional relationship of The focusing control unit 114 may control the position of the focusing lens by selecting one of BDAF processing and PDAF processing or combining both based on the spatial frequency band of the image. The focusing control unit 114 is an example of a control unit. The BDAF process is an example of a first AF process. The PFAF process is an example of the second AF process.

  The focusing control unit 114 selects one of BDAF processing and PDAF processing based on the spatial frequency band of the image derived by the deriving unit 112 and the spatial frequency band predetermined for the PDAF processing. The positional relationship between the imaging surface of the image sensor 120 and the lens 210 may be controlled by combining the two or both. The focusing control unit 114 selects one of BDAF processing and PDAF processing based on the ratio of the spatial frequency band included in the predetermined spatial frequency band among the spatial frequency bands of the image derived by the deriving unit 112. The position relationship between the imaging surface of the image sensor 120 and the lens 210 may be controlled by selecting or combining both.

  The focusing control unit 114 selects the PDAF process when the ratio is equal to or higher than the threshold, and selects the BDAF process when the ratio is smaller than the threshold, thereby determining the positional relationship between the imaging surface of the image sensor 120 and the lens 210. You may control. The focusing control unit 114 selects the PDAF process if the ratio is equal to or more than the first threshold, and combines the BDAF process and the PDAF process if the ratio is smaller than the first threshold and equal to or more than the second threshold. If it is smaller than two threshold values, the positional relationship between the imaging surface of the image sensor 120 and the lens 210 may be controlled by selecting the BDAF processing. The focusing control unit 114 weights each of the distance to the subject specified by the BDAF processing and the distance to the subject specified by the PADF processing, for example, and the position of the focusing lens is based on the weighted distance. You may control The focusing control unit 114 may change the weight of the BDAF process and the PDAF process according to the ratio.

  The focusing control unit 114 compares the spatial frequency band of the image derived by the deriving unit 112 with a predetermined spatial frequency band reproducible by a plurality of pixels generating a phase difference detection signal. The positional relationship between the imaging surface of the image sensor 120 and the lens 210 may be controlled by selecting either or both of the processing and the PDAF processing.

  The focusing control unit 114 is output from the image sensor 120 in which a plurality of pixels for generating a phase difference detection signal and a plurality of other pixels for generating color component signals are arranged in a predetermined array pattern. PDAF processing may be performed based on the phase difference detection signal. The plurality of pixels generating the phase difference detection signal are, for example, the pixel P1 and the pixel P2 shown in FIG. The plurality of other pixels that generate color component signals are, for example, the pixel R, the pixel G, and the pixel B shown in FIG. The predetermined spatial frequency band may be, for example, a spatial frequency band reproducible by the pixel P1 and the pixel P2.

  As shown in FIGS. 7A and 7B, the spatial frequency band reproducible by the pixel R, the pixel G, and the pixel B is, for example, a region 700. On the other hand, a spatial frequency band reproducible by the pixel P1 and the pixel P2 is, for example, a region 702. Region 700 and region 702 each include a low frequency component. The frequency components included in the region 702 have a smaller ratio of high frequency components than the frequency components included in the region 700. Therefore, when the image captured by the image sensor 120 contains many high frequency components, the spatial frequency components reproducible in the pixels P1 and P2 are small, and the focusing control unit 114 can not execute the AF processing with high accuracy by the PDAF processing. There is a case.

  On the other hand, BDAF processing is performed based on the color component signal. That is, the focusing control unit 114 can execute the BDAF processing with high accuracy on an image including the spatial frequency band in the range of the area 700.

  7A and 7B show the relationship between the spatial frequency and the gain of the spatial frequency. As the gain is larger, more spatial frequency components corresponding to the gain are included in the image. In FIGS. 7A and 7B, a region 710 is a region corresponding to the spatial frequency band of the image derived by the deriving unit 112. Region 712 is a region corresponding to the spatial frequency band of the image included in region 702 in the spatial frequency band of the image. The focusing control unit 114 controls the position of the focusing lens by selecting one of BDAF processing and PDAF processing or combining both on the basis of the ratio of the area of the area 712 to the area of the area 710. You may

  For example, as shown in FIG. 7A, the focusing control unit 114 may select PDAF processing to control the position of the focusing lens if the ratio of the area 712 is equal to or greater than a predetermined threshold. On the other hand, as shown in FIG. 7B, when the ratio of the area 712 is smaller than a predetermined threshold, the focusing control unit 114 may select BDAF processing to control the position of the focusing lens.

  The focusing control unit 114 selects the PDAF process if the largest spatial frequency component included in the spatial frequency band of the image derived by the deriving unit 112 is equal to or less than a predetermined spatial frequency component, and the largest space is obtained. If the frequency component is greater than a predetermined spatial frequency component, BDAF processing may be selected to control the position of the focus lens. The predetermined spatial frequency component may be determined based on, for example, the largest spatial frequency component among spatial frequency bands reproducible in the pixels P1 and P2.

  FIG. 8 is a flowchart showing an example of the procedure of the AF process. First, the derivation unit 112 acquires an image captured by the image sensor 120. The derivation unit 112 derives the spatial frequency band of the image. Further, the derivation unit 112 derives a ratio B of the space frequency band in the image to the space frequency band predetermined for the PDAF processing (S200).

  The focusing control unit 114 determines whether the derived ratio B is equal to or greater than a predetermined first threshold Th1 (S202). If the ratio B is equal to or greater than the first threshold Th1, the focusing control unit 114 selects the PDAF process to control the positional relationship between the imaging surface of the image sensor 120 and the lens 210 (S204). On the other hand, if the ratio B is smaller than the first threshold Th1, the focusing control unit 114 selects the BDAF processing to control the positional relationship between the imaging surface of the image sensor 120 and the lens 210 (S206).

  As described above, the focusing control unit 114 switches between the PDAF process and the BDAF process based on the spatial frequency component included in the image used for the AF process. As a result, when the image contains a large amount of high frequency components, the focusing control unit 114 can prevent the accuracy of the AF processing from being lowered by executing the PDAF processing. In addition, when the image does not contain a large amount of high frequency components, the focusing control unit 114 can prevent the AF processing from taking time by executing the BDAF processing. Therefore, according to the imaging apparatus 100 according to the present embodiment, it is possible to suppress an increase in time of AF processing and a decrease in accuracy of AF processing in a well-balanced manner. The low frequency component and the high frequency component may be distinguished on the basis of the first threshold Th.

  FIG. 9 is a flowchart showing another example of the procedure of the AF process. First, the derivation unit 112 derives the spatial frequency band of the image. Further, the derivation unit 112 derives a ratio B of the space frequency band in the image to the space frequency band predetermined for the PDAF processing among the space frequency bands of the image (S300).

  The focusing control unit 114 determines whether the derived ratio B is equal to or greater than a predetermined first threshold Th1 (S302). If the ratio B is equal to or greater than the first threshold Th1, the focusing control unit 114 selects the PDAF process to control the positional relationship between the imaging surface of the image sensor 120 and the lens 210 (S304). On the other hand, if the ratio B is smaller than the first threshold Th1, the focusing control unit 114 determines whether the ratio B is smaller than the first threshold Th1 and is equal to or greater than a predetermined second threshold Th2 (S306).

  If the ratio B is smaller than the first threshold Th1 and equal to or greater than a predetermined second threshold Th2, the focusing control unit 114 combines the PDAF processing and the BDAF processing, and the imaging surface of the image sensor 120 and the lens 210. Control the positional relationship with (S308). For example, the focusing control unit 114 may control the position of the focusing lens based on the position of the focusing lens specified in the PDAF processing and the position of the focusing lens specified in the BDAF processing. The focusing control unit 114 weights each of the positions of the focusing lens specified by the PDAF process and the BDAF process according to the ratio B, and controls the position of the focusing lens based on the weighted distances. Good. The focus control unit 114 may perform weighting on each distance such that the weighting of the distance specified in the PDAF processing is larger than the weight of the distance specified in the BDAF processing as the ratio B is larger. .

  If the ratio B is smaller than the second threshold value, the focusing control unit 114 selects BDAF processing to control the positional relationship between the imaging surface of the image sensor 120 and the lens 210 (S310).

  As described above, based on the spatial frequency component included in the image, the focusing control unit 114 performs the AF process by switching the PDAF process and the BDAF process, or combining the PDAF process and the BDAF process. As a result, when the image contains a large amount of high frequency components, the focusing control unit 114 can prevent the accuracy of the AF processing from being degraded by executing the PDAF processing. In addition, when the image does not contain a large amount of high frequency components, the focusing control unit 114 can execute the BDAF process to prevent the AF process from taking time. When the ratio B is, for example, about 50%, the AF process is executed by combining the PDAF process and the BDAF process, instead of selecting either one of the PDAF process and the BDAF process. As a result, when the ratio B is, for example, about 50%, the accuracy of the AF processing can be improved as compared to the case where the AF processing is performed by only one of the PDAF processing and the BDAF processing. Therefore, according to the imaging apparatus 100 according to the present embodiment, it is possible to suppress an increase in time of AF processing and a decrease in accuracy of AF processing in a well-balanced manner.

  FIG. 10 shows an example of a computer 1200 in which aspects of the present invention may be fully or partially embodied. A program installed on computer 1200 can cause computer 1200 to act as an operation or one or more "parts" of the device according to embodiments of the invention. Alternatively, the program may cause the computer 1200 to execute the operation or one or more “parts”. The program can cause the computer 1200 to execute the process according to the embodiment of the present invention or the steps of the process. Such programs may be executed by CPU 1212 to cause computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

  The computer 1200 according to the present embodiment includes a CPU 1212 and a RAM 1214, which are interconnected by a host controller 1210. Computer 1200 also includes a communication interface 1222, an input / output unit, which are connected to host controller 1210 via an input / output controller 1220. Computer 1200 also includes a ROM 1230. The CPU 1212 operates in accordance with programs stored in the ROM 1230 and the RAM 1214 to control each unit.

  The communication interface 1222 communicates with other electronic devices via a network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores therein a boot program or the like executed by the computer 1200 upon activation, and / or a program dependent on the hardware of the computer 1200. The program is provided via a computer readable recording medium or network such as a CR-ROM, a USB memory or an IC card. The program is installed in the RAM 1214 or ROM 1230, which is also an example of a computer-readable recording medium, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200 and brings about coordination between the programs and the various types of hardware resources. An apparatus or method may be configured by implementing the operation or processing of information in accordance with the use of computer 1200.

  For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 executes the communication program loaded in the RAM 1214, and performs communication processing to the communication interface 1222 based on the processing described in the communication program. You may order. The communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as the RAM 1214 or a USB memory under the control of the CPU 1212 and transmits the read transmission data to the network, or The received data received from the network is written in a reception buffer area or the like provided on the recording medium.

  In addition, the CPU 1212 allows the RAM 1214 to read all or necessary portions of a file or database stored in an external recording medium such as a USB memory, and executes various types of processing on data on the RAM 1214. Good. The CPU 1212 may then write back the processed data to the external storage medium.

  Various types of information such as various types of programs, data, tables, and databases may be stored in the recording medium and subjected to information processing. The CPU 1212 describes various types of operations, information processing, condition judgment, conditional branching, unconditional branching, information retrieval, which are described throughout the present disclosure for data read from the RAM 1214 and specified by a program instruction sequence. Various types of processing may be performed, including / replacement etc, and the results written back to RAM 1214. Further, the CPU 1212 may search for information in a file in a recording medium, a database or the like. For example, when a plurality of entries each having the attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for an entry matching the condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby associate the first attribute satisfying the predetermined condition with the first attribute An attribute value of the second attribute may be acquired.

  The programs or software modules described above may be stored on computer readable storage medium on or near computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer readable storage medium, whereby the program can be transmitted to the computer 1200 via the network. provide.

  The order of execution of each process such as operations, procedures, steps, and steps in the apparatuses, systems, programs, and methods shown in the claims, the specification, and the drawings is particularly "before", "before" It should be noted that it can be realized in any order, unless explicitly stated as etc., and unless the output of the previous process is used in the later process. With regard to the operation flow in the claims, the specification, and the drawings, even if it is described using “first,” “next,” etc. for the sake of convenience, it means that it is essential to carry out in this order. is not.

10 UAV
Reference Signs List 20 UAV main body 30 UAV control unit 32 memory 34 communication interface 40 promotion unit 41 GPS receiver 42 inertia measurement device 43 magnetic compass 44 barometric altimeter 45 temperature sensor 50 gimbal 60 imaging device 100 imaging device 102 imaging unit 110 imaging control unit 112 derivation unit 114 focus control unit 120 image sensor 130 memory 200 lens unit 210 lens 212 lens moving mechanism 220 lens control unit 300 remote control device 1200 computer 1210 host controller 1212 CPU
1214 RAM
1220 I / O controller 1222 communication interface 1230 ROM

Claims (8)

A deriving unit that derives a spatial frequency band of the captured image;
The first AF is performed based on the blur amount of a plurality of images captured in a state in which the positional relationship between the imaging surface and the lens is different based on the spatial frequency band of the image derived by the derivation unit. It has a control unit that controls the positional relationship between the imaging surface and the lens by selecting either one of the processing and the second AF processing that performs autofocus processing in the phase difference method, or by combining both. ,
The control unit is configured such that a ratio of a spatial frequency band included in a spatial frequency band predetermined for the second AF process in the spatial frequency band of the image derived by the derivation unit is equal to or greater than a first threshold In the case where the second AF process is selected and the ratio is smaller than the first threshold and equal to or greater than the second threshold, the first AF process and the second AF process are combined and smaller than the second threshold. An imaging control apparatus that controls the positional relationship between the imaging surface and the lens by selecting the first AF process .   The control unit compares the spatial frequency band of the image derived by the derivation unit with a predetermined spatial frequency band reproducible by a plurality of pixels generating a phase difference detection signal. The imaging control apparatus according to claim 1, wherein the positional relationship between the imaging surface and the lens is controlled by selecting one of the first AF processing and the second AF processing, or combining both. The second AF process is performed based on the phase difference detection signal output from an image sensor in which the plurality of pixels and a plurality of other pixels generating color component signals are arranged in a predetermined array pattern. The imaging control device according to claim 2 . An imaging control apparatus according to any one of claims 1 to 3 .
An imaging device comprising: an image sensor having the imaging surface; and the lens. An imaging device according to claim 4 ;
And a support mechanism for supporting the imaging device. A mobile body mounted and moving with the imaging system according to claim 5 . Deriving a spatial frequency band of the captured image;
A first AF process for performing an autofocus process based on the blur amount of a plurality of images captured in a state in which the positional relationship between the imaging surface and the lens is different based on the spatial frequency band of the derived image Controlling the positional relationship between the imaging surface and the lens by selecting either one of the second AF processing to execute the autofocus processing in the phase difference method or combining the two .
In the controlling step, a ratio of a spatial frequency band included in a spatial frequency band predetermined to the second AF processing among the spatial frequency bands of the image derived in the deriving step is equal to or greater than a first threshold If the second AF process is selected and the ratio is smaller than the first threshold and equal to or greater than the second threshold, the first AF process and the second AF process are combined and smaller than the second threshold In this case, an imaging control method includes controlling the positional relationship between the imaging surface and the lens by selecting the first AF process . A program for causing a computer to function as the imaging control device according to any one of claims 1 to 3 .

Images