WO2021031833A1 - Control device, photographing system, control method, and program - Google Patents

Abstract

A distance measurement sensor for measuring a distance to an object according to reflective light of an optical pulse cannot precisely measure the distance to the object sometimes because of receiving external light other than the reflective light of the optical pulse such as sunlight. A control device comprises a circuit. The circuit is configured to: when a signal satisfies a first condition of the reliability of a shown distance, perform focus control for focusing on an object on the basis of a first target position relationship, determined according to the distance, between a photographing surface of a photographing device and a focusing lens; and when the signal does not satisfy the first condition, perform focus control on the basis of a second target position relationship between the photographing surface and the focusing lens, wherein the second target position relationship is determined according to a blur amount of a first image photographed by the photographing device when the position relationship between the photographing surface and the focusing lens is a first position relationship and a blur amount of a second image photographed by the photographing device when the position relationship between the photographing surface and the focusing lens is a second position relationship.

Description

Control device, camera system, control method and program 【Technical Field】

The invention relates to a control device, a camera system, a control method and a program.

【Background technique】

Patent Document 1 describes that the distance to the target subject is measured based on the reflected light of the light pulse.

[Prior Art Literature]

[Patent Literature]

[Patent Document 1] JP 2006-79074 A

[Content of the invention]

[Technical problems to be solved by the invention]

The distance measuring sensor that measures the distance to the object based on the reflected light of the light pulse may not be able to accurately measure the distance to the object because it receives external light that is not reflected light of the light pulse such as sunlight.

[Technical means used to solve the problem]

The control device according to one aspect of the present invention may be a control device that controls an imaging device including a distance measuring sensor and a focusing lens. The distance measuring sensor includes a light emitting element that emits pulsed light and receives reflections that include pulsed light from an object. A light-receiving element that outputs a signal corresponding to the amount of light received by the light, and measures the distance to the object based on the signal. The control device may include a circuit configured to perform focusing on the object based on the first target positional relationship between the imaging surface of the imaging device and the focus lens determined according to the distance when the signal satisfies the first condition showing the reliability of the distance Focus control. The circuit can be configured as follows: when the signal does not satisfy the first condition, the first image captured by the imaging device when the positional relationship between the imaging surface and the focusing lens is the first positional relationship and the positional relationship between the imaging surface and the focusing lens are The second positional relationship is the second target positional relationship between the imaging surface determined by the respective blur amounts of the second image captured by the imaging device and the focus lens, and the focus control is performed.

The circuit may be configured to perform focus control according to the second target position relationship when the first difference degree showing the degree of difference between the first image and the second image satisfies the second condition showing the reliability of the second target position relationship.

The circuit may be configured to perform focus control according to contrast AF when the first degree of difference does not satisfy the second condition.

The circuit can be configured as follows: when the first degree of difference does not satisfy the second condition, the positional relationship between the imaging surface and the focusing lens is changed from the second positional relationship to the third positional relationship; when the positional relationship between the imaging surface and the focusing lens is shown When it is the third positional relationship, when the second degree of difference between the third image taken by the imaging device and the first image or the second image satisfies the second condition, it is based on the difference between the first image or the second image and the third image The third target positional relationship between the imaging surface and the focusing lens determined by the respective blur amounts of, performs focus control; when the second degree of difference does not meet the second condition, focus control is performed according to contrast AF.

When the signal shows that the amount of light is contained within the preset range, the signal may satisfy the first condition.

The circuit may be configured to: when the signal does not meet the first condition, after acquiring the first image captured by the imaging device when the positional relationship between the imaging surface and the focusing lens is the first positional relationship, the positional relationship between the imaging surface and the focusing lens is converted Is a second positional relationship, and acquires a second image captured by the imaging device when the positional relationship between the imaging surface and the focus lens is the second positional relationship.

The circuit may be configured as follows: when the signal does not satisfy the first condition, after acquiring the first image taken by the imaging device when the positional relationship between the imaging surface and the focusing lens is the first positional relationship, the focusing lens is moved according to the first target positional relationship Move the predetermined distance in the determined direction, thereby moving the positional relationship between the imaging surface and the focusing lens to the second positional relationship.

The camera system according to an aspect of the present invention may include the above-mentioned control device, a distance measuring sensor, and a camera device.

The control method according to an aspect of the present invention may be a control method for controlling an imaging device including a distance measuring sensor and a focus lens. The distance measuring sensor includes a light emitting element that emits pulsed light and receives reflections that include pulsed light from an object. A light-receiving element that outputs a signal corresponding to the amount of light received by the light, and measures the distance to the object based on the signal. The control method may include, when the signal satisfies the first condition showing the reliability of the distance, performing focus control of focusing on the object based on the first target position relationship of the imaging surface of the imaging device and the focus lens determined according to the distance. The control method may include: when the signal does not satisfy the first condition, based on the first image taken by the imaging device when the positional relationship between the imaging surface and the focusing lens is the first positional relationship and the positional relationship between the imaging surface and the focusing lens In the case of the second positional relationship, the second target positional relationship between the imaging surface determined by the respective blur amounts of the second image captured by the imaging device and the focus lens is executed to perform focus control.

The program according to one aspect of the present invention may be a program for causing a computer to function as the above-mentioned control device.

According to an aspect of the present invention, it is possible to avoid meaningless driving of the focus lens, and to perform focus control in a better manner.

In addition, the above summary does not enumerate all the essential features of the present invention. In addition, sub-combinations of these feature groups may also constitute inventions.

【Explanation of drawings】

Fig. 1 is an example of an external perspective view of an imaging system.

FIG. 2 is an example of an external perspective view showing another form of the imaging system.

Fig. 3 is a diagram showing functional blocks of the camera system.

FIG. 4 is a diagram showing an example of a curve showing the relationship between the blur amount and the lens position.

FIG. 5 is a diagram showing an example of the process of calculating the distance to the object based on the blur amount.

FIG. 6 is a diagram for explaining the relationship between the object position, the lens position, and the focal length.

FIG. 7 is a flowchart showing one example of the AF processing procedure of the imaging device.

FIG. 8 is a diagram showing the relationship between the TOF calculation result and the focus lens position.

FIG. 9 is a diagram showing an example of the action of the focus lens when performing AF.

FIG. 10 is a diagram showing an example of the action of the focus lens when performing AF.

FIG. 11 is a diagram showing an example of the action of the focus lens when performing AF.

FIG. 12 is a diagram showing an example of the action of the focus lens when performing AF.

Fig. 13 is a diagram showing an example of the appearance of an unmanned aircraft and a remote control device.

FIG. 14 is a diagram showing an example of the hardware configuration.

【detailed description】

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, all the feature combinations described in the embodiments are not necessarily necessary for the solution of the invention. It is obvious to a person skilled in the art that various changes or improvements can be made to the following embodiments. It is obvious from the description of the claims that all such changes or improvements can be included in the technical scope of the present invention.

The claims, the description, the drawings of the description, and the abstract of the description include matters that are the subject of copyright protection. As long as anyone copies these files as shown in the patent office files or records, the copyright owner will not raise an objection. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams. Here, the blocks may show (1) the stages of the process of performing operations or (2) the "parts" of the device that perform operations. Specific stages and "parts" can be implemented by programmable circuits and/or processors. Dedicated circuits may include digital and/or analog hardware circuits. May include integrated circuits (ICs) and/or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits can include logical AND, logical OR, logical exclusive OR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate array (FPGA), programmable logic array (PLA) ) And other memory components.

The computer-readable medium may include any tangible device that can store instructions to be executed by a suitable device. As a result, the computer-readable medium having instructions stored thereon includes a product including instructions that can be executed to create means for performing operations specified by the flowchart or block diagram. As examples of computer-readable media, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like may be included. As a more specific example of the computer readable medium, it may include floppy (registered trademark) disk, floppy disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blue-lay (registered trademark) Blu-ray discs, memory sticks, integrated circuit cards, etc.

The computer-readable instructions may include any one of source code or object code described in any combination of one or more programming languages. The source code or object code includes traditional procedural programming languages. Traditional procedural programming languages can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or Smalltalk (registered trademark), JAVA (registered trademark) , C++ and other object-oriented programming languages and "C" programming language or similar programming languages. The computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the Internet to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device. The processor or programmable circuit can execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

FIG. 1 is an example of an external perspective view of an imaging system 10 according to this embodiment. The imaging system 10 includes an imaging device 100, a supporting mechanism 200 and a grip 300. The imaging device 100 includes a TOF sensor 160. The support mechanism 200 uses actuators to rotatably support the imaging device 100 around the roll axis, the pitch axis, and the yaw axis. The support mechanism 200 can change or maintain the posture of the imaging device 100 by rotating the imaging device 100 around at least one of the roll axis, the pitch axis, and the yaw axis. The support mechanism 200 includes a roll axis drive mechanism 201, a pitch axis drive mechanism 202, and a yaw axis drive mechanism 203. The supporting mechanism 200 also includes a base 204 for fixing the yaw axis driving mechanism 203. The grip 300 is fixed to the base 204. The holding part 300 includes an operation interface 301 and a display part 302. The imaging device 100 is fixed to the pitch axis driving mechanism 202.

The operation interface 301 accepts instructions for operating the camera device 100 and the supporting mechanism 200 from the user. The operation interface 301 may include a shutter/recording button for instructing the camera device 100 to shoot or record. The operation interface 301 may include a power/function button for instructing to turn on or off the power of the camera 10 and to switch the still image shooting mode or the moving image shooting mode of the camera 100.

The display part 302 can display an image captured by the imaging device 100. The display unit 302 can display a menu screen for operating the imaging device 100 and the supporting mechanism 200. The display unit 302 may be a touch screen display that accepts instructions for operating the imaging device 100 and the supporting mechanism 200.

FIG. 2 is an example of an external perspective view showing another form of the imaging system 10. As shown in FIG. 2, the camera system 10 can be used in a state where a mobile terminal including a display such as a smartphone 400 is fixed to the side of the grip 300. The user holds the grip 300 and takes a still image or a moving image through the imaging device 100. A display such as the smartphone 400 displays still images or moving images of the imaging device 100.

FIG. 3 is a diagram showing functional blocks of the imaging system 10. The imaging device 100 includes an imaging control unit 110, an image sensor 120, a memory 130, a lens control unit 150, a lens driving unit 152, a plurality of lenses 154, and a TOF sensor 160.

The image sensor 120 may be composed of CCD or CMOS. The image sensor 120 is an example of an image sensor used for shooting. The image sensor 120 outputs image data of optical images formed by the plurality of lenses 154 to the imaging control unit 110. The imaging control unit 110 may be constituted by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, or the like.

The imaging control unit 110 follows the operation instruction of the grip 300 to the imaging device 100, and the imaging control unit 110 performs a demosaicing process on the image signal output from the image sensor 120 to generate image data. The imaging control unit 110 stores image data in the memory 130. The imaging control unit 110 controls the TOF sensor 160. The imaging control unit 110 is an example of a circuit. The TOF sensor 160 is a time-of-flight type sensor that measures the distance to an object. The imaging apparatus 100 adjusts the position of the focus lens based on the distance measured by the TOF sensor 160, thereby performing focus control.

The memory 130 may be a computer-readable storage medium, and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 130 stores programs and the like necessary for the imaging control unit 110 to control the image sensor 120 and the like. The memory 130 may be provided inside the housing of the imaging device 100. The grip 300 may include other memory for storing image data captured by the imaging device 100. The holding part 300 may have a slot through which the memory can be detached from the housing of the holding part 300.

The multiple lenses 154 can function as zoom lenses, varifocal lenses, and focus lenses. At least a part or all of the plurality of lenses 154 are configured to be movable along the optical axis. The lens control unit 150 drives the lens driving unit 152 in accordance with a lens control command from the imaging control unit 110 to move one or more lenses 154 in the optical axis direction. The lens control commands are, for example, zoom control commands and focus control commands. The lens driving part 152 may include a voice coil motor (VCM) that moves at least a part or all of the plurality of lenses 154 in the optical axis direction. The lens driving part 152 may include a motor such as a DC motor, a coreless motor, or an ultrasonic motor. The lens driving unit 152 can transmit the power from the motor to at least a part or all of the plurality of lenses 154 via mechanism components such as cam rings and guide shafts, and move at least a part or all of the plurality of lenses 154 along the optical axis.

The imaging device 100 further includes a posture control unit 210, an angular velocity sensor 212, and an acceleration sensor 214. The angular velocity sensor 212 detects the angular velocity of the imaging device 100. The angular velocity sensor 212 detects the respective angular velocities of the imaging device 100 around the roll axis, the pitch axis, and the yaw axis. The posture control unit 210 acquires angular velocity information related to the angular velocity of the imaging device 100 from the angular velocity sensor 212. The angular velocity information may show the respective angular velocities of the camera device 100 around the roll axis, the pitch axis, and the yaw axis. The posture control unit 210 acquires acceleration information related to the acceleration of the imaging device 100 from the acceleration sensor 214. The acceleration information may also show the acceleration of the camera device 100 in each direction of the roll axis, the pitch axis, and the yaw axis.

The angular velocity sensor 212 and the acceleration sensor 214 may be provided in a housing that houses the image sensor 120, the lens 154, and the like. In this embodiment, a mode of the integrated configuration of the imaging device 100 and the support mechanism 200 will be described. However, the supporting mechanism 200 may include a base for detachably fixing the camera device 100. In this case, the angular velocity sensor 212 and the acceleration sensor 214 may be provided outside the housing of the imaging device 100 such as a base.

The posture control unit 210 controls the support mechanism 200 based on angular velocity information and acceleration information to maintain or change the posture of the imaging device 100. The posture control unit 210 controls the support mechanism 200 according to the operation mode of the support mechanism 200 for controlling the posture of the imaging device 100 to maintain or change the posture of the imaging device 100.

The working modes include the following modes: operating at least one of the roll axis driving mechanism 201, the pitch axis driving mechanism 202, and the yaw axis driving mechanism 203 of the support mechanism 200 so that the posture change of the camera device 100 follows the base 204 of the support mechanism 200 The posture changes. The working modes include the following modes: each of the roll axis drive mechanism 201, the pitch axis drive mechanism 202 and the yaw axis drive mechanism 203 of the support mechanism 200 is operated so that the posture change of the camera device 100 follows the posture change of the base 204 of the support mechanism 200 . The working modes include the following modes: each of the pitch axis driving mechanism 202 and the yaw axis driving mechanism 203 of the support mechanism 200 is operated so that the posture change of the camera device 100 follows the posture change of the base 204 of the support mechanism 200. The working modes include the following modes: only the yaw axis driving mechanism 203 is operated so that the posture change of the camera device 100 follows the posture change of the base 204 of the support mechanism 200.

The working mode may include: FPV (First Person View) mode in which the support mechanism 200 is operated to make the posture change of the camera device 100 follow the posture change of the base 204 of the support mechanism 200; and to maintain the posture of the camera device 100 And the fixed mode in which the support mechanism 200 works.

The FPV mode is a mode in which at least one of the roll axis drive mechanism 201, the pitch axis drive mechanism 202, and the yaw axis drive mechanism 203 is operated to make the posture change of the camera 100 follow the posture change of the base 204 of the support mechanism 200. The fixed mode is a mode in which at least one of the roll axis drive mechanism 201, the pitch axis drive mechanism 202, and the yaw axis drive mechanism 203 is operated in order to maintain the current posture of the imaging device 100.

The TOF sensor 160 includes a light emitting unit 162, a light receiving unit 164, a light emitting control unit 166, a light receiving control unit 167, and a memory 168. The TOF sensor 160 is an example of a distance measuring sensor.

The light emitting part 162 includes at least one light emitting element 163. The light emitting element 163 is a device that repeatedly emits high-speed modulated pulsed light such as an LED or laser. The light emitting element 163 may emit pulsed light which is infrared light. The light emission control unit 166 controls the light emission of the light emitting element 163. The light emission control section 166 can control the pulse width of the pulse light emitted from the light emitting element 163.

The light receiving unit 164 includes a plurality of light receiving elements 165 that measure the distance to the subject associated with each of the plurality of regions. The light receiving unit 164 is an example of an image sensor used for distance measurement. The plurality of light receiving elements 165 respectively correspond to each of the plurality of regions. The light receiving element 165 repeatedly receives reflected light of pulsed light from the object. The light receiving element 165 receives light including reflected light of pulsed light from the object, and outputs a signal corresponding to the amount of received light. The light receiving control unit 167 controls the light receiving element 165 to receive light. The light receiving control unit 167 measures the distance to the subject associated with each of the plurality of areas based on the signal output from the light receiving element 165. The light receiving control unit 167 measures the distance to the subject associated with each of the plurality of regions based on the amount of reflected light repeatedly received by the light receiving element 165 in the preset light receiving period. The light receiving control unit 167 can measure the distance to the subject by determining the phase difference between the pulsed light and the reflected light based on the amount of reflected light repeatedly received by the light receiving element 165 in a preset light receiving period. The light receiving unit 164 can measure the distance to the subject by reading the frequency change of the reflected wave. This is called FMCW (Frequency Modulated Continuous Wave) mode.

The memory 168 may be a computer-readable recording medium, and may include at least one of SRAM, DRAM, EPROM, and EEPROM. The memory 168 stores a program necessary for the light emitting control unit 166 to control the light emitting unit 162, a program necessary for the light receiving control unit 167 to control the light receiving unit 164, and the like.

The auto focus (AF) method executed by the imaging device 100 will be described. The imaging device 100 may move the focus lens according to the distance (subject distance) from the imaging device 100 to the subject measured by the TOF sensor 160 to control the positional relationship between the focus lens and the imaging surface of the image sensor 120.

The imaging control unit 110 may perform contrast AF by deriving the contrast evaluation value of the image captured by the imaging device 100 while moving the focus lens, and determining the position of the focus lens when the contrast evaluation value reaches a peak. The imaging control unit 110 may apply a contrast evaluation filter to the image captured by the imaging device 100, thereby deriving the contrast evaluation value of the image. The imaging control unit 110 may determine the position of the focus lens focused on the designated subject based on the contrast evaluation value to perform contrast AF.

As another AF method, there is a method in which the focus lens is moved to determine the blur amount of a plurality of images captured in a state where the positional relationship between the focus lens and the imaging surface of the image sensor 120 is different. In this article, the AF using this method is called the Bokeh Detection Auto Foucus (BDAF) method. Specifically, DFD (Depth From Defocus) calculation is performed in BDAF to perform AF.

For example, the amount of image blur (Cost) can be expressed by the following equation (1) using a Gaussian function. In formula (1), x represents the pixel position in the horizontal direction. σ represents the standard deviation value.

【Formula 1】

FIG. 4 shows an example of a curve representing the relationship between the amount of image blur (Cost) and the focus lens position. C1 is the blur amount of the image acquired when the focus lens is located at x1. C2 is the blur amount of the image acquired when the focus lens is located at x2. It is possible to focus on the subject by aligning the focus lens to the lens position x0 corresponding to the minimum point 502 of the curve 500 determined according to the blur amount C1 and the amount C2 in consideration of the optical characteristics of the lens 154.

Fig. 5 is a flowchart showing an example of a distance calculation process in the BDAF method. The imaging control unit 110 takes a first image and stores it in the memory 130 when the lens 154 and the imaging surface of the image sensor 120 are in a first positional relationship. The imaging control unit 110 moves the lens 154 in the optical axis direction so that the lens 154 and the imaging surface are in a second positional relationship, and the second image is captured by the imaging device 100 and stored in the memory 130 (S201). For example, the imaging control unit 110 changes the positional relationship between the lens 154 and the imaging surface from the first positional relationship to the second positional relationship by moving the focus lens along the optical axis direction. The amount of movement of the lens may be about 10 μm, for example.

Then, the imaging control unit 110 divides the first image into a plurality of regions (S202). The imaging control unit 110 may calculate a feature amount for each pixel in the first image, and divide the first image into a plurality of regions by using a group of pixels having similar feature amounts as one region. The imaging control unit 110 may divide the pixel group set as the range of the AF processing frame in the first image into a plurality of regions. The imaging control unit 110 divides the second image into a plurality of regions corresponding to the plurality of regions of the first image. The imaging control unit 110 calculates the distance to the subject corresponding to the object contained in each of the multiple areas based on the respective blur amounts of the multiple areas of the first image and the respective blur amounts of the multiple areas of the second image ( S203).

In addition, the method of changing the positional relationship between the lens 154 and the imaging surface of the image sensor 120 is not limited to the method of moving the focus lens provided in the lens 154. For example, the imaging control unit 110 may move the entire lens 154 in the optical axis direction. The imaging control unit 110 can move the imaging surface of the image sensor 120 in the optical axis direction. The imaging control unit 110 can move at least a part of the lens included in the lens 154 and the imaging surface of the image sensor 120 along the optical axis direction. The imaging control unit 110 may adopt any method for optically changing the relative positional relationship between the focal point of the lens 154 and the imaging surface of the image sensor 120.

The calculation process of the subject distance will be further explained with reference to FIG. 5. The distance from the principal point of the lens L to the subject 510 (object plane) is set to A, and the distance from the principal point of the lens L to the position (image plane) where the light beam from the subject 510 is formed is set to B, Set the focal length of lens L to F. In this case, the following formula (2) can be used to express the relationship between the distance A, the distance B, and the focal length F according to the lens formula.

[Formula 2]

The focal length F is determined by the position of each lens included in the lens L. Therefore, if the distance B of the light beam imaging from the subject 510 can be determined, the equation (2) can be used to determine the distance A from the principal point of the lens L to the subject 510.

Here, it is assumed that by moving the imaging surface of the image sensor to the lens L side, the positional relationship between the lens L and the imaging surface is changed. As shown in FIG. 6, if there is an imaging surface at a distance D1 from the principal point of the lens L or a distance D2 from the principal point of the lens L, the image of the subject 510 projected on the imaging surface will be Produce blur. The distance B can be determined by calculating the imaged position of the subject 510 according to the blur size (circles of confusion 512 and 514) of the image of the subject 510 projected on the imaging surface, and the distance A can be further determined. That is, considering that the size of the blur (blur amount) is proportional to the imaging surface and the imaging position, the imaging position can be determined based on the difference in the blur amount.

Here, the respective images of the image I _{1 at a} distance D1 from the imaging surface and the image I _{2 at a distance D2 from the imaging surface are blurred.} Regarding the image I ₁ , assuming that the point spread function (Point Spread Function) is PSF ₁ and the subject image is I _d1 , then the image I ₁ can be represented by the following equation (3) by convolution operation.

[Formula 3]

I ₁ ＝PSF ₁ *I _d1 …(3)

The image I ₂ can also be represented by the convolution operation of _{PSF 2.} Suppose the Fourier transform of the subject image is f, and the optical transfer function obtained by _{the Fourier transform of the point spread functions PSF 1} and PSF _{2 is OTF 1} and OTF ₂ , as shown in the following equation (4) ratio.

[Formula 4]

The C value shown in equation (4) is the amount of change in the respective blur amounts of the image at a distance D1 from the principal point of the lens L and the image at a position D2 from the principal point of the lens L, that is, the C value corresponds to The difference between the amount of blur between the image at a position distance D1 from the principal point of the lens L and the image at a position distance D2 from the principal point of the lens L.

In FIG. 6, the case where the positional relationship between the lens L and the imaging surface is changed due to the movement of the imaging surface on the side of the lens L has been described. By moving the focus lens with respect to the imaging surface, the positional relationship between the focal position of the lens L and the imaging surface is changed, so that the amount of blur will also be different. In this embodiment, images with different blur amounts are mainly acquired due to moving the focus lens relative to the imaging surface, and DFD calculations are performed based on the acquired images to obtain DFD calculation values showing the amount of defocus, and based on the DFD calculation values The position target value of the focus lens for focusing on the subject is calculated as the target position relationship between the imaging surface and the focus lens.

As described above, the imaging device 100 performs AF in any one of AF based on the TOF sensor 160 distance measurement (TOF method), AF based on DFD calculation (DFD method), and AF based on a contrast evaluation value (contrast method).

According to the TOF method, the imaging device 100 can derive a target value for focusing on the position of the focus lens of the subject using one image. When the imaging device 100 captures low-brightness and low-contrast images, the TOF method can accurately derive the target value. However, in an environment where sunlight is easy to enter, external light other than the reflected light of the pulsed light may enter the TOF sensor 160, and therefore, the target value may not be accurately derived.

According to the DFD method, the imaging device 100 can derive a target value for focusing on the position of the focus lens of the subject based on two or more images taken at different positions of the focus lens. However, when the imaging device 100 captures low-contrast images such as clouds, sometimes the difference in the amount of blur of two or more images is small, and the target value cannot be accurately derived.

According to the contrast method, the imaging device 100 needs to capture multiple images to determine the position of the focus lens when the contrast evaluation value reaches the peak. Therefore, it takes longer to derive the target value. However, it is highly resistant to disturbances such as mechanical changes caused by temperature and changes in optical components such as lenses, and the target value accuracy is also high.

As described above, the optimal AF method differs according to the environment in which the subject photographed by the imaging device 100 is located. Moreover, by performing AF without moving the focus lens as much as possible, image blur and the like will not occur during the AF processing, and the sense of disharmony caused to the user is reduced. Therefore, according to the present embodiment, the imaging device 100 preferentially executes the AF method, that is, it can perform AF while moving the focus lens as unnecessarily as possible. More specifically, the imaging apparatus 100 performs AF in the TOF method when the reliability of the subject distance calculated in the TOF method is high. When the reliability of the subject distance calculated by the TOF method is not high, the imaging device 100 performs AF in the DFD method. Further, when the reliability of the subject distance calculated by the TOF method and the DFD method is not high, AF is performed in the contrast method.

The imaging control unit 110 determines whether the signal corresponding to the amount of light output from the light receiving element 165 satisfies the first condition showing the reliability of the subject distance. When the signal shows that the amount of light is contained within the preset range, the imaging control section 110 determines that the signal satisfies the first condition. When the signal continues to show that the amount of light is included in the preset range during the preset period, the imaging control section 110 may determine that the signal satisfies the first condition. When the amplitude of the light received by the light receiving element 165 indicated by the signal is above the lower limit threshold, and the A/D converted value of the signal is below the upper threshold, the imaging control unit 110 may determine that the signal satisfies the first condition. When the amplitude of the light received by the light receiving element 165 indicated by the signal is above the threshold and the value of the signal after A/D conversion does not reach saturation, the imaging control unit 110 may determine that the signal satisfies the first condition. That is, when the light received by the light receiving element 165 is not too weak and not too strong with respect to the pulsed light emitted from the light emitting element 163, the imaging control unit 110 determines that the signal satisfies the first condition.

When the signal satisfies the first condition, the imaging control unit 110 performs focus control to focus on the object based on the first target positional relationship between the imaging surface of the imaging device 100 and the focus lens determined from the distance to the object measured by the TOF sensor 160 . When the signal does not satisfy the first condition, the imaging control unit 110 is based on the first image captured by the imaging device 100 when the positional relationship between the imaging surface and the focusing lens is the first positional relationship, and the positional relationship between the imaging surface and the focusing lens is The second positional relationship is the second target positional relationship between the imaging plane determined by the respective blur amounts of the second image captured by the imaging device 100 and the focus lens, and the focus control for focusing on the object is performed. That is, when the signal does not satisfy the first condition, the imaging control section 110 performs focus control in the DFD method.

When the signal does not satisfy the first condition, the imaging control unit 110 may convert the positional relationship between the imaging surface and the focusing lens to the first image after acquiring the first image captured by the imaging device 100 when the imaging surface and the focusing lens are in the first positional relationship. The second position relationship is to acquire a second image captured by the imaging device 100 when the position relationship between the imaging surface and the focus lens is the second position relationship.

When the signal does not satisfy the first condition, the imaging control unit 110 may obtain the first image captured by the imaging device 100 when the imaging surface and the focusing lens are in the first positional relationship, and then determine the orientation of the focusing lens according to the first target positional relationship. Move the preset distance in the direction to make the positional relationship between the imaging surface and the focusing lens move to the second positional relationship. Even when the accuracy of the subject distance calculated by the TOF method is low, the imaging control unit 110 may correctly determine the direction in which the focus lens should move based on the subject distance calculated by the TOF method. Therefore, when the signal does not satisfy the first condition, the imaging control unit 110 may determine the moving direction of the focus lens according to the subject distance calculated in the TOF method.

The imaging control section 110 may determine whether the first degree of difference showing the degree of difference between the first image and the second image satisfies the second condition showing the reliability of the second target positional relationship before performing the focus control in the DFD method. The imaging control unit 110 may determine whether the first degree of difference satisfies the second condition showing the reliability of the DFD method based on the similarity between the first image and the second image. When the degree of similarity between the first image and the second image is lower than the preset threshold, the imaging control unit 110 may determine that the first degree of difference satisfies the second condition. When the difference between the contrast of the first image and the contrast of the second image is greater than a preset threshold, the imaging control unit 110 may determine that the first degree of difference satisfies the second condition. When the difference between the blur amount of the first image and the blur amount of the second image is greater than a preset threshold, the imaging control unit 110 may determine that the first degree of difference satisfies the second condition. Also, when the first degree of difference satisfies the second condition, the imaging control unit 110 may perform focus control to focus on the object according to the second target position relationship derived in the DFD method.

When the first degree of difference does not satisfy the second condition, the imaging control section 110 may perform focus control of focusing on the object through contrast AF.

When the first degree of difference does not satisfy the second condition, the imaging control unit 110 may further transform the positional relationship between the imaging surface and the focus lens from the second positional relationship to the third positional relationship. And, when the positional relationship between the imaging surface and the focus lens is in the third positional relationship, the second degree of difference, which is the degree of difference between the third image captured by the imaging device 100 and the first image or the second image, satisfies the second When the conditions are met, the imaging control unit 110 may perform focus control of focusing on the object based on the third target position relationship between the imaging surface and the focus lens determined according to the respective blur amounts of the first image or the second image and the third image. Also, when the second degree of difference does not satisfy the second condition, the imaging control unit 110 may perform focus control to focus on the object through contrast AF.

FIG. 7 is a flowchart showing one example of the AF processing procedure of the imaging device 100. After the AF process, the imaging control unit 110 causes the imaging device 100 to capture the first image for DFD when the focus lens is at the first position, and acquire the first image for DFD (S100). Then, the imaging control unit 110 determines whether the reliability of the subject distance measured by the TOF sensor 160 is high (S102). If the amount of light indicated by the signal output from the light receiving element 165 is within the preset range, the imaging control section 110 can determine that the reliability of the subject distance measured by the TOF sensor 160 is high.

When the reliability of the subject distance measured by the TOF sensor 160 is high, the imaging control unit 110 calculates the position of the focus lens that focuses on the object based on the TOF result showing the subject distance measured by the TOF sensor 160 Focus position (S104). The imaging control unit 110 can calculate the focus position by referring to the data showing the relationship between the TOF calculation result and the focus lens position as shown in FIG. 8, for example. Then, the imaging control unit 110 drives the focus lens to the focus position of the TOF method (S106). In a case where the reliability of the TOF result is high, as shown in FIG. 9, the imaging control unit 110 drives the focus lens from the AF start position to the focus position.

When the reliability of the TOF result is low, the imaging control unit 110 drives the focus lens to the focus direction indicating the direction toward the focus position calculated from the TOF result by only one depth or less (S108). The imaging control unit 110 moves the focus lens from the first position to the second position. A depth is equivalent to the minimum moving distance of the focus lens that can be used for the point spread function (Point Spread Function) when performing DFD calculations. For example, a depth can be 0.003mm. By driving the focus lens only one depth or less, it is difficult for the user to perceive the change in image blur caused by the driving of the focus lens. Therefore, it is possible to prevent the blur degree of the image displayed on the display from greatly changing due to the driving of the focus lens, thereby causing the user to feel a sense of disharmony.

Then, the imaging control unit 110 causes the imaging device 100 to capture the second image for DFD when the focus lens is at the second position, and acquire the second image for DFD (S110). The imaging control unit 110 performs DFD calculation using the first image and the second image to calculate the focus position (S112). The imaging control unit 110 determines whether the reliability of the focus position derived from the DFD calculation using the first image and the second image is high (S114). When the similarity between the first image and the second image is below the preset threshold, the imaging control unit 110 may determine that the reliability of the focus position derived by the DFD calculation is high. When the difference between the blur amount of the first image and the blur amount of the second image is greater than or equal to the preset threshold, the imaging control unit 110 may determine that the reliability of the focus position derived by the DFD calculation is high.

When it is determined that the reliability of the focus position derived by the DFD calculation is high, the imaging control unit 110 drives the focus lens to the focus position of the DFD method (S116). When it is determined that the reliability of the focus position derived by the DFD calculation is high, as shown in FIG. 10, the imaging control unit 110 drives the focus lens from the AF start position (first position) to the second position. Then, the imaging control unit 110 drives the focus lens to the focus position calculated by the DFD method.

When it is determined that the reliability of the focus position derived by the DFD calculation is low, the imaging control unit 110 further drives the focus lens in the focus direction (S118). The imaging control unit 110 can drive the focus lens only one depth or less. Alternatively, the imaging control unit 110 may drive the focus lens to a boundary position in the contrast AF search range determined by the focus position calculated by the TOF method or the DFD method to perform contrast AF. The imaging control unit 110 drives the focus lens from the second position to the third position. Then, the imaging control unit 110 causes the imaging device 100 to capture a third image for DFD when the focus lens is at the third position, and acquire the third image for DFD (S120). The imaging control unit 110 determines whether the reliability of the focus position derived from the DFD calculation using the second image and the third image is high (S124). The imaging control unit 110 may also determine whether the reliability of the focus position derived from the DFD calculation using the first image and the third image is high.

When the reliability of the focus position derived by the DFD calculation is high, the imaging control unit 110 drives the focus lens to the focus position of the DFD method (S116). As shown in FIG. 11, the imaging control unit 110 drives the focus lens from the AF start position (first position) to the second position, and after further moving to the third position, drives the focus lens until the second image and The focus position derived from the DFD calculation of the third image.

When the reliability of the focus position derived by the DFD calculation is low, the imaging control unit 110 executes contrast AF and acquires the contrast evaluation value of the third image (S126). The imaging control unit 110 searches for the position of the focus lens when the contrast evaluation value reaches the peak, and drives the focus lens in minute steps (S128).

The imaging control unit 110 determines whether the position of the focus lens when the contrast evaluation value reaches the peak is retrieved (S130). If the peak value is not found, the imaging control unit 110 repeats the steps after step S126. If the peak value is retrieved, the imaging control unit 110 drives the focus lens to the focus position determined by the contrast method (S132). As shown in FIG. 12, if the reliability of the focus position of the TOF method and the DFD method is low, the imaging control unit 110 drives the focus lens in a mountain climbing method to retrieve the peak value of the contrast evaluation value. The imaging control unit 110 drives the focus lens to a peak value exceeding the contrast evaluation value, and then drives the focus lens in the reverse direction to drive the focus lens to the focus position of the contrast method.

As described above, according to the present embodiment, when the reliability of the subject distance calculated in the TOF method is high, the imaging device 100 performs AF in the TOF method. When the reliability of the subject distance calculated in the TOF method is not high, the imaging device 100 performs AF in the DFD method. Further, when the reliability of the subject distance calculated by the TOF method and the DFD method is not high, AF is performed in the contrast method. As a result, the imaging device 100 can preferentially execute the AF method, that is, a method capable of moving the focus lens as necessary as possible to perform AF.

The aforementioned imaging device 100 may be mounted on a mobile body. The camera device 100 may be mounted on an unmanned aerial vehicle (UAV) as shown in FIG. 13. The UAV 1000 may include a UAV main body 20, a universal joint 50, a plurality of camera devices 60, and a camera device 100. The universal joint 50 and the camera device 100 are an example of a camera system. UAV1000 is an example of a moving body propelled by a propulsion unit. The concept of moving objects refers to flying objects such as airplanes moving in the air, vehicles moving on the ground, ships moving on water, etc., in addition to UAVs.

The UAV main body 20 includes a plurality of rotors. Multiple rotors are an example of a propulsion section. The UAV main body 20 makes the UAV 1000 fly by controlling the rotation of a plurality of rotors. The UAV body 20 uses, for example, four rotors to make the UAV 1000 fly. The number of rotors is not limited to four. In addition, UAV1000 can also be a fixed-wing aircraft without rotors.

The imaging device 100 is an imaging camera that captures a subject included in a desired imaging range. The universal joint 50 rotatably supports the imaging device 100. The universal joint 50 is an example of a supporting mechanism. For example, the gimbal 50 uses an actuator to rotatably support the imaging device 100 with a pitch axis. The universal joint 50 uses an actuator to further rotatably support the imaging device 100 around the roll axis and the yaw axis, respectively. The gimbal 50 can change the posture of the imaging device 100 by rotating the imaging device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are sensing cameras that photograph the surroundings of the UAV 1000 in order to control the flight of the UAV 1000. The two camera devices 60 can be installed on the nose of the UAV1000, that is, on the front side. In addition, the other two camera devices 60 may be installed on the bottom surface of the UAV1000. The two camera devices 60 on the front side may be paired to function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired to function as a stereo camera. The three-dimensional spatial data around the UAV 1000 can be generated based on the images captured by the plurality of imaging devices 60. The number of imaging devices 60 included in the UAV 1000 is not limited to four. It is sufficient that the UAV1000 includes at least one camera device 60. The UAV1000 may also include at least one camera 60 on the nose, tail, side, bottom and top surfaces of the UAV1000. The viewing angle that can be set in the imaging device 60 may be larger than the viewing angle that can be set in the imaging device 100. The imaging device 60 may have a single focus lens or a fisheye lens.

The remote operation device 600 communicates with the UAV1000 to perform remote operation on the UAV1000. The remote operation device 600 can wirelessly communicate with the UAV1000. The remote operation device 600 transmits to the UAV 1000 instruction information indicating various commands related to the movement of the UAV 1000 such as ascending, descending, accelerating, decelerating, forwarding, retreating, and rotating. The instruction information includes, for example, instruction information for raising the height of the UAV 1000. The indication information may show the height at which the UAV1000 should be located. The UAV 1000 moves to be at the height indicated by the instruction information received from the remote operation device 600. The instruction information may include an ascending instruction to raise the UAV1000. UAV1000 rises while receiving the rise command. When the height of the UAV1000 has reached the upper limit height, even if the ascent command is accepted, the UAV1000 can be restricted from rising.

FIG. 14 shows an example of a computer 1200 that can fully or partially embody various aspects of the present invention. The program installed on the computer 1200 can make the computer 1200 function as an operation associated with the device according to the embodiment of the present invention or one or more "parts" of the device. Alternatively, the program can cause the computer 1200 to perform the operation or the one or more "parts". This program enables the computer 1200 to execute the process or stages of the process involved in the embodiment of the present invention. Such a program may be executed by the CPU 1212, so that the computer 1200 executes specified operations associated with some or all blocks in the flowcharts and block diagrams described in this specification.

The computer 1200 of this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other through a host controller 1210. The computer 1200 further includes a communication interface 1222, an input/output unit, which is connected to the host controller 1210 through the input/output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates in accordance with programs stored in the ROM 1230 and RAM 1214 to control each unit.

The communication interface 1222 communicates with other electronic devices through a network. The hard disk drive can store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores therein a boot program executed by the computer 1200 during operation, and/or a program that depends on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as CR-ROM, USB memory, or IC card, or a network. The program is installed in RAM 1214 or ROM 1230 which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and causes cooperation between the programs and the various types of hardware resources described above. The apparatus or method may be constituted by implementing operations or processing of information according to the use of the computer 1200.

For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 can execute a communication program loaded in the RAM 1214, and based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. The communication interface 1222 reads the transmission data stored in the transmission buffer provided in a recording medium such as RAM 1214 or USB memory under the control of the CPU 1212, and sends the read transmission data to the network or receives the data from the network The received data is written into the receiving buffer provided in the recording medium, etc.

In addition, the CPU 1212 can make the RAM 1214 read all or necessary parts of files or databases stored in an external recording medium such as a USB memory, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 can write the processed data back to the external recording medium.

It is possible to store various types of information such as various types of programs, data, tables, and databases in the recording medium and receive information processing. For the data read from the RAM 1214, the CPU 1212 can perform various types of operations, information processing, conditional judgment, conditional transfer, unconditional transfer, and information retrieval/retrieval/information specified by the instruction sequence of the program described in various places in this disclosure. Replace various types of processing, and write the results back to RAM 1214. In addition, the CPU 1212 can search for information in files, databases, and the like in the recording medium. For example, when a plurality of entries having the attribute value of the first attribute respectively associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 may retrieve the attribute value of the specified first attribute from the multiple entries. The item that matches the condition is read, and the attribute value of the second attribute stored in the item is read to obtain the attribute value of the second attribute associated with the first attribute that meets the preset condition.

The programs or software modules described above may be stored on the computer 1200 or on a computer-readable storage medium near the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium so that the program can be provided to the computer 1200 via the network.

The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above-mentioned embodiments. It is obvious from the description of the claims that all such changes or improvements can be included in the technical scope of the present invention.

It should be noted that the execution order of the actions, sequences, steps, and stages of the devices, systems, programs, and methods shown in the claims, specification, and drawings of the specification, as long as there is no special indication that "before... ", "in advance", etc., and can be implemented in any order as long as the output of the previous processing is not used in the subsequent processing. Regarding the operating procedures in the claims, the specification and the drawings in the specification, the descriptions are made using "first", "next", etc. for convenience, but it does not mean that it must be implemented in this order.

【Symbol Description】

10 Camera system

20 UAV subject

50 universal joint

60 Camera device

100 camera device

110 Camera Control Department

120 Image sensor

130 Memory

150 Lens Control Department

152 Lens Drive

154 Lens

160 sensor

162 Light-emitting part

163 Light-emitting element

164 Light Receiving Department

165 Light receiving element

166 Luminous Control Unit

167 Light Receiving Control Department

168 Memory

200 supporting institutions

201 Rolling shaft drive mechanism

202 Pitch axis drive mechanism

203 Yaw axis drive mechanism

204 Base

210 Posture Control Department

212 Angular velocity sensor

214 Acceleration sensor

300 grip

301 Operation interface

302 Display

400 smart phones

600 remote operation device

1200 Computer

1210 Host Controller

1212 CPU

1214 RAM

1220 Input/Output Controller

1222 Communication interface

1230 ROM

Claims (10)

1. A control device that controls an imaging device including a distance measuring sensor and a focusing lens. The distance measuring sensor includes a light emitting element that emits pulsed light and receives light including reflected light of the pulsed light from an object. And a light-receiving element that outputs a signal corresponding to the amount of light received and measures the distance to the object based on the signal. The control device is characterized by including a circuit, and the circuit is configured as:

   When the signal satisfies the first condition showing the reliability of the distance, based on the first target position relationship between the imaging surface of the imaging device and the focusing lens determined according to the distance, focusing on the Focus control of objects;

   When the signal does not satisfy the first condition, based on the first image taken by the imaging device and the imaging surface when the positional relationship between the imaging surface and the focus lens is the first positional relationship When the positional relationship with the focusing lens is the second positional relationship, the second target positional relationship between the imaging surface and the focusing lens determined by the respective blur amounts of the second image captured by the imaging device is executed.述focus control.
2. The control device according to claim 1, wherein the circuit is configured to: when a first degree of difference showing the degree of difference between the first image and the second image satisfies the display of the second target When the second condition of the reliability of the positional relationship is established, the focus control is executed according to the second target positional relationship.
3. The control device according to claim 2, wherein the circuit is configured to execute the focus control based on contrast AF when the first degree of difference does not satisfy the second condition.
4. The control device according to claim 2, wherein the circuit is configured as:

   When the first degree of difference does not satisfy the second condition, changing the positional relationship between the imaging surface and the focusing lens from the second positional relationship to a third positional relationship;

   When the positional relationship between the imaging surface and the focusing lens is the third positional relationship, the third image taken by the imaging device is different from the first image or the second image. When the second difference degree satisfies the second condition, based on the third target positional relationship between the imaging surface and the focusing lens determined according to the respective blur amounts of the first image or the second image and the third image, Execute the focus control;

   When the second degree of difference does not satisfy the second condition, the focus control is performed according to contrast AF.
5. The control device according to claim 1, wherein when the signal shows that the light amount is contained within a preset range, the signal satisfies the first condition.
6. The control device according to claim 1, wherein the circuit is configured to: when the signal does not satisfy the first condition, obtain the positional relationship between the imaging surface and the focusing lens as the In the first positional relationship, after the first image captured by the imaging device, the positional relationship between the imaging surface and the focusing lens is transformed into the second positional relationship, and the positional relationship between the imaging surface and the The positional relationship of the focus lens is the second image captured by the imaging device in the second positional relationship.
7. The control device according to claim 6, wherein the circuit is configured to obtain the positional relationship between the imaging surface and the focusing lens as the In the first positional relationship, after the first image captured by the imaging device, the focusing lens is moved by a preset distance in a direction determined based on the first target positional relationship, thereby causing the imaging surface to be aligned with the The positional relationship of the focus lens moves to the second positional relationship.
8. A camera system, characterized by comprising: the control device according to any one of claims 1 to 7;

   The ranging sensor; and

   The camera device.
9. A control method, characterized in that it controls an imaging device including a range-finding sensor and a focusing lens, the range-finding sensor including a light emitting element that emits pulsed light and receiving reflected light including the pulsed light from an object The light-receiving element that outputs a signal corresponding to the amount of the received light and measures the distance to the object based on the signal. The control method is characterized in that it includes:

   When the signal satisfies the first condition showing the reliability of the distance, based on the first target position relationship between the imaging surface of the imaging device and the focusing lens determined according to the distance, focusing on the Focus control of objects;

   When the signal does not satisfy the first condition, based on the first image taken by the imaging device when the positional relationship between the imaging surface and the focus lens is the first positional relationship and the When the positional relationship between the surface and the focusing lens is the second positional relationship, the second target positional relationship between the imaging surface and the focusing lens determined by the respective blur amounts of the second image captured by the imaging device is executed The focus control.
10. A program characterized by causing a computer to function as the control device according to any one of claims 1 to 7.

Images