WO2021013143A1

WO2021013143A1 - Apparatus, photgraphic apparatus, movable body, method, and program

Info

Publication number: WO2021013143A1
Application number: PCT/CN2020/103227
Authority: WO
Inventors: 高宫诚
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-07-23
Filing date: 2020-07-21
Publication date: 2021-01-28
Also published as: CN112292712A; JP2021018376A; JP6874251B2

Abstract

The detection accuracy of the distance of a photographed subject can sometimes decrease according to the differences of photographed subjects. Provided is an apparatus, comprising a circuit composed as follows: on the basis of the amount of blurring of a first image included in first photographed images photographed when the imaging surface and the focusing lens of the photographic apparatus are in a first positional relationship state and the amount of blurring of a second image included in second photographed images photographed when the imaging surface and the focusing lens are in a second positional relationship state, acquiring a first predicted value representing the positional relationship between the imaging surface and the focusing lens when focusing on the photographed subject; on the basis of the amount of blurring of a third image included in third photographed images photographed when the imaging surface and the focusing lens are in a third positional relationship state, acquiring a second predicted value representing the positional relationship between the imaging surface and the focusing lens when focusing on the photographed subject; and, on the basis of the difference between the first predicted value and the second predicted value, determining a target value representing the positional relationship between the imaging surface and the focusing lens when focusing on the photographed subject.

Description

Device, camera device, mobile body, method and program

Technical field

The present invention relates to a device, a camera device, a mobile body, a method, and a program.

Background technique

Records the technology of measuring the distance of the subject using DFD method.

[Prior Art Literature]

[Patent Literature]

[Patent Document 1] JP 2013-242617 A

Summary of the invention

Sometimes depending on the subject, the detection accuracy of the subject distance decreases.

The device according to one aspect of the present invention includes a circuit based on the circuit structure: acquiring the first image included in the first captured image captured in a state where the imaging surface of the imaging device and the focusing lens are in a first positional relationship The amount of blur and the amount of blur of the second image included in the second captured image taken with the imaging surface and the focus lens in the second positional relationship. The circuit is configured to obtain a first predicted value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the blur amount of the first image and the blur amount of the second image. The circuit is configured to acquire the blur amount of the third image included in the third captured image captured in the state where the imaging surface and the focus lens are in the third positional relationship. The circuit is configured to obtain a second predicted value representing the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the blur amount of the second image and the blur amount of the third image. The circuit is configured to determine a target value representing the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the difference between the first predicted value and the second predicted value.

The first positional relationship, the second positional relationship, and the third positional relationship may respectively indicate the position of the focus lens relative to the imaging surface. The first predicted value may indicate the first target position of the focus lens relative to the imaging surface when focusing on the subject. The second predicted value may indicate the second target position of the focus lens relative to the imaging surface when focusing on the subject. The circuit may be configured to use the difference between the first target position and the second target position to determine a target value representing the target position of the focus lens relative to the imaging surface when focusing on the subject.

The position of the focusing lens determined according to at least one of the first positional relationship and the second positional relationship is set as the first reference position, and the position of the focusing lens determined according to at least one of the second positional relationship and the third positional relationship is set For the second reference position, the circuit may be configured to use the difference between the first target position and the second target position relative to the difference between the first reference position and the second reference position to determine the relative focus of the focus lens when focusing on the subject. The target value of the target position on the camera surface.

The circuit may be configured to calculate the ratio of the difference between the first target position and the second target position relative to the difference between the first reference position and the second reference position as the change information. The circuit may be configured to calculate the correction value of the second target position by dividing the second target position by the ratio. The circuit may be configured to calculate the position indicated by the correction value relative to the second reference position as the target value.

The circuit may be configured to perform focus control of the imaging device according to the target value.

The circuit may be configured to obtain the blur amount of the first image and the blur amount of the second image. The circuit may be configured to obtain the first predicted value according to the blur amount of the first image and the blur amount of the second image. The circuit may be configured to perform focus control of the imaging device according to the first predicted value when the reliable value of the first predicted value is greater than or equal to the preset first value. The circuit may be configured as follows: when the reliable value of the first predicted value is lower than the preset first value, obtain the blur amount of the third image, and obtain the second prediction according to the blur amount of the second image and the blur amount of the third image Value, and perform focus control of the camera device according to the target value.

The circuit may be configured such that when the reliable value of the first predicted value is lower than the second value lower than the first value, the third image is not acquired, and the focus control of the imaging device according to the first predicted value is not performed. The circuit may be configured as follows: when the reliable value of the first predicted value is lower than the first value and greater than or equal to the second value, the blur amount of the third image is obtained, and the second image is obtained according to the blur amount of the second image and the blur amount of the third image. Two predicted values, and the focus control of the camera device is executed according to the target value.

The device according to one aspect of the present invention includes a circuit based on the circuit structure: acquiring the first image included in the first captured image captured in a state where the imaging surface of the imaging device and the focusing lens are in a first positional relationship The amount of blur and the amount of blur of the second image included in the second captured image taken with the imaging surface and the focus lens in the second positional relationship. The circuit may be configured to obtain a first predicted value representing the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the blur amount of the first image and the blur amount of the second image. The circuit may be configured to acquire the blur amount of the third image included in the third captured image taken with the imaging surface and the focusing lens in the third positional relationship, and the image taken with the imaging surface and the focusing lens in the fourth positional relationship. The blur amount of the fourth image included in the fourth captured image. The circuit may be configured to obtain a second predicted value representing the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the blur amount of the third image and the blur amount of the fourth image. The circuit may be configured to determine a target value representing the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the difference between the first predicted value and the second predicted value.

The imaging device according to an aspect of the present invention may include the above-mentioned device and an image sensor having an imaging surface.

An imaging system according to an aspect of the present invention may include the above-mentioned imaging device and a support mechanism that supports the imaging device in a manner that can control the posture of the imaging device.

The mobile body according to an aspect of the present invention can be moved by mounting the aforementioned imaging device.

The method involved in one aspect of the present invention includes: acquiring the blur amount of a first image contained in a first captured image captured in a state where the imaging surface of the imaging device and the focus lens are in a first positional relationship, and that the imaging surface and the focus lens are The blur amount of the second image included in the second captured image captured in the state of the second positional relationship. The method may include: obtaining a first predicted value based on the blur amount of the first image and the blur amount of the second image, the first predicted value representing the positional relationship between the imaging surface and the focusing lens when focusing on the subject. The method may include: acquiring a blur amount of a third image included in a third image captured in a state where the imaging surface of the imaging device and the focus lens are in a third positional relationship. The method may include: obtaining a second predicted value based on the blur amount of the second image and the blur amount of the third image, the second predicted value representing the positional relationship between the imaging plane and the focusing lens when focusing on the subject. The method may include: determining a target value representing the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the difference between the first predicted value and the second predicted value.

The program related to one aspect of the present invention may be a program for causing a computer to execute the above-mentioned method.

According to an aspect of the present invention, it is sometimes possible to suppress a decrease in the detection accuracy of the subject distance depending on the subject.

In addition, the above summary does not enumerate all the essential features of the present invention. In addition, sub-combinations of these feature groups may also constitute inventions.

Description of the drawings

FIG. 1 is a diagram showing an example of an external perspective view of the imaging device 100.

FIG. 2 is a diagram showing functional blocks of the imaging device 100.

FIG. 3 shows an example of a curve representing the relationship between the amount of image blur (Cost) and the position of the focus lens.

Fig. 4 is a flowchart showing an example of a distance calculation process in the BDAF method.

FIG. 5 is a diagram illustrating the calculation process of the subject distance.

FIG. 6 shows the relationship between the DFD calculation value and the defocus amount for different subjects.

FIG. 7 shows the principle of focus control based on two DFD calculations performed by the imaging control unit 110.

FIG. 8 is a flowchart showing the processing procedure of the focus control executed by the imaging control unit 110.

Figure 9 shows an example of an unmanned aerial vehicle (UAV).

FIG. 10 shows an example of a computer 1200 that embodies aspects of the present invention in whole or in part.

Detailed ways

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, all the feature combinations described in the embodiments are not necessarily necessary for the solution of the invention. It is obvious to a person skilled in the art that various changes or improvements can be made to the following embodiments. It is obvious from the description of the claims that all such changes or improvements can be included in the technical scope of the present invention.

The claims, the description, the drawings of the description, and the abstract of the description include matters that are the subject of copyright protection. As long as anyone makes copies of these files as indicated in the patent office files or records, the copyright owner will not raise an objection. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention can be described with reference to flowcharts and block diagrams. Here, the blocks can represent (1) the stages of the process of performing operations or (2) the "parts" of the device that perform operations. Specific stages and "parts" can be implemented by programmable circuits and/or processors. Dedicated circuits may include digital and/or analog hardware circuits. May include integrated circuits (ICs) and/or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits can include logical AND, logical OR, logical exclusive OR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate array (FPGA), programmable logic array (PLA) ) And other memory components.

The computer-readable medium may include any tangible device that can store instructions to be executed by a suitable device. As a result, the computer-readable medium having instructions stored thereon includes a product including instructions that can be executed to create means for performing operations specified by the flowchart or block diagram. As examples of computer-readable media, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like may be included. As a more specific example of the computer readable medium, it may include floppy (registered trademark) disk, floppy disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (registered trademark) disc, Memory stick, integrated circuit card, etc.

The computer-readable instructions may include any one of source code or object code described in any combination of one or more programming languages. The source code or object code includes traditional procedural programming languages. Traditional procedural programming languages can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or Smalltalk (registered trademark), JAVA (registered trademark) , C++ and other object-oriented programming languages and "C" programming language or similar programming languages. The computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the Internet to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device. The processor or programmable circuit can execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

FIG. 1 is a diagram showing an example of an external perspective view of an imaging device 100 according to this embodiment. FIG. 2 is a diagram showing functional blocks of the imaging device 100 according to this embodiment.

The imaging device 100 includes an imaging unit 102 and a lens unit 200. The imaging unit 102 includes an image sensor 120, an imaging control unit 110, a memory 130, an instruction unit 162, and a display unit 160.

The image sensor 120 may be composed of CCD or CMOS. The image sensor 120 receives light through the lens 210 included in the lens unit 200. The image sensor 120 outputs image data of the optical image captured by the lens 210 to the imaging control unit 110.

The imaging control unit 110 may be constituted by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, or the like. The memory 130 may be a computer-readable recording medium, and may also include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The imaging control unit 110 corresponds to an electric circuit. The memory 130 stores programs and the like necessary for the imaging control unit 110 to control the image sensor 120 and the like. The memory 130 may be provided inside the housing of the imaging device 100. The storage 130 may be configured to be detachable from the housing of the imaging device 100.

The instruction unit 162 is a user interface that accepts instructions to the imaging device 100 from the user. The display unit 160 displays images captured by the image sensor 120, images processed by the imaging control unit 110, various setting information of the imaging device 100, and the like. The display part 160 may be composed of a touch panel.

The imaging control unit 110 controls the lens unit 200 and the image sensor 120. For example, the imaging control unit 110 controls the focal position and focal length of the lens 210. The imaging control unit 110 outputs a control command to the lens control unit 220 included in the lens unit 200 based on the information indicating the instruction from the user, thereby controlling the lens unit 200.

The lens unit 200 has one or more lenses 210, a lens driving unit 212, a lens control unit 220, and a memory 222. In this embodiment, one or more lenses 210 are collectively referred to as “lens 210”. The lens 210 may include a focus lens and a zoom lens. At least a part or all of the lenses included in the lens 210 are arranged to be movable along the optical axis of the lens 210. The lens unit 200 may be an interchangeable lens detachably provided in the imaging unit 102.

The lens driving unit 212 moves at least a part or all of the lens 210 along the optical axis of the lens 210. The lens control unit 220 drives the lens drive unit 212 according to the lens control instruction from the imaging unit 102 to move the entire lens 210 or the zoom lens or the focus lens included in the lens 210 along the optical axis, thereby performing the zoom operation or focus operation. at least one. The lens control commands are, for example, zoom control commands and focus control commands.

The lens driving part 212 may include a voice coil motor (VCM) that moves at least a part or all of the plurality of lenses 210 in the optical axis direction. The lens driving part 212 may include a motor such as a DC motor, a coreless motor, or an ultrasonic motor. The lens driving unit 212 can transmit the power from the motor to at least a part or all of the plurality of lenses 210 via mechanism components such as cam rings and guide shafts, so as to move at least a part or all of the lenses 210 along the optical axis.

The memory 222 stores control values for the focus lens and zoom lens that are moved by the lens drive unit 212. The memory 222 may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory.

The imaging control unit 110 outputs a control command to the image sensor 120 based on the information indicating the user's instruction acquired by the instruction unit 162 and the like to perform control including imaging operation control on the image sensor 120. The imaging control unit 110 acquires an image captured by the image sensor 120. The imaging control unit 110 performs image processing on the image acquired from the image sensor 120 and stores it in the memory 130.

The operation of the imaging control unit 110 in this embodiment will be described. The imaging control unit 110 acquires the amount of blur of the first image contained in the first captured image captured in the state where the imaging surface and the focus lens of the imaging device 100 are in the first positional relationship, and the imaging surface and the focus lens are in the second position The amount of blurring of the second image included in the second captured image captured in the state of the relationship. The imaging control unit 110 acquires a first predicted value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the blur amount of the first image and the blur amount of the second image. The imaging control unit 110 acquires the amount of blurring of the third image included in the third captured image captured with the imaging surface and the focus lens in the third positional relationship. The imaging control unit 110 obtains a second predicted value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the blur amount of the second image and the blur amount of the third image. The imaging control unit 110 determines a target value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the difference between the first predicted value and the second predicted value.

The first positional relationship, the second positional relationship, and the third positional relationship may respectively indicate the position of the focus lens relative to the imaging surface. The first predicted value may indicate the first target position of the focus lens relative to the imaging surface when focusing on the subject. The second predicted value may indicate the second target position of the focus lens relative to the imaging surface when focusing on the subject. In this case, the imaging control unit 110 may use the difference between the first target position and the second target position to determine a target value indicating the target position of the focus lens relative to the imaging surface when focusing on the subject.

The imaging control unit 110 sets the position of the focus lens determined based on at least one of the first positional relationship and the second positional relationship as the first reference position, and sets the position of the focus lens determined based on at least one of the second positional relationship and the third positional relationship as the first reference position. The position is set as the second reference position, and the imaging control unit 110 uses the difference between the first target position and the second target position relative to the difference between the first reference position and the second reference position to determine the focus lens that indicates when focusing on the subject The target value of the target position relative to the imaging surface.

The imaging control unit 110 may calculate the ratio of the difference between the first target position and the second target position to the difference between the first reference position and the second reference position as the change information. The imaging control unit 110 may calculate the correction value of the second target position by dividing the second target position by the ratio. The imaging control unit 110 may calculate the position indicated by the correction value relative to the second reference position as the target value. The imaging control unit 110 may perform focus control of the imaging device 100 based on the target value.

The imaging control unit 110 may acquire the blur amount of the first image and the blur amount of the second image. The imaging control unit 110 may obtain the first predicted value based on the blur amount of the first image and the blur amount of the second image. When the reliable value of the first predicted value is greater than or equal to the preset first value, the imaging control unit 110 may perform focus control of the imaging device 100 according to the first predicted value. When the reliable value of the first predicted value is lower than the preset first value, the imaging control unit 110 obtains the blur amount of the third image, and obtains the second predicted value according to the blur amount of the second image and the blur amount of the third image , And execute the focus control of the imaging device 100 according to the target value.

When the reliable value of the first predicted value is lower than the second value lower than the first value, the imaging control section 110 does not acquire the third image, and does not perform focus control of the imaging device 100 based on the first predicted value. When the reliable value of the first predicted value is lower than the first predicted value and greater than or equal to the second value, the imaging control unit 110 obtains the blur amount of the third image, and obtains the second image based on the blur amount of the second image and the blur amount of the third image. Two predicted values, and the focus control of the imaging device 100 is performed according to the target value.

In addition, the imaging control unit 110 can acquire the blur amount of the first image contained in the first captured image captured in the first positional relationship between the imaging surface and the focus lens of the imaging device 100, and whether the imaging surface and the focus lens are in the The blur amount of the second image included in the second captured image captured in the state of the second positional relationship. The imaging control unit 110 may obtain the first predicted value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the blur amount of the first image and the blur amount of the second image. The imaging control unit 110 can acquire the blur amount of the third image included in the third captured image taken with the imaging surface and the focus lens in the third positional relationship and the image taken with the imaging surface and the focus lens in the fourth positional relationship. The blur amount of the fourth image included in the fourth captured image. The imaging control unit 110 may obtain a second predicted value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the blur amount of the third image and the blur amount of the fourth image. The imaging control unit 110 may determine a target value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject based on the difference between the first predicted value and the second predicted value.

The AF method executed by the imaging device 100 will be described. In order to perform the AF processing, the imaging device 100 determines the distance from the lens 210 to the subject (subject distance). As a method for determining the subject distance, there is the following method: by moving the focus lens, it is determined based on the blur amount of a plurality of images captured in a state where the positional relationship between the focus lens and the light receiving surface of the image sensor 120 is different . Here, the AF using this method is called a blur detection auto focus (Bokeh Detection Auto Foucus: BDAF) method. Specifically, in BDAF, DFD (Depth From Defocus) calculation is performed to perform AF.

For example, the amount of blurring (Cost) of the image can be expressed by the following formula (1) using a Gaussian function. In formula (1), x represents the pixel position in the horizontal direction. σ represents the standard deviation value.

【Formula 1】

FIG. 3 shows an example of a curve representing the relationship between the amount of image blur (Cost) and the position of the focus lens. C1 is the amount of image blur obtained when the focus lens is located at x1. C2 is the amount of image blur obtained when the focus lens is located at x2. By aligning the focus lens at the lens position x0, the object can be focused on, where the lens position x0 corresponds to the minimum point 502 of the curve 500 determined according to the defocus amount C1 and the amount C2 in consideration of the optical characteristics of the lens 210.

Fig. 4 is a flowchart showing an example of a distance calculation process in the BDAF method. The imaging control unit 110 takes a first image and stores it in the memory 130 when the lens 210 and the imaging surface of the image sensor 120 are in a first positional relationship. The imaging control unit 110 moves the lens 210 in the optical axis direction to place the lens 210 and the imaging surface in a second positional relationship, and captures a second image by the imaging device 100 and stores it in the memory 130 (S201). For example, the imaging control unit 110 changes the positional relationship between the lens 210 and the imaging surface from the first positional relationship to the second positional relationship by moving the focus lens along the optical axis direction. The amount of movement of the lens may be, for example, about 10 μm.

Then, the imaging control unit 110 divides the first image into a plurality of regions (S202). The imaging control unit 110 may calculate a feature amount for each pixel in the first image, and divide the first image into a plurality of regions by using a group of pixels having similar feature amounts as one region. The imaging control unit 110 may divide the pixel group set as the range of the AF processing frame in the first image into a plurality of regions. The imaging control unit 110 divides the second image into a plurality of regions corresponding to the plurality of regions of the first image. The imaging control unit 110 calculates, for a plurality of regions, a target corresponding to an object contained in each of the plurality of regions based on the respective blur amounts of the plurality of regions of the first image and the respective blur amounts of the plurality of regions of the second image. The distance of the subject (S203).

In addition, the method of changing the positional relationship between the lens 210 and the imaging surface of the image sensor 120 is not limited to the method of moving the focus lens included in the lens 210. For example, the imaging control unit 110 may move the entire lens 210 in the optical axis direction. The imaging control unit 110 can move the imaging surface of the image sensor 120 in the optical axis direction. The imaging control unit 110 can move at least a part of the lens included in the lens 210 and the imaging surface of the image sensor 120 in the optical axis direction. The imaging control unit 110 may adopt any method as long as the relative positional relationship between the focal point of the lens 210 and the imaging surface of the image sensor 120 is optically changed.

The calculation process of the subject distance will be further explained with reference to FIG. 5. The distance from the principal point of the lens L to the subject 510 (object plane) is set to A, and the distance from the principal point of the lens L to the position (image plane) where the light beam from the subject 510 is formed is set to B, Set the focal length of lens L to F. In this case, the following formula (2) can be used to express the relationship between the distance A, the distance B, and the focal length F according to the lens formula.

[Formula 2]

The focal length F is determined by the position of each lens included in the lens L. Therefore, if it is possible to determine the distance B imaged according to the light beam of the subject 510, the equation (2) can be used to determine the distance A from the principal point of the lens L to the subject 510.

Here, it is assumed that by moving the imaging surface of the image sensor to the side of the lens L, the positional relationship between the lens L and the imaging surface is changed. As shown in FIG. 5, if there is an imaging surface at a distance D1 from the principal point of the lens L or a distance D2 from the principal point of the lens L, the image of the subject 510 projected on the imaging surface will be Produce blur. The imaged position of the subject 510 is calculated according to the blur size (circles of confusion 512 and 514) of the image of the subject 510 projected on the imaging surface, and thus the distance B can be determined, and the distance A can be further determined. That is, considering that the size of the blur (blur amount) is proportional to the imaging surface and the imaging position, the imaging position can be determined from the difference in the blur amount.

Here, each image of the image I ₁ at a position distance D1 from the imaging surface and the image I ₂ at a position distance D2 from the imaging surface is blurred. Regarding the image I ₁ , assuming that the point spread function (Point Spread Function) is PSF ₁ and the subject image is I _d1 , the image I ₁ can be expressed by the following equation (3) through convolution operation.

[Formula 3]

I ₁ ＝PSF ₁ *I _d1 …(3)

Like I ₂ is also expressed by the convolution operation of PSF ₂ . The Fourier transform of the subject image is set to f, and the optical transfer function after Fourier transform of the point image distribution functions PSF ₁ and PSF ₂ is set to OTF ₁ and OTF ₂ , as follows ( 4) As shown, the ratio is obtained.

[Formula 4]

The value C shown in equation (4) corresponds to the amount of change in each blur of the image at a position distance D1 from the principal point of the lens L and an image at a position distance D2 from the principal point of the lens L, that is, the value C corresponds to The difference in the amount of blur between an image at a position distance D1 from the principal point of the lens L and an image at a position distance D2 from the principal point of the lens L.

In FIG. 5, the case where the positional relationship between the lens L and the imaging surface is changed by moving the imaging surface to the side of the lens L has been described. By moving the focus lens relative to the imaging surface, thereby changing the positional relationship between the focal position of the lens L and the imaging surface, a difference in the amount of blur also occurs. In this embodiment, mainly by moving the focus lens relative to the imaging surface to obtain images with different blur amounts, perform DFD calculations based on the acquired images to obtain a DFD calculation value indicating the amount of defocus, and calculate the amount of defocus based on the DFD calculation value. The target value of the position of the focus lens that focuses on the subject.

Fig. 6 shows the relationship between the DFD calculation value and the defocus amount for different subjects. The horizontal axis of the graph 610, the graph 620, and the graph 630 is the defocus amount, and the vertical axis is the DFD calculation value. The unit of the defocus amount and the DFD calculation value is fδ. The circular marks of the graph 610, the graph 620, and the graph 630 respectively indicate a DFD calculation value obtained by the DFD calculation of the two images obtained by changing the position of the focus lens. The solid lines of the graph 610, the graph 620, and the graph 630 represent the ideal value of the DFD calculation.

The graph 610 shows the relationship between the DFD calculation value and the actual defocus amount for the image with the specific object A as the subject. The graph 620 shows the relationship between the DFD calculation value and the defocus amount of an image with a specific object B different from the object A as the subject. The graph 630 shows the relationship between the DFD calculation value and the defocus amount for an image with a specific object C different from the object A and the object B as the subject. In addition, the data point when it is judged to be in the most focused state from the actually captured image is taken as the origin of the graphs of

graphs

610, 620, and 630.

The graph 650 shows the deviation of the ideal value of the DFD calculation value for the image of the object A. The graph 660 shows the deviation of the ideal value of the DFD calculation value for the image of the object B. The graph 670 shows the deviation of the ideal value of the DFD calculation value for the image of the object C.

Among object A, object B, and object C, object A is the object with the highest contrast, and object C is the object with the lowest contrast. The object C is, for example, a cloud. It can be seen from the graph 610, graph 620, graph 630, graph 650, graph 660, and graph 670 that, regarding the image of object A, the error of the DFD calculation value is extremely small. Even when the amount of defocus is large, the DFD calculation value The accuracy is also high. On the other hand, regarding the image of the object C, the error of the DFD calculation value is relatively large, and in particular, the greater the defocus amount, the lower the accuracy of the DFD calculation value. In this way, depending on the object that is the subject, the amount of change in the DFD calculation value with respect to the amount of change in the defocus amount may be smaller than the ideal value of the DFD calculation. In addition, depending on the object as the subject, it may deviate from the ideal value of the DFD calculation.

FIG. 7 shows the principle of focus control based on two DFD calculations performed by the imaging control unit 110. The vertical axis of the graph 710 is the DFD calculation value, and the horizontal axis is the defocus amount. The graph 710 is a graph showing two

data points

711 and 712 used for focus control in the graph 630 of FIG. 6. The graph 720 expresses the data of the graph 710 in units of the number of lens pulses corresponding to the defocus amount. The number of lens pulses indicates the number of pulses provided to the stepping motor driving the lens 210 for focusing control. The number of lens pulses indicates the position of the lens 210 relative to a predetermined reference point in the optical axis direction.

In FIG. 7, the data point 711 represents the DFD operation value and the number of lens pulses obtained by the first DFD operation. The data point 712 represents the DFD operation value and the number of lens pulses obtained by the second DFD operation. The number of lens pulses indicates the reference position of the lens position in order to take an image for each DFD calculation. In one DFD calculation, since two images taken with different lens positions are used, the average value of the lens positions when the images used in one DFD calculation are taken is used as the lens reference position. In addition, the DFD calculation value obtained by the DFD calculation indicates the amount of defocus at the reference position. The DFD calculation value represents the target movement amount of the lens 210 from its reference position to the focused state.

The imaging control section 110 performs the first DFD calculation and the second DFD calculation by making the lens reference positions different from each other. The imaging control unit 110 calculates the amount of change in the DFD calculation value with respect to the amount of change in the lens reference position. This value represents the slope of the straight line 722 connecting the data point 711 and the data point 712. By dividing the second DFD calculation value by the slope value of the straight line 722, the second DFD calculation value is corrected and the lens 210 is moved. Thus, the imaging control unit 110 can be closer to the in-focus state (the number of lens pulses is 2000) than when the second DFD calculation value is not corrected.

The operation of the imaging control unit 110 will be explained using specific numerical values. The imaging control unit 110 captures three images that are image P1, image P2, and image P3 in a state where the lens positions are different from each other, and performs DFD calculation twice using the three images, image P1, image P2, and image P3. Specifically, the imaging control unit 110 performs a first DFD calculation based on the images P1 and P2, and performs a second DFD calculation based on the images P2 and P3.

The lens position when the image P1 is captured is LensP1, the lens position when the image P2 is captured is LensP2, and the lens position when the image P3 is captured is LensP3. Specifically, LensP1=1160, LensP2=1440, and LensP3=1720.

The first DFD operation value acquired by the first DFD operation based on the images P1 and P2 is set as DFD1. Specifically, let DFD1=-514. If the average value of LensP1 and LensP2 is set to DFDLensP1, the lens reference position DFDLensP1 is 1300. The data point 711 of the graph 720 of FIG. 7 represents coordinates (DFDLensP1, DFD1).

The second DFD operation value based on the images P2 and P3 is set to DFD2. Specifically, set DFD2=-311. If the average value of LensP2 and LensP3 is DFDLensP2, then DFDLensP2=1580. The data point 712 of the graph 720 of FIG. 7 represents (DFDLensP2, DFD2).

First, the target position (Peak2) of the lens 210 based only on the second DFD calculation is calculated by the following formula.

Peak2=DFDLensP2-DFD2

Thus, Peak2=1580-(-311)=1891 is obtained. In FIG. 7, since the lens position in the focused state is 2000, an error of -109 pulses has occurred. Therefore, even if the position of the lens 210 is moved to Peak2, it may become in a state of insufficient focus with the subject.

Next, the focus control of the imaging control unit 110 will be described. The imaging control unit 110 calculates the slope of the straight line 722 passing through the data point 711 and the data point 712 by the following formula.

slope=(DFD2-DFD1)/(DFDLensP2-DFDLensP1)...(5)

Thus, slope=0.725 can be obtained.

The imaging control unit 110 uses the value of slope to correct DFD2 as follows to calculate the target position PeakCorrection of the lens 210.

PeakCorrection=DFDLensP2-DFD2/slope

Therefore, PeakCorrection=1580(-311)/0.725=2009. Since the lens position in the focused state is 2000, the error is 9 pulses. From this, it can be seen that the error can be controlled below 1/10 compared with Peak2 calculated based only on the second DFD calculation.

FIG. 8 is a flowchart showing the processing procedure of the focus control executed by the imaging control unit 110. The imaging control unit 110 executes a focus control process 800 and a DFD process 850 in parallel, where the former is implemented by an algorithm for focus control, and the latter is implemented by an algorithm for DFD calculation.

First, in the focus control process 800, the imaging control unit 110 captures the image sensor 120 at the current lens position, and acquires image data of the first image (S802). In the DFD processing 850, the imaging control unit 110 processes the image data of the first image into information necessary for DFD calculation (S852).

In the focus control process 800, the imaging control unit 110 moves the position of the focus lens of the lens 210 by a preset movement amount (S804), causes the image sensor 120 to take an image, and acquires image data of the second image (S806). In the DFD processing 850, the imaging control unit 110 processes the image data of the second image into information required for DFD calculation, performs the first DFD calculation based on the data of the first image and the second image, and calculates the current DFD calculation value. The amount of defocus and the reliability value of the DFD calculation value (S852). The imaging control unit 110 calculates the amount of movement of the focus lens based on the DFD calculation value obtained by the first DFD calculation (S854). The imaging control unit 110 may calculate the reliability value of the DFD calculation value based on the blur amount of the subject in the image. The imaging control unit 110 may calculate the reliability value of the DFD calculation value based on the amount of blur (Cost) of the image represented by the equation (1), for example. The imaging control unit 110 can make the smaller the amount of blur, the smaller the reliability value.

In S856, the imaging control unit 110 compares the reliability value obtained by the first DFD operation with the first threshold value and a second threshold value lower than the first threshold value. The imaging control unit 110 may compare the reliability value acquired through the first DFD operation with a first threshold value or a second threshold value lower than the first threshold value. Specifically, if the reliability value obtained by the first DFD calculation is greater than or equal to the first threshold value, the imaging control unit 110 transfers the process to S812 and drives the lens position to the lens target position indicated by the DFD calculation value. If the reliability value obtained by the first DFD operation is lower than the preset first threshold value and greater than or equal to the second threshold value, the process proceeds to S808 to move the lens position. In addition, when the lens is moved in S808, considering the DFD calculation value calculated in S852, the imaging control unit 110 may determine that the amount of movement of the focus lens does not exceed the focus position. In addition, the imaging control unit 110 processes the image data of the second image into information necessary for DFD calculation (S858).

When the reliability value obtained by the first DFD calculation is lower than the second threshold value, the focus control of the BDAF method is stopped. When the focus control of the BDAF method is stopped, the imaging control unit 110 may perform focus control using a method other than BDAF, such as contrast AF. If the imaging device 100 is in the process of shooting a moving image, the imaging control unit 110 may place the lens 210 in an infinite focus state when the focus control of the BDAF method is stopped. When the imaging device 100 is about to capture a still image, if the BDAF focus control is stopped, the imaging control unit 110 may notify the user of a focus error.

After the focus lens is moved in S808, the imaging control unit 110 causes the image sensor 120 to take an image, and acquires image data of the third image (S810). In the DFD processing 850, the imaging control unit 110 processes the image data of the third image into information required for DFD calculation, performs the second DFD calculation based on the data of the second image and the third image, and calculates the current DFD calculation value. The defocus amount and the reliability value of the DFD calculation value (S858). The imaging control unit 110 is based on the DFD operation value obtained by the first DFD operation of S852, the DFD operation value obtained by the second DFD operation of S858, and the lens when shooting the first image, the second image, and the third image. For the position, the slope is calculated according to formula (5), the DFD calculation value is corrected by the slope, and the movement amount of the focus lens is calculated (S860).

The imaging control unit 110 determines whether the reliability value calculated by the second DFD calculation is greater than or equal to the second threshold value (S862). When the reliability value calculated by the second DFD calculation is lower than the second threshold value, the imaging control unit 110 stops the focus control of the BDAF method. When the reliability value obtained by the second DFD operation is greater than or equal to the second threshold value, the imaging control section 110 moves the process to S812, and moves the focus lens according to the movement amount calculated by S860 (S812). In addition, the imaging control unit 110 processes the image data of the third image into information necessary for DFD calculation (S864).

After the lens is moved in S812, the imaging control unit 110 causes the image sensor 120 to capture an image, and acquires image data of the image for focus checking (S814). In the DFD process 850, the imaging control unit 110 performs a DFD calculation based on the image data of the third image and the focus check image, and calculates the current defocus amount obtained as the DFD calculation value and the reliability value of the DFD calculation (S864). The imaging control unit 110 determines whether it is in focus with the subject based on the focus amount represented by the DFD operation value obtained by the DFD operation in S864 and the reliability value of the DFD operation (S866), and when it is determined to be in focus, completes BDAF In the AF operation of the method, when it is determined that it is not in focus, the process moves to S812, and the focus lens is moved to the target position indicated by the DFD calculation value obtained by the DFD calculation of S864. After that, the operations of moving the focus lens, image capturing for DFD calculation, DFD calculation, and focus determination are repeated until it is determined that it is in a focused state.

As described above, the imaging control unit 110 calculates the amount of movement of the focus lens based on the difference between the DFD calculation value obtained by the first DFD calculation and the DFD calculation value obtained by the second DFD calculation. As a result, even when an object with low DFD arithmetic sensitivity is used as the subject, the AF operation can be performed with high accuracy.

In addition, in the above description, in order to perform two DFD calculations, three imagings are performed. However, in order to perform two DFD calculations, four shots may be performed. For example, the first DFD operation may be performed based on the first image and the second image with different lens positions, and the second DFD operation may be performed based on the third image and the fourth image with different lens positions.

In addition, in the above description, the slope is calculated using the results of two DFD operations. However, the slope can also be calculated using the results of more than three DFD operations.

The aforementioned imaging device 100 may be mounted on a mobile body. The camera device 100 may also be mounted on an unmanned aerial vehicle (UAV) as shown in FIG. 9. The UAV 10 may include a UAV main body 20, a universal joint 50, a plurality of camera devices 60, and a camera device 100. The universal joint 50 and the camera device 100 are an example of a camera system. UAV10 is an example of a mobile body propelled by a propulsion unit. In addition to UAVs, the concept of moving objects also includes flying objects such as other airplanes that move in the air, vehicles that move on the ground, and ships that move on the water.

The UAV main body 20 includes a plurality of rotors. Multiple rotors are an example of a propulsion section. The UAV main body 20 makes the UAV 10 fly by controlling the rotation of a plurality of rotors. The UAV main body 20 uses, for example, four rotors to fly the UAV 10. The number of rotors is not limited to four. In addition, UAV10 can also be a fixed-wing aircraft without rotors.

The imaging device 100 is an imaging camera that captures a subject included in a desired imaging range. The universal joint 50 rotatably supports the imaging device 100. The universal joint 50 is an example of a supporting mechanism. For example, the universal joint 50 uses an actuator to rotatably support the imaging device 100 around the pitch axis. The universal joint 50 uses an actuator to further rotatably support the imaging device 100 around the roll axis and the yaw axis, respectively. The gimbal 50 can change the posture of the camera device 100 by rotating the camera device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are sensing cameras that photograph the surroundings of the UAV 10 in order to control the flight of the UAV 10. The two camera devices 60 can be installed on the nose of the UAV 10, that is, on the front. In addition, the other two camera devices 60 may be provided on the bottom surface of the UAV 10. The two imaging devices 60 on the front side may be paired to function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired to function as a stereo camera. The three-dimensional spatial data around the UAV 10 can be generated based on the images taken by the plurality of camera devices 60. The number of imaging devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one camera device 60. The UAV10 may be equipped with at least one camera 60 on the nose, tail, side, bottom, and top of the UAV10. The viewing angle that can be set in the imaging device 60 may be larger than the viewing angle that can be set in the imaging device 100. The imaging device 60 may have a single focus lens or a fisheye lens.

The remote operation device 300 communicates with the UAV 10 to remotely operate the UAV 10. The remote operation device 300 can wirelessly communicate with the UAV 10. The remote operation device 300 transmits to the UAV 10 instruction information indicating various commands related to the movement of the UAV 10 such as ascending, descending, accelerating, decelerating, forwarding, retreating, and rotating. The instruction information includes, for example, instruction information for raising the height of the UAV 10. The indication information may indicate the height at which the UAV10 should be located. The UAV 10 moves to be located at the height indicated by the instruction information received from the remote operation device 300. The instruction information may include an ascending instruction to raise the UAV10. UAV10 rises while receiving the rise command. When the height of UAV10 has reached the upper limit height, even if the ascending instruction is accepted, the ascent of UAV10 can be restricted.

FIG. 10 shows an example of a computer 1200 that can embody aspects of the present invention in whole or in part. The program installed on the computer 1200 can make the computer 1200 function as an operation associated with the device according to the embodiment of the present invention or one or more "parts" of the device. For example, a program installed on the computer 1200 can make the computer 1200 function as the imaging control unit 110. Alternatively, the program can enable the computer 1200 to perform related operations or related functions of one or more "parts". This program enables the computer 1200 to execute the process or stages of the process involved in the embodiment of the present invention. Such a program may be executed by the CPU 1212, so that the computer 1200 executes specified operations associated with some or all blocks in the flowcharts and block diagrams described in this specification.

The computer 1200 of this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other through a host controller 1210. The computer 1200 further includes a communication interface 1222, an input/output unit, which is connected to the host controller 1210 through the input/output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates in accordance with programs stored in the ROM 1230 and RAM 1214 to control each unit.

The communication interface 1222 communicates with other electronic devices through a network. The hard disk drive can store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores therein a boot program executed by the computer 1200 during operation, and/or a program that depends on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as CR-ROM, USB memory, or IC card, or a network. The program is installed in RAM 1214 or ROM 1230 which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and causes cooperation between the programs and the various types of hardware resources described above. The apparatus or method may be constituted by implementing operations or processing of information according to the use of the computer 1200.

For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 can execute a communication program loaded in the RAM 1214, and based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. The communication interface 1222 reads the transmission data stored in the transmission buffer provided in a recording medium such as RAM 1214 or USB memory under the control of the CPU 1212, and sends the read transmission data to the network or receives the data from the network The received data is written into the receiving buffer provided in the recording medium, etc.

In addition, the CPU 1212 can make the RAM 1214 read all or necessary parts of files or databases stored in an external recording medium such as a USB memory, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 can write the processed data back to the external recording medium.

It is possible to store various types of information such as various types of programs, data, tables, and databases in the recording medium and receive information processing. For the data read from the RAM 1214, the CPU 1212 can perform various types of operations, information processing, conditional judgment, conditional transfer, unconditional transfer, and information retrieval/retrieval/information specified by the instruction sequence of the program described in various places in this disclosure. Replace various types of processing, and write the results back to RAM 1214. In addition, the CPU 1212 can search for information in files, databases, and the like in the recording medium. For example, when a plurality of entries having the attribute value of the first attribute respectively associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 may retrieve the attribute value of the specified first attribute from the multiple entries. And read the attribute value of the second attribute stored in the entry that matches the condition, so as to obtain the attribute value of the second attribute that is associated with the first attribute that meets the predetermined condition.

The programs or software modules described above may be stored on the computer 1200 or on a computer-readable storage medium near the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium so that the program can be provided to the computer 1200 via the network.

The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above-mentioned embodiments. It is obvious from the description of the claims that all such changes or improvements can be included in the technical scope of the present invention.

It should be noted that the operation, sequence, step and stage of the device, system, program and method shown in the claims, specification and drawings of the specification are executed in the order of each process, as long as there is no special indication that "before... ", "in advance", etc., and can be implemented in any order as long as the output of the previous processing is not used in the subsequent processing. Regarding the operating procedures in the claims, the specification and the drawings in the specification, the descriptions are made using "first", "next", etc. for convenience, but it does not mean that it must be implemented in this order.

【Symbol Description】

10 UAV

20 UAV subject

50 universal joint

60 Camera device

100 camera device

102 Camera Department

110 Camera Control Department

120 Image sensor

130 Memory

152 Lens Drive

160 Display

162 Instruction Department

200 lens department

210 Lens

212 Lens Drive

220 Lens Control Department

222 Memory

300 remote operation device

500 curve

502 Very Small Point

510 Subject

512 Circle of Diffusion

610, 620, 630, 650, 660, 670 Graph

710 Chart

711 data points

712 data points

720 charts

722 Straight

800 Focus control processing

850 DFD processing

1200 Computer

1210 Host Controller

1212 CPU

1214 RAM

1220 Input/Output Controller

1222 Communication interface

1230 ROM

Claims

A device, characterized by comprising a circuit based on the circuit structure: to obtain the first image contained in the first image captured in a state where the imaging surface of the imaging device and the focusing lens are in a first positional relationship The amount of blur and the amount of blur of the second image included in the second captured image taken when the imaging surface and the focus lens are in a second positional relationship;

Acquiring, according to the amount of blur of the first image and the amount of blur of the second image, a first predicted value representing the positional relationship between the imaging surface and the focusing lens when focusing on the subject;

Acquiring a blur amount of a third image included in a third camera image taken when the camera surface and the focusing lens are in a third positional relationship;

Acquiring, according to the amount of blur of the second image and the amount of blur of the third image, a second predicted value representing the positional relationship between the imaging surface and the focusing lens when focusing on the subject;

Based on the difference between the first predicted value and the second predicted value, a target value representing the positional relationship between the imaging surface and the focus lens when focusing on a subject is determined.
The device according to claim 1, wherein:

The first positional relationship, the second positional relationship, and the third positional relationship respectively represent the position of the focusing lens relative to the imaging surface,

The first predicted value represents a first target position of the focusing lens relative to the imaging surface when focusing on a subject,

The second predicted value represents a second target position of the focusing lens relative to the imaging plane when focusing on the subject,

The circuit is configured to use the difference between the first target position and the second target position to determine the target value representing the target position of the focusing lens relative to the imaging surface when focusing on a subject .
The device according to claim 2, wherein:

The position of the focusing lens determined according to at least one of the first positional relationship and the second positional relationship is set as a first reference position, and the position of the focusing lens is determined according to the second positional relationship and the third positional relationship. At least one determined position of the focus lens is set as a second reference position, and the circuit is configured to use the first target position and the difference between the first reference position and the second reference position The difference between the second target position determines the target value representing the target position of the focus lens relative to the imaging plane when focusing on the subject.
The device according to claim 3, wherein the circuit-based configuration is:

Calculating the ratio of the difference between the first target position and the second target position relative to the difference between the first reference position and the second reference position as change information,

The correction value of the second target position is calculated by dividing the second target position by the ratio,

The position indicated by the correction value relative to the second reference position is calculated as the target value.
The device according to any one of claims 1 to 4, characterized in that:

The circuit is configured to execute focus control of the imaging device based on the target value.
The device according to claim 5, wherein the circuit-based configuration is:

Acquiring the blur amount of the first image and the blur amount of the second image;

Acquiring the first predicted value according to the blur amount of the first image and the blur amount of the second image;

When the reliable value of the first predicted value is greater than or equal to the preset first value, execute the focus control of the camera device according to the first predicted value;

When the reliable value of the first predicted value is lower than the preset first value, the blur amount of the third image is acquired, according to the blur amount of the second image and the blur of the third image The second predicted value is acquired by a certain amount, and the focus control of the imaging device is executed according to the target value.
The device according to claim 6, wherein the circuit-based configuration is:

When the reliable value of the first predicted value is lower than the second value lower than the first value, the third image is not acquired, and the imaging device based on the first predicted value is not performed. Focus control

When the reliable value of the first predicted value is lower than the first value and greater than or equal to the second value, the blur amount of the third image is acquired, based on the blur amount of the second image and the first The blur amounts of the three images acquire the second predicted value, and perform focus control of the imaging device according to the target value.
A device, characterized by comprising a circuit based on the circuit structure: to obtain the first image contained in the first image captured in a state where the imaging surface of the imaging device and the focusing lens are in a first positional relationship The amount of blur and the amount of blur of the second image included in the second captured image taken when the imaging surface and the focus lens are in a second positional relationship;

Acquiring, according to the amount of blur of the first image and the amount of blur of the second image, a first predicted value representing the positional relationship between the imaging surface and the focusing lens when focusing on the subject;

Obtain the blur amount of the third image included in the third captured image captured in the state where the imaging surface and the focus lens are in the third positional relationship, and the difference that the imaging surface and the focus lens are in the fourth positional relationship The blur amount of the fourth image contained in the fourth camera image taken in the state;

Acquiring, according to the blur amount of the third image and the blur amount of the fourth image, a second predicted value representing the positional relationship between the imaging surface and the focus lens when focusing on the subject;

According to the difference between the first predicted value and the second predicted value, a target value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject is determined.
An imaging device, characterized by comprising: the device according to any one of claims 1 to 4 and 8; and

An image sensor provided with the imaging surface.
A camera system, comprising: the camera device according to claim 9; and

A support mechanism that supports the camera in such a way that the posture of the camera can be controlled.
A mobile body characterized in that it mounts and moves the imaging device according to claim 9.
A method, characterized in that it comprises: acquiring a blur amount of a first image contained in a first captured image captured in a state where the imaging surface of the imaging device and the focusing lens are in a first positional relationship, and the imaging surface and the The amount of blurring of the second image included in the second captured image taken with the focusing lens in the second positional relationship;

Acquiring, according to the amount of blur of the first image and the amount of blur of the second image, a first predicted value representing the positional relationship between the imaging surface and the focusing lens when focusing on the subject;

Acquiring a blur amount of a third image included in a third captured image captured in a state where the imaging surface of the imaging device and the focus lens are in a third positional relationship;

Acquiring, according to the amount of blur of the second image and the amount of blur of the third image, a second predicted value representing the positional relationship between the imaging surface and the focusing lens when focusing on the subject;

According to the difference between the first predicted value and the second predicted value, a target value indicating the positional relationship between the imaging surface and the focus lens when focusing on the subject is determined.
A program which is a program for causing a computer to execute the method according to claim 12.