JP6639155B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing device and an image processing method.

  It has been proposed to acquire distance information to a subject by using a plurality of captured images. Specifically, a correlation amount between parallax and blur between a plurality of images is calculated, and distance information to the subject is acquired based on the calculated correlation amount.

  Patent Literature 1 discloses distance calculation using a depth from defocus (DFD) method. In the cited document 1, a plurality of images having different blurs photographed with different photographing parameters are used, a correlation amount of the blur between the plurality of images is calculated, and distance information to the subject is calculated based on the calculated correlation amount of the blur. To get. Patent Literature 2 discloses that an image is divided into segments based on color information, and that parallax is assumed to be the same within the same segment, and distance information is obtained by calculating a correlation amount within the range of the segment. Have been.

JP 2013-253964 A JP-A-2006-012166

  However, according to the conventional technology, it has not always been possible to acquire distance information satisfactorily.

  An object of the present invention is to provide an image processing apparatus and an image processing method that can acquire distance information well.

According to one aspect of the present invention, an image input unit for inputting a plurality of images captured by an imaging unit, an imaging information acquisition unit for acquiring imaging information when the image is captured, and A distance for determining the size of the evaluation frame based on the included focusing distance, calculating a correlation amount between the plurality of images in the evaluation frame, and calculating distance information based on the correlation amount between the plurality of images. An image processing apparatus having information calculation means is provided.

  According to the present invention, the size of the evaluation frame can be appropriately set because the size of the evaluation frame is determined based on the imaging information that is information indicating the imaging conditions when the image is captured. Since the evaluation frame having an appropriate size is used, the correlation amount can be obtained well, and the distance information can be obtained well.

FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus according to the first embodiment. 5 is a flowchart illustrating an operation of the image processing apparatus according to the first embodiment. 5 is a graph showing the relationship between each piece of imaging information and the size of an evaluation frame. FIG. 5 is a diagram illustrating an example of setting of an evaluation frame size according to a focal length of a lens. FIG. 9 is a diagram for describing a calculation process of a correlation amount when the DFD method is used. FIG. 9 is a diagram for explaining a calculation process of a correlation amount when a stereo method is used. FIG. 9 is a diagram for explaining a calculation process of a correlation amount when a stereo method is used. It is a block diagram showing the composition of the image processing device by a 2nd embodiment. 9 is a flowchart illustrating an operation in the image processing apparatus according to the second embodiment. It is a figure showing the size of an evaluation frame. FIG. 14 is a diagram for explaining the size of an evaluation frame used in the image processing device according to the second embodiment.

[First Embodiment]
An image processing apparatus and an image processing method according to the first embodiment will be described with reference to FIGS.

  When calculating the distance in the depth direction to the subject (subject distance) or information corresponding to the subject based on the amount of correlation between a plurality of images, the difference in the range on the image (on the screen) that is the target of the correlation calculation processing may cause The calculation result of the distance information is different. When the range to be subjected to the correlation operation processing is wide, many textures can be included, and thus the calculated distance information tends to be stable. However, when the range to be subjected to the correlation calculation processing is wide, a plurality of subjects whose distances from the imaging device are different from each other are likely to be included in the range. The competition area tends to increase. On the other hand, when the range to be subjected to the correlation calculation process is narrow, the distance conflict region is reduced, but the texture included in the range is also reduced, so that the calculated distance information tends to be unstable. As described above, when the range to be subjected to the correlation calculation processing is simply set, the distance information cannot always be obtained satisfactorily.

  The image processing apparatus according to the present embodiment determines the size of the evaluation frame for calculating the amount of correlation between a plurality of images based on the imaging information at the time of imaging, thereby making it possible to obtain good distance information. It is assumed that.

  First, the configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram illustrating the configuration of the image processing apparatus according to the present embodiment.

  The image input unit 101 is for inputting a plurality of captured images to the image processing device according to the present embodiment. The image input via the image input unit 101 is input to an evaluation frame selection unit 104 of the distance information calculation unit 103 described later. The image input unit 101 may be an image sensor or an image signal input terminal.

  The imaging information acquisition unit 102 acquires imaging information (camera information) that is information indicating imaging conditions and the like at the time of acquiring a captured image, that is, imaging information at the time of imaging. The imaging information acquired by the imaging information acquisition unit 102 is input to the evaluation frame selection unit 104 of the distance information calculation unit 103.

  The distance information calculation unit 103 is for calculating distance information, and includes an evaluation frame selection unit 104, a correlation amount calculation unit 105, and a distance information generation unit 106.

  The evaluation frame selection unit 104 determines a range to be subjected to a correlation operation, that is, a size of the evaluation frame, based on the imaging information acquired by the imaging information acquisition unit 102.

  The correlation amount calculation unit 105 calculates a correlation amount between a plurality of images in an evaluation frame of the size determined by the evaluation frame selection unit 104.

  Based on the correlation amount calculated by the correlation amount calculation unit 105, the distance information generation unit 106 generates a distance in the depth direction from the imaging device to each subject, or distance information (object distance information) corresponding to the distance in the depth direction. ) Is calculated. The information corresponding to the distance in the depth direction is, for example, an image shift amount based on the correlation amount calculated by the correlation amount calculation unit 105 or a defocus amount based on the correlation amount. The distance information generation unit 106 outputs the calculated distance information via the distance information output unit 107.

  Next, the operation of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 2 is a flowchart illustrating the operation of the image processing apparatus according to the present embodiment. Note that the overall operation of the image processing apparatus according to the present embodiment is controlled by, for example, a CPU (not shown) as control means.

  In step S201, a plurality of images necessary for acquiring distance information are input to the distance information calculation unit 103 via the image input unit 101. More specifically, a plurality of images input via the image input unit 101 are input to the evaluation frame selection unit 104 of the distance information calculation unit 103. As described above, the image input unit 101 may be an image sensor or an image signal input terminal. When the image input unit 101 is an image sensor, a plurality of images acquired using the image sensor are input to the distance information calculator 103. When the image input unit 101 is an image input terminal, a plurality of images input via the image input terminal are input to the distance information calculation unit 103.

  In step S202, imaging information obtained when each of the plurality of images is captured is acquired by the imaging information acquisition unit 102, and the imaging information acquired by the imaging information acquisition unit 102 is input to the distance information calculation unit 103. More specifically, the imaging information acquired by the imaging information acquisition unit 102 is input to the evaluation frame selection unit 104 of the distance information calculation unit 103. Examples of the imaging information acquired by the imaging information acquisition unit 104 include a focal length of a lens, a focusing distance, an aperture of a lens, a size of a subject, a camera shake amount, distortion information, ISO sensitivity, and the like.

  In step S203, an evaluation used when calculating a correlation amount between a plurality of images based on the plurality of images input through the image input unit 101 and the imaging information acquired by the imaging information acquiring unit 102. The size of the frame is determined by the evaluation frame selection unit 104. The correlation amount between a plurality of images is, for example, a change amount of blur between a plurality of images having different focus positions.

  As described later, the size of the evaluation frame used when calculating the correlation amount between a plurality of images is determined by comprehensively determining the size of the evaluation frame calculated for each piece of imaging information. The size of the evaluation frame calculated for each piece of imaging information will be described below.

  FIG. 3 is a graph showing the relationship between each piece of imaging information and the size of the evaluation frame. FIG. 3A shows the relationship between the focal length of the lens and the size of the evaluation frame, and FIG. 3B shows the relationship between the focusing distance and the size of the evaluation frame. FIG. 3C shows the relationship between the aperture of the lens and the size of the evaluation frame, and FIG. 3D shows the relationship between the size of the subject and the size of the evaluation frame. FIG. 3E shows the relationship between the amount of camera shake and the size of the evaluation frame. FIG. 3F shows the relationship between the distortion information and the size of the evaluation frame, more specifically, the relationship between the image height position and the image height position. The relationship with the size of the evaluation frame is shown. FIG. 3G shows the relationship between the ISO sensitivity and the lower limit of the size of the evaluation frame. The vertical axis in each of FIGS. 3A to 3F indicates the size of the evaluation frame, and the vertical axis in FIG. 3G indicates the lower limit of the size of the evaluation frame.

  First, the determination of the size of the evaluation frame based on the focal length of the lens will be described. FIG. 4 is a diagram illustrating an example of determining the size of the evaluation frame based on the focal length of the lens. FIGS. 4A and 4B show captured images when the focal length of the lens is short, and FIGS. 4C and 4D show images captured when the focal length of the lens is long. An image is shown. The evaluation frame is shown using a thick line in each drawing of FIG.

  As shown in FIG. 4A, when the focal length of the lens is short, many subjects appear in the captured image, so that many subjects at different distances from the imaging device are included in the evaluation frame. Tends to be included. Therefore, when the focal length of the lens is short, as shown in FIG. 4B, the size of the evaluation frame is set to be small, and the number of subjects that can be located in the evaluation frame is reduced, so that the distance Conflict is unlikely to occur, and accurate distance information is obtained.

  On the other hand, as shown in FIG. 4C, when the focal length of the lens is long, the number of subjects appearing in the captured image decreases, and the number of subjects included in the evaluation frame tends to decrease. . For this reason, when the focal length of the lens is long, as shown in FIG. 4D, by setting the size of the evaluation frame to be large, stable distance information, that is, distance information with small variation can be obtained. To do.

  For example, the size of the evaluation frame is determined according to the focal length of the lens so as to satisfy the relationship shown in FIG. That is, the size of the evaluation frame is determined such that the size of the evaluation frame increases as the focal length of the lens increases. For the size of the evaluation frame, a lower limit and an upper limit are set. If the size of the evaluation frame is too small, the texture included in the evaluation frame is too small, and it is difficult to obtain stable distance information. Further, if the size of the evaluation frame is too large, the number of textures included in the evaluation frame becomes too large, which causes an increase in the amount of calculation when calculating the correlation amount. For these reasons, a lower limit and an upper limit are set for the size of the evaluation frame. Then, within the range between the lower limit and the upper limit, the size of the evaluation frame is determined according to the focal length of the lens.

  Next, determination of the size of the evaluation frame based on the focusing distance will be described with reference to FIG. When the focusing distance is short, that is, when the distance to the subject is short, the size of the subject to be shot tends to increase, and the number of subjects appearing in the captured image tends to decrease. For this reason, when the focusing distance is short, the variation of the distance information is reduced by setting the size of the evaluation frame to be large. On the other hand, when the focusing distance is long, the size of the subject to be photographed tends to be small, and the number of subjects included in the captured image tends to increase. Therefore, when the focusing distance is long, the accuracy of the distance information is improved by setting the size of the evaluation frame to a small size and reducing the number of subjects located in the evaluation frame. For example, the size of the evaluation frame is determined according to the focusing distance so as to satisfy the relationship shown in FIG. That is, the size of the evaluation frame is determined such that the size of the evaluation frame decreases as the focusing distance increases.

  Next, determination of the size of the evaluation frame based on the F value will be described with reference to FIG. When the aperture of the lens is on the open side, that is, when the F-number of the lens is small, the background is blurred, and there is a tendency that the number of objects that can be recognized as subjects is reduced in the background. For this reason, when the F-number of the lens is small, the size of the evaluation frame is set to be large, thereby reducing the variation in the distance information. On the other hand, when the aperture of the lens is on the aperture side, that is, when the F-number is large, blurring of the background is small, so that many objects recognizable as subjects at different distances are likely to be included in the background. For this reason, when the F value is large, the size of the evaluation frame is set to a small value, thereby reducing the number of subjects located in the evaluation frame and improving the accuracy of the distance information. For example, the size of the evaluation frame is determined according to the F value so as to satisfy the relationship shown in FIG. That is, the size of the evaluation frame is determined such that the size of the evaluation frame decreases as the F-number increases.

  Next, the setting of the evaluation frame based on the size of the subject will be described with reference to FIG. When obtaining information on the size of the subject, for example, recognition information can be used. Specifically, when a person is recognized as the recognition information, the size of the subject is determined based on the range in which the person is recognized. When the size of the subject is large, the number of subjects appearing in the captured image tends to be small. For this reason, when the size of the subject is large, the size of the evaluation frame is set to be large, thereby reducing the variation in the distance information. On the other hand, when the size of the subject is small, many subjects are likely to be included in the captured image. For this reason, when the size of the subject is small, the size of the evaluation frame is set to be small, so that the number of subjects located in the evaluation frame is reduced, and the accuracy of the distance information is improved. For example, the size of the evaluation frame is determined according to the size of the subject so as to satisfy the relationship shown in FIG. That is, the size of the evaluation frame is determined such that the size of the evaluation frame increases as the size of the subject increases.

  Next, setting of an evaluation frame based on the camera shake amount will be described with reference to FIG. When obtaining information on the camera shake amount, for example, a value of a gyro sensor can be used. If the size of the evaluation frame is small even though the camera shake amount is large, a correct correspondence cannot be found due to the influence of the positional shift, and a highly accurate correlation amount cannot be calculated. For this reason, when the camera shake amount is large, the influence of the camera shake is reduced by setting the size of the evaluation frame to a large value. On the other hand, when the camera shake amount is small, a correct correspondence can be detected even if the size of the evaluation frame is small. For this reason, when the camera shake amount is small, the size of the evaluation frame is set to a small value, thereby reducing the number of subjects located in the evaluation frame and improving the accuracy of the distance information. For example, the size of the evaluation frame is determined according to the amount of camera shake so as to satisfy the relationship shown in FIG. That is, the size of the evaluation frame is determined so that the size of the evaluation frame increases as the camera shake amount increases.

  Next, setting of an evaluation frame based on distortion information will be described with reference to FIG. Since the captured image is affected by the aberration, the magnitude of the distortion differs depending on the image height position. Therefore, it is preferable to change the size of the evaluation frame according to the image height position. For example, when barrel distortion occurs, a subject appears larger in a place where the image height position is high than in a place where the image height position is low. Therefore, when the image height position is high, the size of the evaluation frame is set to be large, so that many textures are included in the evaluation frame, and a highly accurate correlation amount is obtained. On the other hand, at a position where the image height position is low, the accuracy of the distance information is improved by setting the size of the evaluation frame to a small value. For example, the size of the evaluation frame is determined according to the image height position so as to satisfy the relationship shown in FIG. That is, the size of the evaluation frame is determined such that the size of the evaluation frame increases as the image height position increases.

  Next, setting of an evaluation frame based on ISO sensitivity will be described with reference to FIG. When the image is captured with high sensitivity, a large amount of noise is included in the captured image. Therefore, if the size of the evaluation frame is small, a correct correlation amount cannot be calculated due to the influence of noise. For this reason, when shooting with high sensitivity, the accuracy of the correlation amount is improved by setting the size of the evaluation frame to be large. On the other hand, when the image is captured with low sensitivity, the captured image does not include much noise. Therefore, even if the size of the evaluation frame is set to a small value, a highly accurate correlation amount can be obtained. For example, the size of the evaluation frame is determined according to the ISO sensitivity so as to satisfy the relationship shown in FIG. That is, the lower limit of the size of the evaluation frame increases as the ISO sensitivity increases. The lower limit m of the size of the evaluation frame when shooting with high sensitivity is larger than the lower limit n of the size of the evaluation frame when shooting with low sensitivity.

Thus, the size of the evaluation frame is calculated for each piece of imaging information. Then, the size of the evaluation frame used for calculating the correlation amount is calculated by comprehensively determining these using the size of the evaluation frame calculated for each piece of imaging information. Specifically, for example, by performing weighted addition using the following equation (1), the size S _{out of the} evaluation frame used when calculating the correlation amount is determined. In the equation (1), the alpha _n, which indicates the weight of each imaging information, S _n indicates the size of the evaluation frame calculated for each imaging information.

  As a specific example, for example, a case where the focal length of the lens is long and the aperture of the lens is on the open side will be described. When the focal length of the lens is long, the number of subjects appearing in the captured image tends to decrease. Also, when the aperture of the lens is on the open side, the background is blurred, and there is a tendency that the number of objects that can be recognized as subjects is reduced in the background. Therefore, when the focal length of the lens is long and the aperture is on the open side, the size of the evaluation frame is set to be large. Even when the size of the evaluation frame is set to be large, recognizable subjects located at different distances are hardly included in the evaluation frame, so that highly accurate distance information can be obtained.

  As a specific example, a case where the focal length of the lens is long and the aperture of the lens is on the aperture side will be described. When the focal length of the lens is long, the number of subjects appearing in the captured image tends to decrease. Also, when the aperture of the lens is on the aperture side, the number of objects that can be recognized as subjects tends to increase in the background. For this reason, even when the focal length of the lens is long, the size of the evaluation frame is set to be smaller when the aperture of the lens is on the aperture side than when the aperture of the lens is on the open side. You. As a result, a good correlation amount can be obtained, and highly accurate distance information can be obtained.

In step S204, the correlation amount between the plurality of images is calculated by the correlation amount calculation unit 105. When calculating the correlation amount between a plurality of images, the evaluation frame of the size S _out determined in step S203 is used. When calculating a correlation amount using the DFD method as described in Patent Literature 1, a plurality of images having different focal positions are used, and a change amount of blur between the plurality of images is set as a correlation amount. calculate.

  The calculation process of the correlation amount using the DFD method will be described with reference to FIG. FIG. 5 is a diagram for explaining a calculation process of a correlation amount when the DFD method is used. FIG. 5A shows a characteristic 501 of an edge strength in a focus image (reference image) obtained in a state where a subject is focused, and an edge strength in a captured image obtained in a state where a focus position is changed with respect to a focus state. The characteristic 502 of FIG. The horizontal axis indicates the defocus amount, and the vertical axis indicates the edge intensity. By using these two edge strength characteristics 501 and 502, the correlation amount of blur between a plurality of images can be calculated. FIG. 5B is a graph showing the characteristics of the amount of blur correlation between a plurality of images. In FIG. 5B, a characteristic 503 of the amount of blur correlation between a plurality of images is shown using a thick solid line.

  In step S205, distance information is generated by the distance information generation unit 106 based on the correlation amount calculated in step S204. Specifically, distance information is generated by calculating a defocus amount on the image plane based on the correlation amount characteristic 503 calculated in step S204. The defocus amount calculated by such a method is a distance on the image plane. By converting the distance on the image plane into the object plane distance using the formula of the lens, it is also possible to obtain distance information to the subject.

  In the present embodiment, the case where the DFD system is used has been described as an example. However, the present invention is not limited to this. For example, a stereo system may be used.

The case where the stereo system is used will be described below.
When the stereo method is used, a plurality of images having parallax are input via the image input unit 101 in step S201. FIG. 6 is a diagram for explaining a calculation process of a correlation amount when the stereo method is used. FIG. 6A shows a reference image, and FIG. 6B shows an image whose imaging position is different from the reference image. Since the photographing positions are different from each other, parallax occurs in these multiple images.

  Steps S202 and S203 are the same as in the case where the DFD method is used, and thus description thereof is omitted.

In Step S204, the correlation amount between the plurality of images is calculated by the correlation amount calculation unit 105 using the evaluation frame of the size S _out determined in Step 203. The correlation amount between a plurality of images is a difference value between images calculated for obtaining a parallax amount. When the stereo method is used, the correlation amount between a plurality of images is calculated, for example, as follows. In the reference image, the evaluation frame 602 is set as shown in FIG. On the other hand, in an image whose shooting position is different from the reference image, the evaluation frame 612 is gradually shifted in the direction of the arrow. While gradually shifting the evaluation frame 612, the correlation amount between the image whose imaging position is different from the reference image and the reference image is calculated.

FIG. 7 is a diagram for explaining a calculation process of the correlation amount when the stereo method is used. FIG. 7A shows the first image, and FIG. 7B shows data in the evaluation frame of the first image. FIG. 7C shows the second image, and FIG. 7D shows data in the evaluation frame of the second image. Note that the evaluation frame is shown using a thick solid line in FIG. The integrated value of the absolute value of the difference between the data value in the evaluation frame as shown in FIG. 7B and the data value in the evaluation frame as shown in FIG. 7D, that is, the SAD (Sum of Absolute Difference) is Is calculated. Specifically, the integrated value R _{SAD of the} absolute difference value is calculated as the correlation amount using the following equation (2). Note that f (i, j) is a pixel value at each coordinate (i, j) in the evaluation frame of the first image shown in FIG. 7A, and g (i, j) is It is a pixel value at each coordinate (i, j) in the second evaluation frame shown in (c).

  Here, the case where the correlation amount is calculated using the SAD that is the integrated value of the absolute difference values has been described as an example, but the present invention is not limited to this. If the point having the highest correlation (shift amount, image shift amount) can be determined from the obtained correlation amount, the correlation amount may be calculated by another method.

  In step S205, the distance information is calculated by the distance information generating unit 106 based on the correlation amount calculated in step S204. The correlation amount calculated in step S204 has, for example, a characteristic as shown in FIG. The horizontal axis of FIG. 6C is the shift amount of the evaluation frame, and the vertical axis is the correlation amount. The point at which the calculated correlation amount is the highest is defined as the point at which the correlation is the highest, and the shift amount at that time is defined as the parallax amount (image shift amount) between a plurality of images. The distance information is generated by calculating the defocus amount on the image plane using the pixel pitch of the image sensor and the conversion coefficient determined according to the lens for the parallax amount. In this embodiment, the defocus amount is generated as the distance information. However, as described above, the subject distance converted from the defocus amount to one-to-one based on the image shift amount and the state of the imaging optical system, as described above. May be used as the distance information.

  As described above, according to the present embodiment, since the size of the evaluation frame is determined based on the imaging information at the time of imaging, the size of the evaluation frame can be appropriately set. Since the evaluation frame having an appropriate size is used, the correlation amount can be obtained well, and the distance information can be obtained well.

[Second embodiment]
An image processing apparatus and an image processing method according to the second embodiment will be described with reference to FIGS. The same components as those of the image processing apparatus according to the first embodiment shown in FIGS. 1 to 7 are denoted by the same reference numerals, and description thereof will be omitted or simplified.
The image processing apparatus according to the present embodiment uses a plurality of evaluation frames having different sizes when calculating the correlation amount.

The configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 8 is a block diagram illustrating the configuration of the image processing apparatus according to the present embodiment.
The components 801 to 807 of the image processing apparatus according to the present embodiment are the same as the components 101 to 107 of the image processing apparatus according to the first embodiment described above with reference to FIG. The image processing apparatus according to the first embodiment is different from the image processing apparatus according to the first embodiment in that the processing by the correlation amount calculation unit 805 and the processing by the distance information generation unit 806 are repeatedly performed using a plurality of evaluation frames having different sizes. It is different from the device.

  The operation of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart illustrating the operation of the image processing apparatus according to the present embodiment.

  First, steps S901 to S902 are the same as steps S201 to S202 described above with reference to FIG.

  In step S903, a plurality of types of evaluation frames having different sizes, that is, P types of evaluation frames having different sizes are determined as follows. That is, first, the size of the evaluation frame is calculated for each piece of imaging information as in the first embodiment. Then, the size of the evaluation frame calculated for each piece of imaging information is classified based on a predetermined criterion. FIG. 10 is a diagram illustrating the size of the evaluation frame calculated for each piece of imaging information. FIG. 10 shows representative values after classification. FIG. 10 shows, as an example, a case where the data is classified into three types of L × L, M × M, and N × N (L <M <N). Note that A <L <M <N.

  The correlation amount calculation processing (step S904) and the distance information generation processing (step S905) are repeatedly performed using the evaluation frames of the P types determined in step S903. That is, in the present embodiment, hierarchical processing is performed by sequentially using evaluation frames of different sizes. FIG. 11 is a diagram for explaining the size of the evaluation frame used in the present embodiment. FIG. 11A shows the size of the evaluation frame 1102 used in the first correlation calculation process. In the first correlation calculation process (step S904), for example, an L × L size evaluation frame 1102 is used. Then, distance information is generated based on the correlation amount calculated in the first correlation calculation process (step S905). Then, a variation of the distance information is calculated for the generated distance information, and a second correlation calculation process is performed on the area where the distance information varies. FIG. 11B shows the size of the evaluation frame 1112 used in the second correlation calculation process. In the second correlation calculation process (step S904), for example, an evaluation frame 1112 having a size of M × M is used. Then, distance information is generated based on the correlation amount calculated in the second correlation calculation process (step S905). Then, the variation of the distance information is calculated for the generated distance information, and the third correlation calculation process is performed on the region where the distance information varies. FIG. 11C shows the size of the evaluation frame 1122 used in the third correlation calculation process. In the third correlation calculation process (step S904), for example, an evaluation frame 1122 having a size of N × N is used. Then, distance information is generated based on the correlation amount calculated in the third correlation calculation process (step S905).

  Meanwhile, in step 906, it is determined whether the correlation amount calculation processing and the distance information generation processing have been performed P times. If the correlation amount calculation processing and the distance information generation processing have not been performed P times (NO in step S906), steps S904 and S905 are repeated. On the other hand, when the correlation amount calculation processing and the distance information generation processing have been performed P times (YES in step S906), the calculation processing of the distance information is completed.

  As described above, in the present embodiment, the correlation amount is sequentially calculated using the plurality of types of evaluation frames having different sizes, and the distance information is generated based on the calculated correlation amount. Therefore, according to the present embodiment, it is possible to acquire distance information more favorably.

  The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

101 image input unit 102 imaging information acquisition unit 103 distance information calculation unit 104 evaluation frame selection unit 105 correlation amount calculation unit 106 distance information generation unit 107 distance information output unit

Claims (14)

Image input means for inputting a plurality of images taken by the imaging means,
Imaging information acquisition means for acquiring imaging information when the image is captured,
Determine the size of the evaluation frame based on the focusing distance included in the imaging information, calculate the amount of correlation between the plurality of images in the evaluation frame, to the subject based on the amount of correlation between the plurality of images And a distance information calculating means for calculating distance information corresponding to the distance in the depth direction of the image processing apparatus. The image processing apparatus according to claim 1, wherein the size of the evaluation frame is determined by weighted addition of the sizes of the plurality of evaluation frames determined according to each of the plurality of pieces of imaging information. . The imaging information further includes a focal length of the lens,
The distance information calculating unit may be configured to determine the size of the evaluation frame such that the size of the evaluation frame is larger when the focal length of the lens is a first distance than when the focal length of the lens is a second distance shorter than the first distance. The image processing apparatus according to claim 1, wherein a size of the frame is determined. It said distance information calculating unit, when the focusing distance is a first distance, the evaluation frame sizes size Kunar so than when the a first distant second distance than the distance, the evaluation The image processing apparatus according to claim 1, wherein a size of the frame is determined. The imaging information further includes a size of the subject,
The distance information calculation means is configured such that when the size of the subject is a first size, the size of the evaluation frame is larger than when the size of the subject is a second size smaller than the first size. The image processing device according to claim 1, wherein a size of the evaluation frame is determined. The imaging information further includes an F value of the lens,
The distance information calculation means may be configured to determine the size of the evaluation frame so that the F-number of the lens is smaller than the first value when the F-number of the lens is a first value. The image processing apparatus according to claim 1, wherein a size of the frame is determined. The imaging information further includes an ISO sensitivity,
The distance information calculating unit may be configured to perform the evaluation so that a lower limit of the size of the evaluation frame is larger when the ISO sensitivity is the first sensitivity than when the ISO sensitivity is the second sensitivity lower than the first sensitivity. The image processing apparatus according to claim 1, wherein a size of the frame is determined. The imaging information further includes a camera shake amount,
The distance information calculating means may determine that the size of the evaluation frame is larger when the camera shake amount is a first value than when the shake amount is a second value smaller than the first value. The image processing apparatus according to claim 1, wherein the size is determined. The imaging information further includes an image height position,
The distance information calculating unit may be configured to determine that the size of the evaluation frame is larger when the image height position is the first position than when the image height position is the second position lower than the first position. The image processing device according to claim 1, wherein the size of the image processing device is determined. The image processing apparatus according to any one of claims 1 to 9, wherein the correlation amount between the plurality of images is a blur correlation amount between the plurality of images having different focus positions. The image processing device according to any one of claims 1 to 9, wherein the correlation amount between the plurality of images is a parallax correlation amount between the plurality of images having different imaging positions. The distance information calculating unit determines the plurality of evaluation frames having different sizes in accordance with each of a plurality of pieces of imaging information, and calculates the correlation amount between the plurality of images in the plurality of evaluation frames having different sizes. The image processing device according to claim 1, wherein the distance information is calculated based on the correlation amount between the plurality of images. Inputting a plurality of images taken by the imaging means;
Acquiring imaging information at the time of capturing the image,
Determining the size of the evaluation frame based on the focusing distance included in the imaging information;
Calculating a correlation amount between the plurality of images in the evaluation frame,
Calculating distance information corresponding to the distance in the depth direction to the subject based on the correlation amount between the plurality of images. On the computer,
Inputting a plurality of images taken by the imaging means;
Acquiring imaging information at the time of capturing the image,
Determining the size of the evaluation frame based on the focusing distance included in the imaging information;
Calculating a correlation amount between the plurality of images in the evaluation frame,
Calculating distance information corresponding to a distance in the depth direction to the subject based on the correlation amount between the plurality of images.

Images