JP2017053710A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide image processing means which can acquire distance information properly.SOLUTION: An image processing apparatus includes: first filter means which applies a first filter to a first image and a second image different in a shooting condition from the first image; first score value calculation means of calculating a first score value which is the amount of correlation of blur between the first image with the first filter applied thereto and the second image with the first filter applied thereto; second filter means which applies a second filter, which is different in property from the first filter, to the first and second images; second score value calculation means of calculating a second score value which is the amount of correlation of blur between the first image with the second filter applied thereto and the second image with the second filter applied thereto; and reliability evaluation means of evaluating reliability of the first score value, on the basis of the first and second score values.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus and an image processing method.

  Conventionally, a depth from defocus (DFD) method as described in Patent Document 1 has been proposed as a method of acquiring the distance of a shooting scene from an image acquired by an imaging apparatus. In the DFD method, a plurality of images with different blurs are acquired by controlling the shooting conditions (shooting parameters) of the imaging optical system, and the size and image of the blur in each image are obtained using the pixels to be measured and surrounding pixels. The amount of blur correlation between the two is calculated. Since the size of the blur and the correlation amount of the blur change according to the distance of the subject in the image, the distance can be calculated using these relationships.

  For example, in Patent Document 2, when a distance map is generated using the DFD method, a range of distances to be calculated is specified, shooting conditions for each image are determined based on the specified range, and the determined shooting is performed. A technique for calculating a distance using a plurality of images photographed under conditions is disclosed. Patent Document 3 acquires an image focused on the subject, an image focused on the near side of the subject, and an image focused on the back side of the subject. It is disclosed that a wider range measurement is performed by using the image by the DFD method.

JP-A-01-167610 JP 2014-145725 A JP 2014-130131 A

  However, in the techniques disclosed in Patent Documents 1 to 3, there are cases where distance information cannot be acquired satisfactorily.

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of obtaining distance information satisfactorily.

  According to an aspect of the present invention, a first filter unit that applies a first filter to a first image and a second image having different shooting conditions from the first image, A first score value that is a correlation value of blur between the first image to which the first filter is applied and the second image to which the first filter is applied; Score value calculating means, second filter means for applying a second filter having different characteristics from the first filter to the first image and the second image, respectively, Second score value calculation that calculates a second score value that is a blur correlation amount between the first image to which the filter is applied and the second image to which the second filter is applied Means, and the first score value and the second score value. There, the image processing apparatus is provided which is characterized by having a reliability evaluating means for evaluating the reliability of the first score value.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and image processing method which can acquire distance information favorably can be provided.

1 is a block diagram illustrating a configuration of an image processing device according to a first embodiment of the present invention. It is a block diagram which shows the structure of the image processing part of the image processing apparatus by 1st Embodiment of this invention. It is a flowchart which shows the process performed by the image processing apparatus by 1st Embodiment of this invention. It is a flowchart which shows the production | generation process of the reliability map performed by the image processing apparatus by 1st Embodiment of this invention. It is a figure which shows the imaging conditions at the time of acquiring the some image from which a focus state differs. It is a graph which shows BPF power at the time of applying BPF to a focus image and a defocus image. It is a graph which shows the characteristic of the DFD score value of a 1st step. It is a graph which shows the characteristic of the DFD score value of a 2nd step. It is a graph which shows the characteristic of the DFD score value obtained by applying 1st BPF and 2nd BPF from which a pass band differs, respectively. It is a graph which shows a threshold value. It is a graph which shows reliability. It is a figure which shows the imaging conditions at the time of acquiring the some image from which a focus state differs. It is a graph which shows BPF power at the time of applying BPF to a focus image, a 1st defocus image, and a 2nd defocus image. It is a graph which shows the characteristic of a DFD score value. It is a block diagram which shows the structure of the image processing part of the image processing apparatus by 2nd Embodiment of this invention. It is a flowchart which shows the process performed by the image processing apparatus by 2nd Embodiment of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
An image processing apparatus and an image processing method according to a first embodiment of the present invention will be described with reference to FIGS.
The image processing apparatus according to the present embodiment calculates a correlation amount of blur between a plurality of images with different focus states by the DFD method, and calculates distance information based on the calculated correlation amount of blur, thereby calculating a position in the image. Generate a map of distance information for each. In the following description, a map of distance information for each position in the image is referred to as a distance map. Also, the blur correlation amount calculated by the DFD method is referred to as a DFD score value.

  In the present embodiment, a distance map is generated by calculating a DFD score value by the DFD method based on two images of a focus image and a defocus image. Further, in the present embodiment, the calculated DFD score value is positioned in the folding region 803 as will be described later based on two DFD score values obtained by applying two BPFs having different pass characteristics. It is determined whether or not. In the present embodiment, a reliability map indicating whether or not the distance map is correctly calculated is generated together with the distance map.

  First, the configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the present embodiment.

As illustrated in FIG. 1, the image processing apparatus 105 according to the present embodiment includes an imaging optical system 101, an imaging element 102, an image processing unit 103, and a focus drive control unit 104.
Here, a case where the image processing apparatus 105 according to the present embodiment has an imaging function, that is, a case where the image processing apparatus 105 according to the present embodiment is an imaging apparatus will be described as an example, but the present invention is not limited thereto. It is not something. For example, the image processing apparatus 105 according to the present embodiment may not have an imaging function.
Here, a case where the image processing apparatus 105 according to the present embodiment includes the imaging optical system 101 will be described as an example. However, the image processing apparatus 105 according to the present embodiment does not include the imaging optical system 101. Also good. That is, the imaging optical system 101 may be detachable from the image processing apparatus 105 according to the present embodiment.

As the imaging optical system 101, an imaging optical system that is generally used when capturing an image can be used, which may be a single-focus imaging optical system or a variable-focus imaging optical system. Good. The imaging optical system 101 can adjust the magnification, focus position, light amount, and the like of the subject image that reaches the image sensor 102.
The image sensor 102 disposed on the image plane is a CCD image sensor, a CMOS image sensor, or the like that converts the luminous flux of the subject that has passed through the imaging optical system 101 into an electrical signal by photoelectric conversion. The image sensor 102 is not limited to an image sensor having a color filter, and may be a monochrome image sensor or a three-plate image sensor.
The image processing unit 103 performs not only normal signal processing but also processing for generating a distance map from a plurality of images with different focus states using the DFD method. Details of the processing performed by the image processing unit 103 will be described later.
The focus drive control unit 104 drives the imaging optical system 101 so that the subject to be imaged is in focus. The focus drive control unit 104 can also perform focus drive for acquiring a plurality of images with different focus states.
Thus, the image processing apparatus 105 according to the present embodiment is configured.

  Next, an outline of processing using the DFD method performed by the image processing unit 103 of the image processing apparatus 105 according to the present embodiment will be described. The basic concept of the DFD method is described in Patent Document 1 and Patent Document 2, and will be omitted here.

  First, photographing for performing processing using the DFD method will be described with reference to FIG. It is a figure which shows the imaging conditions at the time of acquiring the some image from which a focus state differs. A position 502 is a position of the imaging optical system 101 when the subject 501 is focused, that is, a focus position. When the imaging optical system 101 is located at the focus position 502, the point 505 is an imaging point, and an image in a state where the subject 501 is in focus is formed on the imaging surface 504 of the imaging element 102. The image obtained in this way is referred to as a focus image in the present embodiment.

  On the other hand, when the imaging optical system 101 is moved to the position 503, the subject 501 is not in focus, the point 506 becomes an imaging point, and an image with the subject 501 defocused is connected to the imaging surface 504 of the imaging element 102. Imaged. The image obtained in this way is referred to as a defocused image in this embodiment.

  In this embodiment, the amount D by which the imaging optical system 101 is moved from the position 502 to the position 503 to acquire the focus image and the defocus image is referred to as a focus bracket amount (FB amount). In the present embodiment, processing using the DFD method is performed using the focus image and the defocus image thus captured as input images.

  Next, a method for calculating a DFD score value calculated using the DFD method will be described with reference to FIG. FIG. 6 is a graph showing the BPF power when a band pass filter (BPF) is applied to each of the focus image and the defocus image. The horizontal axis indicates the defocus amount, and the vertical axis indicates the BPF power. The BPF is a filter having a characteristic of passing only a predetermined band. BPF power indicates the intensity of the amplitude at the output of the BPF, and is the absolute value of the result of applying the BPF at the coordinates of the object.

  The magnitude of the output signal of a focus image first BPF processing section (first filter means) 203 described later corresponds to the BPF power when the first BPF is applied to the focus image. Also, the magnitude of the output signal of the second BPF processing unit (second filter means) 204 described later corresponds to the BPF power when the second BPF is applied to the focus image. The magnitude of the output signal of a first BPF processing unit (first filter means) 205 described later corresponds to the BPF power when the first BPF is applied to the defocused image. The magnitude of the output signal of a second BPF processing unit (second filter means) 206 for defocused image, which will be described later, corresponds to the BPF power when the second BPF is applied to the defocused image.

  The BPF power is not limited to the absolute value of the application result of the BPF at the target coordinates, and the absolute value of the application result of the BPF is integrated or within a range determined around the target coordinates. It may be an average.

  As shown in FIG. 6, the characteristic curve of the BPF power of the focus image shows a peak when the defocus amount is zero. On the other hand, the BPF power characteristic curve of the defocused image shows a peak when the defocus amount is + D. Further, the BPF power characteristic curve of the focus image and the BPF power characteristic curve of the defocus image intersect at a position where the defocus amount is + D / 2. That is, when the defocus amount is + D / 2, the BPF power of the focus image is equal to the BPF power of the defocus image.

  In the present embodiment, the DFD score value is calculated based on the ratio between the BPF power of the focus image and the BPF power of the defocus image. Here, for convenience of explanation, calculation of the DFD score value will be described in two stages. The “first stage DFD score value” shown below is different from the “first DFD score value” described later. Further, the “second stage DFD score value” shown below is different from a “second DFD score value” described later.

First, the DFD score value in the first stage will be described with reference to FIG. FIG. 7 is a graph showing characteristics of the first stage DFD score value. The horizontal axis indicates the defocus amount, and the vertical axis indicates the DFD score value. In FIG. 7, the first-stage DFD score value is indicated by a thick dotted line. As can be seen from FIG. 7, the DFD score value of the first stage can take a value in the range from 0.0 to 1.0. If the BPF power value of the focus image is f_bpf (x, y) and the BPF power value of the defocus image is df_bpf (x, y) with respect to the coordinates (x, y), the first stage DFD score value dfdscore '(X, y) is expressed by the following equation (1).

  The DFD score value of the first stage is 1.0 when the defocus amount is + D / 2, approaches 0.0 as the defocus amount moves away from + D / 2, and further moves away from + D / 2. It shows characteristics that approach 1.0.

Next, the DFD score value in the second stage will be described with reference to FIG. FIG. 8 is a graph showing the characteristics of the second stage DFD score value. The horizontal axis indicates the defocus amount, and the vertical axis indicates the DFD score value. In FIG. 8, the DFD score value of the second stage is shown using a thick broken line. The second-stage DFD score value as shown in FIG. 8 is the final DFD score value. The second stage DFD score value dfdscore (x, y) at the coordinates (x, y) is expressed by the following equation (2). In Expression (2), f_bpf (x, y) is the value of the BPF power of the focus image, df_bpf (x, y) is the value of the BPF power of the defocus image, and dfdscore ′ (x, y y) is the first stage DFD score value.

  As shown in FIG. 8, the second stage DFD score value ranges from 0.0 to 2.0, and is defocused with respect to the first stage DFD score value described above with reference to FIG. The characteristic is such that the right half is turned upside down at the position where the amount is + D / 2. The final distance map is calculated by converting the final DFD score value into the defocus amount by using the second-stage DFD score value calculated in this way as the final DFD score value.

As can be seen from FIG. 8, an area 803 showing the same DFD score value is generated even though the defocus amount is different. Such a region 803 occurs because, as can be seen from FIG. 6, the BPF power characteristic curve is folded. For example, since the DFD score value at the position 801 and the DFD score value at the position 802 indicate the same DFD score value, a correct defocus amount is not always obtained based on the DFD score value. In this embodiment, the region 803 in which the DFD score values are the same despite the different defocus amounts is referred to as a DFD score value folding region in this embodiment. The image processing apparatus according to the present embodiment evaluates the reliability of the calculated DFD score value by determining whether or not the calculated DFD score value is located in the folded area 803.
The above is the outline of the processing using the DFD method.

  Next, processing performed by the image processing unit 103 of the image processing apparatus according to the present embodiment will be described with reference to FIGS. FIG. 2 is a block diagram illustrating the configuration of the image processing unit of the image processing apparatus according to the present embodiment.

  As shown in FIG. 2, the image processing unit 103 includes a focus image input unit (first image input unit) 201 and a defocus image input unit (second image input unit) 202. The image processing unit 103 further includes a first BPF processing unit for focus image (first filter) 203 and a second BPF processing unit for focus image (second filter) 204. The image processing unit 103 further includes a defocused image first BPF processing unit (first filter) 205 and a defocused image second BPF processing unit (second filter) 206. The image processing unit 103 further includes a first DFD score calculation unit (first score value calculation unit) 207 and a second DFD score calculation unit (second score value calculation unit) 208. The image processing unit 103 further includes a distance map generation unit (distance information calculation unit) 209 and a reliability map generation unit (reliability evaluation unit) 210.

  FIG. 3 is a flowchart showing the operation of the image processing unit 103 of the image processing apparatus 105 according to the present embodiment.

  In step S <b> 301, the focus image is input to the focus image input unit 201, and the defocus image is input to the defocus image input unit 202.

  In step S302, a process of applying a first BPF having a predetermined passband characteristic to the focus image and the defocused image input in step S301, that is, a first BPF process is performed. Specifically, the first BPF process is performed by the focus image first BPF processing unit 203 on the focus image input via the focus image input unit 201. The first BPF process is performed by the first BPF processing unit 205 for the defocused image on the defocused image input via the defocused image input unit 202.

  In step S303, a process of applying a second BPF having a passband characteristic different from that of the first BPF to the focus image and the defocused image input in step S301, that is, a second BPF process is performed. Done. Specifically, the second BPF process is performed on the focus image input via the focus image input unit 201 by the second BPF processing unit 204 for the focus image. The second BPF process is performed by the second BPF processing unit 206 for defocused images on the defocused image input via the defocused image input unit 202. Note that the passband of the second BPF is preferably located on the lower frequency side than the passband of the first BPF. The reason for using the first BPF and the second BPF having such a pass band will be described later.

  In step S304, a first DFD score map is calculated by calculating a first DFD score value for each coordinate position based on the focus image and the defocused image on which the first BPF process has been performed in step S302. Is generated. Since the calculation method of the first DFD score value has been described above using Expression (1) and Expression (2), description thereof is omitted here.

  In step S305, a second DFD score map is calculated by calculating a second DFD score value for each coordinate position based on the focus image and the defocused image that have been subjected to the second BPF process in step S303. Is generated. The method for calculating the second DFD score value is the same as that in step S304, and thus the description thereof is omitted here.

  In step S306, a distance map is generated by converting the value of the first DFD score map generated in step S304 into a defocus amount. When converting the value of the first DFD score map into the defocus amount, for example, a table as shown in FIG. 8 is used. Note that the table used when converting the value of the first DFE score map into the defocus amount does not include the characteristic curve in the folded region 803.

In step S307, a reliability map indicating the reliability of each coordinate value in the distance map generated in step S306 is generated based on the first DFD score map and the second DFD score map. A method for generating the reliability map will be described later.
The processing performed by the image processing unit 103 has been described above.

  Next, the reliability map generation method performed in step S307 of FIG. 3 will be described in detail.

  In the present embodiment, by using the relationship between the characteristics of the DFD score value obtained by applying the first BPF and the characteristics of the DFD score value obtained by applying the second BPF, the following is performed. To generate a reliability map.

  First, the concept of generating a reliability map will be described with reference to FIG. FIG. 9 is a graph showing the characteristics of the DFD score values obtained by applying the first BPF and the second BPF having different passbands. The passband of the second BPF is located on the lower side than the passband of the first BPF. For this reason, as shown in FIG. 9, the characteristic curve 902 of the second DFD score value to which the second BPF is applied is compared with the characteristic curve 901 of the first DFD score value to which the first BPF is applied. Thus, the range that the DFD score value can take is narrowed. On the other hand, the characteristic curve 902 of the second DFD score value to which the second BPF is applied is compared with the characteristic curve 901 of the first DFD score value to which the first BPF is applied in the folded region 903. No area, that is, a range where a correct DFD score value is obtained becomes wide.

  The characteristic curve 902 of the second DFD score value is not suitable for generating a distance map because the resolution of the DFD score value with respect to the defocus amount is low, but the characteristic curve 901 of the first DFD score value is folded back. This can be used to determine the region 903. In the present embodiment, the reliability map is generated by utilizing such characteristics. For example, the characteristic curve 901 of the first DFD score value includes points 904 and 905 where the DFD score value is the same even though the defocus amount is different. Here, a case will be described as an example in which it is determined whether or not these points 904 and 905 are located in the folded region 903.

  A point 904 on the characteristic curve 901 of the first DFD score value is located in a region that is not in the folded region 903. A point on the characteristic curve 902 of the second DFD score value corresponding to the point 904 is a point 906. On the other hand, a point 905 on the characteristic curve 901 of the first DFD score value is located in the folded region 903. A point on the characteristic curve 903 of the second DFD score value corresponding to the point 903 is a point 907. The DFD score value at point 907 is significantly different from the DFD score value at point 906. Therefore, depending on whether or not the acquired second DFD score value is close to the DFD score value at the point 906, it is the DFD score value at the point 904 in the region that is not within the folded region 903 or within the folded region 903. The DFD score value at the point 905 can be determined. Thereby, it is possible to determine whether or not the reliability of the acquired first DFD score value is high.

Based on the above, the reliability map generation processing performed in step S307 will be described with reference to FIG. FIG. 4 is a flowchart showing a reliability map generation process performed in the image processing unit 103 of the image processing apparatus 105 according to the present embodiment.
In the reliability map generation process, the upper left coordinate of the input image is set to (0, 0), and the process is sequentially performed from the upper left to the lower right.

  In step S401, processing for initializing the coordinates (x, y) to be processed to (0, 0), which is the upper left coordinates, is performed.

  In step S402, a threshold TH (x, y) is calculated based on the first DFD score value dfdscore1 (x, y) corresponding to the coordinates (x, y) to be processed in the first DFD score map. The threshold value TH can be calculated using, for example, a table. FIG. 10 is a graph showing threshold values. The horizontal axis indicates the first DFD score value dfdscore1, and the vertical axis indicates the threshold value TH. The threshold value is set based on a second DFD score value assumed from the first DFD score value in an area that is not the folded area 903. In FIG. 10, the threshold characteristic is simply expressed using a broken line, but the threshold characteristic may not be represented by a broken line.

In step S403, the second DFD score value dfdscore2 (x, y) corresponding to the coordinate (x, y) to be processed in the second DFD score map and the threshold TH (x, y) calculated in step S402. The difference value Δdiff (x, y) is calculated. The difference value Δdiff (x, y) can be calculated based on the following equation (3).

  Here, the difference value Δdiff (x, y) is obtained based on Expression (3), but the present invention is not limited to this. For example, the absolute value of the difference between the second DFD score value dfdscore2 (x, y) and the threshold value TH (x, y) may be obtained, or the second DFD score value dfdscore2 (x, y) and You may make it obtain | require ratio with threshold value TH (x, y).

  In step S404, based on the difference value Δdiff (x, y) calculated in step S403, the reliability Relia (x, y) that is the value of the reliability map is calculated. In the present embodiment, the reliability Relia can be calculated based on a table. FIG. 11 is a graph showing the reliability. The horizontal axis shows the difference value Δdiff, and the vertical axis shows the reliability Relia. As can be seen from FIG. 11, the reliability Relia decreases as the difference value Δdiff increases from 0. In addition, in FIG. 11, the table expressed simply using the broken line is illustrated, but the table may not be represented by the broken line. Alternatively, the table may show different characteristics in the positive direction and the negative direction.

In step S405, it is determined whether or not the processing from step S402 to step S404 has been completed for all coordinates. If the process for all coordinates is completed (YES in step S405), the reliability map generation process is terminated. On the other hand, when the process for all coordinates has not been completed (NO in step S405), the process proceeds to step S406. In step S406, the coordinates (x, y) to be processed are updated, and the process proceeds to step S402.
In this way, the reliability map generation process is performed.

  In this embodiment, the case where the distance map and the reliability map are output in parallel has been described as an example. However, the present invention is not limited to this. For example, the value of the distance map may be changed according to the value of the reliability map. For example, when the value of the reliability map is low, the value of the distance map may indicate a defocus amount corresponding to infinity.

  As described above, according to the present embodiment, the first DFD score value that is the amount of blur correlation between the plurality of images subjected to the process of applying the first filter is calculated. In addition, a second DFD score value that is a correlation amount of blur between a plurality of images subjected to the process of applying the second filter is calculated. Then, the reliability of the first DFD score value is evaluated based on the first DFD score value and the second DFD score value. Specifically, based on the difference between the threshold value preset based on the second DFD score value assumed from the first DFD score value and the second DFD score value, the first DFD score value Evaluate reliability. For this reason, according to the present embodiment, it is possible to provide an image processing apparatus and an image processing method capable of obtaining distance information satisfactorily.

[Second Embodiment]
An image processing apparatus and an image processing method according to the second embodiment of the present invention will be described with reference to FIGS. The same components as those in the image processing apparatus and the image processing method according to the first embodiment shown in FIGS. 1 to 11 are denoted by the same reference numerals, and description thereof will be omitted or simplified.

  The image processing apparatus 105 according to the second embodiment uses three images with different focus states. As the three images having different focus states, for example, one focus image and two defocus images having different focus bracket amounts are used. Then, a distance map is generated by calculating two DFD score values based on the focus image and each of the two defocus images using the DFD method. Note that it is the same as that of the image processing apparatus 105 according to the first embodiment in that it is possible to determine whether or not the calculated DFD score value is within the aliasing region and that a reliability map is generated.

  First, shooting performed to acquire an image in the present embodiment will be described with reference to FIG. FIG. 12 is a diagram illustrating imaging conditions when a plurality of images having different focus states are acquired. In this embodiment, one focus image and two defocus images are acquired. A position 1201 is the position of the imaging optical system 101 when the subject 501 is focused, that is, the focus position. When the imaging optical system 101 is positioned at the focus position 1201, the point 1204 becomes an imaging point, and an image in a state where the subject 501 is focused is formed on the imaging surface 504 of the imaging element 102. In this way, a focus image is obtained.

  On the other hand, when the imaging optical system 101 is moved to the position 1202, the subject 501 is not focused, the point 1205 becomes an imaging point, and an image with the subject 501 defocused is connected to the imaging surface 504 of the imaging element 102. Imaged. In this way, a defocus image with a focus bracket amount of D is obtained.

  Further, even when the imaging optical system 101 is moved to the position 1203, the subject 501 is not focused, the position 1206 becomes an imaging point, and an image in a state where the subject 501 is defocused is an imaging surface of the imaging element 102. An image is formed at 504. In this way, a defocused image with the focus bracket amount D ′ is obtained.

  Here, it is assumed that the focus bracket amount D ′ is smaller than the focus bracket amount D. The reason why the focus bracket amount D ′ is smaller than the focus bracket amount D will be described later. In the following description, a defocus image with a focus bracket amount D is referred to as a first defocus image, and a defocus image with a focus bracket amount D ′ is referred to as a second defocus image.

  Next, the BPF power when the BPF is applied to the focus image, the first defocus image, and the second defocus image will be described with reference to FIG. FIG. 13 is a graph showing the BPF power when the BPF is applied to the focus image, the first defocus image, and the second defocus image. Since the focus image and the first defocus image have been described above with reference to FIG. 6 in the first embodiment, description thereof is omitted here. As can be seen from FIG. 13, the BPF power of the second defocused image shows a peak when the defocus amount is + D ′. Further, the BPF power of the second defocus image and the BPF power of the focus image intersect at a position where the defocus amount is + D ′ / 2.

  Next, the difference between the DFD score value characteristic between the first defocus image and the focus image and the DFD score value characteristic between the second defocus image and the focus image will be described with reference to FIG. To do. FIG. 14 is a graph showing the characteristics of the DFD score value. The method for calculating the DFD score value has been described in the first embodiment with reference to FIG. In FIG. 14, the characteristic curve 1401 of the DFD score value between the first defocused image and the focused image is shown using a thick dotted line. In FIG. 14, a characteristic curve 1402 of the DFD score value between the second defocused image and the focused image is shown using a thick one-dot chain line. Since the characteristic curve 1401 of the DFD score value between the first defocus image and the focus image has been described in the first embodiment with reference to FIG. 7, the description thereof is omitted here. The characteristic curve 1402 of the DFD score value between the second defocus image and the focus image is 1.0 when the defocus amount is + D ′ / 2. The characteristic curve 1402 has such characteristics that the defocus amount gradually approaches 0.0 as the distance from + D ′ / 2 gradually increases, and approaches 1.0 as the defocus amount further increases from + D ′ / 2. Show.

  The focus bracket amount D ′ of the second defocus image is smaller than the focus bracket amount D of the first defocus image. For this reason, the rate of change of the ratio between the BPF power of the second defocus image and the BPF power of the focus image is higher than the rate of change of the ratio of the BPF power of the first defocus image and the BPF power of the focus image. Be gentle. Therefore, the slope of the characteristic curve 1402 of the DFD score value between the second defocus image and the focus image is the slope of the characteristic curve 1401 of the DFD score value between the first defocus image and the focus image. It becomes moderate compared to. In the DFD score value between the second defocus image and the focus image, the DFD score value is correctly calculated as compared with the DFD score value between the first defocus image and the focus image. The area where it is widened. In the present embodiment, by using such characteristics, the calculated DFD score value is included in the folded region based on the DFD score values calculated between the plurality of defocus images and the focus image. It is determined whether or not it is located.

  Next, the configuration of the image processing apparatus 105 according to the present embodiment will be described. FIG. 15 is a block diagram illustrating the configuration of the image processing unit 103 of the image processing apparatus 105 according to the present embodiment.

  The image processing unit 103 includes a focus image input unit 1501, a first defocus image input unit 1502, and a second defocus image input unit 1503. The image processing unit 103 further includes a focus image BPF processing unit 1504, a first defocus image BPF processing unit 1505, and a second defocus image BPF processing unit 1506. The image processing unit 103 further includes a first DFD score calculation unit 1507, a second DFD score calculation unit 1508, a distance map generation unit 1509, and a reliability map generation unit 1510.

  Next, processing performed by the image processing unit 103 of the image processing apparatus 105 according to the present embodiment will be described with reference to FIGS. 15 and 16. FIG. 16 is a flowchart showing operations performed in the image processing unit 103 of the image processing apparatus 105 according to the present embodiment.

  In step S1601, the focus image is input to the focus image input unit 1501, the first defocus image is input to the first defocus image input unit 1502, and the second defocus image is input to the second defocus image. This is input to the unit 1503.

  In step S1602, a process of applying a BPF having a characteristic of allowing a predetermined band to pass, that is, a BPF process is performed on the focus image, the first defocus image, and the second defocus image. Specifically, the focus image BPF processing unit 1504 performs BPF processing on the focus image input via the focus image input unit 1501. Also, the first defocus image BPF processing unit 1505 performs BPF processing on the first defocus image input via the first defocus image input unit 1502. In addition, the second defocus image BPF processing unit 1506 performs BPF processing on the second defocus image input via the second defocus image input unit 1503.

  In step S1603, a first DFD score is calculated by calculating a first DFD score value corresponding to each coordinate position based on the focus image and the first defocus image that have been subjected to the BPF process in step S1602. Generate a map. Since the calculation method of the first DFD score value is the same as that of the first embodiment, description thereof is omitted here.

  In step S1604, a second DFD score is calculated by calculating a second DFD score value corresponding to each coordinate position based on the focus image and the second defocused image that have been subjected to the BPF process in step S1602. Generate a map. The method for calculating the second DFD score value is the same as the method for calculating the first DFD score value in step S1603, and thus the description thereof is omitted.

  In step S1605, a distance map is generated by converting the value of the first DFD score map generated in step S1603 into a defocus amount. Since the method of converting the value of the first DFD score map into the defocus amount is the same as the method described in the first embodiment, the description thereof is omitted here.

  In step S1606, a reliability map indicating the reliability of each coordinate value of the distance map generated in step S1605 is generated based on the first DFD score map and the second DFD score map. A method similar to the method described in the first embodiment can be applied to the reliability map generation method. That is, also in the present embodiment, as in the first embodiment, the difference between the threshold value set in advance based on the second DFD score value assumed from the first DFD score value and the second DFD score value To evaluate the reliability of the first DFD score value.

  In this way, processing is performed in the image processing unit 103 of the image processing apparatus 105 according to the present embodiment.

  As described above, according to the present embodiment, the first DFD score value that is the correlation amount of the blur between the focus image and the first defocus image is calculated, and the focus image, the second defocus image, A second DFD score value, which is the amount of correlation between blurs, is calculated. Then, the reliability of the first DFD score value is evaluated based on the first DFD score value and the second DFD score value. For this reason, according to this embodiment, it is possible to provide an image processing apparatus and an image processing method that can acquire distance information satisfactorily.

[Modified Embodiment]
As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  For example, in the above embodiment, the case where the distance information acquired by the image processing apparatus 105 is the defocus amount has been described as an example, but the present invention is not limited to this. For example, the distance information acquired by the image processing apparatus 105 may be a distance to the subject. In this case, a table indicating the relationship between the DFD score value and the distance to the subject is created in advance, and the calculated DFD score value may be converted into the distance to the subject using the table. .

  In the first embodiment, the case where a focus image and a defocus image are used as a plurality of images having different focus states has been described as an example. However, the present invention is not limited to this. For example, a plurality of defocus images having different focus states may be used as the plurality of images having different focus states.

  In the second embodiment, the case where one focus image and two defocus images are used as three images having different focus states has been described as an example. However, the present invention is not limited to this. For example, three defocused images with different focus states may be used as the three images with different focus states.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101 ... Imaging optical system 102 ... Imaging element 103 ... Image processing part 104 ... Focus drive control part 105 ... Image processing apparatus 201 ... Focus image input part 202 ... Defocus image input part 203 ... First BPF processing part 204 for focus images ... second BPF processing unit for focus image 205 ... first BPF processing unit for defocused image 206 ... second BPF processing unit for defocused image 207 ... first DFD score calculation unit 208 ... second DFD score Calculation unit 209 ... Distance map generation unit 210 ... Reliability map generation unit 1501 ... Focus image input unit 1502 ... First defocus image input unit 1503 ... Second defocus image input unit 1504 ... Focus image BPF processing unit 1505 ... First defocus image BPF processing unit 1506... Second defocus image PF processor 1507 ... first DFD score calculator 1508 ... second DFD score calculating unit 1509 ... distance map generation section 1510 ... reliability map generation unit

Claims (14)

First filter means for respectively applying a first filter to a first image and a second image having different shooting conditions from the first image;
A first score value that is a correlation value of blur between the first image to which the first filter is applied and the second image to which the first filter is applied Score value calculating means for
Second filter means for applying a second filter having different characteristics from the first filter to the first image and the second image, respectively;
A second score value that is a blur correlation amount between the first image to which the second filter is applied and the second image to which the second filter is applied; Score value calculating means for
An image processing apparatus comprising: a reliability evaluation unit that evaluates reliability of the first score value based on the first score value and the second score value. The first score value calculating means applies the first filter to the first image and the first output signal obtained by applying the first filter to the first image. Calculating the first score value based on a ratio to the magnitude of the second output signal obtained by applying the filter of
The second score value calculation means includes a magnitude of a third output signal obtained by applying the second filter to the first image, and the second value for the second image. The image processing apparatus according to claim 1, wherein the second score value is calculated based on a ratio with a magnitude of a fourth output signal obtained by applying the filter. The image processing apparatus according to claim 1, wherein a pass band of the second filter is located on a lower frequency side than a pass band of the first filter. First score value calculating means for calculating a first score value that is a blur correlation amount between the first image and a second image in which the first image has different shooting conditions;
A second score value that is a blur correlation amount between the first image and a third image that has a shooting condition different from any of the first image and the second image; A score value calculating means;
An image processing apparatus comprising: a reliability evaluation unit that evaluates reliability of the first score value based on the first score value and the second score value. The first score value calculating means applies the filter to the second image and the magnitude of the first output signal obtained by applying the filter to the first image. Calculating the first score value based on a ratio to the magnitude of the obtained second output signal;
The second score value calculating means is based on a ratio between the magnitude of the first output signal and the magnitude of the third output signal obtained by applying the filter to the third image. The image processing apparatus according to claim 4, wherein the second score value is calculated. The second image is an image defocused with respect to the first image by a first focus bracket amount;
6. The third image according to claim 4 or 5, wherein the third image is an image defocused with respect to the first image by a second focus bracket amount smaller than the first focus bracket amount. The image processing apparatus described. The reliability evaluation unit is configured to determine the first score based on a difference between a threshold value set in advance based on the second score value assumed from the first score value and the second score value. The image processing apparatus according to claim 1, wherein reliability of the value is evaluated. The image processing apparatus according to claim 1, further comprising a distance information calculation unit that calculates distance information based on the first score value. The image processing apparatus according to claim 8, wherein the distance information is a defocus amount or a distance to a subject. The distance information calculation means outputs the distance information calculated based on the first score value according to the reliability of the first score value evaluated by the reliability evaluation means, The image processing apparatus according to claim 8, wherein it is determined whether to output information different from the distance information calculated based on the first score value. Applying a first filter to a first image and a second image having different shooting conditions from the first image; and
Calculating a first score value that is a correlation amount of blur between the first image to which the first filter is applied and the second image to which the first filter is applied; ,
Applying a second filter having different characteristics from the first filter to the first image and the second image, respectively.
Calculating a second score value that is a correlation amount of blur between the first image to which the second filter is applied and the second image to which the second filter is applied; ,
An image processing method comprising: evaluating the reliability of the first score value based on the first score value and the second score value. Calculating a first score value, which is a blur correlation amount between the first image and the second image in which the first image has different shooting conditions;
Calculating a second score value, which is a blur correlation amount between the first image and a third image having a shooting condition different from any of the first image and the second image;
An image processing method comprising: evaluating the reliability of the first score value based on the first score value and the second score value. On the computer,
Applying a first filter to a first image and a second image having different shooting conditions from the first image; and
Calculating a first score value that is a correlation amount of blur between the first image to which the first filter is applied and the second image to which the first filter is applied; ,
Applying a second filter having different characteristics from the first filter to the first image and the second image, respectively.
Calculating a second score value that is a correlation amount of blur between the first image to which the second filter is applied and the second image to which the second filter is applied; ,
A program for executing the step of evaluating the reliability of the first score value based on the first score value and the second score value. On the computer,
Calculating a first score value, which is a blur correlation amount between the first image and the second image in which the first image has different shooting conditions;
Calculating a second score value, which is a blur correlation amount between the first image and a third image having a shooting condition different from any of the first image and the second image;
A program for executing the step of evaluating the reliability of the first score value based on the first score value and the second score value.

Images