JP2017138199A - Image processing device, imaging device, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a distance image having high stability and high resolution from a distance image having different resolution.SOLUTION: An image processing device for generating a third distance image on the basis of a first distance image and a second distance image having resolution lower than that of the first distance image includes: representative value determination means for selecting an area of the first distance image corresponding to an object pixel of the second distance image, and determining a representative value of distance values of the first distance image in the area; correction value determination means for determining a correction value on the basis of a weighted average between the representative value and a distance value of the second distance image in the object pixel; and generation means for generating a third distance image with a correction value obtained by correcting the distance value of each pixel in the area of the second distance image on the basis of the correction value as a pixel value.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an image processing apparatus, and more particularly to a technique for generating a high-resolution distance image by combining distance images with different resolutions.

  Conventionally, a depth from focus (DFD) method has been proposed as a method for acquiring the distance of a shooting scene from an image acquired by an imaging device. In the DFD method, a plurality of images with different blurs are acquired by controlling shooting parameters of the imaging optical system, and the magnitude and correlation amount of each other are calculated. Since the size of the blur and the correlation amount change according to the distance of the subject in the image, the distance can be calculated using the relationship. Distance measurement by the DFD method can generate a distance image with the resolution of an image used for measurement, and can generate an appreciation image and a distance image from the same viewpoint.

  A technique for acquiring a distance image by using a phase difference method used in an autofocus (AF) technique in an imaging apparatus and arranging pixels for distance measurement in an imaging element has been proposed. Since many pixels dedicated to distance measurement cannot be arranged in the image sensor, the resolution of the obtained distance image is low (the number of pixels is small).

  Many active distance measurement techniques such as Time of Flight (TOF) method have also been proposed. The active method enables stable distance measurement even for a low-contrast subject, but the resolution of the generated distance image is low (the number of pixels is small).

  As a distance measuring method using such a different distance measuring method, there is Patent Document 1. In Patent Document 1, distance images of different distance measurement methods are selected by selecting a highly reliable defocus amount or subject distance from the results of distance measurement using the phase difference method and the results of distance measurement using the DFD method. Proposals have been made to synthesize.

  Further, in Patent Document 2, a proposal has been made to increase the speed of distance measurement by the phase difference method using the distance measurement result of the TOF method as an initial value. Furthermore, proposals have been made to avoid indefiniteness of the TOF method based on the distance measurement result of the phase difference method.

  In Patent Document 3, it is proposed to use a distance measurement result by the TOF method for an area that cannot be measured by the stereo method using the stereo method and the TOF method.

JP 2014-63142 A JP 2008-8687 A JP 2008-116309 A

  When synthesizing distance images with different resolutions as in Patent Document 1, if the distance measurement method is switched for each area, an area in which the distance value changes in a block shape due to the influence of the low-resolution distance image. It can happen.

In Patent Document 2, the TOF method is used for speeding up the distance measurement of the phase difference method.
Since the final distance measurement performance depends on the phase difference method, the distance measurement may not be stable for a subject with low contrast.

  In Patent Document 3, an area that cannot be measured by the stereo method is supplemented by a distance measurement result by the TOF method, and the distance measurement method is switched and integrated. Therefore, an area that could not be measured in the high-resolution distance image by the stereo method may become a uniform distance by using the distance measurement result of the low-resolution TOF method.

  As described above, in any document, it cannot be said that the effect of obtaining a high-accuracy and high-resolution range image by combining different ranging methods is not sufficiently achieved.

  An object of the present invention is to generate a distance image having both accuracy and resolution from distance images of different resolutions obtained by different distance measurement methods.

In order to solve the above-mentioned problem, a first aspect of the present invention generates a third distance image based on a first distance image and a second distance image having a lower resolution than the first distance image. An image processing apparatus that
Representative value determining means for selecting a region of the first distance image corresponding to the target pixel of the second distance image and determining a representative value of the distance value of the first distance image in the region;
Correction value determining means for determining a correction value based on a weighted average of the representative value and the distance value of the second distance image in the target pixel;
Generating means for generating a third distance image having a distance value obtained by correcting the distance value of each pixel in the region of the second distance image based on the correction value as a pixel value;
It is characterized by comprising.

A second aspect of the present invention is an image processing method for generating a third distance image based on a first distance image and a second distance image having a lower resolution than the first distance image,
A representative value determining step of selecting a region of the first distance image corresponding to the target pixel of the second distance image and determining a representative value of the distance value of the first distance image in the region;
A correction value determining step for determining a correction value based on a weighted average of the representative value and the distance value of the second distance image in the target pixel;
Generating a third distance image having a distance value obtained by correcting the distance value of each pixel in the region based on the correction value as a pixel value;
It is characterized by including.

  According to the present invention, it is possible to generate a distance image having both accuracy and resolution from distance images having different resolutions obtained by different distance measurement methods.

1 is a diagram illustrating a configuration of an imaging apparatus according to a first embodiment. Explanatory drawing of the pixel arrangement | positioning in the image pick-up element in 1st embodiment. The figure which shows the structure of the imaging pixel which concerns on 1st embodiment. The figure which shows the structure of the pixel for distance measurement which concerns on 1st embodiment. 5 is a flowchart showing a flow of imaging processing in the first embodiment. The figure explaining the distance measurement of the DFD system in 1st embodiment. The flowchart which shows the flow of the distance correction process in 1st embodiment. The figure explaining the distance correction process in 1st embodiment. The figure which shows the structure of the image pick-up element which concerns on the modification in 1st embodiment. The figure which shows the structure of the imaging device which concerns on the modification in 1st embodiment. The flowchart which shows the flow of distance measurement in 2nd embodiment.

(First embodiment)
<System configuration>
FIG. 1A is a system configuration diagram of the imaging apparatus according to the first embodiment. The imaging apparatus 1 includes an imaging optical system 10, an imaging element 11, a control unit 12, a signal processing unit 13, a distance measurement unit 14, a memory 15, an input unit 16, a display unit 17, and a storage unit 18.

  The imaging optical system 10 is an optical system that includes a plurality of lenses and forms incident light on the image plane of the imaging element 11. As the imaging optical system 10, a variable focus optical system is used, and automatic focusing is possible by the autofocus function of the control unit 12.

  The control unit 12 is a functional unit that controls each unit of the imaging apparatus 1. Functions of the control unit 12 include, for example, automatic focusing by autofocus (AF), change of focus position, change of F value (aperture), image capture, control of shutter and flash (both not shown), input There are control of the unit 16, the display unit 17, and the storage unit 18.

  The signal processing unit 13 is a functional unit that performs processing on a signal output from the image sensor 11. Specifically, analog signal A / D conversion, noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, color correction, and the like are performed. The digital image data output from the signal processing unit 13 is temporarily stored in the memory 15, and then displayed on the display unit 17, recorded (stored) in the storage unit 18, and output to the distance measurement unit 14 and the like. Is performed.

  The distance measuring unit 14 is a function that calculates the distance in the depth direction to the subject in the image. As shown in FIG. 1B, the distance measuring unit 14 includes a phase difference type distance image generating unit 141, a DFD type distance image generating unit 142, a representative value determining unit (representative value determining means) 143, a correction value determining unit ( Correction value determination means) 144 and generation unit (generation means) 147. The generation unit 147 includes a correction unit 145 and a smoothing processing unit 146. The distance measuring unit 14 may be configured by a logic circuit, may be configured by a computer (processor) and a program, or may be configured by a combination thereof. Details of the distance measuring unit 14 will be described later.

  The input unit 16 is an interface that is operated by the user to input information and change settings on the imaging apparatus 1. For example, dials, buttons, switches, touch panels, etc. can be used.

  The display unit 17 is a display unit configured by a liquid crystal display, an organic EL display, or the like. The display unit 17 is used for composition confirmation at the time of photographing, browsing of photographed / recorded images, display of various setting screens and message information, and the like.

  The storage unit 18 is a non-volatile storage medium that stores captured image data, parameter data used by the imaging apparatus 1, and the like. The storage unit 18 is preferably a high-capacity storage medium that can read and write at high speed. For example, a flash memory can be suitably used.

<Configuration of image sensor>
FIG. 2 is a diagram illustrating the configuration of the image sensor 11. One imaging pixel 21 of the image sensor 11 is composed of a 2 × 2 Bayer array pixel group. Further, the distance measurement pixels 22H and 22V are regularly and sparsely arranged in the Bayer array. In FIG. 2, one set of distance measurement pixels is arranged for a 10 × 10 square area. The distance measurement pixels 22H divided in the horizontal direction and the distance measurement pixels 22V divided in the vertical direction are alternately arranged for each block.

  FIGS. 3A and 3B are diagrams for explaining the imaging pixel 21. As shown in FIG. 3A, each photographing pixel 21 is composed of a set of 4 pixels in 2 rows and 2 columns. The photographing pixel 21 is a pixel having G (green) spectral sensitivity as a diagonal pixel, and the remaining two pixels having one pixel having R (red) and B (blue) spectral sensitivity. It has a Bayer arrangement arranged one by one.

  FIG. 3B illustrates an S-S ′ cross-sectional view in FIG. A microlens ML is disposed above each pixel 21, and a color filter CL is disposed below the microlens ML. The color filter is a filter that transmits each wavelength of RGB in each pixel 21 and is arranged in the Bayer arrangement as described above. Below the color filter is a wiring layer CL, which is a wiring layer for forming signal lines for transmitting various signals in the image sensor. A photoelectric conversion unit PD, which is a photoelectric conversion element, is disposed below the photoelectric conversion unit PD.

  The distance measurement pixels 22H and 22V for performing distance detection by the pupil division phase difference method are arranged in the imaging pixels arranged in the Bayer array in this way. In the distance measurement pixels 22H and 22V, among the pixels in 2 rows and 2 columns of the Bayer array, the G pixel is left for photographing, and the distance measurement pixels are arranged at the R pixel position and the B pixel position.

  The reason why the G pixel is left as a photographing pixel is that the sensitivity characteristic of human vision has a peak at 555 nm, and thus the sensitivity is high in the G wavelength range. Further, the visual characteristics are more sensitive to luminance information than color information, and luminance information is acquired mainly by G pixels, and color information is acquired mainly by R pixels and B pixels. This is because it is difficult to be recognized.

  4A and 4B show an example of a 2 × 2 pixel array including distance measurement pixels 22H that perform pupil division in the horizontal direction, and FIG. The cross-sectional view taken along the line SS ′ in FIG. The pixel DA and the pixel DB in FIG. 4A are distance measurement pixels, and the filled portions in the pixel DA and the pixel DB are openings of the wiring layer CL. The light irradiated to the distance measurement pixel enters the photoelectric conversion unit PD through the opening. When the distance measurement pixels are not used for image generation, it is preferable to arrange a transparent film filter.

  As shown in FIG. 4B, the basic structure of the pixels DA and DB is the same as that of the photographing pixel 21. The difference is that the opening corresponding to the pixel DA and the opening corresponding to the pixel DB are arranged shifted in the horizontal direction with respect to the center of the microlens ML. As a result, light incident on the pixel DA becomes a light beam that has passed through the left exit pupil EPA of the photographic lens L, and light incident on the pixel DB becomes a light beam that has passed through the right exit pupil EPB of the photographic lens L. That is, pupil division by the image sensor is performed.

  The structure of the distance measurement pixel 22V that divides the pupil in the vertical direction is the same as the structure of the distance measurement pixel 22H except that the opening is shifted in the vertical direction, and thus the description thereof is omitted.

  By using the image sensor 11 shown in the present embodiment, a DFD range image and a phase difference range image can be acquired from the same imaging plane.

<Distance measurement processing>
Next, distance measurement processing performed by the imaging apparatus 1 will be described in detail with reference to the drawings. FIG. 5A is a flowchart showing the basic flow of distance measurement processing.

  When the user operates the input unit 16 to instruct the execution of distance measurement and starts shooting (step S11), the control unit 12 performs shooting control and executes the following shooting. FIG. 5B is a flowchart showing details of the photographing process S11.

  When the photographing process starts, first, the control unit 12 executes autofocus (AF) and automatic exposure control (AE), and determines the focus position, aperture (F number), and shutter speed (step S111). These shooting parameters can also be input by the user. Thereafter, photographing is performed in step S <b> 112, and the control unit 12 captures an image from the image sensor 11 and temporarily stores it in the memory 15.

  When the first image capturing is completed, the control unit 12 changes the image capturing parameters (step S113). The imaging parameter to be changed is at least one of an aperture (F number), a focus position, and a focal length. The parameter value or change amount after the change may be a value stored in advance, or may be determined based on information input by the user through the input unit 16. In this embodiment, the focus position is changed and a second image is taken. For example, the first image is shot so as to focus on the main subject, and the second image is shot while changing the focus position so that the main subject is blurred.

  When the change of the shooting parameter is completed, the process proceeds to step S114, and the second shooting is performed by the control unit 12.

  When the photographing of the two images is completed, two images (A image and B image) with parallax are generated from the distance measurement pixels in the first photographed image (step S115). Specifically, the subject image acquired by the pixel DA is an A image, and the subject image acquired by the pixel DB is a B image.

  When the above shooting is completed, the signal processing unit 13 processes the shot and generated images so as to be images suitable for distance measurement, and temporarily stores them in the memory 15. Specifically, development processing is performed, but it is necessary to avoid edge enhancement processing or the like so that blur does not change during signal processing. In addition, a luminance image or a specific color image is suitable as an image used for the subsequent distance measurement processing. At this time, at least one of the captured images may be signal-processed as an ornamental image and stored in the memory 15.

<Distance measurement method by phase difference method>
Next, the distance image generation unit 141 of the distance measurement unit 14 performs distance measurement by the phase difference method (step S12) to generate a phase difference method distance image (second distance image) _DPD . The distance measurement process S12 by the phase difference method will be described with reference to the flowchart of FIG.

ここで、Ａ（ｉ）はＡ像の像信号、Ｂ（ｉ）はＢ像の像信号、ｉは画素番号、ｋはＡ像
とＢ像の相対シフト量である。ｍは、相関値Ｃ（ｋ）の演算に用いる対象画素範囲を表している。 In the phase difference method, the defocus amount Def of the subject image is detected by detecting the image shift amount r between the A image and the B image. Therefore, the distance image generation unit 141 obtains the detection of the image shift amount r by the correlation calculation between the A image and the B image (step S121). A known method can be used for the correlation calculation. For example, the correlation value C (k) is calculated by Equation 1, and the image shift amount r can be calculated from k where C (k) = 0.

Here, A (i) is an image signal of A image, B (i) is an image signal of B image, i is a pixel number, and k is a relative shift amount between the A image and the B image. m represents the target pixel range used for the calculation of the correlation value C (k).

ここでｗは基線長（射出瞳ＥＰＡと射出瞳ＥＰＢの中心間隔）であり、Ｚは撮像素子から射出瞳までの距離である。 Next, the distance image generation unit 141 calculates the defocus amount Def by substituting the calculated image shift amount r into Equation 2 (step S122).

Here, w is a base line length (a center distance between the exit pupil EPA and the exit pupil EPB), and Z is a distance from the image sensor to the exit pupil.

  By performing the same processing on the entire surface of the A image and the B image used for distance measurement, a distance image having a defocus amount as a pixel value can be generated. In addition, the calculated defocus amount can be converted into a distance to the subject using a parameter at the time of shooting, and a distance image can be generated.

  For example, as shown in FIG. 2, one pixel for distance measurement is arranged for 10 rows and 10 columns. Therefore, the distance image calculated from the acquired A image and B image is the same as the captured image. The distance image has a small number of pixels. When this distance image is enlarged to the number of pixels of the image sensor (the number of pixels of the captured image or the distance image of the DFD method), the same distance value is obtained in the block region R of 10 rows and 10 columns, and the distance is large but the resolution is low. It becomes an image.

<Distance measurement method by DFD method>
Next, the distance image generation unit 142 of the distance measurement unit 14 performs distance measurement by the DFD method (step S13) to generate a DFD method distance image (first distance image) D _DFD . The distance measurement process S13 using the DFD method will be described with reference to FIGS. 6 (A) to 6 (C).

撮像素子までの距離ｖ、焦点距離ｆ、Ｆ値は既知なのでぼけの半径σを求めれば物体距離を求めることができる。 The DFD method is a method of measuring a distance from a difference in blur between two images. FIG. 6A illustrates the DFD method. When the blur radius is σ, the distance to the image sensor is v, the focal length is f, and the F value is F, the object distance s is expressed by Equation 3.

Since the distance v to the image sensor, the focal length f, and the F value are known, the object distance can be obtained by obtaining the blur radius σ.

  Here, two images with different blurs are acquired by changing the shooting conditions, and distance information can be calculated from the difference in blurs. The imaging condition to be changed is preferably at least one of the size of the aperture, the focus position, and the focal length. Since the size of the blur increases as the focus position is minimized and defocusing is performed, the defocus amount can be calculated from the difference in blur between the two images.

  In the present embodiment, a method for acquiring distance information from two images taken by changing the focus position will be described as an example. FIG. 6B is a flowchart of distance measurement by the DFD method.

First, the coordinates on the image for calculating the distance information are determined, and a local region around the determined coordinate position in the two images is selected (step S131). It is assumed that the same subject image appears in the area selected here. If there is a shift in the image due to movement such as camera shake during the two-image shooting, it is necessary to execute the alignment process between the two images in advance. By shifting the region selection one pixel at a time and setting a local region over the entire image, it is possible to calculate a high-resolution distance image having the same number of pixels as the input image.

収差が無くデフォーカスした場合のぼけ方がフォーカス位置の前と後で同じ場合、像側においてフォーカスを移動して撮影した２つのフォーカス位置の中間でぼけが同等となり、その位置での相関が最も高い値となる。この中間位置から離れるに従い、二画像のぼけ方は変化し相関が低下していく。つまり、ぼけの大きさが同じ位置をピークとしてその位置から離れるに従い相関が低下する。相関値はデフォーカスによるぼけに応じた値となるため、相関値が分かれば対応するデフォーカス量を知ることができ、フォーカス位置からの相対的な距離を算出することが可能となる。 Next, in the correlation value calculation step S132, calculates a local region Ir ₁ of the first image _{I 1} selected in the region selection step S131, the correlation between the local region Ir ₂ of the second image _{I 2} in the formula 4 .

If there is no aberration and the defocus is the same before and after the focus position, the blur is equivalent between the two focus positions taken by moving the focus on the image side, and the correlation at that position is the most. High value. As the distance from the intermediate position increases, the blurring of the two images changes and the correlation decreases. That is, the correlation decreases with increasing distance from the position where the blur is at the same position. Since the correlation value is a value corresponding to the blur due to defocus, if the correlation value is known, the corresponding defocus amount can be known, and the relative distance from the focus position can be calculated.

  Next, in the distance information calculation step S133, the correlation value calculated in the correlation calculation step S132 is converted into an image distance (defocus amount) from the focus position. The relationship between the correlation value and the defocus amount is as shown in FIG. A different conversion table or conversion expression for converting the correlation value into the defocus amount is held and converted into the defocus amount. The conversion table or the conversion formula is obtained in advance by measurement or by simulation using optical design information. In the case of simulation, a representative wavelength is selected and the result of simulation using that wavelength is applied. Or you may average and use the result of multiple wavelengths.

In the present embodiment, Formula 4 has been described as an example of the calculation method, but any formula can be used as long as it can determine the blur relationship between two images, and is not limited to this formula. If the relationship between the output value according to the calculation and the focus position on the image plane is known, conversion to a relative distance is possible. As another calculation example, the following Expression 5 is given.

ここでＦはフーリエ変換を表し、ＯＴＦは光学伝達関数、Ｓは撮影シーンのフーリエ変換結果を表している。数式６では、２つの撮影条件における光学伝達関数の比が得られ、予め光学系の設計データからこの値のデフォーカスによる変化を知ることができるため変換が可能となる。 Further, Formula 6 is given as an example of calculation using the evaluation value in the frequency space by performing Fourier transform.

Here, F represents the Fourier transform, OTF represents the optical transfer function, and S represents the Fourier transform result of the shooting scene. In Expression 6, the ratio of the optical transfer function under the two imaging conditions is obtained, and the change due to defocus of this value can be known from the design data of the optical system in advance, so that conversion is possible.

  Through the processes in steps S12 and S13 described above, distance information can be acquired as an image by two distance measurement methods. The distance information may be an absolute distance to the subject or a relative distance (defocus amount) from the focus position to the subject. However, since distance correction is performed in the subsequent processing, the two distance information must be the same information, and it is necessary to unify the absolute distance to the subject or the defocus amount. In the present embodiment, the following description will be given using an example in which the defocus amount is acquired as an image.

<Distance correction>
In the distance correction step S14, by integrating the distance image (a first distance image) D _DFD by the distance image (a second distance image) D _PD and the DFD using the phase difference method, high stability high resolution range image (Third distance image) D3 is generated. In the present embodiment, the distance image D distance value at each pixel in the _DFD (pixel value), by correcting on the basis of the distance image D _PD and the distance image D _DFD, generates a distance image D3. In the present embodiment, the distance image D3 is also referred to as a corrected distance image.

Before describing specific processing contents, first, distance images obtained by the two distance measurement methods according to the present embodiment are compared from the viewpoint of resolution. In the distance image _DPD by the phase difference method, since the pixels for distance measurement are sparsely arranged in the image sensor 11 as shown in FIG. 2, the resolution in the image plane is lower than that of the distance image _DDFD . Distance image D _PD the distance image D _DFD same number of pixels as if the enlarged so that the (number of pixels of the image sensor) is a range image having the same distance values in blocks. FIG. 8A shows an example of the one-dimensional direction of the distance measurement result by the phase difference method. On the other hand, the distance image D _DFD by the DFD method is subjected to correlation calculation for each pixel, and a distance is obtained for each pixel. Accordingly, the distance image _DDFD has the same number of pixels as the number of pixels of the image sensor 11 as shown in FIG. That is, the distance image D _DFD according to the DFD method is a distance image with higher definition (higher resolution) than the distance image D _PD according to the phase difference method.

  Next, the two methods are compared from the viewpoint of the stability of distance measurement. In the case of the DFD method in which the correlation value of the selected area in the two images is directly associated with the defocus amount as in Expression 4, the DFD method tends to be easily influenced by noise and the subject, and the stability of the distance measurement is low. . On the other hand, in the case of the phase difference method, a search based on the correlation value is performed when calculating the image shift amount, and a region where the most compared region is the same is obtained from the change in the correlation value. is there.

Utilizing these advantages and disadvantages, ranging using the distance image D _PD by highly stable phase difference method, by correcting the high distance image D _DFD of screen resolution by the DFD, high stability and high screen Make both resolutions compatible.

  FIG. 7 shows a flowchart showing the flow of processing in the distance correction step S14.

First, in step S141, the representative value determining unit 143 selects one pixel from the distance image _DPD . In step S142, the range of the distance image D _DFD corresponding to the selection target pixel of the distance image D _PD is selected. As described above, towards the distance image _{D PD} is it has a lower resolution than the distance image _{D DFD,} 1 pixel of the distance image _{D PD} corresponds to a plurality of pixels of the distance image _{D DFD.} In the present embodiment, one pixel of the distance image _{D PD} corresponds to a region of 10 pixels × 10 pixels of the distance image _{D DFD.}

In step S143, the representative value determining unit 143 determines a representative value of the defocus amount (distance value) of the distance image D _DFD (high resolution distance image) in the selected region. Hereinafter, this representative value is also referred to as a representative defocus amount DH _rep . The representative defocus amount DH _rep is preferably an average value or a median value of the defocus amounts in the region.

In step S144, the correction value determination unit 144 acquires the defocus amount DL (distance value) at the selected pixel of the distance image D _DFD (low resolution distance image).

The region of the distance image D _DFD corresponding to the pixel RPi of the distance image D _PD in FIG. 8A is a region RDi shown in FIG. Dashed lb in FIG. 8 (B) shows a representative amount of defocus in each region of the distance image D _DFD, dashed ld shows the defocus amount at the corresponding pixel of the distance image D _PD. Generally, these two defocus amounts are different values.

ここで、αは位相差方式による距離計測の安定度とＤＦＤ方式による距離計測の安定度の比を表す。２つの安定度の和が１となるように規格化した場合に、αが位相差方式による距離計測の安定度であり、１−αがＤＦＤ方式による距離計測の安定度である。 In step S145, the correction value determination section 144, a distance image based on the defocus amount _{DH rep} obtained from the representative defocus amount DL and the distance image _{D PD} obtained from the _{D DFD,} to determine a correction value DC according to Equation 7 .

Here, α represents the ratio between the stability of distance measurement by the phase difference method and the stability of distance measurement by the DFD method. When normalization is performed such that the sum of two stability levels is 1, α is the stability of distance measurement by the phase difference method, and 1-α is the stability of distance measurement by the DFD method.

In the present embodiment, the correction value DC is from the distance image _D weighted average value based on the distance measurement stability of the defocus amount DL representative defocus amount _{DH rep} and the distance image _{D PD} of _DFD, representative data of the distance image _{D DFD} It is defined as a value obtained by subtracting the focus amount DH _rep .

  The stability of distance measurement is determined by the distance measurement method and shooting conditions. For example, the stability for each distance meter side method can be determined according to the correct answer rate and the error amount when a known distance is measured by two distance measurement methods. In addition, when taking into account the conditions at the time of shooting, it is possible to measure a known distance using various distance measurement methods under various shooting conditions, and to determine the stability based on the correct answer rate and the error amount for each shooting condition. In the latter case, the correction value determination unit 144 employs the stability α corresponding to the shooting conditions.

  The distance measurement by the phase difference method is performed at a lower spatial frequency than the DFD method, and further, the optimum image shift amount is selected by the correlation calculation at the time of calculating the image shift amount, so that the stability is higher than that of the DFD method. Therefore, the stability α of the phase difference method is a value larger than 0.5.

In step S146, the correction unit 145 corrects the DFD method distance image by adding the defocus amount correction value DC calculated in step S141 to the defocus amount of each pixel in the region RD _i . Specifically, when the distance value before correction of the distance image _DDFD is D, the corrected distance value D ′ is determined by the following Equation 8.

By this correction, the representative defocus amount in the region RD _i approaches the defocus amount in the block region RP _i , and the difference between the broken line lb and the dotted line ld becomes smaller than that in FIG. 8B as shown in FIG.

At this time, if there is a large difference in the correction value DC between the block area to be corrected and the block area adjacent thereto (adjacent area), a large step may occur at the boundary of the block area. Therefore, when the difference in the correction value DC between the block area to be corrected and the adjacent block area is equal to or greater than the threshold value, the correction unit 145 replaces the correction value in the correction target area with the adjacent area instead of the correction according to Equation 8. It is also preferable to perform correction by adding a weighted average of the correction values. In this case, this makes it possible to suppress a sudden step between the regions. A predetermined value can be adopted as the weighting coefficient in this weighted average.

In step S147, it is determined whether the above processing has been performed for all pixels (regions). If there are unprocessed pixels, the processing returns to step S141 and the above processing is repeated. By performing the above processing on all the pixels, the DFD distance image _DDFD is corrected, and a corrected distance image D3 is obtained. The correction processing image D3 is a distance image having a distance value obtained by correcting the distance value D of each pixel of the distance image _DDFD based on the defocus amount correction value DC as a pixel value.

  Since the obtained correction distance image D3 has a different defocus amount correction value for each block region, a step of the defocus amount occurs between adjacent blocks as shown in FIG. 8C. Therefore, in step S148, the smoothing processing unit 146 performs an averaging process of, for example, 4 × 4 pixels along the block region boundary. Thereby, the step of the defocus amount caused by the correction becomes inconspicuous. By this process, the step at the block boundary becomes inconspicuous as shown in FIG. 8D, and a smooth distance image can be generated.

  By performing the above correction processing, it is possible to correct a distance image with high resolution using a distance image with high stability, and obtain a distance image with high resolution and high stability. The calculated distance image may be output as it is, or may be converted into an object distance and output using the focal distance and the object-side focus distance.

  According to the present embodiment, the stability information is synthesized by synthesizing the distance information image with high stability of distance measurement and low resolution within the image sensor surface and the distance information image with low stability but high resolution. It becomes possible to generate a distance information image having both resolutions.

<Modification 1>
As another form of the imaging element 11, it is good also as a structure as shown to FIG. 9 (A)-9 (C). FIG. 9A shows a pixel group 201 and a pixel group 202, which are pixels of 2 rows × 2 columns. In the pixel group 201, a green pixel 201Gv is arranged in the diagonal direction, and a red pixel 201Rv and a blue pixel 201Bv are arranged in the other diagonal direction. Similarly, in the pixel group 202, a green pixel 202Gh is arranged diagonally, and a red pixel 202Rh and a blue pixel 2020Bh are arranged on the other side. These two types of pixel groups are arranged on the image sensor 11 in a checkered pattern.

FIG. 9B is a schematic cross-sectional view taken along the line VV ′ in the pixel group 201. Each pixel in the pixel group 201 includes a micro lens 201ML, a color filter 201CF, and a photoelectric conversion unit 201A.
201B. FIG. 9C is a schematic cross-sectional view of HH ′ in the pixel group 202. Each pixel in the pixel portion 202 includes a microlens 202ML, a color filter 202CF, and photoelectric conversion units 202A and 202B. In FIGS. 9B and 9C, the light receiving surface 210 is an xy surface (+ z side surface) on the light incident side of the photoelectric conversion unit.

In the imaging device 11 of this modification, two photoelectric conversion units are arranged in one pixel (each pixel such as a green pixel), and the microlens 201ML is arranged so that the light receiving surface 210 and the exit pupil are optically conjugate. , 202ML power is set. With such an arrangement, the photoelectric conversion units 201A and 201B (and 202A and 202B) can receive light beams that have passed through different regions of the exit pupil. In the pixel group 201 and the pixel group 202, the arrangement of the two photoelectric conversion units is rotated by 90 °. With such a configuration, in the pixel group 201 and the pixel group 202, a light beam that has passed through an area obtained by dividing the exit pupil in different directions is received by the photoelectric conversion unit. As a result, the subject distance can be detected for subjects whose contrast changes in different directions between the pixel group 201 and the pixel group 202.

  In this modification, image signals generated from the light received by the two photoelectric conversion units in each of the pixel groups 201 and 202 are acquired as an A image and a B image, respectively. Since the A and B images have the same number of resolutions as the pixels of the image sensor, it is possible to acquire a high-definition distance image. The calculation of the distance image here is the same process as in step S12. However, when distance measurement is performed using a phase difference method using an image with high resolution, there is a high possibility that noise or an image shift amount calculation error due to a periodic subject will occur when calculating the image shift amount. Therefore, in this modification, in addition to the phase difference method distance measurement using the A image and the B image as they are, the phase difference method distance measurement using the low-frequency component image obtained by reducing the A image and the B image is also performed. Do. When distance measurement is performed using the reduced A and B images, noise and image shift amount calculation errors due to periodic subjects are reduced.

  For the above reason, in this modification, two distance images having different resolutions are obtained. One is high resolution but low stability, the other is low resolution but high stability. Therefore, also in this modification, a high-resolution and high-stability distance image can be obtained by correcting a high-resolution distance image using a low-resolution distance image by the same process as described above. Specifically, the low-resolution distance image is treated as a distance image by the phase difference method in the first embodiment, and the high-resolution distance image is treated as a distance image by the DFD method in the first embodiment. Just do it.

  As described above, even when an imaging element capable of measuring a distance by the phase difference method is used on the entire surface, it is possible to generate a high-definition distance image with higher stability.

<Modification 2>
As another modification, it is possible to employ a configuration that performs distance measurement by the TOF method (Time-of-Flight method), which is an active distance measurement method, and distance measurement by the DFD method. FIG. 10 shows a configuration of the imaging apparatus 2 according to this modification. In the imaging apparatus 2 according to this modification, an illumination unit 19, a hot mirror 20, and an imaging element 21 are added to the configuration of the imaging apparatus 1 of the first embodiment. In addition, the image sensor 11 may be a commonly used image sensor that does not include pixels for distance measurement.

  In this configuration, the control unit 12 controls the illumination unit 19 during imaging to illuminate the modulated near infrared light. In order to separate near infrared light and visible light, a hot mirror 20 is installed in the optical path. Illuminated near-infrared light is reflected by the subject, collected by the imaging optical system, and only the near-infrared light is reflected by the hot mirror 20 to form an image on the image sensor 21. On the other hand, visible light passes through the hot mirror 20 and forms an image on the image sensor 11.

  The image sensor 12 performs exposure four times at a timing shifted by 90 ° with respect to the modulated illumination light, and calculates the phase difference of the reflected light from the subject. The distance to the subject can be calculated by using the calculated phase difference, modulation frequency fm, and speed of light.

  An active distance measurement method such as the TOF method generally has a low dependency on a subject and can perform stable distance measurement. On the other hand, the number of pixels for distance measurement is small, and only a distance image with the number of pixels about VGA can be obtained. Therefore, it is possible to generate a high-definition distance image with high stability by correcting the distance image based on the DFD method in the distance correction step S14 using the distance image based on the TOF method.

The DFD-type distance image and the TOF-type distance image obtained by the imaging apparatus of the present modification are images of different viewpoints. Therefore, it is preferable that the distance measurement unit 14 includes a distance image deformation unit (deformation unit) that performs a deformation process on one or both distance images so that the two distance images have the same viewpoint. .

  Although the TOF method has been described here as a representative example of the active distance measurement method, the same processing can be performed using an active distance measurement method such as another laser scan method (laser scanner method).

(Second embodiment)
Next, a second embodiment of the present invention will be described. In the second embodiment, reliability information (image) representing the reliability of distance information (image) is calculated simultaneously when two distance measurements are executed, and distance correction is performed using the reliability image. Is different from the first embodiment. Since the configuration of the imaging apparatus according to this embodiment is the same as that of the first embodiment, description will be made using the same reference numerals.

  Hereinafter, differences in processing from the first embodiment will be described. FIGS. 11A and 11B are diagrams illustrating the flow of each distance measurement method in the second embodiment. FIG. 11A is a flowchart of the distance measurement S12 by the phase difference method. Compared to the first embodiment (FIG. 5A), a reliability information calculation step S123 is added.

  In the reliability information calculation step S123, distance measurement reliability information indicating how reliable the calculated distance information is is calculated. The reliability information is calculated using the luminance information and contrast information of the image used for calculating the distance information, the number of correlation value peaks when calculating the image shift amount, and the like, and is a value that varies depending on the subject.

  When the image displacement amount is calculated, if the image is blacked out or whiteout in the calculation range, the correct image displacement amount cannot be calculated. Therefore, the luminance value (luminance information) of the subject image is used as an index of reliability information. be able to.

  Further, when the contrast of the image is low in the calculation range, an error occurs in the calculation of the peak value in the correlation calculation at the time of calculating the image shift amount, so that the contrast information can be used as an index of reliability information.

  The number of correlation value peaks at the time of calculating the image shift amount is a large number of correlation value peaks if there is a periodic subject in the calculation range at the time of image shift amount calculation. At this time, since the smallest (large) peak is not necessarily the correct image shift position, it can be considered that the reliability of distance measurement is low when a large number of correlation value peaks occur. Therefore, the periodicity information of the subject image can be used as an index of reliability information.

  FIG. 11B is a flowchart of distance measurement by the DFD method. Compared to the first embodiment (FIG. 8B), a reliability information calculation step S134 is added as described above. As an index for calculating reliability information in the DFD method, the luminance value of the subject image, the contrast of the subject image, the periodicity information of the subject, the motion information of the subject at the time of imaging, the distance measurement range information, the distance edge information, the positional deviation information, etc. Is available.

  Since the difference in blur cannot be detected correctly when blackout or whiteout occurs in the area used for calculation, the luminance value (luminance information) of the subject image can be used as an index of reliability information.

  If the contrast is low, the difference in blur cannot be detected correctly, and the influence of noise becomes dominant in distance measurement. Therefore, the contrast information of the subject image can also be used as an index of reliability information.

  The distance measurement range information is information indicating a range in which distance measurement can be performed with high accuracy. It can be determined that the reliability of the area outside the range specified by the distance measurement range information is low. For example, when distance measurement is performed using Equation 4, when the blur increases, the difference between the blurs of the two images becomes difficult to distinguish, and the correlation increases again. Since this area is different from the area with high correlation near the focus, it is necessary to make the area unreliable.

  In the DFD method, since it is necessary to change the shooting parameters to acquire two images, a positional shift occurs between the two images due to different shooting timings. By performing the alignment process, it is possible to compare the blur in the same area, but it is difficult to match the movement of the subject. For this reason, motion information such as the sum of squares of differences and the sum of absolute differences for each pixel in the calculation area of two images is calculated, and if the motion information is equal to or greater than a threshold, it is determined that the positional deviation is large and the reliability is lowered. Can do.

  As described above, there are many methods for obtaining the reliability information by the distance measurement method, and other methods than those described here can be considered. However, the method for calculating the reliability information is not limited here. The reliability information may be binary information indicating whether the calculated distance information is reliable or not, or may be multi-level reliability information.

  Further, the calculated plural types of reliability information may be integrated and output as one reliability information. At the time of integration, it is desirable to combine the information used for calculating reliability information in consideration of the magnitude of influence on distance measurement. For example, if either one of the blacked out areas or overexposed areas occupies the calculation area, the distance cannot be measured by any distance measurement method including the active method. Integrate information. In the phase difference method, a contrast reduction or a periodic subject has the largest influence as an error factor in distance measurement. In the DFD method, the influence of the position shift due to the difference in the photographing timing has the next highest influence after blackout and whiteout, followed by the influence of the distance measurement range and contrast.

  Next, distance correction using distance information and reliability information calculated by each distance measurement method will be described. The basic flow of the distance correction process is the same as the flowchart shown in FIG. However, the operation of the correction value calculation step S145 is different.

In the first embodiment, the average value or the median value is used when calculating the representative defocus amount in the region RD _i selected in the _DFD distance image D _DFD . In the present embodiment, the weighted average value of the defocus amount DH using the reliability information CM _DFD calculated for each pixel of the distance image D _DFD as represented by Expression 9 is used as the representative defocus amount DH _rep .

When the reliability information CM _DFD is binary information, the representative defocus amount DH _rep is calculated using only reliable defocus amounts. When the reliability information CM _DFD is multi-value information, The representative defocus amount DH _rep is calculated by the weighted average. The correction value DC is calculated by substituting this representative defocus amount DH _rep into Equation 7.

As a result, the influence of the value estimated to have a large distance measurement error in the distance image _DDFD calculated by the DFD method can be suppressed, and a more accurate correction value can be obtained by calculating a more accurate representative defocus amount. Calculation is possible.

  In the correction using Expression 9, correction is performed using only the reliability information of the DFD method, but a method using the reliability information of the phase difference method will be described below.

First, the reliability information CM _PD of the pixel RP _i is acquired in the distance image measured by the phase difference method, and the reliability information of the region RD _i corresponding to the block region RP _i from the distance image similarly measured by the DFD method. Get CM _DFD . In the range image D _DFD of the DFD method, the defocus amount and the reliability information CM _DFD are not constant in the region RD _i . Therefore, the average value of the reliability information of the region RD _i is calculated as the representative reliability information CM _{DFD_REP} (CM _{D
FD_REP} = Average (CM _DFD )). The ratio β of the phase difference reliability information is calculated by Equation 10 using the DFD representative reliability information CM _{DFD_REP} and the phase difference reliability information CM _PD .

  The ratio β of the reliability information of the phase difference method is a value normalized so that the maximum value is 1 and the minimum value is 0, and the sum of the ratios of the reliability information of the phase difference method and the DFD method is 1. .

ここで、距離計測方式の安定度αは被写体とは無関係に決定されるもので主に撮像装置側（距離計測装置側）で決定される値であり、信頼度情報の比率βは被写体側に依存して決定される値である。 In the present embodiment, the defocus amount correction value DC is calculated using Equation 11 obtained by transforming Equation 7 using the reliability information ratio β and stability α. However, the stability α is assumed to satisfy 1 ≧ α> 0.5 similarly to Equation 7.

Here, the stability α of the distance measurement method is determined regardless of the subject and is a value mainly determined on the imaging device side (distance measurement device side), and the reliability information ratio β is on the subject side. It is a value determined depending on.

Also, the defocus amount correction value DC can be calculated by Expression 12 using only the reliability information ratio β.

  In this embodiment, the case where the distance measurement result and reliability information of the phase difference method and the DFD method are used is shown as an example, but the same information is also used in other distance measurement methods using reliability information according to the characteristics of each method. Correction can be performed, and the distance measurement method is not limited.

According to the present embodiment, in addition to the stability of the distance measurement method that does not depend on the subject, it is possible to calculate a more accurate defocus amount correction value by using the reliability information of the distance measurement that depends on the subject. It becomes. Thereby, it is possible to generate distance information with higher accuracy, higher stability, and higher resolution.

<Modification>
The description of each embodiment is an exemplification for explaining the present invention, and the present invention can be implemented with appropriate modifications or combinations without departing from the spirit of the invention. For example, the present invention can be implemented as an imaging apparatus that includes at least a part of the above processing, or can be implemented as a distance measuring apparatus that does not include an imaging unit. Moreover, it can also implement as a distance measurement method, and can also be implemented as an image processing program which makes a distance measurement apparatus perform the said distance measurement method. The above processes and means can be freely combined and implemented as long as no technical contradiction occurs.

  In the above description, a high-resolution and high-stability distance image is generated by correcting the distance value at each pixel of the high-resolution distance image. However, it is only necessary to integrate a high-resolution distance image and a low-resolution distance image to generate a high-resolution and high-stability distance image, for example, a new distance image while retaining the high-resolution and low-resolution distance images. May be generated. That is, the method for generating a high-resolution and high-stability distance image is not limited to the process of correcting an existing distance image.

  In addition, the elemental technologies described in the embodiments may be arbitrarily combined.

  When the present invention is implemented as an image processing device that integrates distance information acquired by different distance measuring devices, it is necessary to add a positioning process between different distance information images. A highly stable and high-resolution distance information image can be generated by executing the distance correction means of the present invention on the two distance information images that have been aligned.

<Example of implementation>
The above-described distance measurement technique of the present invention can be preferably applied to, for example, an imaging apparatus such as a digital camera or a digital camcorder, or an image processing apparatus or computer that performs image processing on image data obtained by the imaging apparatus. Further, the present invention can also be applied to various electronic devices (including mobile phones, smartphones, slate terminals, and personal computers) incorporating such an imaging device or image processing device.

  In the description of the embodiment, the configuration in which the distance measurement function is incorporated in the imaging apparatus main body is shown. However, the distance measurement may be performed by other than the imaging apparatus. For example, a distance measurement function may be incorporated into a computer having an imaging device, and the computer may acquire an image captured by the imaging device and calculate the distance. Alternatively, a distance measurement function may be incorporated in a computer that can be accessed via a wired or wireless network, and the computer may acquire a plurality of images via the network and perform distance measurement.

  The obtained distance information can be used, for example, for various image processing such as image segmentation, generation of stereoscopic images and depth images, and emulation of blur effects.

  It should be noted that the specific mounting on the device can be either software (program) mounting or hardware mounting. For example, various processes for achieving the object of the present invention are realized by storing a program in a memory of a computer (microcomputer, FPGA, etc.) built in an imaging apparatus or an image processing apparatus and causing the computer to execute the program. May be. It is also preferable to provide a dedicated processor such as an ASIC that implements all or part of the processing of the present invention with a logic circuit.

  For this purpose, the program is stored in the computer from, for example, various types of recording media that can serve as the storage device (ie, computer-readable recording media that holds data non-temporarily). Provided to. Therefore, any of the computer (including devices such as CPU and MPU), the method, the program (including program code and program product), and the computer-readable recording medium that holds the program in a non-temporary manner is used in the present invention. Included in the category.

14: Distance measurement unit 143: Representative value determination unit 144: Correction value determination unit 147: Generation unit

Claims (19)

An image processing device that generates a third distance image based on a first distance image and a second distance image having a lower resolution than the first distance image,
Representative value determining means for selecting a region of the first distance image corresponding to the target pixel of the second distance image and determining a representative value of the distance value of the first distance image in the region;
Correction value determining means for determining a correction value based on a weighted average of the representative value and the distance value of the second distance image in the target pixel;
Generating means for generating a third distance image having a distance value obtained by correcting the distance value of each pixel in the region of the second distance image based on the correction value as a pixel value;
An image processing apparatus comprising: The correction value determining means uses the weighting factor based on the ratio between the stability of the distance measurement of the first distance image and the stability of the distance measurement of the second distance image to determine the representative value and the target pixel. A weighted average with the distance value of the second distance image is obtained, and a value obtained by subtracting the representative value from the weighted average is determined as the correction value.
The image processing apparatus according to claim 1. The correction value determining means is a weight based on a ratio between a representative reliability of the distance value in each pixel of the first distance image in the region and a reliability of the distance value in the target pixel of the second distance image. A weighted average of the representative value and the distance value of the second distance image in the target pixel is obtained using a coefficient, and a value obtained by subtracting the representative value from the weighted average is determined as the correction value.
The image processing apparatus according to claim 1. The representative value determining means determines a median value or an average value of distance values of the first distance image in the region as the representative value.
The image processing apparatus according to claim 1. The representative value determining means determines the representative value using the reliability of the distance value in each pixel of the first distance image;
The image processing apparatus according to claim 1. The representative value determining means uses a weighted average value of the distance values of the first distance image in the region, using the reliability of the distance value in each pixel of the first distance image as a weighting factor. As determined,
The image processing apparatus according to claim 5. The reliability is determined using at least one of the contrast of the subject image, the luminance value of the subject image, the periodicity information of the subject, the subject and the movement information at the time of imaging, and the distance measurement range information.
The image processing apparatus according to claim 3. The generating means generates a third distance image having a distance value as a pixel value, which is a value obtained by adding the correction value to the distance value of each pixel in the region of the second distance image.
The image processing apparatus according to claim 1. The generation means includes a correction value determined for the area and a correction value determined for an adjacent area adjacent to the area in the distance value of each pixel in the area of the second distance image. Generating a third distance image having a distance value as a pixel value obtained by adding a weighted average value of
The image processing apparatus according to claim 1. The generation means performs a smoothing process on a boundary of an adjacent region corresponding to an adjacent pixel of the second distance image of the third distance image.
The image processing apparatus according to claim 1. The first distance image and the second distance image are acquired from the same imaging surface.
The image processing apparatus according to claim 1. The first distance image is a distance image in which a distance is detected based on a difference in blur from a plurality of images shot under different shooting conditions.
The image processing apparatus according to claim 1. The second distance image is a distance image in which a distance is detected based on a phase difference between a plurality of images having parallax.
The image processing apparatus according to claim 1. The second distance image is a distance image acquired by the Time of Flight method.
The image processing apparatus according to claim 1. The second distance image is a distance image acquired by a laser scanning method.
The image processing apparatus according to claim 1. Deformation means for deforming at least one of the first distance image and the second distance image so that the first distance image and the second distance image are images of the same viewpoint,
The image processing apparatus according to claim 1. An image sensor;
A distance image generating means for generating the first distance image and the second distance image from an image captured by the image sensor;
The image processing apparatus according to any one of claims 1 to 16,
An imaging apparatus comprising: An image processing method for generating a third distance image based on a first distance image and a second distance image having a lower resolution than the first distance image,
A representative value determining step of selecting a region of the first distance image corresponding to the target pixel of the second distance image and determining a representative value of the distance value of the first distance image in the region;
A correction value determining step for determining a correction value based on a weighted average of the representative value and the distance value of the second distance image in the target pixel;
Generating a third distance image having a distance value obtained by correcting the distance value of each pixel in the region based on the correction value as a pixel value;
Including an image processing method.   The program which makes a computer perform each step of the method of Claim 18.

Images