JP6700818B2 - Image processing device, imaging device, and image processing method - Google Patents

Description

  The present invention relates to an image processing device, and more particularly to a technique for generating a high resolution range image by combining range images having different resolutions.

  Conventionally, a depth from focus (DFD) method has been proposed as a method of acquiring the distance of a shooting scene from an image acquired by an imaging device. In the DFD method, a plurality of images with different blurs are acquired by controlling the shooting parameters of the image pickup optical system, and the sizes of blurs and the amounts of correlation are calculated. Since the size of the blur and the correlation amount change according to the distance of the subject in the image, the distance can be calculated using the relationship. Distance measurement by the DFD method can generate a distance image at the resolution of an image used for measurement, and can generate a viewing image and a distance image from the same viewpoint.

  A technique has been proposed in which a phase difference method used in an autofocus (AF) technique in an image pickup apparatus is used to obtain a distance image by arranging pixels for distance measurement in an image pickup device. Since many pixels dedicated to distance measurement cannot be arranged in the image sensor, the resolution of the obtained distance image is low (the number of pixels is small).

  In addition, many active distance measurement techniques such as the Time of Flight (TOF) method have been proposed. The active method enables stable distance measurement even for a subject with low contrast, but the resolution of the generated distance image is low (the number of pixels is small).

  As a distance measuring method using such a different distance measuring method, there is Patent Document 1. In Patent Document 1, from the result of distance measurement by the phase difference method and the result of distance measurement by the DFD method, by selecting a highly reliable defocus amount or subject distance, distance images of different distance measurement methods can be obtained. Proposals for synthesis have been made.

  Further, in Patent Document 2, a proposal is made to speed up the distance measurement by the phase difference method using the TOF method distance measurement result as an initial value. Further, a proposal has been made to avoid the uncertainty of the TOF method based on the distance measurement result of the phase difference method.

  Patent Document 3 proposes to use the distance measurement result by the TOF method in the area where the distance cannot be measured by the stereo method using the stereo method and the TOF method.

JP, 2014-63142, A JP, 2008-8687, A JP, 2008-116309, A

  When the distance measurement method is switched for each area when the distance images having different resolutions are combined as in Japanese Patent Laid-Open No. 2004-242, there are areas where the distance value changes in blocks due to the influence of the low-resolution distance image. It can happen.

In Patent Document 2, the TOF method is used to speed up the distance measurement by the phase difference method.
Since the final distance measurement performance depends on the phase difference method, the distance measurement may not be stable for a subject with low contrast.

  In Patent Document 3, the area that could not be measured by the stereo method is supplemented by the distance measurement result by the TOF method, and the distance measurement method is switched and integrated. Therefore, there is a possibility that a region that cannot be measured in the high-resolution range image of the stereo method will have a uniform distance by using the distance measurement result of the low-resolution TOF method.

  As described above, it cannot be said in any of the documents that the effect of combining different distance measuring methods to obtain a highly accurate and high resolution range image can be sufficiently achieved.

  An object of the present invention is to generate a distance image having both accuracy and resolution from distance images having different resolutions obtained by different distance measuring methods.

In order to solve the above problems, a first aspect of the present invention is to generate a third range image based on a first range image and a second range image having a lower resolution than the first range image. An image processing device for
Representative value determining means for selecting a region of the first distance image corresponding to the target pixel of the second distance image, and determining a representative value of distance values of the first distance image in the region.
Correction value determining means for determining a correction value based on a weighted average of the representative value and the distance value of the second distance image at the target pixel;
A generating means for generating a third range image having a distance value corrected as a pixel value based on the distance value of each pixel before Symbol region on the correction value,
Is provided.

A second aspect of the present invention is an image processing method for generating a third distance image based on a first distance image and a second distance image having a lower resolution than the first distance image,
A representative value determination step of selecting a region of the first distance image corresponding to the target pixel of the second distance image, and determining a representative value of distance values of the first distance image in the region.
A correction value determination step of determining a correction value based on a weighted average of the representative value and the distance value of the second distance image in the target pixel,
A generation step of generating a third distance image having a distance value obtained by correcting the distance value of each pixel in the area based on the correction value,
It is characterized by including.

  According to the present invention, it is possible to generate a distance image having both accuracy and resolution from distance images having different resolutions obtained by different distance measuring methods.

FIG. 3 is a diagram showing a configuration of an image pickup apparatus according to the first embodiment. FIG. 3 is an explanatory diagram of pixel arrangement in the image sensor according to the first embodiment. FIG. 3 is a diagram showing a configuration of an image pickup pixel according to the first embodiment. The figure which shows the structure of the pixel for distance measurement which concerns on 1st embodiment. 3 is a flowchart showing the flow of shooting processing according to the first embodiment. The figure explaining distance measurement of the DFD method in a first embodiment. 6 is a flowchart showing a flow of distance correction processing in the first embodiment. The figure explaining the distance correction process in 1st embodiment. FIG. 6 is a diagram showing a configuration of an image sensor according to a modified example of the first embodiment. The figure which shows the structure of the imaging device which concerns on the modification in 1st embodiment. The flowchart which shows the flow of the distance measurement in 2nd embodiment.

(First embodiment)
<System configuration>
FIG. 1A is a system configuration diagram of the image pickup apparatus according to the first embodiment. The image pickup apparatus 1 includes an image pickup optical system 10, an image pickup element 11, a control unit 12, a signal processing unit 13, a distance measuring unit 14, a memory 15, an input unit 16, a display unit 17, and a storage unit 18.

  The image pickup optical system 10 is an optical system that includes a plurality of lenses and forms incident light on the image plane of the image pickup element 11. A variable focus optical system is used as the imaging optical system 10, and automatic focusing is possible by the autofocus function of the control unit 12.

  The control unit 12 is a functional unit that controls each unit of the imaging device 1. The functions of the control unit 12 include, for example, automatic focusing by autofocus (AF), focus position change, F value (aperture) change, image capture, shutter and flash (neither is shown) control, and input. There are controls for the unit 16, the display unit 17, and the storage unit 18.

  The signal processing unit 13 is a functional unit that processes a signal output from the image sensor 11. Specifically, A/D conversion of analog signals, noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, color correction, etc. are performed. The digital image data output from the signal processing unit 13 is temporarily stored in the memory 15, and then displayed on the display unit 17, recorded (stored) in the storage unit 18, output to the distance measuring unit 14, etc. Is processed.

  The distance measuring unit 14 has a function of calculating the distance in the depth direction to the subject in the image. As shown in FIG. 1B, the distance measuring unit 14 includes a phase difference distance image generating unit 141, a DFD method distance image generating unit 142, a representative value determining unit (representative value determining unit) 143, a correction value determining unit ( The correction value determining unit) 144 and the generating unit (generating unit) 147 are included. The generation unit 147 includes a correction unit 145 and a smoothing processing unit 146. The distance measuring unit 14 may be configured by a logic circuit, a computer (processor) and a program, or a combination thereof. Details of the distance measuring unit 14 will be described later.

  The input unit 16 is an interface that a user operates to input information and change settings for the imaging device 1. For example, dials, buttons, switches, touch panels, etc. can be used.

  The display unit 17 is a display unit including a liquid crystal display or an organic EL display. The display unit 17 is used for confirming the composition at the time of photographing, viewing the photographed/recorded images, displaying various setting screens, message information, and the like.

  The storage unit 18 is a non-volatile storage medium that stores captured image data, parameter data used in the imaging device 1, and the like. As the storage unit 18, it is preferable to use a large-capacity storage medium that can read and write at high speed. For example, a flash memory can be preferably used.

<Structure of image sensor>
FIG. 2 is a diagram illustrating the configuration of the image sensor 11. One image-capturing pixel 21 of the image sensor 11 is composed of a pixel group in a Bayer array of 2 rows and 2 columns. Further, the distance measuring pixels 22H and 22V are regularly and sparsely arranged in the Bayer array. In FIG. 2, one set of distance measurement pixels is arranged in a square area of 10 rows and 10 columns. The distance-measuring pixels 22H that are pupil-divided in the horizontal direction and the distance-measuring pixels 22V that are pupil-divided in the vertical direction are arranged alternately for each block.

  FIG. 3A and FIG. 3B are diagrams for explaining the imaging pixel 21. As shown in FIG. 3A, each photographing pixel 21 is composed of a set of 4 pixels in 2 rows and 2 columns. In the imaging pixel 21, pixels having a spectral sensitivity of G (green) are arranged in diagonal pixels, and one pixel having a spectral sensitivity of R (red) and B (blue) is one pixel in the remaining two pixels. Each has a Bayer array.

  FIG. 3B is a sectional view taken along the line S-S′ in FIG. A microlens ML is arranged above each pixel 21, and a color filter CL is arranged below it. The color filter is a filter that transmits each wavelength of RGB in each pixel 21, and is arranged in the Bayer array as described above. Below the color filter, there is a wiring layer CL, which is a wiring layer that forms signal lines for transmitting various signals in the image sensor. A photoelectric conversion unit PD, which is a photoelectric conversion element, is disposed below the photoelectric conversion unit PD.

  Distance measuring pixels 22H and 22V for performing distance detection by the pupil division phase difference method are arranged in the photographing pixels arranged in the Bayer array in this way. In the distance measuring pixels 22H and 22V, among the pixels in 2 rows and 2 columns of the Bayer array, G pixels are left for imaging, and distance measuring pixels are arranged at R pixel positions and B pixel positions.

  The reason why the G pixel is left as a pixel for photographing is that the sensitivity characteristic of human vision has a peak at 555 nm and thus has high sensitivity to the G wavelength region. The visual characteristics are more sensitive to the luminance information than the color information, and the luminance information is mainly acquired by the G pixel and the color information is mainly acquired by the R pixel and the B pixel. Therefore, leaving the G pixel causes deterioration of the image quality. This is because it is hard to be recognized.

  4A and 4B show an example of a pixel array of 2 rows and 2 columns including distance measurement pixels 22H that perform pupil division in the horizontal direction, and FIG. 4B shows FIG. FIG. 3A is a cross-sectional view taken along line S-S′ of FIG. Pixels DA and pixels DB of FIG. 4A are pixels for distance measurement, and the filled portions of the pixels DA and DB are openings of the wiring layer CL. The light with which the distance measuring pixel is irradiated enters the photoelectric conversion unit PD through this opening. If the pixels for distance measurement are not used for image generation, it is preferable to dispose a transparent film filter.

  As shown in FIG. 4B, the basic structure of the pixels DA and DB is the same as the structure of the imaging pixel 21. The difference is that the opening corresponding to the pixel DA and the opening corresponding to the pixel DB are arranged so as to be displaced in the horizontal direction with respect to the center of the microlens ML. As a result, the light incident on the pixel DA becomes a light flux that has passed through the left exit pupil EPA of the taking lens L, and the light that enters the pixel DB becomes a light flux that has passed through the right exit pupil EPB of the taking lens L. That is, pupil division is performed by the image sensor.

  The structure of the distance measuring pixel 22V that performs pupil division in the vertical direction is the same as the structure of the distance measuring pixel 22H except that the opening is shifted in the vertical direction, and therefore description thereof will be omitted.

  By using the image sensor 11 according to the present embodiment, it is possible to acquire a DFD type range image and a phase difference type range image from the same image pickup surface.

<Distance measurement processing>
Next, the distance measurement processing performed by the imaging device 1 will be described in detail with reference to the drawings. FIG. 5A is a flowchart showing the basic flow of distance measurement processing.

  When the user operates the input unit 16 to instruct execution of distance measurement and start photographing (step S11), the control unit 12 performs photographing control and executes the following photographing. FIG. 5B is a flowchart showing details of the photographing process S11.

  When the photographing process is started, first, the control unit 12 executes auto focus (AF) and automatic exposure control (AE) to determine the focus position, aperture (F number), and shutter speed (step S111). Note that these shooting parameters can also be input by the user. After that, shooting is performed in step S112, and the control unit 12 captures an image from the image sensor 11 and temporarily stores it in the memory 15.

  When the shooting of the first image is completed, the control unit 12 changes the shooting parameters (step S113). The shooting parameter to be changed is at least one of aperture (F number), focus position, and focal length. The changed parameter value or the changed amount may be a value stored in advance, or may be determined based on the information input by the user through the input unit 16. In the present embodiment, the focus position is changed to capture the second image. For example, the first image is taken so that the main subject is in focus, and the second image is taken while changing the focus position so that the main subject is blurred.

  When the change of the shooting parameter is completed, the process proceeds to step S114, and the control unit 12 shoots the second image.

  When the photographing of the two images is completed, two images (A image and B image) with parallax are generated from the pixels for distance measurement in the first photographed image (step S115). Specifically, the subject image acquired by the pixel DA is the A image, and the subject image acquired by the pixel DB is the B image.

  When the above shooting is completed, the signal processing unit 13 processes the shot and generated images so as to be images suitable for distance measurement, and temporarily stores them in the memory 15. Specifically, development processing is performed, but edge enhancement processing or the like must be avoided so that blur does not change during signal processing. Further, a luminance image or a specific color image is suitable for the image used for the distance measurement process thereafter. At this time, at least one of the taken images may be signal-processed as an ornamental image and stored in the memory 15.

<Distance measuring method by phase difference method>
Next, the distance image generation unit 141 of the distance measurement unit 14 executes the distance measurement by the phase difference method (step S12) and generates the distance image by the phase difference method (second distance image) D _PD . The distance measurement processing S12 using the phase difference method will be described with reference to the flowchart in FIG.

ここで、Ａ（ｉ）はＡ像の像信号、Ｂ（ｉ）はＢ像の像信号、ｉは画素番号、ｋはＡ像
とＢ像の相対シフト量である。ｍは、相関値Ｃ（ｋ）の演算に用いる対象画素範囲を表している。 In the phase difference method, the defocus amount Def of the subject image is detected by detecting the image shift amount r between the A image and the B image. Therefore, the distance image generation unit 141 obtains the detection of the image shift amount r by the correlation calculation of the A image and the B image (step S121). A known method can be used for the correlation calculation. For example, the correlation value C(k) can be calculated by Expression 1, and the image shift amount r can be calculated from k where C(k)=0.

Here, A(i) is the image signal of the A image, B(i) is the image signal of the B image, i is the pixel number, and k is the relative shift amount between the A image and the B image. m represents the target pixel range used for the calculation of the correlation value C(k).

ここでｗは基線長（射出瞳ＥＰＡと射出瞳ＥＰＢの中心間隔）であり、Ｚは撮像素子から射出瞳までの距離である。 Next, the distance image generation unit 141 calculates the defocus amount Def by substituting the calculated image shift amount r into the mathematical expression 2 (step S122).

Here, w is a base line length (center interval between the exit pupil EPA and the exit pupil EPB), and Z is a distance from the image sensor to the exit pupil.

  By performing similar processing on the entire surfaces of the A image and the B image used for distance measurement, it is possible to generate a distance image having a defocus amount as a pixel value. It is also possible to convert the calculated defocus amount into a distance to a subject using a parameter at the time of shooting and generate a distance image.

  Since one pixel for distance measurement is arranged for 10 rows and 10 columns as shown in FIG. 2, for example, the distance image calculated from the acquired A image and B image is The distance image has a small number of pixels. When this range image is expanded to the number of pixels of the image pickup element (the number of pixels of the range image of the captured image or the DFD method), the same distance value is obtained in the block region R of 10 rows and 10 columns, and the number of pixels is large but the resolution is low. It becomes an image.

<Distance measuring method by DFD method>
Next, the distance image generation unit 142 of the distance measurement unit 14 executes the distance measurement by the DFD method (step S13) to generate the DFD method distance image (first distance image) D _DFD . The distance measurement processing S13 by the DFD method will be described with reference to FIGS. 6(A) to 6(C).

撮像素子までの距離ｖ、焦点距離ｆ、Ｆ値は既知なのでぼけの半径σを求めれば物体距離を求めることができる。 The DFD method is a method of measuring a distance from the difference in blur between two images. FIG. 6A is a diagram illustrating the DFD method. When the radius of blur is σ, the distance to the image sensor is v, the focal length is f, and the F value is F, the object distance s is expressed by Equation 3.

Since the distance v to the image pickup element, the focal length f, and the F value are known, the object distance can be obtained by obtaining the blur radius σ.

  Here, two images with different blurs can be acquired by changing the shooting conditions, and distance information can be calculated from the difference in blurs. It is preferable that the shooting condition to be changed is at least one of the size of the aperture, the focus position, and the focal length. Since the size of the blur increases as the focus position is minimized and the focus is defocused, the defocus amount can be calculated from the difference in blur between the two images.

  In the present embodiment, a method of changing the focus position and acquiring distance information from two captured images will be described as an example. FIG. 6B is a flowchart of distance measurement by the DFD method.

First, the coordinates on the image for which the distance information is calculated are determined, and the local area around the determined coordinate position in the two images is selected (step S131). It is assumed that the same subject image is captured in the area selected here. If the images are misaligned due to movement such as camera shake during the two-image shooting, it is necessary to perform the alignment process between the two images in advance. By shifting the region selection pixel by pixel and setting a local region over the entire image, it is possible to calculate a high-resolution range image having the same number of pixels as the input image.

収差が無くデフォーカスした場合のぼけ方がフォーカス位置の前と後で同じ場合、像側においてフォーカスを移動して撮影した２つのフォーカス位置の中間でぼけが同等となり、その位置での相関が最も高い値となる。この中間位置から離れるに従い、二画像のぼけ方は変化し相関が低下していく。つまり、ぼけの大きさが同じ位置をピークとしてその位置から離れるに従い相関が低下する。相関値はデフォーカスによるぼけに応じた値となるため、相関値が分かれば対応するデフォーカス量を知ることができ、フォーカス位置からの相対的な距離を算出することが可能となる。 Next, in the correlation value calculation step S132, calculates a local region Ir ₁ of the first image _{I 1} selected in the region selection step S131, the correlation between the local region Ir ₂ of the second image _{I 2} in the formula 4 ..

If there is no aberration and the blurring is the same before and after the focus position, the blurring is the same between the two focus positions photographed with the focus moved on the image side, and the correlation at that position is the best. It becomes a high value. As the distance from this intermediate position increases, the blurring of the two images changes and the correlation decreases. That is, the correlation is reduced as the position where the blur size is the same is a peak and the position is farther from the position. Since the correlation value is a value according to the blur due to defocus, if the correlation value is known, the corresponding defocus amount can be known, and the relative distance from the focus position can be calculated.

  Next, in the distance information calculation step S133, the correlation value calculated in the correlation calculation step S132 is converted into an image distance (defocus amount) from the focus position. The relationship between the correlation value and the defocus amount is as shown in FIG. A different conversion table or conversion formula for converting the correlation value to the defocus amount is held and converted to the defocus amount. The conversion table or conversion formula is obtained in advance by measurement or by simulation using optical design information. In the case of simulation, a representative wavelength is selected and the result of simulation using that wavelength is applied. Alternatively, the results of a plurality of wavelengths may be averaged and used.

In the present embodiment, the calculation method has been described by using the formula 4, but the formula is not limited to this formula as long as the formula can determine the blur relationship between two images. If the relationship between the output value according to the calculation and the focus position on the image plane is known, conversion to a relative distance is possible. As another calculation example, the following formula 5 and the like can be given.

ここでＦはフーリエ変換を表し、ＯＴＦは光学伝達関数、Ｓは撮影シーンのフーリエ変換結果を表している。数式６では、２つの撮影条件における光学伝達関数の比が得られ、予め光学系の設計データからこの値のデフォーカスによる変化を知ることができるため変換が可能となる。 Further, as an example of the calculation by the Fourier transform and the evaluation value in the frequency space, Formula 6 can be cited.

Here, F represents a Fourier transform, OTF represents an optical transfer function, and S represents a Fourier transform result of a shooting scene. In Equation 6, the ratio of the optical transfer functions under the two shooting conditions can be obtained, and the change due to defocus of this value can be known from the design data of the optical system in advance, so that conversion is possible.

  The distance information can be acquired as an image by the two distance measuring methods by the processing in steps S12 and S13 described above. The distance information may be an absolute distance to the subject or a relative distance (defocus amount) from the focus position to the subject. However, since distance correction is performed in the subsequent processing, the two pieces of distance information must be the same information, and it is necessary to unify the absolute distance to the subject or the defocus amount. In the present embodiment, the following description will be given by taking the case where the defocus amount is acquired as an image as an example.

<Distance correction>
In the distance correction step S14, by integrating the distance image (a first distance image) D _DFD by the distance image (a second distance image) D _PD and the DFD using the phase difference method, high stability high resolution range image (Third distance image) D3 is generated. In the present embodiment, the distance image D distance value at each pixel in the _DFD (pixel value), by correcting on the basis of the distance image D _PD and the distance image D _DFD, generates a distance image D3. In the present embodiment, the distance image D3 is also referred to as a corrected distance image.

Before explaining specific processing contents, first, distance images obtained by the two distance measuring methods according to the present embodiment are compared from the viewpoint of resolution. In the distance image D _PD by the phase difference method, the pixels for distance measurement are sparsely arranged in the image sensor 11 as shown in FIG. 2, and therefore the resolution in the image plane is lower than that of the distance image D _DFD . When the distance image D _PD is enlarged to have the same number of pixels (the number of pixels of the image sensor) as the distance image D _DFD , the distance image has the same distance value in block units. FIG. 8A shows an example of the one-dimensional direction of the distance measurement result by the phase difference method. On the other hand, in the distance image D _DFD by the DFD method, the correlation calculation is performed for each pixel, and the distance is obtained for each pixel. Therefore, the distance image D _DFD has the same number of pixels as the number of pixels of the image sensor 11 as shown in FIG. That is, the distance image D _DFD by the DFD method is a higher-definition (higher resolution) distance image than the distance image D _PD by the phase difference method.

  Next, we compare the two methods from the viewpoint of stability of distance measurement. When the DFD method is a method in which the correlation value of the selected area in the two images is directly associated with the defocus amount as in Expression 4, the DFD method tends to be easily affected by noise and the subject, and the stability of distance measurement is low. . On the other hand, in the case of the phase difference method, a search is performed by the correlation value when calculating the image shift amount, and the area to be most compared is obtained from the change in the correlation value. is there.

Utilizing these advantages and disadvantages, by using the range image D _PD by the phase difference method with high ranging stability and correcting the range image D _DFD with high in-screen resolution by the DFD method, high stability and high in-screen Make resolution compatible.

  FIG. 7 is a flowchart showing the flow of the process of the distance correction step S14.

First, in step S141, the representative value determination unit 143 selects one pixel from the distance image D _PD . In step S142, the area of the distance image D _DFD corresponding to the selection target pixel of the distance image D _PD is selected. As described above, towards the distance image _{D PD} is has a lower resolution than the distance image _{D DFD,} 1 pixel of the distance image _{D PD} corresponds to a plurality of pixels of the distance image _{D DFD.} In the present embodiment, one pixel of the distance image D _PD corresponds to the area of 10 pixels×10 pixels of the distance image D _DFD .

In step S143, the representative value determination unit 143 determines the representative value of the defocus amount (distance value) of the distance image D _DFD (high-resolution distance image) in the selected area. Hereinafter, this representative value is also referred to as a representative defocus amount DH _rep . The representative defocus amount DH _rep is preferably the average value or the median value of the defocus amounts in the area.

In step S144, the correction value determination unit 144 acquires the defocus amount DL (distance value) in the selected pixel of the distance image D _DFD (low resolution distance image).

The area of the distance image D _DFD corresponding to the pixel RPi of the distance image D _PD of FIG. 8A is the area RDi shown in FIG. 8B. The broken line lb in FIG. 8B indicates the representative defocus amount in each area of the distance image D _DFD , and the broken line ld indicates the defocus amount at the corresponding pixel of the distance image D _PD . Generally, these two defocus amounts have different values.

ここで、αは位相差方式による距離計測の安定度とＤＦＤ方式による距離計測の安定度の比を表す。２つの安定度の和が１となるように規格化した場合に、αが位相差方式による距離計測の安定度であり、１−αがＤＦＤ方式による距離計測の安定度である。 In step S145, the correction value determination unit 144 determines the correction value DC according to Equation 7 based on the representative defocus amount DL obtained from the distance image D _DFD and the defocus amount DH _rep obtained from the distance image D _PD. ..

Here, α represents the ratio of the stability of distance measurement by the phase difference method and the stability of distance measurement by the DFD method. When normalized so that the sum of two stability is 1, α is the stability of distance measurement by the phase difference method, and 1-α is the stability of distance measurement by the DFD method.

In the present embodiment, the correction value DC is calculated from the weighted average value based on the distance measurement stability of the representative defocus amount DH _rep of the distance image D _{DFD and} the defocus amount DL of the distance image D _{PD to} the representative defocus amount of the distance image D _DFD . It is defined as a value obtained by subtracting the focus amount DH _rep .

  The stability of distance measurement depends on the distance measurement method and shooting conditions. For example, the stability for each rangefinder-side method can be determined according to the correct answer rate and the amount of error when a known distance is measured by the two distance measuring methods. Further, in the case of considering the conditions at the time of shooting, the known distances can be measured under various shooting conditions by the two distance measuring methods, and the stability can be determined based on the correct answer rate and the error amount for each shooting condition. In the latter case, the correction value determination unit 144 adopts the stability α according to the conditions at the time of shooting.

  The distance measurement by the phase difference method is more stable than the DFD method because it is calculated at a lower spatial frequency than the DFD method and the optimum image deviation amount is selected by the correlation calculation when calculating the image deviation amount. Therefore, the stability α of the phase difference method becomes a value larger than 0.5.

Next, in step S146, the correction unit 145 corrects the distance image of the DFD method by adding the defocus amount correction value DC calculated in step S141 to the defocus amount of each pixel in the region RD _i . Specifically, assuming that the distance value before correction of the distance image D _DFD is D, the corrected distance value D′ is determined by the following formula 8.

By this correction, the representative defocus amount of the region RD _i becomes closer to the defocus amount of the block region RP _i , and the difference between the broken line lb and the dotted line ld becomes smaller than that in FIG. 8B as shown in FIG. 8C.

At this time, if there is a large difference in the correction value DC between the block area to be corrected and the block area (adjacent area) adjacent thereto, a large step may occur at the boundary of the block areas. Therefore, when the difference in the correction value DC between the block area to be corrected and the adjacent block area is greater than or equal to the threshold value, the correction unit 145 replaces the correction value of the correction target area with the adjacent area, instead of the correction according to Expression 8. It is also preferable to perform the correction by adding the weighted average of the correction values. In this case, this can suppress a sharp step between the regions. A predetermined value can be adopted as the weighting coefficient in this weighted average.

In step S147, it is determined whether the above process has been performed for all pixels (regions). If there is an unprocessed pixel, the process returns to step S141 and the above process is repeated. By performing the above process on all the pixels, the distance image D _DFD of the DFD method is corrected, and the corrected distance image D3 is obtained. The correction processing image D3 is a distance image having a distance value obtained by correcting the distance value D of each pixel of the distance image D _DFD based on the defocus amount correction value DC as a pixel value.

  In the obtained corrected distance image D3, since the defocus amount correction value differs for each block area, a step of the defocus amount occurs between adjacent blocks as shown in FIG. 8C. Therefore, in step S148, the smoothing processing unit 146 performs an averaging process of, for example, 4×4 pixels along the block region boundary. As a result, the step of the defocus amount caused by the correction becomes inconspicuous. By this processing, the step at the block boundary becomes inconspicuous as shown in FIG. 8D, and a smooth distance image can be generated.

  By performing the above correction processing, it is possible to correct a range image with high resolution using a range image with high stability and obtain a range image with high resolution and high stability. Note that the calculated distance image may be output as it is, or may be output after being converted into an object distance using the focal length and the focus distance on the object side.

  According to the present embodiment, by combining a distance information image having a high stability of distance measurement and a low resolution in the image sensor plane and a distance information image having a low stability but a high resolution, the stability is improved. It is possible to generate a distance information image that is compatible with the resolution.

<Modification 1>
As another form of the image pickup element 11, a configuration as shown in FIGS. 9A to 9C may be adopted. FIG. 9A shows a pixel group 201 and a pixel group 202 having 2 rows×2 columns of pixels. In the pixel group 201, green pixels 201Gv are arranged in a diagonal direction, and red pixels 201Rv and blue pixels 201Bv are arranged in the other diagonal direction. Similarly, in the pixel group 202, green pixels 202Gh are arranged diagonally, and red pixels 202Rh and blue pixels 2020Bh are arranged on the other side. These two types of pixel groups are arranged on the image sensor 11 in a checkered pattern.

FIG. 9B is a schematic cross-sectional view taken along line VV′ of the pixel group 201. Each pixel in the pixel group 201 includes a microlens 201ML, a color filter 201CF, and a photoelectric conversion unit 201A.
201B is provided. FIG. 9C is a schematic sectional view taken along line HH′ in the pixel group 202. Each pixel in the pixel portion 202 includes a microlens 202ML, a color filter 202CF, and photoelectric conversion units 202A and 202B. Further, in FIGS. 9B and 9C, the light receiving surface 210 is an xy surface (+z side surface) on the light incident side of the photoelectric conversion unit.

In the image sensor 11 of this modified example, two photoelectric conversion units are arranged in one pixel (each pixel such as a green pixel), and the microlens 201ML is arranged so that the light receiving surface 210 and the exit pupil have an optically conjugate relationship. , 202 ML power is set. With such an arrangement, the photoelectric conversion units 201A and 201B (and 202A and 202B) can respectively receive the light fluxes that have passed through the regions having different exit pupils. Further, in the pixel group 201 and the pixel group 202, the arrangement of the two photoelectric conversion units is rotated by 90°. With such a configuration, in the pixel group 201 and the pixel group 202, the light flux that has passed through the regions obtained by dividing the exit pupil in different directions is received by the photoelectric conversion unit. This allows the pixel group 201 and the pixel group 202 to detect the subject distance with respect to the subject whose contrast changes in different directions.

  In this modification, the image signals generated from the light received by the two photoelectric conversion units in each of the pixel groups 201 and 202 are acquired as an A image and a B image, respectively. Since the A image and the B image have the same number of resolutions as the pixels of the image sensor, it is possible to obtain a high-definition range image. The calculation of the distance image here is the same as that in step S12. However, when performing distance measurement by the phase difference method using an image with high resolution, there is a high possibility that noise or an image shift amount calculation error due to a periodic subject will occur when calculating the image shift amount. Therefore, in this modification, in addition to the phase difference distance measurement using the A image and the B image as they are, the phase difference distance measurement using the low frequency component image obtained by reducing the A image and the B image is also performed. To do. When the distance is measured with the reduced A image and B image, the image shift amount calculation error due to noise or a periodic subject is reduced.

  For the above reason, in this modification, two range images having different resolutions can be obtained. One has high resolution but low stability, and the other has low resolution but high stability. Therefore, also in this modification, a high-resolution range image can be obtained by correcting the high-resolution range image using the low-resolution range image by the same processing as described above. Specifically, the low-resolution range image is treated as a phase-difference range image in the first embodiment, and the high-resolution range image is treated as a DFD system range image in the first embodiment to perform similar distance correction processing. I'll do it.

  As described above, even when the image pickup device capable of measuring the distance by the phase difference method is used on the entire surface, it is possible to generate a highly accurate distance image with higher stability.

<Modification 2>
As another modification, it is possible to adopt a configuration in which the distance measurement by the TOF method (Time-of-Flight method) which is an active distance measurement method and the distance measurement by the DFD method are performed. FIG. 10 shows the configuration of the image pickup apparatus 2 according to this modification. The imaging device 2 according to the present modification example includes an illumination unit 19, a hot mirror 20, and an imaging element 21 in addition to the configuration of the imaging device 1 according to the first embodiment. The image sensor 11 may be a generally used image sensor in which pixels for distance measurement are not arranged.

  In this configuration, the illumination unit 19 is controlled by the control unit 12 at the time of photographing to illuminate the modulated near infrared light. A hot mirror 20 is installed in the optical path to separate near infrared light and visible light. The illuminated near-infrared light is reflected by the subject and condensed by the imaging optical system, and only the near-infrared light is reflected by the hot mirror 20, and an image is formed on the imaging device 21. On the other hand, visible light passes through the hot mirror 20 and is imaged on the image sensor 11.

  The image sensor 12 performs four exposures at a timing with the phase shifted by 90° with respect to the modulated illumination light, and calculates the phase difference of the reflected light from the subject. It is possible to calculate the distance to the subject by using the calculated phase difference, modulation frequency fm, and speed of light.

  An active distance measuring method such as the TOF method generally has little dependence on a subject and can perform stable distance measurement. On the other hand, the number of pixels for distance measurement is small, and only a distance image having a pixel number of VGA is obtained. Therefore, by correcting the distance image of the DFD method in the distance correction step S14 using the distance image of the TOF method, it is possible to generate a highly accurate distance image with high stability.

The DFD type range image and the TOF type range image obtained by the image pickup apparatus of the present modification are images of different viewpoints. Therefore, it is preferable that the distance measuring unit 14 includes a distance image transforming unit (transforming unit) that performs transforming processing on one of the distance images or both of the distance images so that the two distance images have the same viewpoint. ..

  Here, the TOF method is used as a representative example of the active distance measuring method, but the same processing can be performed using an active distance measuring method such as another laser scanning method (laser scanner method).

(Second embodiment)
Next, a second embodiment of the present invention will be described. In the second embodiment, the reliability information (image) indicating the reliability of the distance information (image) is calculated at the same time when the two distance measurements are executed, and the distance correction is executed using the reliability image. Is different from the first embodiment. The configuration of the image pickup apparatus according to the present embodiment is the same as that of the first embodiment, and therefore the description will be given using the same reference numerals.

  Hereinafter, differences in processing from the first embodiment will be described. 11(A) and 11(B) are diagrams showing the flow of each distance measurement method in the second embodiment. FIG. 11A is a flowchart of the distance measurement S12 by the phase difference method. Compared with the first embodiment (FIG. 5A), a reliability information calculation step S123 is added.

  In the reliability information calculation step S123, the reliability information of the distance measurement indicating how reliable the calculated distance information is is calculated. This reliability information is calculated using the brightness information and contrast information of the image used for calculating the distance information, the number of correlation value peaks at the time of calculating the image shift amount, and the value that changes depending on the subject.

  When the image shift amount is calculated, if the image has blackout or whiteout in the calculation range, the correct image shift amount cannot be calculated, and therefore the brightness value (brightness information) of the subject image is used as an index of reliability information. be able to.

  Further, when the contrast of the image is low in the calculation range, an error occurs in the calculation of the peak value in the correlation calculation when calculating the image shift amount, so that the contrast information can be used as an index of the reliability information.

  Regarding the number of correlation value peaks when calculating the image shift amount, a large number of correlation value peaks occur when a periodic subject exists in the calculation range when calculating the image shift amount. At this time, the smallest (largest) peak is not always the correct image shift position, and therefore, when a large number of correlation value peaks occur, it can be considered that the reliability of distance measurement is low. Therefore, the periodicity information of the subject image can be used as an index of the reliability information.

  FIG. 11B is a flowchart of distance measurement by the DFD method. Compared to the first embodiment (FIG. 8B), the reliability information calculation step S134 is added in the same manner as above. As the index for calculating the reliability information in the DFD method, the brightness value of the subject image, the contrast of the subject image, the periodicity information of the subject, the movement information of the subject at the time of imaging, the distance measurement range information, the distance edge information, the positional deviation information, etc. Is available.

  Since the difference in blur cannot be correctly detected when blackening or whiteout occurs in the area used for calculation, the brightness value (brightness information) of the subject image can be used as an index of reliability information.

  If the contrast is low, the difference in blur cannot be correctly detected, and the influence of noise becomes dominant in the distance measurement. Therefore, the contrast information of the subject image can also be used as an index of the reliability information.

  The distance measurement range information is information indicating a range in which distance measurement can be accurately performed. It can be determined that the reliability of the area outside the range specified by the distance measurement range information is low. For example, when the distance is measured using Expression 4, when the blur becomes large, it becomes difficult to distinguish the difference between the two images, and the correlation becomes high again. Since the area is different from the area having a high correlation near the focus, it needs to be an unreliable area.

  In the DFD method, since it is necessary to change two shooting parameters to acquire two images, a difference in shooting timing causes a positional shift between the two images. By performing the alignment processing, it is possible to compare the blurs in the same area, but it is difficult to match the movement of the subject. Therefore, the motion information such as the sum of squared differences and the sum of absolute differences for each pixel in the calculation area of the two images is calculated, and when the motion information is equal to or larger than the threshold value, it is determined that the positional deviation is large and the reliability is lowered. You can

  As described above, there are many methods of obtaining reliability information by the distance measurement method, and other methods than the method described here are conceivable, but the method of calculating reliability information is not limited here. The reliability information may be binary information indicating whether the calculated distance information is reliable or not, or may be multi-valued and multi-level reliability information.

  Further, the calculated plurality of types of reliability information may be integrated and output as one reliability information. At the time of integration, it is desirable to combine the information used to calculate the reliability information in consideration of the magnitude of the influence on the distance measurement. For example, if one of the black areas or white areas occupies the entire calculation area, the distance cannot be measured by any of the distance measuring methods including the active method, so that the weight of the reliability information is increased and the reliability is calculated by the weighted average. Integrate information. In the phase difference method, the next decrease in contrast and the periodic subject have a great influence as error factors in distance measurement. In the DFD method, the influence of the positional shift due to the difference in the shooting timing has the next largest influence after the blackout and the whiteout, and the influence of the distance measurement range and the contrast continues thereafter.

  Next, the distance correction using the distance information and the reliability information calculated by each distance measuring method will be described. The basic flow of the distance correction processing is the same as the flowchart shown in FIG. 7. However, the operation of the correction value calculation step S145 is different.

In the first embodiment, the average value or the median value is used when calculating the representative defocus amount in the region RD _i selected in the _DFD range image D _DFD . In the present embodiment, the weighted average value of the defocus amounts DH using the reliability information CM _DFD calculated for each pixel of the distance image D _DFD as a weighting coefficient as in Expression 9 is calculated as the representative defocus amount DH _rep .

When the reliability information CM _DFD is binary information, the representative defocus amount DH _rep is calculated using only the reliable defocus amount. When the reliability information CM _DFD is multivalued information, The representative defocus amount DH _rep is calculated by the weighted average. The correction value DC is calculated by substituting the representative defocus amount DH _rep into Expression 7.

As a result, it is possible to suppress the influence of a value that is estimated to have a large distance measurement error in the distance image D _DFD calculated by the DFD method, and it is possible to calculate a more accurate representative defocus amount to obtain a correct correction value. Calculation is possible.

  In the correction using Expression 9, the correction is performed using only the reliability information of the DFD method, but a method of using the reliability information of the phase difference method will be described below.

First, the reliability information CM _PD of the pixel RP _i in the distance image measured by the phase difference method is acquired, and the reliability information of the area RD _i corresponding to the block area RP _i is also obtained from the distance image measured by the DFD method. Get CM _DFD . In the range image D _DFD of the DFD method, the defocus amount and the reliability information CM _DFD are neither constant in the region RD _i . Therefore, the average value of the reliability information of the region RD _i is calculated as the representative reliability information CM _{DFD_REP} (CM _{D
FD_REP} = Average (CM _DFD )). Using the representative reliability information CM _{DFD_REP of the} DFD method and the reliability information CM _PD of the phase difference method, the ratio β of the reliability information of the phase difference method is calculated by Expression 10.

  The ratio β of the reliability information of the phase difference method has a maximum value of 1 and a minimum value of 0, and is a value normalized so that the sum of the ratios of the reliability information of the phase difference method and the DFD method is 1. .

ここで、距離計測方式の安定度αは被写体とは無関係に決定されるもので主に撮像装置側（距離計測装置側）で決定される値であり、信頼度情報の比率βは被写体側に依存して決定される値である。 In the present embodiment, the defocus amount correction value DC is calculated by Expression 11 which is a modification of Expression 7 using the reliability information ratio β and the stability α. However, it is assumed that the stability α satisfies 1≧α>0.5 similarly to the mathematical expression 7.

Here, the stability α of the distance measurement method is determined irrespective of the subject and is mainly a value determined by the imaging device side (distance measuring device side), and the reliability information ratio β is determined by the subject side. It is a value that is determined depending on.

Further, the defocus amount correction value DC can be calculated by Expression 12 using only the ratio β of the reliability information.

  In the present embodiment, the case where the distance measurement result and the reliability information of the phase difference method and the DFD method are used has been described as an example, but other distance measurement methods also use the reliability information according to the characteristics of each method and the same. It is possible to make a correction, and the distance measuring method is not limited.

According to the present embodiment, in addition to the stability of the distance measurement method that does not depend on the subject, by using the reliability information of the distance measurement that depends on the subject, a more accurate correction value of the defocus amount can be calculated. Becomes As a result, it is possible to generate more accurate distance information with high stability and high resolution.

<Modification>
It should be noted that the description of each embodiment is an exemplification for explaining the present invention, and the present invention can be implemented by being appropriately modified or combined without departing from the spirit of the invention. For example, the present invention can be implemented as an imaging device including at least a part of the above processing, or can be implemented as a distance measuring device that does not have an imaging unit. Further, it may be implemented as a distance measuring method, or may be implemented as an image processing program that causes the distance measuring device to execute the distance measuring method. The above processes and means can be freely combined and implemented as long as no technical contradiction occurs.

  In the above description, by correcting the distance value in each pixel of the high-resolution distance image, a high-resolution distance image with high stability is generated. However, the high-resolution range image and the low-resolution range image may be integrated to generate a high-resolution and high-stability range image. For example, the high-resolution range image and the low-resolution range image may be retained and a new range image may be retained. May be generated. That is, the method of generating a high-resolution and highly stable range image is not limited to the process of correcting an existing range image.

  In addition, the element technologies described in the embodiments may be combined arbitrarily.

  When the present invention is implemented as an image processing device that integrates distance information acquired by different distance measuring devices, it is necessary to add alignment processing between different distance information images. By executing the distance correction means of the present invention for the two distance information images that have been aligned, a highly stable and high resolution distance information image can be generated.

<Implementation example>
The distance measuring technique of the present invention described above can be preferably applied to, for example, an image pickup apparatus such as a digital camera or a digital camcorder, or an image processing apparatus or a computer that performs image processing on image data obtained by the image pickup apparatus. Further, the present invention can be applied to various electronic devices (including mobile phones, smartphones, slate type terminals, personal computers) that incorporate such an imaging device or image processing device.

  In the description of the embodiments, the configuration in which the distance measurement function is incorporated in the image pickup apparatus main body has been described, but the distance measurement may be performed by a device other than the image pickup device. For example, the distance measurement function may be incorporated in a computer having an imaging device, and the computer may acquire an image captured by the imaging device to calculate the distance. Further, it is also possible to incorporate a distance measurement function into a computer that can access the network by wire or wirelessly, and the computer can acquire a plurality of images via the network and measure the distance.

  The obtained distance information can be used for various image processing such as image area division, stereoscopic image and depth image generation, and emulation of blur effect.

  It should be noted that the specific implementation on the device may be either software (program) implementation or hardware implementation. For example, by storing a program in a memory of a computer (microcomputer, FPGA, etc.) built in an imaging device or an image processing device and causing the computer to execute the program, various processes for achieving the object of the present invention are realized. You may. It is also preferable to provide a dedicated processor such as an ASIC that realizes all or part of the processing of the present invention by a logic circuit.

  For this purpose, the program may be stored in the computer through a network or from various types of recording media that can serve as the storage device (that is, a computer-readable recording medium that holds data non-temporarily). Provided to. Therefore, the computer (including devices such as CPU and MPU), the method, the program (including program code and program product), and the computer-readable recording medium holding the program non-temporarily are all included in the present invention. It is included in the category of.

14: distance measurement unit 143: representative value determination unit 144: correction value determination unit 147: generation unit

Claims (19)

An image processing device for generating a third distance image based on a second distance image having a lower resolution than the first distance image and the first distance image,
Representative value determining means for selecting a region of the first distance image corresponding to the target pixel of the second distance image, and determining a representative value of distance values of the first distance image in the region.
Correction value determining means for determining a correction value based on a weighted average of the representative value and the distance value of the second distance image at the target pixel;
A generating means for generating a third range image having a distance value corrected as a pixel value based on the distance value of each pixel before Symbol region on the correction value,
An image processing apparatus comprising: The correction value determining means uses the weighting factor based on the ratio of the stability of distance measurement of the first distance image and the stability of distance measurement of the second distance image to the representative value and the target pixel. Obtaining a weighted average with the distance value of the second distance image, the value obtained by subtracting the representative value from the weighted average is determined as the correction value,
The image processing apparatus according to claim 1. The correction value determining means is a weight based on a ratio of a representative reliability of a distance value in each pixel of the first distance image in the area and a reliability of a distance value in the target pixel of the second distance image. A weighted average of the representative value and the distance value of the second distance image in the target pixel is obtained using a coefficient, and the value obtained by subtracting the representative value from the weighted average is determined as the correction value.
The image processing apparatus according to claim 1. The representative value determining means determines a median value or an average value of the distance values of the first distance image in the region as the representative value,
The image processing apparatus according to claim 1. The representative value determining means determines the representative value also using reliability of a distance value in each pixel of the first distance image,
The image processing device according to claim 1. The representative value determination means uses the reliability of the distance value in each pixel of the first distance image as a weighting coefficient, and the weighted average value of the distance values of the first distance image in the region is the representative value. Decide as,
The image processing apparatus according to claim 5. The reliability is determined using at least one of the contrast of the subject image, the brightness value of the subject image, the periodicity information of the subject, the movement information at the time of the subject and imaging, and the distance measurement range information.
The image processing apparatus according to any one of claims 3, 5, and 6. It said generating means generates a third distance image has a value obtained by adding the correction value to the distance value of each pixel before Symbol region distance value as the pixel value,
The image processing apparatus according to claim 1. Said generating means, the distance value of each pixel before Symbol area, adding the weighted average value of the correction value determined for the adjacent region adjacent to the correction value and the area determined for the region Generate a third distance image having the distance value as the pixel value,
The image processing apparatus according to claim 1. The generation means performs a smoothing process on a boundary of an adjacent region of the third distance image corresponding to an adjacent pixel of the second distance image,
The image processing apparatus according to claim 1. The first distance image and the second distance image are acquired from the same imaging surface,
The image processing device according to claim 1. The first distance image is a distance image in which a distance is detected based on a difference in blur from a plurality of images photographed under different photographing conditions,
The image processing apparatus according to claim 1. The second distance image is a distance image whose distance is detected based on the phase difference of a plurality of images with parallax,
The image processing device according to claim 1. The second range image is a range image acquired by the Time of Flight method.
The image processing device according to claim 1. The second distance image is a distance image acquired by a laser scanning method,
The image processing device according to claim 1. In order that the first distance image and the second distance image are images of the same viewpoint, a deforming unit that deforms at least one of the first distance image and the second distance image is further provided.
The image processing device according to claim 1. An image sensor,
Distance image generating means for generating the first distance image and the second distance image from the image captured by the image sensor,
An image processing apparatus according to any one of claims 1 to 16,
An imaging device comprising: An image processing method for generating a third distance image based on a second distance image having a lower resolution than the first distance image and the first distance image,
A representative value determination step of selecting a region of the first distance image corresponding to the target pixel of the second distance image, and determining a representative value of distance values of the first distance image in the region.
A correction value determination step of determining a correction value based on a weighted average of the representative value and the distance value of the second distance image in the target pixel,
A generation step of generating a third distance image having a distance value obtained by correcting the distance value of each pixel in the area based on the correction value,
An image processing method including:   A program that causes a computer to execute each step of the method according to claim 18.

Images