JP2013044844A - Image processing device and image processing method - Google Patents

Abstract

The present invention relates to subject distance estimation accuracy when measuring a subject distance using a plurality of images photographed in a plurality of focus states and a reference image having a focus range in all of the plurality of focus states. The problem is to lower.
In order to solve the above-described problem, the present invention includes a subject distance measurement image selection unit 104 that selects which of the n images is used to measure the distance to the subject. The subject distance measurement image selection unit 104 selects an image in a focus state that makes an appropriate pair with the reference image, and measures the subject distance. This provides an image processing apparatus capable of estimating the subject distance with high accuracy.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus and an image processing method for measuring a subject distance from a plurality of photographed images photographed in a plurality of focus states.

  If the depth of the three-dimensional scene, that is, the subject distance can be measured simultaneously with the image taken by the camera, various applications in image display / conversion / recognition are possible. For example, if a single photographed image and the subject distance to this image are obtained, an image viewed from different viewpoints can be generated in a pseudo manner based on the principle of triangulation, and it corresponds to stereo or multiple viewpoints. A three-dimensional image can be generated. Further, if the image is divided based on the subject distance, it is possible to cut out only a subject existing at a specific distance or adjust the image quality.

  The main methods for measuring the subject distance in a non-contact manner can be broadly divided into the following two methods: an active method and a passive method. The first active method is to irradiate infrared rays, ultrasonic waves, lasers, etc., and measure the subject distance based on the time until the reflected wave returns and the angle of the reflected wave. In general, when this method is used, it is possible to measure with high accuracy when the subject distance is short, but there is a problem that an active irradiation / light receiving device that is not necessary for a normal camera is required. In addition, when the subject is far away, if the output level of the irradiation device is low, the irradiation light that reaches the subject becomes weak and the measurement accuracy of the subject distance decreases, while on the other hand, if the output level of the irradiation device is high In addition to the problem that the power consumption increases, there is also a problem that the environment in which the laser can be used is limited because there is a safety problem when using a laser. On the other hand, the second method is to measure the subject distance using only the image taken by the camera.

  There are many passive methods, and one of them is a method called Depth from Defocus (hereinafter referred to as DFD) that uses a correlation value of the amount of blur generated in a captured image. In general, the amount of blur that occurs in a captured image is uniquely determined for each camera according to the relationship between the focus state at the time of shooting and the subject distance. In the DFD, the relationship between the subject distance and the correlation value of the blur amount generated in the photographed image is measured by photographing a subject at a known subject distance in a plurality of focus states in advance using this characteristic. Thus, if shooting is performed in a plurality of focus states in actual shooting, the subject distance can be measured by calculating the correlation value of the blur amount between images (see Patent Document 1).

  In the method disclosed in Patent Document 1, two images taken in different focus states are used. Since these two images are taken in different focus states, there is a property that the blur amount of the subject existing at a specific distance differs between the two images, and the subject distance is measured using this property. . As an example, let us consider a case where two subjects are photographed at a short distance and a long distance focus respectively. In this case, the subject is clearly visible in one image shot with the focus on the short distance side, while the subject is blurred in the other image shot with the focus on the long distance side. It is reflected in the state. The subject distance is measured by using a correlation value between the difference in how the subject is reflected, the subject distance measured in advance, and the blur amount generated in the captured image.

Special table 2009-505104

Flexible Depth of Field Photography, H.C. Nagahara, S .; Kuthirmal, C.I. Zhou, S .; K. Nayer, European Conference on Computer Vision (ECCV), Oct, 2008

  In the method disclosed in Patent Document 1, the depth of field of the captured image is narrow unless the aperture is greatly reduced. Since the distance distance in which an image is not blurred is determined by the depth of field, the depth of field is an important factor that determines the distance that can be measured by the DFD that uses the difference in blur amount for each distance. Therefore, in order to obtain a long subject distance that can be measured by DFD, that is, to increase the depth of field, it is necessary to capture a large number of images. This offsets the advantage of DFD that distance measurement is possible with a small number of images. In addition, it is possible to obtain a reference image from a small number of images by greatly reducing the aperture, but since the amount of incident light decreases, to compensate for this, either increase the sensitivity of the image sensor or increase the exposure time. Is necessary. However, the former causes an increase in noise of the photographed image, and the latter causes blurring of the subject, which disturbs the spectral component of the subject, and as a result, the measurement accuracy of the subject distance decreases. For this reason, when measuring a wide range of subject distances from two or more images, it is necessary to use a large number of images.

  However, when distance measurement is performed using a plurality of images in the conventional method, it is necessary to use a plurality of images with different blur amounts at the measured distance, but the subject distance is unknown in advance and the subject exists. There is a problem that distance measurement is difficult because an optimal image to be selected cannot be selected.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus and an image processing method capable of stably measuring a distance from a plurality of photographed images having different blurs.

  In order to solve the above problems, the present invention is an image processing apparatus for measuring a subject distance from a plurality of captured images captured in a plurality of focus states, wherein n images (however, n ≧ 2) having an in-focus range in all of a plurality of focus states captured by the imaging unit, an in-focus range control unit that changes the in-focus position and depth of field of the imaging unit, and the imaging unit A reference image generation unit that generates a first reference image, a subject distance measurement image selection unit that selects a distance measurement image to be used for measuring a distance to a subject among n images in the previous period, and the distance measurement And a distance measuring unit that measures the distance to the subject from the degree of blur of the image for use and the first reference image.

  According to the present invention, it is possible to obtain an image with a wider focus range than usual without reducing the aperture by the focus range control unit, and thus it is possible to measure the subject distance from three images. In addition, before measuring the detailed subject distance, an image suitable for subject measurement is selected for each pixel of the image, and the subject distance is determined using two images: an image suitable for subject measurement and an image with a wide focus range. Therefore, it is possible to measure the subject distance with high accuracy even between images in different focus states.

1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment of the present invention. Schematic diagram showing the state of light collection when the focus is changed The flowchart which shows the flow of the process of Embodiment 1 of this invention. The figure which shows the distance measurement result of the object which exists in short distance FIG. 2 is a block diagram showing a configuration of an image processing apparatus according to a second embodiment of the present invention. The flowchart which shows the flow of a process of Embodiment 2 of this invention. Diagram showing processing of block matching method

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
FIG. 1 is a diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention. The image processing apparatus 100 includes an imaging unit 101, a focusing range control unit 102, a reference image generation unit 103, a subject distance measurement image selection unit 104, and a subject distance measurement unit 105.

  The imaging unit 101 includes a lens unit in which a lens for collecting light rays is incorporated and an imaging element such as a CCD or a CMOS, and has a function of capturing an image of a subject and outputting the image.

  The focusing range control unit 102 has a function of controlling the lens unit included in the imaging unit 101 and controlling the focusing position and the depth of field. Specifically, the control is performed by operating an autofocus mechanism incorporated in the lens unit with a specific pattern or switching a specific optical element.

  The reference image generation unit 103 generates a reference image in which a state in which there is no blur due to the lens is estimated from one or a plurality of images having different in-focus positions and depths of field by the operation of the in-focus range control unit 102.

  The subject distance measurement image selection unit 104 selects a plurality of images having different in-focus positions and depths of field, and combinations of images suitable for subject distance measurement for each pixel or area of the image from the reference image. .

  The subject distance measurement unit 105 measures the subject distance based on the DFD technique using the image selected by the subject distance measurement image selection unit 104.

  The data flow between the blocks in FIG. 1 showing the configuration of the first embodiment will be described below.

  The imaging unit 101 captures the first image, the third image, and the second image in which the focus position is changed based on the focus range control information output from the focus range control unit 102.

  The focus range control unit 102 outputs focus range control information such as a start position and an end position of the focus position range when shooting the second image to the imaging unit 101 to control the focus range.

  The reference image generation unit 103 generates a reference image in which no lens blur is estimated from the second image captured by the imaging unit 101, and outputs the reference image to the subject distance measurement image selection unit 104.

  The subject distance measurement image selection unit 104 performs subject distance measurement for each pixel or region of the image from the first and third images captured by the imaging unit 101 and the reference image generated by the reference image generation unit 103. A suitable image combination is selected to generate image selection information. Then, in addition to the first image, the image, the third image, and the reference image, the generated image selection information is output to the subject distance measurement unit 105.

  The subject distance measurement unit 105 measures the subject distance based on the DFD method using the image selected by the image selection information generated by the subject distance measurement image selection unit 104.

  Next, a method for extending the depth of field of a captured image will be described. A general depth of field is defined as follows. First, the hyperfocal distance (Hyperfocal Distance) will be described. The hyperfocal distance is a distance that is determined to be in focus from the distance to infinity when the focus is adjusted to that distance. The focal length of the lens is f, and the F number of the lens is F, where c is the size of the permissible circle of confusion representing the smallest detectable blur, the hyperfocal distance h is approximated by the following equation (1).

  The depth of field when focusing on the distance s is expressed by the following formula 2, where Dn is the depth of field ahead of s and Df is the depth of field behind.

  From the above formula, when the focal length is fixed, the width of the depth of field can be changed only by the diaphragm.

  On the other hand, various methods for expanding the depth of field without reducing the aperture have been proposed, which is referred to as extended depth of field (hereinafter referred to as EDOF). Hereinafter, a specific method of EDOF will be described.

  The simplest EDOF method is a method of taking a plurality of images while gradually shifting the in-focus position, and extracting and synthesizing the in-focus portions from these images.

  On the other hand, Non-Patent Document 1 discloses a technique for realizing the same effect as combining a large number of images by changing the focus during exposure.

  FIG. 2 is a diagram schematically showing the state of light collection when the focus is changed. As shown in FIG. 2, the position of the image sensor that focuses on the subject distance a1 is b1, and the position of the image sensor that focuses on the subject position a2 is b2. According to the lens formula, the position of the image sensor that focuses on the distance between a1 and a2 is always between b1 and b2. Accordingly, when the image sensor moves from b1 to b2 during exposure, the blur gradually increases from the point where it first focused on a1, and these blurs are integrated and overlapped in the image. On the other hand, the focus is gradually increased from the point where the blur is large for a2, and these blurs are superimposed on the image.

  This will be described in more detail using mathematical expressions. It is assumed that a point spread function (Point Spread Function, hereinafter referred to as PSF) representing the blur state of the optical system has a shape represented by the following Equation 3.

  Where f is the focal length of the optical system, a is the diameter of the aperture of the optical system, u is the subject distance, v is the image plane position determined by the lens formula 1 / f = 1 / u + 1 / v, and Δv is from v The amount of movement of the image plane, R is the distance from the blur center, and g is a constant.

  Since Equation 3 is a function of the image plane movement amount Δv, when Δv changes from time 0 to T according to a certain function V (t), the finally obtained PSF can be defined as Equation 4 below.

  Here, if V (t) is expressed as constant velocity motion, that is, V (t) = v0 + st, Equation 4 can be solved as Equation 5 below.

  Here, erfc (x) is a complementary error function. If u >> f, that is, the subject distance is sufficiently larger than the focal length of the optical system, Equation 5 can be considered to be a constant value regardless of u. That is, it means that an image having a constant blur can be obtained regardless of the subject distance. By changing the start point v + V (0) and the end point v + V (T) of the image plane position, the range of the subject distance where the blur is constant can be changed.

  Next, a DFD method using a reference image will be described. Between the reference image R and the captured image I, the relationship shown by the following formula 6 is established.

  Here, h represents the PSF at the position (x, y), and d (x, y) represents the subject distance at the position (x, y). Also, * in the formula represents a convolution operation. For example, as shown in Equation 3, since the PSF varies depending on the subject distance, when there are subjects at a plurality of different distances, an image in which a different PSF is convoluted for each image position is obtained as a captured image.

  Here, PSFs corresponding to the subject distances d1, d2,..., Dn are assumed to be h (x, y, d1), h (x, y, d2), ..., h (x, y, dn). For example, when the subject existing in the reference image R (x, y) is at the distance d1, the captured image I (x, y) is obtained by convolution of R (x, y) with H (x, y, d1). Equally, when a PSF corresponding to another object distance is convoluted with R (x, y), a difference from the observed image occurs. Therefore, D (x, y) is obtained by comparing the difference between the image obtained by convolving each PSF with R (x, y) and the photographed image in order, and examining the distance corresponding to the PSF that minimizes the difference. I can do it. Specifically, it is obtained by the following equation (7).

  Actually, in order to reduce the influence of noise contained in the captured image I, the image is divided into blocks, the sum of errors in the block is obtained, and the distance that minimizes the error is set as the distance of the entire block. By performing this, it is possible to perform distance measurement more stably. Note that the distance measurement may be performed by performing Fourier transform on Equations 6 and 7 and performing comparison in the frequency domain. In the frequency domain, the convolution operations in Equations 6 and 7 are converted into multiplications, and distance measurement can be performed at higher speed.

  In the present invention, the subject distance is measured by applying these EDOF and DFD. Flow of processing for measuring subject distance from a total of three images, two images shot in a different focus state from a reference image with an extended depth of field by the image processing apparatus according to the first embodiment of the present invention. Will be described with reference to FIG. In the following description, a method for changing the focus during the exposure time is used as a method for extending the depth of field.

  First, the imaging unit 101 captures a first image and a third image having different focus ranges (steps S101 and S105). For the sake of explanation, the in-focus position of the first image is assumed to be in front of the in-focus position of the third image. Furthermore, in addition to the first image and the third image, a second image that is captured by changing the in-focus position by the in-focus position control unit 102 is captured. The second image is taken while changing the in-focus position from the start of changing the in-focus position (step S102) to the end of the in-focus position change (step S104) (step S103). Specifically, after shooting the first image, the focus range control unit 2 moves the focus position at a constant speed, and moves the focus position by a predetermined interval to capture the second image. The shooting position of each image will be described on FIG. 2. The first image is shot at the position a1, the second image is shot while changing the focus position from a1 to a2, and the third image is shot at the position a2. . The second image photographed in this way has a uniform blur within the focus position change range. The blur that is uniform with respect to the subject distance contributes to the generation of the reference image, and the blur according to the subject distance contributes to the distance measurement by the DFD. Here, it is described that the focus position is not changed in the first and third images. However, the present invention is not limited to this case, and the focus position may be changed during shooting of the first and third images. Further, the moving speed of the focus during the exposure of the second image may not be constant.

  Next, a reference image is generated from the second image by the reference image generation unit 103 (step S106). The reference image is obtained by deconvolution of the PSF with respect to the photographed image from Equation 6. Originally, the subject distance D (x, y) needs to be known in order to correctly obtain the reference image. However, since the depth of field is extended in the second image, the subject distance in the extended range is determined. PSF is constant. By performing a deconvolution operation with one type of PSF corresponding to uniform blur on the second image, a reference image focused on the entire range from a1 to a2 is obtained. As a deconvolution algorithm, a known method such as a Wiener filter can be used.

  As described above, it is possible to measure a wide range of distances by using a reference image having a wide depth of field in addition to the first and third images.

  Here, when the subject distance is measured from the three images of the first and third images and the reference image, Equation 7 is expanded to 3 as Equation 8.

  Thereby, DFD using three or more images becomes possible. However, if the expansion is performed as shown in Equation 8, the results of the images I1 and I3 captured at different in-focus positions are added, which is an impediment to obtaining an accurate subject distance. .

  Therefore, the subject distance measurement image selection unit 104 according to the present embodiment determines which of the first image and the third image is appropriate when measuring the subject distance using the reference image of the second image. Select (step S107). In order to obtain a highly accurate subject distance, the subject distance is obtained after selecting which one of the first image and the third image is appropriate for each pixel or region in advance.

  The subject distance measurement image selection unit 104 generates an image representing the blur of the reference image in each focus state by convolving the blur parameter of each focus state that has been measured in advance with the reference image. Then, a difference value between a plurality of images taken by the imaging unit and an image representing blurring of the reference image in each focus state is calculated. Further, the minimum value of the difference value is obtained for each pixel or specific area, one image having the minimum value is selected for each pixel or specific area, and distance measurement is performed using the reference image and the selected single image. I do. Specific examples will be described below.

  First, the reference image and the first image I1, and the reference image and the third image I3, respectively, as shown in Equation 7, an image and a captured image obtained by convolving the reference image R (x, y) with PSF: h for each distance d The difference values of I1 and I3 are calculated in order. The calculation results are stored in D1 and D3, respectively, as shown in Equation 9.

  FIG. 4 illustrates an example of the difference values of D1 and D3 at the coordinates x and y positions of a subject existing at a short distance, with the horizontal axis being d. In the case of a subject existing at a short distance, an image obtained by convolving the reference image R with a distance h that matches the subject distance matches the captured image. For this reason, I1 is focused on the near side of I3, D1 takes the minimum value on the near side as shown in FIG. 4A, and the gradient change of the difference value is large. On the other hand, since I3 is in focus behind I1, and the subject existing at a short distance is blurred at the stage of shooting, D3 is a specific subject distance as shown in FIG. 4B. There is no such thing as a minimum value, and the gradient change of the difference value becomes small. By measuring this property, it is possible to select an image suitable for subject measurement in each pixel or region. Although the selection from two images has been described, the above properties may be measured and the most suitable image selected in the same manner when selecting from a plurality of images. Further, when the most suitable image among the plurality of images does not exhibit the above-described properties, the subject distance may be measured by performing expansion as shown in Equation 8.

  Finally, the subject distance measuring unit 105 uses the reference image generated in step S106 and Im (x, y) generated in step S107 using the first and third images, according to the equation 7, the distance map D (x , Y) is calculated (step S108). Specifically, in step S107, which image is to be used in units of pixels or in units of regions, for example, 0 is the first image, and if it is, the image selection information indicating the third image is Im (x, y). The distance map D (x, y) is calculated by changing which image is used for each pixel or each region as shown in Equation 10. As a result, the subject distance can be measured more suitably.

  As described above, in this embodiment, distance measurement can be performed with high accuracy from a small number of images by using a reference image with an extended depth of field. Furthermore, before measuring the detailed subject distance, an image suitable for subject measurement is selected for each pixel of the image, and the subject distance is determined using two images, an image suitable for subject measurement and an image with a wide focusing range. Since measurement is possible, the subject distance can be measured with high accuracy even between images in different focus states.

  Note that a change in focus during exposure can be realized by diverting a standard autofocus mechanism, so that no special mechanism is required.

  Note that the imaging unit in the present embodiment may include n imaging elements arranged so as to have different optical path lengths and a beam splitter that divides a light beam into each of the n imaging elements. Further, the focus range control unit in the present embodiment may extend the depth of field by changing the focus position at a substantially constant speed during exposure.

  Note that the imaging unit in the present embodiment includes n imaging elements, optical axis changing means for changing the optical axis direction of the light beam, and driving for directing the direction of the optical axis toward one of the imaging elements. Means may be provided. In addition, the focus range control unit in the present embodiment may change the focus position at a substantially constant speed during exposure, and operate the driving unit with a predetermined pattern to change the optical axis direction.

(Embodiment 2)
Hereinafter, the second embodiment of the present invention will be described by omitting the same contents as those of the first embodiment.

  The image processing apparatus and the image processing method in the present embodiment will be described with reference to FIG. Hereinafter, description of the same operation as that of the first embodiment will be omitted, and only different operation will be described.

  The image processing apparatus 200 according to the present embodiment includes an imaging unit 201, a focusing range control unit 202, a reference image generation unit 203, a subject distance measurement image selection unit 204, a subject distance measurement unit 205, and a motion amount estimation unit 206. . In the present embodiment, a motion amount estimation unit 206 is added as a new configuration to the configuration of the first embodiment.

  The motion amount estimation unit 206 estimates the amount of motion that occurs between the three images of the first image and the third image captured by the imaging unit 201 and the reference image generated from the second image by the reference image generation unit 203. Do.

  The difference between the data flow between the blocks in FIG. 5 showing the configuration of the second embodiment and the configuration of the first embodiment will be described below.

  The imaging unit 201 outputs the first image and the third image to the motion estimation unit 206 and the subject distance measurement image selection unit 204.

  The reference image generation unit 203 generates a reference image from the second image output by the imaging unit 201 and outputs the reference image to the motion estimation unit 206 and the subject distance measurement image selection unit 204.

  The motion amount estimation unit 206 estimates the motion amount and outputs it to the subject distance measurement image selection unit 204 as motion amount evaluation value information.

  The subject distance measurement image selection unit 204 includes the first third image output from the imaging unit 201, the reference image output from the reference image generation unit 203, and the motion amount evaluation value information output from the motion amount estimation unit 206. Then, image selection information is generated. Then, the image selection information, the motion amount evaluation value information, the reference image, and the first and third images are output to the subject distance measurement unit 205.

  Here, the operation of the motion amount estimation unit 206 will be described. As in the image processing apparatus 200, in the method of taking a plurality of images in order by controlling the focus adjustment mechanism, the shooting timing of the images is different. If the camera moves during shooting, the position of the subject is shifted between images. In the measurement of the subject distance using the DFD, it is necessary to compare the correlation value of the blur amount with respect to the same pixel between a plurality of images taken in a plurality of focus states. Cannot be accurately performed, and an error occurs in the measured subject distance. As described above, since the measurement accuracy of the subject distance is lowered due to the displacement of the subject, it is necessary to correct the displacement of the subject that has occurred between the images. The motion amount estimation unit 206 estimates the subject position shift.

  Flow of processing for measuring subject distance from a total of three images, two images photographed in a different focus state from a reference image with an extended depth of field by the image processing apparatus according to the second embodiment of the present invention. Will be described with reference to FIG. Note that, as a method for extending the depth of field, a method of changing the focus during the exposure time will be described. Hereinafter, among the steps of FIG. 6, the same steps as those of FIG. 2 are denoted by the same reference numerals, and the description thereof will be omitted.

  In step S109, the motion amount estimation unit 206 performs motion estimation in order to correct the positional deviation of the subject. A block matching method can be used to estimate the amount of motion. Here, processing of the block matching method will be described with reference to FIG.

  The block matching method is a method for estimating the amount of motion between images for each block region of a specific size, and has the highest correlation with the image of the block region set in one image (hereinafter referred to as a search source image). The amount of motion is estimated by specifying a region from the other image (hereinafter referred to as a search destination image). As shown in FIG. 7, first, an attention block area composed of a plurality of pixels is set in the search source image. The size of the block area can be arbitrarily set such as 8 × 8 pixels or 16 × 16 pixels. Next, a search area is set in the search destination image. This search area indicates a range in which an area having the highest correlation with the target block area in the search source image is searched, and is preferably set at a position close to the position of the target block area in the search source image. Next, a block area having the same size as the target block area in the search source image is cut out from the search area of the search destination image, and evaluation values rx, y representing the correlation of the images are calculated based on Equation 11.

  Here, x and y are coordinate positions obtained by cutting out the block area from the search destination image, (x, y) is a relative coordinate position in the block area, and f (x, y) is set in the search source image. The pixel value of the target block area, gx, y (x, y) represents the pixel value of the block area cut out from the search destination image. Evaluation values rx and y based on Equation 11 are calculated while shifting the coordinate positions x and y for cutting out the block area from the search area of the search destination image. It means that the higher the evaluation value rx, y, the more the coordinate position x, y deviates from the accurate displacement amount. A coordinate position where the evaluation values rx, y are the smallest is obtained from x, y in the block area. The relative displacement between this coordinate position and the coordinate position of the block area of interest in the search source image represents the amount of motion between images. By performing this process on all the block areas in the search source image, the amount of motion can be estimated for the entire image. The amount of motion estimated between the reference images of the first image and the second image is stored in Sx1 (x, y) and Sy1 (x, y), and the evaluation value based on Equation 11 is stored in r1x and y. The motion amount estimated between the reference images of the third image and the second image is stored in Sx3 (x, y) and Sy3 (x, y), and stored in the evaluation value r3x, y based on Equation 11. Although the description has been made using the block matching method as the motion amount estimation method, the method is not limited to the block matching method, and other methods such as optical flow estimation may be used.

  In step S109, the position shift of the subject generated between the images using the estimated motion amounts Sx1 (x, y), Sy1 (x, y), Sx3 (x, y), and Sy3 (x, y) is detected. After correcting, subject distance measurement is performed. Equation 10 is changed to Equation 12 using Sx1 (x, y), Sy1 (x, y), Sx3 (x, y), and Sy3 (x, y), and subject distance measurement is performed using Equation 12. I do.

  Note that Im (x, y) obtained in step S107 may be calculated using Equation 9 as described above, or calculated using the evaluation values r1x, y, r3x, y calculated in step S109. You may do it.

  In the case of calculating using the evaluation values r1x, y, r3x, y calculated by the motion amount estimation unit, the image selection information Im (x, y) is used by utilizing the difference in blur between the first image and the third image. ). As already described, since the in-focus position of the first image is set in front of the in-focus position of the third image, the subject positioned at the near distance in the first image is the third image. Compared to this, it is clearly visible. In addition, the subject located at a near distance appears as clearly as the first image on the reference image generated from the second image. On the other hand, in the third image, the subject located at a near distance appears blurred. Therefore, when r1x, y and r3x, y in step S109 are calculated for the pixel on the subject located at the near distance, the reference image generated from the second image and the third image have different blurring evaluation values. r3x, y has a higher value than r1x, y, and Im (x, y) can be determined by measuring the difference between r1x, y and r3x, y. Similarly, since r1x, y has a higher value than r3x, y in a subject pixel located at a distant distance, Im (x, y) is obtained by measuring the difference between r1x, y and r3x, y. y) can be determined.

  For example, if the evaluation value at a certain position x, y on the image has a relationship r3x, y−r1x, y> sh when a certain threshold value sh is set, the subject existing at x, y is in front. Since Im (x, y) = 0, a value indicating the first image is recorded. On the other hand, if r1x, y−r3x, y> sh, the subject existing in x and y is considered to be on the back side, and therefore Im (x, y) = 1 is pointed to the third image. Record the value. When the difference between r1x, y and r3x, y is less than sh, the object distance may be measured by performing expansion as shown in Equation 8.

  Further, the calculation may be performed using both Equation 9 and the evaluation values r1x, y, r3x, y calculated in step S109. When both are used, the fact that the subject at a certain position x, y on the image is present on the front side or the back side is indicated by Equation 9 and the evaluation values r1x, y, r3x, y calculated in step S109 are both indicated. In this case, it may be recorded as Im (x, y) = 0 or Im (x, y) = 1.

  In this way, by using the motion amount estimation unit 206 to estimate the subject's positional deviation that occurred between the images, the subject's positional deviation is suppressed in images taken in a plurality of focus states, and the subject distance is measured. Things will be possible.

  The image processing apparatus and the image processing method according to the present invention can measure the subject distance with high accuracy from a plurality of photographed images photographed in a plurality of focus states even when the movement of the subject or a change in the photographing direction occurs. To. These configurations are useful, for example, in fields such as consumer or commercial imaging devices (digital still cameras, video cameras).

101, 201 Imaging unit 102, 202 Focusing range control unit 103, 203 Reference image generation unit 104, 204 Subject distance measurement image selection unit 105, 205 Subject distance measurement unit 206 Motion amount estimation unit

Claims (9)

An image processing apparatus for measuring a subject distance from a plurality of captured images captured in a plurality of focus states,
An imaging unit that captures n images (where n ≧ 2) in a plurality of focus states;
A focusing range control unit for changing a focusing position and a depth of field of the imaging unit;
A reference image generation unit that generates a first reference image having a focusing range in all of a plurality of focus states photographed by the imaging unit;
A subject distance measurement image selection unit that selects a distance measurement image used to measure the distance to the subject among the n images in the previous period;
A distance measuring unit that measures the distance to the subject from the degree of blur of the distance measurement image and the first reference image;
An image processing apparatus. The image processing apparatus includes:
Furthermore, a motion amount estimation unit that estimates a motion amount between the distance measurement image and the first reference image is provided.
The image processing apparatus according to claim 1. The distance is corrected in such a manner that the amount of motion generated between the distance measurement image and the first reference image is canceled out, and the pixel positions in the distance measurement image and the first reference image are matched. Measure the distance to the subject from the degree of blur of the measurement image and the first reference image,
The image processing apparatus according to claim 2. The subject distance measurement image selection unit
Generating a second reference image representing the blur of the first reference image in each focus state by convolving the blur parameter of each focus state measured in advance with the first reference image;
Calculating a difference value between the n images captured by the imaging unit and the second reference image;
Find the minimum difference value for each pixel or specific area,
Select one image with the minimum value for each pixel or specific area,
Distance measurement is performed using the first reference image and the selected one image.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The subject distance measurement image selection unit
Selecting an image to be used for subject measurement from a group of n blurred images based on an evaluation value when obtaining the amount of movement;
The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus. The subject distance measurement image selection unit
Selecting an image for measuring a subject distance in a specific region or pixel unit at the time of image selection from n images captured by the imaging unit;
The image processing apparatus according to claim 4, wherein the image processing apparatus is an image processing apparatus. The imaging unit includes n imaging elements arranged to have different optical path lengths, and a beam splitter that divides a light beam into each of the n imaging elements,
The focus range control unit extends the depth of field by changing the focus position at a substantially constant speed during exposure.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The imaging unit includes n imaging elements, an optical axis changing unit that changes an optical axis direction of a light beam, and a driving unit that directs the direction of the optical axis toward one of the imaging elements.
The focusing range control unit changes a focus position at a substantially constant speed during exposure, operates the driving unit with a predetermined pattern, and changes an optical axis direction.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Measuring subject distance from a plurality of captured images captured in a plurality of focus states;
Photographing n images (where n ≧ 2) in a plurality of focus states;
Changing the focus position and depth of field;
Generating a first reference image having a focus range in all of the plurality of focus states;
Selecting a distance measurement image to be used for measuring the distance to the subject from the n images in the previous period;
Measuring the distance to the subject from the degree of blur of the distance measurement image and the first reference image,
Image processing method.

Images