JP2014155071A5 - - Google Patents

Description

The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program, and more particularly to a technique for generating an image focused on an arbitrary subject distance from output data after shooting.

In recent years, in an imaging device such as a digital camera, information on a light field, that is, an intensity distribution of light and a traveling direction are recorded as output data at the time of photographing, and an image focused on an arbitrary subject distance from the output data after recording Techniques for generating are proposed. In such an imaging device, light incident from various directions is separated by forming light beams that have passed through different pupil regions of the imaging lens on each pixel (photoelectric conversion element) of the imaging device via a microlens array. The method of recording is used (Patent Documents 1 and 2).

In order to achieve the above object, an image processing apparatus of the present invention is characterized by having the following configuration. Specifically, the image processing apparatus is a plurality of image signals obtained by capturing the same subject at different viewpoints at the same time, and each pixel is any one of a plurality of predetermined color components. An acquisition means for acquiring an image signal indicating the signal intensity of the color component; and a specifying means for specifying pixels corresponding to the same subject image existing at the same subject distance among the plurality of image signals acquired by the acquisition means; Generating means for generating, for each of the corresponding pixels specified by the specifying means, the signal intensity of the color component that does not exist in the pixel by using the signal intensity of the color component of the other pixels among the corresponding pixels; It is characterized by that.

・ "Reconstructed image"
An image generated from LF data and focused on an arbitrary focal plane position. The images of a plurality (pupil division number) in particular generated by the pixels that have passed through the same pupil area from the LF data, an image of an object existing in the object distance generated by executing positioning to match the corresponding An image obtained by adding (superimposing) pixel values of pixels to be processed.

In S304, the camera control unit 101 selects a luminance image (attention luminance image) from the luminance image of the number of pupil divisions generated in S302, and for each region obtained by dividing the image between image signals with other luminance images. Detect phase difference. That is, in this step, the camera control unit 101 acquires the defocus amount between the target luminance image and the other luminance image for the subject image existing in each region obtained by dividing the target luminance image by the phase difference detection method. When the subject can be regarded as a Lambertian surface that does not have a strong specular reflection component, the image corresponding to the subject in the luminance image does not depend on the divided pupil region through which the luminous flux passes, that is, the same luminance value regardless of the incident direction of the luminous flux. Show. For this reason, the camera control unit 101 can grasp the position where the subject existing at the same subject distance exists in each RAW image.

In the present embodiment, a luminance image is generated by extracting pixels one by one from the pixel group assigned to each microlens 20. The luminance image generated in this way corresponds to the light flux obtained in a so-called stop state that passes through a limited pupil region obtained by dividing the imaging optical system 202 into the number of pixels assigned to the microlens 20. Yes. That is, since the depth of field is deep, phase differences can be detected for subject images existing at various subject distances. However, the luminance image generated for detecting the phase difference of each subject in the implementation of the present invention is not limited to this, and luminance images corresponding to at least two types of pupil regions may be used for phase difference detection.

Each pixel in one horizontal line of the reconstructed image is generated by integrating pixel values of appropriate pixels from these five types of horizontal lines. The pixels used for integration differ according to the subject distance to which the generated reconstructed image corresponds (the distance corresponding to the subject in focus in the reconstructed image).

For example, for the subject existing at the subject distance focused at the time of photographing the LF data, the same coordinate is set in the pixel group corresponding to each microlens 20 regardless of the pupil region of the imaging optical system 202 through which the light beam passes. Imaged. In other words, since the subject is in focus, even if an image is generated for each pupil region, there is no phase difference in the subject image. Therefore, when generating a reconstructed image focused on the subject distance set at the time of shooting, each pixel adds up pixels in the same positional relationship in the pixel group corresponding to the microlens 20, that is, FIG. The pixel groups 502 arranged in the vertical direction are integrated for generating one pixel. That is, the pixel group 502 is a pixel group through which a light beam corresponding to the same portion of the subject image focused at the focal position at the time of imaging passes. In the figure, only the pixel group 502 is shown by using a frame. However, in order to generate one horizontal line of the reconstructed image focused on the focal position set at the time of shooting, other pixels aligned on the horizontal axis are also generated. Similarly, pixel groups arranged in the vertical direction may be integrated.

On the other hand, in the horizontal line generated corresponding to the pixel 21 and the horizontal line generated corresponding to the pixel 25 in FIG. When generating a configuration image, the pixel group 503 shown in FIG. 5 is integrated to generate one pixel. That is, since there is no phase difference at the subject distance focused on the subject , the pixel 504 and the pixel 505 of the pixel group 503 are the same in the reconstructed image focused on the subject distance as illustrated. It corresponds to the luminous flux of the subject image. In the figure, only the pixel group 503 is shown by using a frame. However, in order to generate one horizontal line of the reconstructed image focused on the focus position set at the time of shooting, a similar inclination ( Pixel groups included in a frame having (determined by the phase difference of the image) may be integrated.

It should be noted that the inclination of the frame indicating the pixel group accumulated in the generation of the reconstructed image is whether the position of the focal plane corresponding to the generated reconstructed image is far or small from the focus position set at the time of shooting the LF data The sign varies depending on the type.

For example, as shown in FIGS. 7A and 7B, a case where subjects 701 and 702 having different distances from the digital camera 100 during shooting are included in the angle of view will be considered. Further, it is assumed that RAW-LF data regarding the situation is acquired with a setting for focusing on a subject at an arbitrary distance 703 that is between the subject 701 and the subject 702. At this time, as in the case of generating the reconstructed image, the RAW image for each divided pupil region generated from the RAW-LF data is applied to each pixel by paying attention to a subject that has no phase difference at a specific subject distance. The signal strength of the missing color component can be interpolated.

As described above, the image processing apparatus of the present embodiment can generate LF data that can reconstruct an image that has been favorably color-reproduced. Specifically, the image processing apparatus is a plurality of image signals obtained by imaging the same subject at different positions at the same time, and each pixel is any one of a plurality of predetermined color components. An image signal indicating the signal intensity of the color component is acquired. Then, a pixel corresponding to the same subject image existing at the same subject distance is identified from among the obtained plurality of image signals, and the signal intensity of the color component that does not exist in the pixel is determined for each of the corresponding pixels. It is generated using the signal intensity of the color component of other pixels.

Claims (12)

A plurality of image signals obtained by imaging the same subject from different viewpoints at the same time, each image signal indicating the signal intensity of any one of a plurality of predetermined color components Obtaining means for obtaining
A specifying unit for specifying pixels corresponding to the same subject image existing at the same subject distance among the plurality of image signals acquired by the acquiring unit;
Generating means for generating, for each of the corresponding pixels specified by the specifying means, a signal intensity of a color component that does not exist in the pixel using a signal intensity of the color component of another pixel of the corresponding pixels; And an image processing apparatus.   The image processing apparatus according to claim 1, wherein the generation unit uses the signal strength of the color component of the other pixel as the signal strength of the color component that does not exist in the pixel.   The image processing apparatus according to claim 1, wherein the generation unit interpolates a signal intensity of a color component that does not exist in the pixel using a signal intensity of the color component of the other pixel.   The generation unit is configured to determine, for a pixel included in one of the corresponding pixels, the pixel specified by the specifying unit from the image signal photographed in the vicinity of the photographing position of the one image signal. The image processing apparatus according to claim 3, wherein interpolation is performed using signal strength. 5. The specifying unit specifies pixels corresponding to the same subject image existing at the same subject distance by detecting an image phase difference between the plurality of image signals. The image processing apparatus according to any one of the above.   The plurality of image signals are original image signals imaged by an imaging device, and each of the pixels corresponds to a light flux having a different combination of a pupil region and an incident direction that have passed through the imaging optical system of the imaging device, The pixel image is obtained by combining pixels corresponding to a light beam that has passed through the same pupil region from an original image signal indicating the signal intensity of any one of a plurality of color components. The image processing apparatus according to any one of 1 to 5.   The image processing apparatus according to claim 1, further comprising a recording unit that records an image signal in which a signal intensity of a color component that does not exist in the pixel is generated by the generation unit. Imaging means for imaging the same subject at different viewpoints at the same time, and outputting a plurality of image signals each of which indicates the signal intensity of any one of a plurality of color components,
A specifying unit for specifying pixels corresponding to the same subject image existing at the same subject distance among the plurality of image signals output by the imaging unit;
Generating means for generating, for each of the corresponding pixels specified by the specifying means, a signal intensity of a color component that does not exist in the pixel using a signal intensity of the color component of another pixel of the corresponding pixels; An imaging apparatus comprising: The acquisition means of the image processing apparatus is a plurality of image signals obtained by imaging the same subject from different viewpoints at the same time, and each pixel has a color selected from a plurality of predetermined color components An acquisition step of acquiring an image signal indicating the signal intensity of the component;
A specifying step in which the specifying unit of the image processing device specifies pixels corresponding to the same subject image existing at the same subject distance among the plurality of image signals acquired in the acquiring step;
For each of the corresponding pixels specified in the specifying step, the generation unit of the image processing device uses the signal intensity of the color component of the other pixels among the corresponding pixels to determine the signal intensity of the color component that does not exist in the pixel. And a generation step of generating using the intensity. A control method for an image processing apparatus. The imaging means of the imaging device images the same subject at different viewpoints at the same time, and outputs a plurality of image signals each of which indicates the signal intensity of one of the plurality of color components. An imaging process to
A specifying step of specifying a pixel corresponding to the same subject image existing at the same subject distance among the plurality of image signals output in the imaging step;
For each of the corresponding pixels specified in the specifying step, the generation unit of the imaging device determines the signal intensity of the color component that does not exist in the pixel and the signal intensity of the color component of the other pixels of the corresponding pixels. And a generation step of generating using the imaging method.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 7.   The program for making a computer perform each process of the control method of the imaging device of Claim 10.