JP2014155071A - Image processing system, imaging apparatus, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate LF data that are able to reconfigure a suitably reproduced image.SOLUTION: An image processing system acquires a plurality of image signals obtained by imaging the same subject at the same time at different positions, and acquires an image signal indicating the signal intensity of one of predetermined color components whose pixels have been predetermined. Among the acquired image signals, a pixel corresponding to the same subject image present at the same focal point is specified. For each of the corresponding pixels, the signal intensity of a color component that is not present in the pixel is generated using the signal intensity of the color component of other pixel among the corresponding pixels.

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program, and more particularly to a technique for generating an image focused on an arbitrary focal length from output data after shooting.

  In recent years, in an imaging apparatus such as a digital camera, information on a light field, that is, an intensity distribution of light and a traveling direction are recorded as output data at the time of photographing, and an image focused on an arbitrary focal length from the output data after recording. Techniques for generating are proposed. In such an imaging device, light incident from various directions is separated by forming light beams that have passed through different pupil regions of the imaging lens on each pixel (photoelectric conversion element) of the imaging device via a microlens array. The method of recording is used (Patent Documents 1 and 2).

  On the other hand, in order to acquire a full-color image signal for one pixel in the image sensor, for example, a photoelectric conversion element to which filters of each color of RGB are applied needs to be provided in one pixel. However, when a plurality of photoelectric conversion elements are provided in one pixel, the number of photoelectric conversion elements used increases. For this reason, in order to reduce the number of photoelectric conversion elements, filters of different colors are generally used for adjacent pixels of the imaging element, such as a so-called Bayer array. A color component that has not been acquired for one pixel is generated by interpolation from the output signals of surrounding pixels, and a method of generating full-color image signals for all pixels (demosaicing) is used.

JP 2007-004471 A JP 2009-124213 A

  Also in Patent Documents 1 and 2 described above, a demosaicing process is performed for a final output image. Specifically, in Patent Document 1, filters of different colors are applied to adjacent pixels in a pixel group assigned to one microlens, and demosaicing processing is performed in the pixel group. In Patent Document 2, filters of the same color are applied to pixel groups assigned to one microlens, and filters of different colors are applied to assigned pixel groups of adjacent microlenses in the microlens array. Yes. And in patent document 2, a demosaicing process is performed about the image of a Bayer arrangement | sequence produced | generated when the signal of the pixel of the same pixel position is combined according to a positional relationship from the pixel group allocated to each micro lens.

  However, the images obtained by the demosaicing process in Patent Documents 1 and 2 may not have a suitable image quality due to the interpolation method.

  In the demosaicing process applied in Patent Document 1, for the pixels in the pixel group assigned to one microlens, a signal of a color component that is insufficient by interpolation is generated from the signals of the peripheral pixels. However, as described above, the light beams formed on the pixel groups assigned through one microlens are light beams that have passed through different pupil regions. That is, the image signal output from each of the pixel groups assigned to one microlens does not correspond strictly because the light ray direction is different, and the signal of the adjacent pixel corresponding to the subject different from the pixel to be interpolated is obtained. There was a possibility that interpolation would be performed.

  In the demosaicing process applied in Patent Document 2, in an image generated by combining signals of pixels having the same relative positional relationship with a microlens from a group of pixels assigned to each microlens, The missing color component signal is interpolated from the pixel signal. In this case, unlike Patent Document 1, all the pixels of the generated image are generated by a light beam that has passed through the same pupil region. However, in the generated image, when using peripheral pixel signals to generate color component signals that do not exist in one pixel, reference is made to signals corresponding to different subjects, particularly in regions where edges and fixed patterns exist. And false colors and moire could occur.

  The present invention has been made in view of the above-described problems, and provides an image processing apparatus, an imaging apparatus, a control method, and a program that generate LF data that can reconstruct an image that has been favorably color-reproduced. Objective.

  In order to achieve the above object, an image processing apparatus of the present invention is characterized by having the following configuration. Specifically, the image processing apparatus is a plurality of image signals obtained by imaging the same subject at different positions at the same time, and each pixel is any one of a plurality of predetermined color components. An acquisition means for acquiring an image signal indicating the signal intensity of the color component; and a specifying means for specifying pixels corresponding to the same subject image existing at the same focal length among the plurality of image signals acquired by the acquisition means; Generating means for generating, for each of the corresponding pixels specified by the specifying means, the signal intensity of the color component that does not exist in the pixel by using the signal intensity of the color component of the other pixels among the corresponding pixels; It is characterized by that.

  With such a configuration, according to the present invention, it is possible to generate LF data that can reconstruct an image that has been favorably color-reproduced.

1 is a block diagram showing functional configurations of a digital camera 100 and a lens 200 included in a camera system according to an embodiment of the present invention. The figure for demonstrating the detailed structure of the imaging part 102 which concerns on embodiment of this invention. The flowchart which illustrated LF data generation processing performed in digital camera 100 concerning an embodiment of the present invention. The figure which showed the hardware constitutions which implement | achieve the phase difference detection between the luminance images which concern on embodiment of this invention A diagram for explaining the principle of generating a reconstructed image The figure which illustrated distribution of the color filter applied in image sensor 109 concerning an embodiment of the present invention The figure for demonstrating the principle of the demosaicing process which concerns on embodiment of this invention The figure for demonstrating the other optical system which can apply this invention

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. Note that an embodiment described below is an example of an image processing apparatus that includes a camera system having an imaging device that can acquire a plurality of image signals obtained by imaging the same subject at different positions at the same time. An example to which the present invention is applied will be described. However, the present invention can be applied to any device capable of acquiring a plurality of image signals obtained by imaging the same subject at different positions at the same time.

  In this specification, the following terms are defined and described.

・ "RAW-LF data"
An original image signal output from the imaging unit 102 included in the digital camera 100 of the present embodiment. Each pixel of the image signal indicates a signal intensity corresponding to a light beam having a different combination of the pupil region and the incident direction of the imaging optical system 202 that has passed. Each pixel of the image signal indicates the signal intensity for any one of a plurality of predetermined color components as will be described later. The RAW-LF data is also called ray space information.

・ "LF data"
An image signal obtained by applying demosaicing processing to RAW-LF data. Each pixel of the LF data has signal intensity information for all of a plurality of predetermined color components. Each pixel of the LF data corresponds to a light beam having the same combination of the corresponding pixel of the RAW-LF data, the pupil region, and the incident direction, and the LF data may be referred to as ray space information.

・ "Reconstructed image"
An image generated from LF data and focused on an arbitrary focal plane position. Specifically, for multiple images (number of pupil divisions) generated from pixels that passed through the same pupil region from LF data, alignment is performed so that the images of the subject existing at the generated focal length match. An image obtained by adding (superimposing) pixel values of pixels to be processed.

《Camera system configuration》
FIG. 1 is a block diagram showing functional configurations of a digital camera 100 and a lens 200 included in a camera system according to an embodiment of the present invention.

<Configuration of digital camera 100>
The camera control unit 101 is a CPU, for example, and includes a ROM and a RAM (not shown). The camera control unit 101 reads out an operation program for each block of the digital camera 100 from the ROM, develops the program in the RAM, and executes the program, thereby controlling the operation of each block of the digital camera 100. In addition, the camera control unit 101 sends information on the focal position determined by analysis of the image signal output from the imaging unit 102 and information on the aperture value corresponding to the determined exposure setting to the lens 200 via the electrical contact 107. Send.

  The imaging unit 102 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 102 performs exposure and readout of each photoelectric conversion element (pixel) included in the imaging device based on the timing signal generated by the camera control unit 101 based on the set exposure setting. Then, the imaging unit 102 outputs an analog image signal of the obtained RAW-LF data to the image processing unit 103. Specifically, the imaging unit 102 photoelectrically converts an optical image formed on the light receiving surface of the imaging element by the imaging optical system 202 of the lens 200, and outputs an analog image signal.

  Further, in the digital camera 100 of the present embodiment, a microlens array 108 in which microlenses 20 are arranged in a lattice pattern as shown in FIG. One microlens 20 of the microlens array 108 is associated with a plurality of photoelectric conversion elements (pixels) of the image sensor 109 as shown in FIG. The light beam that has entered through the imaging optical system 202 of the lens 200 is divided into pupils by being imaged by the microlenses 20 on the pixels of the associated image sensor 109. That is, the light flux that has passed through the pupil region of the imaging optical system 202 corresponding to the associated pixel position is imaged on each pixel of the associated image sensor 109. In the example of FIG. 2B, 5 × 5 = 25 pixels are associated with one microlens 20, and the number of pupil divisions of the imaging optical system 202 is 25.

  FIG. 2C is a diagram illustrating a correspondence relationship between a pixel associated with one microlens and a pupil region of an exit pupil of the imaging optical system 202 through which a light beam formed on each pixel passes. For the sake of simplicity in the example of FIG. 2C, it is assumed that five pixels 21 to 25 in the horizontal direction with respect to one microlens 20 are arranged side by side on a horizontal line passing through the center of the microlens 20. . At this time, each pixel is designed by the microlens 20 so as to be conjugate with the pupil regions 31 to 35 on the exit pupil plane. In the example of FIG. 2C, the pixel 21 has a pupil region 31, the pixel 22 has a pupil region 32, the pixel 23 has a pupil region 33, the pixel 24 has a pupil region 34, and the pixel 25 has a conjugate relationship with the pupil region 35.

  Further, in the image sensor 109 of the present embodiment, it is assumed that any one of a plurality of predetermined color component filters is applied to each photoelectric conversion element of the image sensor. For example, when R, G, and B filters used in a so-called Bayer array are applied, the R component and B component filters are applied to the same number of pixels in the entire image sensor, and the G component filters are twice as many as those. Applied to the other pixels. The arrangement of the filters for the respective color components will be described in detail in the description of the demosaicing process described later.

  The image processing unit 103 performs predetermined image processing on the analog image signal of the RAW-LF data output from the imaging unit 102. Specifically, the image processing unit 103 performs A / D conversion processing, white balance adjustment processing, gamma correction processing, demosaicing processing, and the like on the input analog image signal. In the present embodiment, the image processing unit 103 also performs processing for generating a reconstructed image from the obtained LF data. In addition, the image processing unit 103 performs compression processing on RAW-LF data, LF data, reconstructed image data, audio data, and the like generated in each process described above according to a predetermined encoding method. The image processing unit 103 records RAW-LF data, LF data, reconstructed image data, audio data, and the like generated in each step described above in a memory or a recording medium according to a mode or a user instruction, or a predetermined number. Convert to the format and output to the outside. Various image processes may be realized by a dedicated circuit or the like.

  The memory 104 includes a memory element and a processing circuit that reads from and writes to the memory element. The memory 104 outputs to the storage element and stores an image to be output to the display unit 105. The memory 104 stores encoded images, moving images, audio data, and the like.

  The display unit 105 is a display device provided in the digital camera 100 such as an LCD. The display unit 105 displays a reconstructed image or the like generated from LF data obtained by imaging.

  The operation detection unit 106 detects an operation performed on a user interface such as a release button of the digital camera 100. Specifically, when the operation detection unit 106 detects that the user has operated, for example, a release button, the operation detection unit 106 outputs a control signal corresponding to the operation to the camera control unit 101.

<Configuration of lens 200>
The lens control unit 201 is a CPU, for example, and incorporates a ROM and a RAM (not shown). The lens control unit 201 controls the operation of each block by reading the operation program of each block included in the lens 200 stored in the ROM, developing the program in the RAM, and executing the program. When the lens control unit 201 receives information on the focal position and the aperture value from the camera control unit 101 via the electrical contact 107, the lens control unit 201 transmits the information to the lens driving unit 203 and drives the corresponding optical member of the imaging optical system 202. Let

  The imaging optical system 202 includes a lens group, a diaphragm, and the like that the lens 200 has. In the present embodiment, the imaging optical system 202 includes at least a focus lens, a shift lens, and a diaphragm. The lens driving unit 203 performs drive control of the focus lens, the shift lens, and the diaphragm of the imaging optical system 202 according to the information input from the lens control unit 201. Note that a camera shake detection sensor (not shown) is connected to the lens control unit 201 of this embodiment, and the lens driving unit 203 drives the shift lens according to the output of the sensor input from the lens control unit 201.

<< LF data generation processing >>
A specific process of the LF data generation process performed by the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The process corresponding to the flowchart can be realized by the camera control unit 101 reading, for example, a corresponding processing program stored in the ROM, developing it in the RAM, and executing it. The LF data generation process will be described as being started when shooting is performed using the digital camera 100, for example.

  In step S <b> 301, the image processing unit 103 applies an A / D conversion process to the analog image signal of the RAW-LF data output from the imaging unit 102 by shooting under the control of the camera control unit 101, and RAW-LF. A digital image signal of data is generated.

  In step S <b> 302, under the control of the camera control unit 101, the image processing unit 103 combines the RAW-LF data with pixels that have passed through the same pupil region according to the positional relationship, and generates a RAW image having the number of pupil divisions. That is, in the present embodiment, the image processing unit 103 outputs an image signal output from a pixel of an image sensor associated with each microlens of the microlens array 108 from a pixel having the same relative positional relationship with the microlens. By combining them, a raw image having the number of pupil divisions is generated. Specifically, in the example of FIG. 2B, 5 × 5 pixels associated with the microlenses 20a, b, c, and d are represented by (1, 1), (1, 2),. , (5, 5), and each image is generated using only the pixels classified into the same coordinates from the RAW-LF data. That is, in the example of FIG. 2B, 25 types of images are generated. Since the images generated in this manner correspond to images that have passed through different pupil regions, they are RAW images obtained by photographing the same subject at different positions at the same time.

  In step S <b> 303, the image processing unit 103 generates a luminance image corresponding to each of the generated RAW images for the number of pupil divisions under the control of the camera control unit 101. For example, the luminance image may be generated by weighting and adding pixel values of neighboring R, G, and B component pixels for each pixel, or only the G component pixel is used as a representative value. Also good.

  Depending on the distribution of the filters applied to each pixel, there may be a combination of color components that cannot generate a luminance image. In this case, the number of luminance images generated in this step may be smaller than the number of pupil divisions.

  In S304, the camera control unit 101 selects a luminance image (attention luminance image) from the luminance image of the number of pupil divisions generated in S302, and for each region obtained by dividing the image between image signals with other luminance images. Detect phase difference. That is, in this step, the camera control unit 101 acquires the defocus amount between the target luminance image and the other luminance image for the subject image existing in each region obtained by dividing the target luminance image by the phase difference detection method. When the subject can be regarded as a Lambertian surface that does not have a strong specular reflection component, the image corresponding to the subject in the luminance image does not depend on the divided pupil region through which the luminous flux passes, that is, the same luminance value regardless of the incident direction of the luminous flux. Show. For this reason, the camera control unit 101 can grasp the position where the subject existing at the same focal length exists in each RAW image.

  FIG. 4 shows an example in which the processing of this step is realized as hardware. As shown in FIG. 4A, a target luminance image 401 to be compared and another luminance image 402 are input to a comparator 403 that detects a phase difference. Then, the comparator 403 searches for the position where each region obtained by dividing the target luminance image 401 exists in the other luminance image 402 while changing the phase, that is, while moving the region to be compared. FIGS. 4B, 4C, and 4D show the signal level distributions of the target luminance image 401 and the other luminance image 402 in the line in the search direction when searching while changing the phase. It shows a gap. Here, FIG. 4C shows an example of a case where it is determined that the phase of the target luminance image 401 and the other luminance image 402 are matched, although some variation occurs due to the influence of noise in the search target region. Yes.

  It should be noted that the combination of images for detecting the phase difference from the luminance image of the number of pupil divisions may be limited to images in which the pupil region through which the light beam has passed has the same horizontal coordinate or vertical coordinate. That is, the images classified into the coordinates (1, 1) and (1, 5) or the images classified into the coordinates (1, 1) and (5, 1) in the classification of FIG. Also good. These images are images having a horizontal parallax and a vertical parallax, respectively.

  In addition, since the epipolar constraint occurs between the images having the selected parallax, the search direction can be limited. That is, between images having a horizontal parallax relationship, phase difference detection may be performed on a region 412 in another luminance image 402 having the same vertical coordinates as the region 411 to be searched for the target luminance image 401. In addition, between images having a vertical parallax relationship, phase difference detection may be performed on a region 412 in another luminance image 402 having the same horizontal coordinates as the region 411 to be searched for the target luminance image 401.

For example, as illustrated in FIG. 4A, the comparator 403 has the smallest phase difference in the region 411 of the target luminance image 401 while moving the region 412 from the same position of the other luminance image 402 in the x direction. Search for a location. The search for the position where the phase difference becomes the smallest may be performed, for example, by searching for a position where the integral value SAD (Sum of Absolute Difference) of the difference absolute value is minimum. In the SAD, the pixel in the area 411 of the target luminance image 401 is A (i, j), and the other luminance image 402 is set in a position to be moved by Δx in the x direction from the coordinates of the area 411. Let B (i + Δx, j) be the pixel of
Can be calculated. At this time, Δx at which SAD is minimized corresponds to a phase difference at which SAD is minimized. The search need not be performed in units of pixels.

  In the present embodiment, a luminance image is generated by extracting pixels one by one from the pixel group assigned to each microlens 20. The luminance image generated in this way corresponds to the light flux obtained in a so-called stop state that passes through a limited pupil region obtained by dividing the imaging optical system 202 into the number of pixels assigned to the microlens 20. Yes. That is, since the depth of field is deep, the phase difference can be detected for subject images existing at various focal lengths. However, the luminance image generated for detecting the phase difference of each subject in the implementation of the present invention is not limited to this, and luminance images corresponding to at least two types of pupil regions may be used for phase difference detection.

  In step S305, the camera control unit 101 causes the image processing unit 103 to perform demosaicing processing in consideration of the phase difference detected in step S304.

(Reconstructed image generation principle)
Here, before explaining the demosaicing process of the present embodiment, an outline of generating a reconstructed image from LF data obtained after the demosaicing process will be described with reference to FIG. FIG. 5 shows an image pixel generated from LF data by connecting pixels corresponding to each of the pixels 21 to 25 shown in FIG. 2B for the microlenses 20 arranged in the horizontal direction in the microlens array 108. A row (horizontal 1 line × 5 types) is shown.

  In the figure, the horizontal axis corresponds to the horizontal coordinate of the microlens 20 in the microlens array 108, and the pixels arranged in the horizontal direction in the image corresponding to the divided pupil region are shown in the coordinate order. That is, the illustrated line 501 represents one horizontal line of an image formed by a light beam that has passed through one pupil region. In the figure, the vertical axis corresponds to the horizontal pixel position in the pixel group corresponding to one microlens 20, that is, the spatial coordinates of the pupil region. In this example, generation of one horizontal line of the reconstructed image will be described, but it will be easily understood that other horizontal lines and the vertical direction may be processed in the same manner.

  Each pixel in one horizontal line of the reconstructed image is generated by integrating pixel values of appropriate pixels from these five types of horizontal lines. The pixels used for integration differ according to the focal length corresponding to the generated reconstructed image (focal length corresponding to the subject in focus in the reconstructed image).

  For example, with respect to a subject existing at the focal length that was in focus at the time of shooting LF data, the same coordinates are used in the pixel group corresponding to each microlens 20 regardless of the pupil region of the imaging optical system 202 through which the light beam passes. Imaged. In other words, since the subject is in focus, even if an image is generated for each pupil region, there is no phase difference in the subject image. For this reason, when a reconstructed image focused on the focal length set at the time of shooting is generated, each pixel adds up pixels in the same positional relationship in the pixel group corresponding to the microlens 20, that is, FIG. The pixel groups 502 arranged in the vertical direction are integrated for generating one pixel. That is, the pixel group 502 is a pixel group through which a light beam corresponding to the same portion of the subject image focused at the focal length at the time of imaging passes. In the figure, only the pixel group 502 is shown using a frame. However, in order to generate one horizontal line of a reconstructed image focused on the focal length set at the time of shooting, other pixels arranged on the horizontal axis are also used. Similarly, pixel groups arranged in the vertical direction may be integrated.

  On the other hand, in the horizontal line generated corresponding to the pixel 21 and the horizontal line generated corresponding to the pixel 25 in FIG. When generating a configuration image, the pixel group 503 shown in FIG. 5 is integrated to generate one pixel. That is, since there is no phase difference at the focal length at which the subject is in focus, the pixel 504 and the pixel 505 in the pixel group 503 are identical in focus in the reconstructed image focused on the focal length as illustrated. It corresponds to the luminous flux of the subject image. In the figure, only the pixel group 503 is shown by using a frame. However, in order to generate one horizontal line of a reconstructed image focused on the focal length set at the time of shooting, a similar inclination ( The pixel groups included in the frame having (determined by the phase difference of the image) may be integrated.

  It should be noted that the inclination of the frame indicating the pixel group accumulated in the generation of the reconstructed image is whether the position of the focal plane corresponding to the generated reconstructed image is far or small from the focal length set at the time of shooting the LF data. The sign varies depending on.

(Principle of demosaicing process)
Next, a demosaicing process performed by the image processing unit 103 of the digital camera 100 according to the present embodiment will be described with reference to the drawings.

  For example, the distribution of the color filter applied to each pixel of the image sensor 109 in the imaging unit 102 of the present embodiment is as shown in FIG. 6 (a) and 6 (b), it is assumed that 5 × 5 pixels are assigned to each microlens 20 as in FIG. 2 (b).

  In the example of FIG. 6A, a color filter different from the adjacent pixel is applied to each pixel of the image sensor 109 in accordance with a so-called Bayer array. That is, as shown in the figure, in the image sensor 109, a row in which the G component filter and the R component filter are repeatedly applied in the row direction, and a B component filter and a G component filter are repeatedly applied in the row direction. Are repeatedly provided in the column direction. By arranging the color filters to be applied in this way, a Bayer array is formed in the RAW image for each divided pupil region. That is, as shown in FIG. 6A, 2 × 2 pixels in the RAW image generated by extracting the pixels classified into the positions corresponding to the pixels 21 in FIG. When attention is paid to the group, a Bayer array is formed as in the pixel group 601. Similarly, when attention is paid to the 2 × 2 pixel group in the RAW image generated by extracting the pixels classified in the position corresponding to the pixel 25 in FIG. Thus, a Bayer array is obtained.

  Note that in the example of FIG. 6A, the pixels of the image sensor 109 assigned to the microlens 20 need to be odd rows × odd columns. That is, only when the pupil area is divided into an odd row × odd column area, the RAW image for each divided pupil area can be a Bayer array.

  On the other hand, in the example of FIG. 6B, there is no condition for the pixels of the image sensor 109 assigned to the lens 200. In the example of FIG. 6B, an example is shown in which different color filters are applied to adjacent microlenses 20 or different color filters are applied to pixels assigned to adjacent microlenses 20. As shown in the figure, by defining the color filter applied to each pixel group corresponding to the microlens 20 according to the Bayer array, the 2 × 2 pixel group noted in the RAW image for each divided pupil region is the pixel A Bayer array is formed as in the groups 611 and 612.

  In the following description, the color filter applied to each pixel of the image sensor 109 is described as being set as shown in FIGS. 6A and 6B, but the present invention is not limited to this. It is not limited. In other words, the distribution of the filters of the respective color components in the image sensor 109 may be such that pixels indicating signal intensities of different color components are adjacent when RAW images corresponding to the same pupil region are generated.

  Using such a RAW image having a Bayer array for each divided pupil region, the image processing unit 103 acquires the signal strength of the insufficient color component for each pixel.

  For example, as shown in FIGS. 7A and 7B, a case where subjects 701 and 702 having different distances from the digital camera 100 during shooting are included in the angle of view will be considered. Further, it is assumed that RAW-LF data regarding the situation is acquired with a setting for focusing on a subject at an arbitrary distance 703 that is between the subject 701 and the subject 702. At this time, as in the case of generating the reconstructed image, the RAW image for each divided pupil region generated from the RAW-LF data is applied to each pixel by paying attention to the subject having no phase difference at a specific focal length. The signal strength of the missing color component can be interpolated.

  FIGS. 7C and 7D show the horizontal line of the RAW image for the divided pupil region corresponding to the pixels 21 to 25 in FIG. 2B, paying attention to the G component, as in FIG. ing.

  For example, for the subject 701, a phase difference of four pixels is detected in the horizontal line generated corresponding to the pixel 21 and the horizontal line generated corresponding to the pixel 25. At this time, when the pixel group 710 as shown in FIG. 7C corresponds to the same part of the image of the subject 701, the pixels 711 to 715 included in the pixel group 710 are in principle the same G component. Will have. That is, for example, by interpolating the G component signals of the pixels 711 and 713 and the pixels 713 and 715 for each of the pixels 712 and 714 that do not show the G component signal intensity, an unnatural signal is generated. Occurrence of color interpolation can be avoided. That is, for at least the subject 701, when a reconstructed image focused on the subject is generated, it is possible to reduce the occurrence of false color and moire in the image of the subject in the reconstructed image. At this time, the pixels used for interpolation are pixels that are close to the viewpoint position, that is, pixels that are close to the pupil region through which the light beam has passed, so that more natural color interpolation that is not affected by shielding or the like can be achieved.

  Further, for example, for the subject 702, it is assumed that a phase difference of four pixels is detected in the opposite direction to the subject 701 in the horizontal line generated corresponding to the pixel 21 and the horizontal line generated corresponding to the pixel 25. At this time, when the pixel group 720 as shown in FIG. 7D corresponds to the same part of the image of the subject 702, the pixels 721 to 725 included in the pixel group 720 have the same G component in principle. Will have. That is, for example, by interpolating the G component signals of the pixels 721 and 723 and the pixels 723 and 725 for each of the pixels 722 and 724 that do not show the G component signal intensity, an unnatural signal is generated. Occurrence of color interpolation can be avoided. That is, for at least the subject 702, when a reconstructed image focused on the subject is generated, the generation of false colors and moire in the image of the subject in the reconstructed image can be reduced.

  As described above, in the image processing unit 103 according to the present embodiment, for a subject for which the phase difference has been detected, the signal intensity of the color component that is insufficient in the pixel corresponding to the subject image is obtained, and the light flux for the same image is obtained. Interpolation can be performed by using the imaged signals of other pixels.

  In the present embodiment, the signal intensity of a color component that is insufficient for a specific pixel has been described as being generated from signals of a plurality of pixels on which light fluxes for the same image are formed. However, in principle, these pixels should exhibit the same signal strength, so the signal strength of the color component of any pixel corresponding to the same image is the signal strength of the missing color component. May be substituted.

  For a subject whose phase difference could not be detected, in the RAW image for each of the divided pupil regions, the missing color component using the pixels around the pixel corresponding to the subject, as in the conventional method. What is necessary is just to interpolate a signal strength. At this time, it is possible to reduce the occurrence of false colors and moire to some extent by interpolating in advance the color components of the subject whose phase difference has been detected.

  After the demosaicing process in S305 is completed, the camera control unit 101 records the output LF data on a recording medium (not shown) and completes the LF data generation process.

  In the present embodiment, in the demosaicing process, whether or not the interpolation of the missing color component is performed in the RAW image for the same divided pupil region depending on whether or not the phase difference of the subject can be detected is divided. Explained as a thing. However, when interpolation is performed using pixels of the RAW image for other divided pupil regions, the subject to be detected may be a Lambertian plane in at least two luminance images to be detected for phase difference as described above. It becomes a condition. For this reason, when the minimum value of SAD is larger than a predetermined value when phase difference detection is performed and the degree of similarity cannot be regarded as a Lambertian plane, the RAW for the same divided pupil region is obtained for the subject. Interpolation may be performed using peripheral pixels in the image.

  The present invention can also be applied to other optical systems as shown in FIG. FIG. 8 is a diagram schematically showing a state in which a light beam from an object (subject) forms an image on the image sensor 109. FIG. 8A corresponds to the optical system described in FIG. 2, and is an example in which the microlens array 108 is disposed in the vicinity of the imaging surface of the imaging optical system 202. FIG. 8B shows an example in which the microlens array 108 is arranged closer to the object than the imaging plane of the imaging optical system 202. FIG. 8C shows an example in which the microlens array 108 is arranged on the side farther from the object than the imaging plane of the imaging optical system 202.

  In FIG. 8, the same reference numerals are assigned to configurations common to those in FIG. 2, and duplicate descriptions are omitted. Reference numeral 51 denotes an object plane, and 51a and 51b are arbitrary points on the object plane. Reference numeral 52 denotes a pupil plane of the imaging optical system 202, and 61, 62, 71, 72, 73, 81, 82, 83, and 84 denote specific microlenses on the microlens array 108, respectively.

  In FIGS. 8B and 8C, a virtual image sensor 109a and a virtual microlens array 108a are shown in order to clarify the correspondence with FIG. 8A. Also, a light beam passing from the point 51a on the object plane through the pupil regions 31 and 33 on the pupil plane 52 is indicated by a solid line, and a light beam passing from the point 51b on the object plane to the pupil regions 31 and 33 on the pupil plane 52 is indicated by a broken line. did.

  In the example of FIG. 8A, as described with reference to FIG. 3, the microlens array 108 is disposed in the vicinity of the imaging surface of the imaging optical system 202, so that the imaging device 109 and the pupil plane 52 of the imaging optical system are conjugated. Are in a relationship. Further, the object plane 51 and the microlens array 108 are in a conjugate relationship. For this reason, the light beam from the point 51a on the object plane 51 reaches the microlens 61, the light beam from the point 51b reaches the microlens 62, and the light beams that have passed through the pupil regions 31 to 35 correspond to the microlens. Reach the selected pixel.

  In the example of FIG. 8B, the light beam from the imaging optical system 202 is imaged by the microlens array 108, and the imaging element 109 is provided on the imaging surface. By arranging in this way, the object plane 51 and the image sensor 109 have a conjugate relationship. The light beam that has passed through the pupil region 31 of the pupil plane 52 from the point 51a on the object plane 51 reaches the micro lens 71, and the light beam that has passed through the pupil region 33 on the pupil plane from the point 51a reaches the micro lens 72. Further, the light beam that has passed through the pupil region 31 of the pupil plane 52 from the point 51b on the object plane 51 reaches the micro lens 72, and the light beam that has passed through the pupil region 33 of the pupil plane 52 from the point 51b reaches the micro lens 73. . The light beam that has passed through each microlens reaches a pixel provided so as to correspond to the microlens. Thus, the light beam from the object plane forms an image at different positions on the imaging surface of the image sensor 109 according to the emission position and the passing pupil region. If these are rearranged in the position on the virtual imaging surface 50, the same information as the information obtained on the imaging surface of FIG. 8A can be obtained. That is, information on the pupil region (incident angle) that has passed and the position on the image sensor 109 can be obtained.

  In the example of FIG. 8C, the microlens array 108 re-images the light beam from the imaging optical system 202 (referred to as re-imaging because the light beam once imaged is diffused is imaged). The imaging surface of the image sensor 109 is disposed on the re-imaging plane. By arranging in this way, the object plane 51 and the image sensor 109 have a conjugate relationship. The light beam that has passed through the pupil region 31 of the pupil plane 52 from the point 51a on the object plane 51 reaches the micro lens 82, and the light beam that has passed through the pupil region 33 of the pupil plane 52 from the point 51a reaches the micro lens 81. Further, the light beam that has passed through the pupil region 31 of the pupil plane 52 from the point 51b on the object plane 51 reaches the micro lens 84, and the light beam that has passed through the pupil region 33 of the pupil plane 52 from the point 51b reaches the micro lens 83. The light beam that has passed through each microlens reaches a pixel provided so as to correspond to the microlens.

  Similarly to the case of FIG. 8B, if the pixel signals obtained by the image sensor 109 are rearranged to the positions on the virtual image pickup surface 50, the information obtained on the image pickup surface of FIG. Information can be obtained. That is, information on the pupil region (incident angle) that has passed and the position on the image sensor 109 can be obtained.

  FIG. 8 shows a configuration example in which pupil division is performed using the microlens array 108 (phase modulation element) to acquire position information and angle information of a light beam. Other configurations can be used as long as they can acquire (equivalent to limiting). For example, a configuration in which a pattern mask (gain modulation element) configured by repeating basic patterns is used instead of the microlens array 108 may be used. The present invention can also be applied to a multi-view optical system (camera array) as shown in FIG. In FIG. 8D, each camera of each multi-view optical system has an imaging optical system disposed at a position corresponding to the pupil regions 31 to 35 in FIGS. 8A to 8C, and each imaging element 109a to 109c. Let the image form. Thus, the image obtained from each image sensor has a different shooting position, and corresponds to the RAW image for each divided pupil region described above. For this reason, by specifying the pixels on which the same subject image is formed in accordance with the positional relationship of each camera, it is possible to suitably interpolate the signal strength of the color component that is insufficient for each pixel.

  As described above, the image processing apparatus of the present embodiment can generate LF data that can reconstruct an image that has been favorably color-reproduced. Specifically, the image processing apparatus is a plurality of image signals obtained by imaging the same subject at different positions at the same time, and each pixel is any one of a plurality of predetermined color components. An image signal indicating the signal intensity of the color component is acquired. Then, among the acquired image signals, a pixel corresponding to the same subject image existing at the same focal length is specified, and for each of the corresponding pixels, the signal intensity of the color component not existing in the pixel is determined. It is generated using the signal intensity of the color component of other pixels.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

A plurality of image signals obtained by imaging the same subject at different positions at the same time, each image signal indicating the signal intensity of any one of a plurality of predetermined color components Obtaining means for obtaining
Specifying means for specifying pixels corresponding to the same subject image existing at the same focal length among the plurality of image signals acquired by the acquiring means;
Generating means for generating, for each of the corresponding pixels specified by the specifying means, a signal intensity of a color component that does not exist in the pixel using a signal intensity of the color component of another pixel of the corresponding pixels; And an image processing apparatus.   The image processing apparatus according to claim 1, wherein the generation unit uses the signal strength of the color component of the other pixel as the signal strength of the color component that does not exist in the pixel.   The image processing apparatus according to claim 1, wherein the generation unit interpolates a signal intensity of a color component that does not exist in the pixel using a signal intensity of the color component of the other pixel.   The generation unit is configured to determine, for a pixel included in one of the corresponding pixels, the pixel specified by the specifying unit from the image signal photographed in the vicinity of the photographing position of the one image signal. The image processing apparatus according to claim 3, wherein interpolation is performed using signal strength.   5. The specifying unit specifies pixels corresponding to the same subject image existing at the same focal length by detecting a phase difference of the images between the plurality of image signals. The image processing apparatus according to any one of the above.   The plurality of image signals are original image signals imaged by an imaging device, and each of the pixels corresponds to a light flux having a different combination of a pupil region and an incident direction that have passed through the imaging optical system of the imaging device, The pixel image is obtained by combining pixels corresponding to a light beam that has passed through the same pupil region from an original image signal indicating the signal intensity of any one of a plurality of color components. The image processing apparatus according to any one of 1 to 5.   The image processing apparatus according to claim 1, further comprising a recording unit that records an image signal in which a signal intensity of a color component that does not exist in the pixel is generated by the generation unit. Imaging means for imaging the same subject at different positions at the same time, and outputting a plurality of image signals each of which indicates the signal intensity of any one of a plurality of predetermined color components;
Specifying means for specifying pixels corresponding to the same subject image existing at the same focal length among the plurality of image signals output by the imaging means;
Generating means for generating, for each of the corresponding pixels specified by the specifying means, a signal intensity of a color component that does not exist in the pixel using a signal intensity of the color component of another pixel of the corresponding pixels; An imaging apparatus comprising: A plurality of image signals obtained by the acquisition unit of the image processing apparatus capturing images of the same subject at different positions at the same time, and each pixel has a color selected from a plurality of predetermined color components An acquisition step of acquiring an image signal indicating the signal intensity of the component;
A specifying step in which the specifying unit of the image processing device specifies pixels corresponding to the same subject image existing at the same focal length among the plurality of image signals acquired in the acquiring step;
For each of the corresponding pixels specified in the specifying step, the generation unit of the image processing device uses the signal intensity of the color component of the other pixels among the corresponding pixels to determine the signal intensity of the color component that does not exist in the pixel. And a generation step of generating using the intensity. A control method for an image processing apparatus. The imaging means of the imaging device images the same subject at different positions at the same time, and outputs a plurality of image signals each of which indicates the signal intensity of one of the plurality of color components. An imaging process to
A specifying step of specifying a pixel corresponding to the same subject image existing at the same focal length among the plurality of image signals output in the imaging step;
For each of the corresponding pixels specified in the specifying step, the generation unit of the imaging device determines the signal intensity of the color component that does not exist in the pixel and the signal intensity of the color component of the other pixels of the corresponding pixels. And a generation step of generating using the imaging method.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 7.   The program for making a computer perform each process of the control method of the imaging device of Claim 10.