JP5904281B2 - Image processing method, image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing method, an image processing device, an imaging device, and an image processing program.

2. Description of the Related Art An imaging apparatus that generates a plurality of parallax images having parallax with each other using a single imaging optical system is known.
[Prior art documents]
[Patent Literature]
[Patent Document 1] Japanese Patent Application Laid-Open No. 2003-7994

  Patent Document 1 does not disclose a specific method for actually generating a high-definition stereoscopic color image from imaging data when a new stereoscopic imaging method using a single camera is employed. In general, in 2D color image generation when 2D imaging is performed with one camera, a technique is known in which high-definition 2D images are obtained by performing color interpolation and then performing noise removal and edge enhancement. However, in the new 3D imaging method, it is unknown what procedure can be used to finally obtain a high-definition stereoscopic image. For example, in this type of imaging apparatus, a higher-resolution parallax image may be generated using the generated parallax image. In this case, if filtering processing such as edge enhancement and noise removal is individually performed on the plurality of generated parallax images, moiré / false in each parallax image instead due to insufficient sampling density of each parallax pixel. May cause resolution.

  An image processing method according to a first aspect of the present invention includes a non-parallax pixel including an aperture mask that generates a viewpoint in a reference direction, which includes a plurality of pixels having one aperture mask in one pixel, and a reference direction. An imaging device comprising a pixel array in which at least three types of pixels are arranged: a left parallax pixel having an aperture mask that generates a parallax in the left direction and a right parallax pixel having an aperture mask that generates a parallax in the right direction with respect to a reference direction A first image obtained by capturing a subject image through a single optical system in a different pixel at the same time, a viewpoint in the reference direction, a viewpoint in the left direction, and a viewpoint in the right direction, and an image related to the viewpoint in the left direction; An image processing method for converting to an image related to a right-direction viewpoint, wherein a procedure for generating a temporary left viewpoint image for each pixel using a pixel value of a left parallax pixel of the first image, right A procedure for generating a provisional right viewpoint image for each pixel using the pixel value of the difference pixel, and a procedure for generating a reference viewpoint image for each pixel using the pixel value of at least the non-parallax pixel of the first image; , Based on the procedure for performing edge enhancement processing on the reference viewpoint image to generate the edge-enhanced reference viewpoint image, the edge-enhanced reference viewpoint image, the temporary left viewpoint image, and the temporary right viewpoint image, Each pixel includes a procedure for generating an image relating to the left viewpoint and an image relating to the right viewpoint.

  An image processing method according to a second aspect of the present invention includes a non-parallax pixel including an aperture mask that generates a viewpoint in a reference direction, the pixel including a plurality of pixels each having one aperture mask per pixel, and the reference direction. An imaging device comprising a pixel array in which at least three types of pixels are arranged: a left parallax pixel having an aperture mask that generates a parallax in the left direction and a right parallax pixel having an aperture mask that generates a parallax in the right direction with respect to a reference direction A first image obtained by capturing a subject image through a single optical system in a different pixel at the same time, a viewpoint in the reference direction, a viewpoint in the left direction, and a viewpoint in the right direction, and an image related to the viewpoint in the left direction; An image processing method for converting to an image related to a right-direction viewpoint, wherein a procedure for generating a temporary left viewpoint image for each pixel using a pixel value of a left parallax pixel of the first image, right A procedure for generating a temporary right viewpoint image for each pixel using the pixel value of the difference pixel, a procedure for generating a reference viewpoint image for each pixel using the pixel value of the non-parallax pixel of the first image, Each step is performed based on a procedure for generating a noise-removed reference viewpoint image by performing noise removal processing on the reference viewpoint image, and the noise-removed reference viewpoint image, the temporary left viewpoint image, and the temporary right viewpoint image. The pixel includes a procedure for generating an image related to the left viewpoint and an image related to the right viewpoint.

  The image processing apparatus according to the third aspect of the present invention includes first parallax image data corresponding to a viewpoint shifted in the first direction with respect to the reference direction, and a second direction opposite to the first direction with respect to the reference direction. A parallax image data acquisition unit that acquires the second parallax image data corresponding to the viewpoint shifted to, and the resolution corresponding to the reference direction and higher than the spatial frequency resolution of the first parallax image data and the second parallax image data A reference image data acquisition unit that acquires the reference image data, a filter processing unit that performs filtering processing of at least one of edge adjustment and noise removal on the reference image data, reference image data subjected to filtering processing, Using the first parallax image data and the second parallax image data, the third parallax image data corresponding to the viewpoint shifted in the first direction and the fourth corresponding to the viewpoint shifted in the second direction. And a parallax image data generating unit that generates a difference image data.

  An imaging device according to a fourth aspect of the present invention includes an imaging device that outputs at least one of first parallax image data and second parallax image data, and the above-described image processing device.

  An image processing program according to a fifth aspect of the present invention includes: first parallax image data corresponding to a viewpoint shifted in the first direction with respect to the reference direction; and a second direction opposite to the first direction with respect to the reference direction. A parallax image data step for acquiring the second parallax image data corresponding to the viewpoint shifted to the first position, and a resolution corresponding to the reference direction and higher than the spatial frequency resolution of the first parallax image data and the second parallax image data. A reference image data step for obtaining reference image data, a filter processing step for performing filtering processing of at least one of edge adjustment and noise removal on the reference image data, reference image data subjected to filtering processing, and first parallax Using the image data and the second parallax image data, the third parallax image data corresponding to the viewpoint shifted in the first direction and the view shifted in the second direction Fourth to perform the parallax image data generating step of generating the parallax image data to the computer that corresponds to.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

1 is a diagram illustrating a configuration of a digital camera 10. FIG. 2 is a diagram illustrating a cross-sectional configuration of an image sensor 100. FIG. It is a figure explaining the concept of defocus in a pixel without parallax. It is a figure explaining the concept of defocus in a parallax pixel. It is a figure which shows the light intensity distribution of a parallax pixel and a parallax pixel. It is a figure explaining the opening shape of the opening part 104 in case there are two types of parallax pixels. It is a figure which shows the resolution in the frequency space of a Bayer type | mold G parallax pixel array and this arrangement | sequence. It is a figure explaining the moire of a parallax image. It is a figure which shows the pixel arrangement | sequence figure and the resolution in the frequency space of this arrangement | sequence. It is a figure which shows the pixel arrangement | sequence figure and the resolution in the frequency space of this arrangement | sequence. It is a figure which shows an example of the variation of the repeating pattern 110 in case there are two types of parallax pixels. It is a figure for demonstrating calculation of a gain value. It is a figure for demonstrating gain correction. It is a figure for demonstrating the production | generation of a temporary parallax image. It is a figure for demonstrating calculation of a gain value. It is a figure for demonstrating local gain correction. It is a figure explaining interpolation of G component. It is a figure explaining the production | generation process of color parallax plane data. It is a figure which shows an example of the interpolation of a pixel value.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  The digital camera according to the present embodiment, which is one form of the image processing apparatus and the imaging apparatus, is configured to generate a left viewpoint image and a right viewpoint image for one scene by one shooting. Each image having a different viewpoint is called a parallax image.

  FIG. 1 is a diagram illustrating the configuration of the digital camera 10. The digital camera 10 includes a photographic lens 20 as a photographic optical system, and guides a subject light beam incident along the optical axis 21 to the image sensor 100. The photographing lens 20 may be an interchangeable lens that can be attached to and detached from the digital camera 10. The digital camera 10 includes an image sensor 100, a control unit 201, an A / D conversion circuit 202, a memory 203, a drive unit 204, an image processing unit 205, a memory card IF 207, an operation unit 208, a display unit 209, an LCD drive circuit 210, and an AF. A sensor 211 is provided.

  As shown in the figure, the direction parallel to the optical axis 21 toward the image sensor 100 is defined as the + Z-axis direction, and the direction toward the back of the sheet on the plane orthogonal to the Z-axis is defined as the + X-axis direction and the upward direction on the sheet + Y-axis. . Regarding the relationship with composition in photographing, the X axis is the horizontal direction and the Y axis is the vertical direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

  The taking lens 20 is composed of a plurality of optical lens groups, and forms an image of a subject light flux from the scene in the vicinity of its focal plane. In FIG. 1, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens arranged in the vicinity of the pupil. The image sensor 100 is disposed near the focal plane of the photographic lens 20. The image sensor 100 is an image sensor such as a CCD or CMOS sensor in which a plurality of photoelectric conversion elements are two-dimensionally arranged. The image sensor 100 is controlled in timing by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202.

  The A / D conversion circuit 202 converts the image signal output from the image sensor 100 into a digital image signal and outputs the digital image signal to the memory 203. The image processing unit 205 performs various image processing using the memory 203 as a work space, and generates image data. In particular, the image processing unit 205 includes an interpolation processing unit 231, a reference image data generation unit 232, a filter processing unit 233, and a parallax image data generation unit 234.

  The interpolation processing unit 231 generates left parallax image data for the left viewpoint and right parallax image data for the right viewpoint based on the output of the image sensor 100. The left parallax image data and the right parallax image data generated by the interpolation processing unit 231 are data used for processing of the reference image data generation unit 232 and the parallax image data generation unit 234. Since the left parallax image data and the right parallax image data themselves generated by the interpolation processing unit 231 are not output as final left parallax image data and right parallax image data, the left parallax image data and the right parallax image are not output. The data can be said to be temporary parallax image data for generating final parallax image data. The spatial frequency resolution of the left parallax image data and right parallax image data generated by the interpolation processing unit 231 is lower than the spatial frequency resolution of the left parallax image data and right parallax image data generated by the parallax image data generation unit 234.

  The reference image data generation unit 232 generates reference image data using the pixel values of the left parallax image data and the right parallax image data generated by the interpolation processing unit 231. Details of the reference image data will be described later. The spatial frequency resolution of the reference image data is higher than the spatial frequency resolution of the left parallax image data and the right parallax image data generated by the interpolation processing unit 231.

  The filter processing unit 233 performs at least one of edge adjustment and noise removal filtering processing on the reference image data generated by the reference image data generation unit 232. In the following description, edge enhancement will be mainly described as an example of edge adjustment.

  The parallax image data generation unit 234 uses the left parallax image data and the right parallax image data generated by the interpolation processing unit 231, and the reference image data subjected to the filtering process by the filter processing unit 233 to use the parallax image data generation unit Higher-resolution left-parallax image data and higher-resolution right-parallax image data than the left-parallax image data and right-parallax image data generated by H.234 are generated.

  The image processing unit 205 also has general image processing functions such as adjusting image data according to the selected image format. The generated image data is converted into a display signal by the LCD drive circuit 210 and displayed on the display unit 209. The data is recorded on the memory card 220 attached to the memory card IF 207.

  The AF sensor 211 is a phase difference sensor in which a plurality of distance measuring points are set for the subject space, and detects the defocus amount of the subject image at each distance measuring point. A series of shooting sequences is started when the operation unit 208 receives a user operation and outputs an operation signal to the control unit 201. Various operations such as AF and AE accompanying the imaging sequence are executed under the control of the control unit 201. For example, the control unit 201 analyzes the detection signal of the AF sensor 211 and executes focus control for moving a focus lens that constitutes a part of the photographing lens 20.

  Next, the configuration of the image sensor 100 will be described in detail. FIG. 2 is a schematic diagram illustrating a cross section of the image sensor 100.

  The imaging element 100 is configured by arranging a microlens 101, a color filter 102, an aperture mask 103, a wiring layer 105, and a photoelectric conversion element 108 in order from the subject side. The photoelectric conversion element 108 is configured by a photodiode that converts incident light into an electrical signal. A plurality of photoelectric conversion elements 108 are two-dimensionally arranged on the surface of the substrate 109.

  An image signal converted by the photoelectric conversion element 108, a control signal for controlling the photoelectric conversion element 108, and the like are transmitted and received through the wiring 106 provided in the wiring layer 105. In addition, an opening mask 103 having openings 104 arranged in a one-to-one correspondence with each photoelectric conversion element 108 and repeatedly arranged two-dimensionally is provided in contact with the wiring layer. As will be described later, the opening 104 is shifted for each corresponding photoelectric conversion element 108 so that the relative position is precisely determined. As will be described in detail later, parallax occurs in the subject light beam received by the photoelectric conversion element 108 by the action of the opening mask 103 including the opening 104.

  On the other hand, the aperture mask 103 does not exist on the photoelectric conversion element 108 that does not generate parallax. In other words, it can be said that an aperture mask 103 having an aperture 104 that does not limit the subject luminous flux incident on the corresponding photoelectric conversion element 108, that is, allows the entire incident luminous flux to pass therethrough is provided. Although no parallax is generated, the aperture 107 formed by the wiring 106 defines the incident light flux that is incident, so the wiring 106 is regarded as an aperture mask that allows the entire incident light flux that does not cause parallax to pass. You can also The opening mask 103 may be arranged separately and independently corresponding to each photoelectric conversion element 108, or may be formed collectively for a plurality of photoelectric conversion elements 108 in the same manner as the manufacturing process of the color filter 102. .

  The color filter 102 is provided on the opening mask 103. The color filter 102 is a filter provided in a one-to-one correspondence with each photoelectric conversion element 108, which is colored so as to transmit a specific wavelength band to each photoelectric conversion element 108. In order to output a color image, it is only necessary to arrange at least two types of color filters that are different from each other. However, in order to obtain a higher quality color image, it is preferable to arrange three or more types of color filters. For example, a red filter (R filter) that transmits the red wavelength band, a green filter (G filter) that transmits the green wavelength band, and a blue filter (B filter) that transmits the blue wavelength band may be arranged in a grid pattern. The color filter may be not only a combination of primary colors RGB but also a combination of YCM complementary color filters.

  The microlens 101 is provided on the color filter 102. The microlens 101 is a condensing lens for guiding more incident subject light flux to the photoelectric conversion element 108. The microlenses 101 are provided in a one-to-one correspondence with the photoelectric conversion elements 108. In consideration of the relative positional relationship between the pupil center of the taking lens 20 and the photoelectric conversion element 108, the optical axis of the microlens 101 is shifted so that more subject light flux is guided to the photoelectric conversion element 108. It is preferable. Furthermore, the arrangement position may be adjusted so that more specific subject light beam, which will be described later, is incident along with the position of the opening 104 of the opening mask 103.

  As described above, one unit of the aperture mask 103, the color filter 102, and the microlens 101 provided in one-to-one correspondence with each photoelectric conversion element 108 is referred to as a pixel. In particular, a pixel provided with the opening mask 103 that generates parallax is referred to as a parallax pixel, and a pixel that is not provided with the opening mask 103 that generates parallax is referred to as a non-parallax pixel. For example, when the effective pixel area of the image sensor 100 is about 24 mm × 16 mm, the number of pixels reaches about 12 million.

  Note that in the case of an image sensor with good light collection efficiency and photoelectric conversion efficiency, the microlens 101 may not be provided. In the case of a back-illuminated image sensor, the wiring layer 105 is provided on the side opposite to the photoelectric conversion element 108. Further, if the opening 104 of the opening mask 103 has a color component, the color filter 102 and the opening mask 103 can be formed integrally. Note that the color filter 102 is not provided when a monochrome image signal may be output.

  In the present embodiment, the opening mask 103 and the wiring 106 are provided separately, but the wiring 106 may serve the function of the opening mask 103 in the parallax pixels. That is, a prescribed opening shape is formed by the wiring 106, and the incident light beam is limited by the opening shape to guide only a specific partial light beam to the photoelectric conversion element 108. In this case, the wiring 106 that forms the opening shape is preferably closest to the photoelectric conversion element 108 in the wiring layer 105.

Further, the opening mask 103 may be formed by a permeation blocking film provided to overlap the photoelectric conversion element 108. In this case, the opening mask 103 is formed, for example, by sequentially laminating a SiN film and a SiO ₂ film to form a permeation blocking film and removing a region corresponding to the opening 104 by etching.

  Next, the concept of defocusing when the parallax Lt pixel and the parallax Rt pixel receive light will be described. First, it is a figure explaining simply the concept of defocus in a pixel without parallax. FIG. 3 is a diagram for explaining the concept of defocus in a pixel without parallax. As shown in FIG. 3A, when an object point that is a subject exists at the focal position, the subject luminous flux that reaches the image sensor light receiving surface through the lens pupil is steep with the pixel at the corresponding image point as the center. The light intensity distribution is shown. That is, if non-parallax pixels that receive the entire effective luminous flux that passes through the lens pupil are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity Drops rapidly.

  On the other hand, as shown in FIG. 3B, when the object point deviates from the focal position, the subject luminous flux exhibits a gentle light intensity distribution on the light receiving surface of the image sensor as compared with the case where the object point exists at the focal position. . That is, the output value at the pixel of the corresponding image point is lowered, and a distribution having output values up to the peripheral pixels is shown.

  Further, as shown in FIG. 3C, when the object point further deviates from the focal position, the subject luminous flux exhibits a gentler light intensity distribution on the image sensor light receiving surface. In other words, the output value at the pixel of the corresponding image point further decreases, and a distribution having output values up to the surrounding pixels is shown.

  FIG. 4 is a diagram for explaining the concept of defocusing in the parallax pixels. The parallax Lt pixel and the parallax Rt pixel receive the subject luminous flux that arrives from one of the two parallax virtual pupils set as the optical axis target as a partial region of the lens pupil. In this specification, a method of capturing a parallax image by receiving subject light fluxes that arrive from different virtual pupils in a single lens pupil is referred to as a monocular pupil division imaging method.

  As shown in FIG. 4 (a), when an object point that is a subject exists at the focal position, steep light centering on the pixel of the corresponding image point, regardless of the parallax virtual pupil, regardless of the parallax virtual pupil. The intensity distribution is shown. If the parallax Lt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Further, even when the parallax Rt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. That is, even if the subject luminous flux passes through any parallax virtual pupil, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Match each other.

  On the other hand, as shown in FIG. 4B, when the object point deviates from the focal position, the peak of the light intensity distribution indicated by the parallax Lt pixel corresponds to the image point as compared to the case where the object point exists at the focal position. Appearing at a position away from the pixel in one direction, and its output value decreases. In addition, the width of the pixel having the output value is increased. The peak of the light intensity distribution indicated by the parallax Rt pixel appears at a position away from the pixel corresponding to the image point in the opposite direction to the one direction in the parallax Lt pixel and at an equal distance, and the output value similarly decreases. Similarly, the width of the pixel having the output value is increased. That is, the same light intensity distribution that is gentler than that in the case where the object point exists at the focal position appears at an equal distance from each other. Further, as shown in FIG. 4C, when the object point further deviates from the focal position, the same light intensity distribution that is more gentle than the state of FIG. . That is, it can be said that the amount of blur and the amount of parallax increase as the object point deviates from the focal position.

  When the change of the light intensity distribution explained in FIG. 3 and the change of the light intensity distribution explained in FIG. 4 are respectively graphed, they are expressed as shown in FIG. In the figure, the horizontal axis represents the pixel position, and the center position is the pixel position corresponding to the image point. The vertical axis represents the output value of each pixel. Since this output value is substantially proportional to the light intensity, it is shown as the light intensity in the figure.

  FIG. 5A is a graph showing changes in the light intensity distribution described in FIG. A distribution curve 1801 represents the light intensity distribution corresponding to FIG. 3A and shows the steepest state. The distribution curve 1802 represents the light intensity distribution corresponding to FIG. 3B, and the distribution curve 1803 represents the light intensity distribution corresponding to FIG. Compared with the distribution curve 1801, it can be seen that the peak value gradually decreases and has a broadening.

  FIG. 5B is a graph showing changes in the light intensity distribution described in FIG. A distribution curve 1804 and a distribution curve 1805 represent the light intensity distribution of the parallax Lt pixel and the light intensity distribution of the parallax Rt pixel in FIG. 4B, respectively. As can be seen from the figure, these distributions have a line-symmetric shape with respect to the center position. Further, a combined distribution curve 1806 obtained by adding them shows a similar shape to the distribution curve 1802 of FIG. 3B which is in a defocus state equivalent to that of FIG.

  A distribution curve 1807 and a distribution curve 1808 respectively represent the light intensity distribution of the parallax Lt pixel and the light intensity distribution of the parallax Rt pixel in FIG. As can be seen from the figure, these distributions are also symmetrical with respect to the center position. Further, a combined distribution curve 1809 obtained by adding them shows a similar shape to the distribution curve 1803 in FIG. 3C which is in a defocus state equivalent to that in FIG.

  FIG. 6 is a diagram illustrating the opening shape of the opening 104 when there are two types of parallax pixels. In FIG. 6A, the shape of the opening 104l of the parallax Lt pixel and the shape of the opening 104r of the parallax Rt pixel are respectively obtained by dividing the shape of the opening 104n of the non-parallax pixel by the center line 322. The example which is the same is shown. That is, in FIG. 6A, the area of the opening 104n of the non-parallax pixel is the sum of the area of the opening 104l of the parallax Lt pixel and the area of the opening 104r of the parallax Rt pixel. In the present embodiment, the opening 104n of the non-parallax pixel is referred to as a full-opening opening, and the opening 104l and the opening 104r are referred to as half-opening openings. In addition, when the opening is located in the center of the photoelectric conversion element, the opening is directed to the reference direction. The opening 104l of the parallax Lt pixel and the opening 104r of the parallax Rt pixel are displaced in opposite directions with respect to a virtual center line 322 passing through the center (pixel center) of the corresponding photoelectric conversion element 108. Yes. Accordingly, the opening 104l of the parallax Lt pixel and the opening 104r of the parallax Rt pixel each generate parallax in one direction with respect to the reference direction and in the other direction opposite to the one direction.

  FIG. 6B shows the light intensity distribution when the object point is deviated from the focal position in the pixel having each opening shown in FIG. In the figure, the curve Lt corresponds to the distribution curve 1804 in FIG. 5B, and the curve Rt corresponds to the distribution curve 1805 in FIG. 5B. A curve N corresponds to a pixel without parallax, and shows a similar shape to the combined distribution curve 1806 in FIG. Each of the openings 104n, 104l, and 104r has an aperture stop function. Therefore, the blur width of the non-parallax pixel having the opening 104n having an area double the opening 104l (opening 104r) is the Lt pixel and the parallax Rt pixel indicated by the combined distribution curve 1806 in FIG. It is about the same as the blur width of the added curve.

  FIG. 6C shows an example in which the shape of the opening 104l of the parallax Lt pixel, the shape of the opening 104r of the parallax Rt pixel, and the shape of the opening 104c of the parallax C pixel are all the same shape. Yes. Here, the parallax C pixel is a pixel having no eccentricity. Strictly speaking, the parallax C pixel is a parallax pixel that outputs a parallax image, in that only the subject luminous flux having the central portion of the pupil as a partial region is guided to the photoelectric conversion element 108. However, here, a pixel having an opening corresponding to the reference direction is defined as a non-parallax pixel. Accordingly, as the reference direction, the parallax C pixel in FIG. 6C having an opening at the center of the photoelectric conversion element is the pixel without parallax, as in the pixel without parallax in FIG. 6A. In addition, the opening 104l, the opening 104r, and the opening 104c are half the area of the opening 104n illustrated in FIG. As in the case of FIG. 6A, each of the openings 104 l and 104 r is in contact with a virtual center line 322 that passes through the center (pixel center) of the photoelectric conversion element 108.

  FIG. 6D shows the light intensity distribution when the object point is deviated from the focal position in the pixel having each opening shown in FIG. In the figure, the curve Lt corresponds to the distribution curve 1804 in FIG. 5B, and the curve Rt corresponds to the distribution curve 1805 in FIG. 5B. Each of the openings 104c, 104l, and 104r has a function of an aperture stop. Therefore, the blur width of the parallax C pixel having the opening 104c having the same shape and the same area as the opening 104l and the opening 104r is approximately the same as the blur width of the parallax Lt pixel and the parallax Rt pixel. As described above, in the monocular pupil division imaging method, the parallax is included in the blur and the same optical image as the 2D imaging is acquired at the in-focus position as compared with the normal binocular stereoscopic imaging. is there.

  Patent Document 1 discloses a color / parallax arrangement formed by a combination of a color filter and a pixel arrangement in which only left and right parallax pixels are arranged. For example, a color / parallax arrangement in which pixels without parallax are arranged in addition to the left and right parallax pixels can be used. Imaging with these color / parallax arrangements can be called stereoscopic imaging with a monocular pupil division imaging system.

  As a method of generating a stereoscopic image, the image processing unit 205 generates a left parallax image by collecting sampling points (grid points) of only the left parallax pixels and interpolating the empty grid points, and sets sampling points of only the right parallax pixels. It is conceivable to generate a right parallax image by collecting and interpolating vacancies. However, the method of interpolating the left and right parallax images inevitably has a problem that a resolution exceeding the sampling resolution limit of each parallax pixel cannot be obtained.

  On the other hand, in the case of an array in which pixels with no parallax of all apertures coexist, a 2D image without parallax is once generated as an intermediate image (hereinafter also referred to as “2D intermediate image”). It is possible to have a resolution up to the Nyquist frequency equal to the resolution limit when all pixels are sampled under a certain condition. That is, as described above, as a characteristic characteristic of the monocular pupil division imaging method, the point spread function of the subject image at the in-focus position is the same point spread function for all the non-parallax pixels, the left parallax pixels, and the right parallax pixels. Become. For this reason, the same subject image as the 2D dedicated sensor is captured in the vicinity of the focus position, and the resolution can be derived to the maximum.

  Using the 2D intermediate image thus obtained, a low-resolution left parallax image and a low-resolution right parallax image obtained by independently temporarily interpolating left and right parallax pixels are subjected to parallax modulation described later to generate a 2D intermediate image. By superimposing the high-frequency components, a high-resolution left parallax image and a high-resolution right parallax image exceeding the sampling limit of each parallax pixel can be obtained.

  It should be noted here that if a high-resolution left parallax image and a high-resolution right parallax image obtained by parallax modulation also become a subject image in an out-of-focus area out of the focus position, the 2D intermediate image While high-frequency resolution is reflected, high-frequency components that existed in the original low-resolution left-parallax image and low-resolution right-parallax image above the resolution limit appear as low-frequency moire. It is a problem. On the other hand, such a moire component does not appear in the 2D intermediate image even in the out-of-focus region. That is, the 2D intermediate image can stably achieve high resolution.

  Therefore, in view of such a fact, in order to perform edge enhancement processing with a 3D image, it can be concluded as follows. First, instead of individually performing edge enhancement processing on the left parallax image and right parallax image, once a high-resolution 2D intermediate image is generated, edge enhancement processing is performed on the 2D image. Then, a 3D image subjected to edge enhancement processing is generated by applying parallax modulation. Through such an image processing procedure, a high-definition and natural edge-enhanced stereoscopic image can be obtained without enhancing aliasing moire that exceeds the sampling limit of the parallax pixels.

  FIG. 7 is a diagram illustrating a Bayer-type G parallax pixel array and the resolution in the frequency space of this array. In the arrangement shown in FIG. 7, the left parallax pixel (Lt), the right parallax pixel (Rt), and the non-parallax pixel (N) are mixed. The color filter array has a Bayer array structure, and one G pixel is provided with a left-opening parallax pixel, and the other G pixel is provided with a right-opening parallax pixel as an opening mask. Further, the R pixel and the B pixel are provided with a pixel with no parallax of all openings. This corresponds to the arrangement described in the first embodiment described later. FIG. 7 also shows a frequency space diagram (k space diagram) representing the sampling resolution limit corresponding to each color component and each parallax. However, there is a relationship of k = 2πf between the resolution frequency f [lines / mm] and the wave number k.

  A monochrome subject (eg, a circular zone plate) at the in-focus position can be resolved to the outermost Nyquist frequency limit when the aforementioned 2D intermediate image is generated. On the other hand, in the out-of-focus area, the resolution frequency is basically limited to the inner square area.

  FIG. 8 is a diagram illustrating the moire of the parallax image. 8A shows an image without parallax, FIG. 8B shows a left parallax image, and FIG. 8C shows a right parallax image. The images shown in FIGS. 8B and 8C are generated through a process excluding the part of edge enhancement processing shown in Embodiment 1 described later. When a 3D image is obtained through a 2D intermediate image for a subject image slightly deviated from the above-described in-focus position, as shown in FIGS. The component appears in the left and right parallax images. Therefore, if edge enhancement is performed separately for the left parallax image and the right parallax image, the moire component of the false resolution is emphasized and the image quality is impaired. If edge enhancement is performed on the 2D intermediate image, high-definition 3D edge enhancement processing can be performed without enhancing the moire component.

  As the color / parallax arrangement in which the left parallax pixel (Lt), the right parallax pixel (Rt), and the non-parallax pixel (N) are mixed, other arrangements may be used. For example, an array in which parallax pixels described in Embodiment 2 described later have a sparse density, a parallax pixel described in Embodiment 3 described later in a sparse density, and a monochrome array can also be used.

  In the embodiment described below, as described above, a high-resolution 2D intermediate image is first generated, and the generated 2D intermediate image is superimposed on the 3D image to obtain a high-resolution output image for both 2D and 3D. A technique has been introduced in which color information of parallax pixels and non-parallax pixels is referred to each other and their correlation is used.

  Next, noise removal will be described. The higher the resolution of the image, the higher the accuracy of the noise removal filter, the more it can be determined whether the region needs to store the edge structure. As a result, in the region where the edge structure is to be stored, it is difficult to exclude information regarding the image structure. Therefore, as in the case of edge enhancement, it is desirable to perform noise removal processing on a high-resolution 2D intermediate image. A known edge-preserving smoothing filter can be used as the filter.

  By the way, in the case of noise removal, unlike the case of edge enhancement, a noise fluctuation component also appears in the low-parity left parallax image and right parallax image that have been temporarily interpolated. In order to prevent noise from being propagated at the stage of parallax modulation, it is desirable to remove noise fluctuation components at the stage of low-resolution left parallax image and right parallax image before parallax modulation.

  However, when considering the structure of a sparse parallax pixel array as in Embodiments 2 and 3 to be described later, the number of non-parallax pixels is the largest, and the sparse parallax pixels exist only at the positions where they are skipped. For this reason, the interpolated value calculated by the average interpolation between the points between them has a characteristic that it cannot originally contain a high-frequency noise component. Accordingly, in the case of a sparse parallax pixel array, it is only necessary to perform noise removal processing on only the intermediate 2D image as long as it is about the common high sensitivity region of about ISO 800 to ISO 6400. Details will be described in a fourth embodiment described later. In the ultra-high sensitivity region of ISO 12800 to ISO 409600, the fluctuation component is conspicuous even if it is a sparse parallax pixel. Therefore, it is necessary to remove the fluctuation component.

  However, it is not necessary to perform the process of removing the fluctuation component for all pixels of the actual resolution. A noise component is extracted from a reduced image downsampled to such an extent that only one left parallax pixel or right parallax pixel sampled by the image sensor is included in one pixel. It is only necessary to return to the resolution and perform the subtraction process. For example, a multi-resolution noise removal technique disclosed in Japanese Patent Laid-Open No. 2006-309749 by the same inventor as the present applicant can be used. Specifically, the process of reducing the high resolution to the low resolution sequentially by multi-resolution conversion, extracting the noise components at each of those resolutions, and sequentially integrating them to return to the real resolution noise components is performed on the high resolution side. It is sufficient to omit several steps. Therefore, for the temporarily interpolated parallax image, it is possible to omit all processing on the high resolution side, which has the largest computation scale of noise removal processing. Therefore, the calculation can be performed very simply, that is, in the case of software at high speed and in the case of hardware with a small circuit scale.

<Embodiment 1>
--- Bayer G parallax pixel array, edge enhancement ---
The image processing procedure is approximately as follows.
1) Color / parallax multiplexed mosaic image data input 2) Global gain balance correction of color / parallax mosaic image 3) Temporary parallax image generation 4) Generation of parallax-free color mosaic image by right and left local illuminance distribution correction (Local・ Gain balance correction
5) Generation of reference image without parallax 6) Edge enhancement processing for reference image without parallax 7) Generation of actual parallax image 8) Conversion to output color space

1) Color / parallax multiplexed mosaic image data input Single plate mosaic image with color and parallax multiplexed as shown in FIG. 7: M (x, y)
It is assumed that the gradation is a linear gradation output by A / D conversion. That is, it has a pixel value proportional to the amount of light. This is sometimes called RAW data.

  In this embodiment, an arithmetic average is employed. In this way, a mosaic image in which the left parallax pixel is corrected with one gain coefficient and the right parallax pixel is corrected with one gain coefficient is output as M ′ (x, y). Note that this step can be realized at the same time by performing only the local gain correction in step 4, and may be omitted in some cases.

3) Generation of temporary parallax image A left parallax image and a right parallax image with low spatial frequency resolution are generated.
Performs simple average interpolation in the G color plane, where only the left parallax pixels are collected. Linear interpolation is performed according to the ratio of distances using adjacent pixel values. Similarly, simple average interpolation in the G color plane in which only the right parallax pixels are collected is performed. That, Lt _mosaic (x, y) Lt from (x, y) and, Rt _mosaic (x, y) from the Rt (x, y) to produce a.
Temporary left parallax image: Lt (x, y)
Temporary right parallax image: Rt (x, y)
When creating the temporary left parallax image Lt (x, y) and the temporary right parallax image Rt (x, y), high-definition may be performed by introducing direction determination in the signal plane.

4) Generation of parallax-free color mosaic image by correcting left and right illumination distribution (local gain balance correction)
Next, in accordance with the same idea as the global gain correction performed in step 1, local gain correction is performed in units of pixels to match the illuminance of the left parallax pixel in the screen and the right parallax pixel in the screen. Then, a new Bayer surface with gain matching is created. This is equivalent to replacing with an average value, and a Bayer surface in which the parallax disappears is created. This is written as M _N (x, y).
In this case as well, there are two types of target value setting methods to be used as reference points for each pixel: a method of selecting an arithmetic mean and a method of selecting a geometric mean.

The process of applying local gain correction to each pixel is actually only necessary to substitute the average value of each pixel obtained first. In this sense, it can be said that the local gain correction is a modulation process for eliminating the parallax. In this embodiment, an arithmetic average is employed. In this way, the Bayer plane data is rewritten using the average value of the left viewpoint image and the right viewpoint image as the non-parallax pixel value of the new G pixel position, and the Bayer plane image M _N (x, y) without parallax is output. To do.

5) Generation of reference image without parallax Nyquist frequency equivalent to the number of pixels of the sensor using the conventional color interpolation technique from the Bayer plane M _N (x, y) where the illuminance balance of the G component is aligned and parallax disappears It is possible to generate a color image with no resolution and without parallax as an intermediate image. For example, as the best example of the known Bayer interpolation technique, there is an interpolation algorithm shown in US publication 2010/021853 of the same inventor as the present applicant. This technology includes technology USP6,836,572 that increases the resolution of direction determination by the same inventor and the same inventor to resolve vertical and horizontal Nyquist frequencies, technology USP7,236,628 that increases the resolution in the diagonal direction when calculating interpolation values, color Applicable to adaptive false color countermeasure technology USP7,565,007 that improves the resolution of direction judgment and adaptive false color countermeasure technology USP7,391,903 that uses the color gradient judgment method and technology that increases resolution of direction judgment The best high performance demosaic technology used has been introduced.

  In the following, not all of them are shown, but only the part that increases the vertical and horizontal Nyquist resolution and diagonal resolution of the G component responsible for luminance and the part that uses color difference interpolation to increase the resolution of the R and B components are described. To do.

5-1) Transition to gamma space by gradation conversion For the purpose of performing the above-described high-resolution Bayer interpolation, gradation conversion that further realizes a uniform noise space is performed, and the gamma space for interpolation (image processing) Prediction of the interpolated value in (space). This is the method introduced by USP 7,957,588, the same inventor as the present applicant.

Assume that the input signal is x, the output signal is y, and both the gradation of the input signal and the gradation of the output signal are defined in the range [0, 1]. The gradation curve (gamma curve) is defined so that the input / output characteristics pass through (x, y) = (0, 0) and (1, 1). Assuming that the maximum value of the actual input gradation X is Xmax and the maximum value of the output gradation Y is Ymax, x = X / Xmax, y = Y / Ymax, and gradation conversion is
Is done by.
Here, the gradation characteristic of y = f (x) is
It is. The positive offset value ε is set to a larger value as the imaging condition with higher sensitivity increases the dark current noise component.

5-2) Color interpolation process For the color interpolation process, only the simple process shown in USP 7,957,588 (WO2006 / 006373) of the same inventor as the present applicant is copied once again. However, (x, y) will be described using the symbol [i, j]. Further, the G component on the _MN plane after gradation conversion is referred to using the symbol G, and the R and B components are referred to using the symbol Z.

  In step S4, the CPU performs an interpolation process as follows. Here, a pixel having R component color information is referred to as an R pixel, a pixel having B component color information is referred to as a B pixel, and a pixel having G component color information is referred to as a G pixel. The signal value of the R component corresponding to the pixel indicated by i, j] is R [i, j], the signal value of the G component is G [i, j], and the signal value of the B component is B [i, j]. I will represent it.

(Direction judgment)
The CPU determines the vertical similarity CvN [i, j] and the horizontal similarity ChN [i, j] for the pixel (R pixel or B pixel) that is not the G pixel indicated by the pixel position [i, j]. Are calculated by the following equations (3) and (4), respectively.
However, Z [i, j] is the signal value of the R component or B component indicated by the pixel position [i, j]. The first term is the similarity between the same colors that represents the similarity between the same colors at two pixel intervals, and the second term is the similarity between the different colors that represents the similarity between the different colors at the adjacent pixel intervals. The similarity between different colors has the ability to resolve vertical and horizontal Nyquist frequencies.

The absolute value of the first term in the above formulas (3) and (4) detects the rough directionality by comparing the G color components. The absolute values of the second and third terms of the above formulas (3) and (4) detect fine similarities that cannot be detected by the first term. The CPU calculates the vertical similarity and the horizontal similarity obtained by the above formulas (3) and (4) for each coordinate, and calculates the vertical and horizontal similarity in the target coordinates [i, j]. Based on the following equation (5), the direction of similarity is determined.

  However, Th is a determination threshold value used for the purpose of avoiding erroneous determination due to noise included in the signal value, and is changed according to the ISO sensitivity. HV [i, j] indicates the direction of similarity with respect to the pixel position [i, j]. When HV [i, j] = 0, the vertical and horizontal directions are similar, and when HV [i, j] = 1, the vertical direction Similar, HV [i, j] =-1 is similar in the horizontal direction.

(G interpolation)
Based on the determined similar direction, the CPU interpolates the G component using the unevenness information of the R component or the B component. That is, information on whether or not to interpolate to an external dividing point that cannot be predicted only by interpolation of the inner G component of the surrounding G component, information on the same color component that is located in the vicinity of information on other color components existing at the interpolation target position And determining whether the image structure is convex upward or downward. That is, the information of the high frequency component obtained by sampling other color components is superimposed on the interpolation target color component. The G color complement is calculated by the following formulas (6) and (9) when the vertical direction is similar to the position [i, j] of the center R pixel shown in FIG. 4 of WO2006 / 006373, for example. When the directions are similar, calculation is performed using the following equations (7) and (10). The pixel position when G color interpolation is performed on the position of the B pixel is shown in FIG. 5 of WO2006 / 006373.

  However, Z [i, j] is the signal value of the R component or B component indicated by the pixel position [i, j]. By adding a correction term of the second derivative by the other color components of the second term to the average value of the color components to be interpolated in the first term, the spatial resolution in the oblique direction is increased.

  The first term in the above equation (9) is calculated from the G component signal values G [i, j-1] and G [i, j + 1] arranged vertically with respect to the pixel position [i, j]. Represents an average value. The second term in the above equation (9) is the amount of change calculated from the R component signal values R [i, j], R [i, j-2] and R [i, j + 2] arranged vertically. Represent. By adding the amount of change in the R component signal value to the average value of the G component signal value, the G component interpolation value G [i, j] is obtained. Since such interpolation can be predicted except for the internal dividing point of the G component, it will be referred to as extrapolation for convenience.

  The above equation (10) performs extrapolation in the horizontal direction using the signal values of the pixels arranged horizontally with respect to the pixel position [i, j], as in the case of the extrapolation in the vertical direction described above. When the similar direction is classified as both the vertical and horizontal directions, the CPU calculates the G color interpolation values by the above formulas (9) and (10), and takes the average of the two calculated G color interpolation values. Let it be a G color interpolation value.

(R interpolation)
For R color interpolation, pixel positions [i + 1, j], [i, j + 1], [i + 1, j other than the R pixel position [i, j] shown in FIG. +1] is calculated by the following equations (11) to (13), respectively. At this time, unevenness information of G component signal values (FIG. 7 of WO2006 / 006373) corresponding to all pixel positions obtained by the G interpolation described above is used.

  The first term in the above equations (11) to (13) represents an average value calculated from the R component signal value adjacent to the coordinates to be subjected to the R component interpolation, and the first term in the above equations (11) to (13). The second term represents the amount of change calculated from the coordinates subject to R component interpolation and the G component signal value adjacent to the coordinates. That is, similarly to the extrapolation performed in the G interpolation, the change amount of the G component signal value is added to the average value of the R component signal values to obtain an R component interpolation value. This is equivalent to a method in which a color difference Cr = RG is generated at the R position and average interpolation is performed in this color difference plane.

(B interpolation)
For the B component interpolation, interpolation processing is performed in the same manner as the R component. For example, for pixel positions [i + 1, j], [i, j + 1], [i + 1, j + 1] other than the position [i, j] of the B pixel shown in FIG. 8 of WO2006 / 006373 Are calculated by the following equations (14) to (16), respectively. At this time, the unevenness information of the G component signal values (FIG. 9 of WO2006 / 006373) corresponding to all the pixel positions obtained by the G interpolation described above is used.

  According to the above formulas (14) to (16), the amount of change in the G component signal value is added to the average value of the B component signal value to obtain the B component interpolation value. This is equivalent to a method of generating a color difference Cb = B−G at the B position and performing an average interpolation in the color difference plane. Since the sample frequency of the R component and the B component is lower than that of the G component, the high frequency component of the G component signal value is reflected using the color difference RG and the color difference BG. Therefore, such interpolation for the chroma component will be referred to as color difference interpolation for convenience.

6) Edge enhancement processing for reference image without parallax 6-1) Color space conversion processing RGB color images without parallax obtained by Bayer interpolation in step 5 are converted to R _N ^Γ (x, y), G _N ^Γ (x, y ), B _N ^Γ (x, y). These are RGB data represented by the gradation of the interpolated gamma space. These RGB data are converted to the YCbCr space of the color system expressed by luminance and color difference.
Y (x, y) = 0.2990R _N ^Γ (x, y) + 0.5870G _N ^Γ (x, y) + 0.1140B _N ^Γ (x, y)
Cb (x, y) =-0.1684R _N ^Γ (x, y) -0.3316G _N ^Γ (x, y) + 0.5000B _N ^Γ (x, y)
Cr (x, y) = 0.5000R _N ^Γ (x, y) -0.4187G _N ^Γ (x, y) -0.0813B _N ^Γ (x, y)

6-2) Edge enhancement processing Edge enhancement processing is performed on the luminance Y plane.
Y '(x, y) = Y (x, y) + k * △ Y (x, y)
Here, Δ represents a Laplacian filter representing a second order differential. The constant k is a parameter for adjusting the degree of edge enhancement. As a Laplacian filter, for example, the following filter coefficients can be considered, but the present invention is not limited to this.

6-3) Inverse color space conversion processing From YCbCr space to RGB space using edge-enhanced luminance component Y '(x, y) and raw color difference components Cb (x, y), Cr (x, y) return. It is only necessary to multiply the inverse matrix of step 6-1). Since the definition is the same as that used in JPEG, the description is omitted here.

6-4) Transition to the original linear gradation space by inverse gradation conversion
The RGB color planes subjected to Bayer interpolation and edge enhancement are subjected to inverse gradation conversion in step 5-1 to return to linear gradation RGB data.
Thus, the obtained RGB color image without parallax is represented by R _N (x, y), G _N (x, y), and B _N (x, y). These are RGB data expressed in linear gradation.

7) Generation of actual parallax image Temporary left parallax image Lt (x, y) generated in step 3 and low-parity color image R _N (x, y) generated as intermediate processing in step 5 ), G _N (x, y), B _N (x, y), the left-parallax color image R _Lt (x, y), G _Lt (x, y), B Generate _Lt (x, y). Similarly, a temporary right parallax image Rt (x, y) generated in step 3 with low resolving power and a color image R _N (x, y), G _N ( x, y), B _N (x, y), and the right resolution parallax color image R _Rt (x, y), G _Rt (x, y), B _Rt (x, y) ) Is generated. This can be called parallax modulation because the displacement processing is realized by superimposing the parallax components of the temporary parallax image.

  There are two possible parallax modulation methods: a method using an arithmetic average as a reference point and a method using a geometric average as a reference point. Both can obtain a parallax modulation effect, but when the aperture mask of pixels without parallax of the image sensor is full aperture, a method using an arithmetic average as a reference point is adopted, and the aperture mask of pixels without parallax is a pixel with parallax. It is better to adopt a method using the geometric mean as a reference point when the same half aperture is used. Therefore, in this embodiment, a method using an arithmetic mean as a reference point is used.

  Thus, the parallax modulation calculation formula defined in step 7 and the parallax extinction calculation formula (local gain balance correction) for correcting illuminance unevenness between the left and right defined in step 4 are just in inverse relationship. It can be seen that modulation is applied by multiplying. Therefore, in step 7, it works in the direction in which the parallax is applied, and in step 4, it works in the direction in which the parallax disappears.

8) Conversion to output color space The high resolution non-parallax intermediate color images R _N (x, y), G _N (x, y), B _N (x, y) obtained in this way and the high resolution Left parallax color image R _Lt (x, y), G _Lt (x, y), B _Lt (x, y), High resolution right parallax color image R _Rt (x, y), G _Rt (x , y) and B _Rt (x, y) are subjected to color matrix conversion and gamma conversion from the camera RGB of the spectral characteristics of the sensor to the standard sRGB color space, and output as an image in the output color space. As described above, high-definition 2D images and 3D images with edge enhancement are generated.

<Additional description of Embodiment 1>
FIG. 11 is a diagram illustrating an example of the repetitive pattern 110 when there are two types of parallax pixels. The coordinate system of the digital camera 10 is defined by the X axis, the Y axis, and the Z axis. However, in the image sensor 100, the x axis is defined in the right direction and the y axis is defined in the lower direction with reference to the left and upper end pixels. . In the example of the figure, the same four pixels as the Bayer array are used as the repeated pattern 110. In the repeating pattern 110, the effective pixel area of the image sensor 100 is periodically arranged in the vertical and horizontal directions. In other words, the imaging device 100 uses a repetitive pattern 110 indicated by a thick line in the drawing as a basic lattice. The R pixel and the B pixel are non-parallax pixels, and the Gb pixel is assigned to the parallax L pixel and the Gr pixel is assigned to the parallax R pixel. In this case, the opening 104 is defined so that the parallax Lt pixel and the parallax Rt pixel included in the same repetitive pattern 110 receive the light beam emitted from the same minute region when the subject is in the in-focus position. . Further, in this example, as described with reference to FIG. 6A, it is assumed that the non-parallax pixel is the opening portion of the full opening, and the parallax Lt pixel and the parallax Rt pixel are the opening portions of the half opening. Note that the pixel pitch is a.

  In the example of the figure, Gb pixels and Gr pixels, which are green pixels with high visibility, are used as parallax pixels, so that it is expected to obtain a parallax image with high contrast. In addition, since the Gb pixel and the Gr pixel which are the same green pixels are used as the parallax pixels, it is easy to perform a conversion operation from these two outputs to an output having no parallax, and the output of the R pixel and the B pixel which are non-parallax pixels is high. High-quality 2D image data can be generated.

  FIG. 7 (lower stage) is a diagram illustrating the resolution related to the spatial frequency of the image captured by the image sensor in which the repetitive pattern 110 illustrated in FIG. 11 is employed. In FIG. 7 (lower stage), the resolution related to the spatial frequency is described in the k-space (k-space) of the wave number k represented by k = 2πf. However, f shows a frequency. The frequency resolution region is described by a first Brillouin zone representing a unit cell (Wigner-Zeilz cell) in a reciprocal lattice space.

Assuming that the pixel pitch is a as described above, if the color filter and the aperture mask are not arranged, the captured image is the Nyquist frequency k _x = [− π / a, + π / a] surrounded by a dotted line, k _y = [− π / a, + π / a]. That is, the range surrounded by the dotted line is the limit resolution frequency of the image. However, in the present embodiment, the color filter and the aperture mask are arranged so as to overlap one sensor surface. Since the information captured by one sensor surface is constant, the amount of information is reduced by dividing the functions. For example, when the parallax pixels are formed by the opening mask, the number of pixels without parallax is relatively reduced, and thus the amount of information obtained by the pixels without parallax is reduced. The same applies to the color filter, and the amount of individual information is reduced by the amount of R, G, B.

Therefore, when focusing on an image of a specific color in a specific aperture mask, the limit resolution frequency of the image does not reach the inherent Nyquist frequency. Specifically, as shown in the drawing, for example, the G component image G _{Lt at} the left viewpoint is an area half the original Nyquist frequency with respect to both the k _x axis and the k _y axis, k _x = [ -π / (2a), + π / (2a)], k y = [- π / (2a), + π / (2a)] not only have a range of resolution of. Image G _Rt of the G component of the right-side _perspective, the image R _N of the R component of no parallax intermediate _viewpoint, the image B _N of the B component of intermediate viewpoint no disparity is the same.

Accordingly, even if an RGB color image of the left viewpoint and an RGB color image of the right viewpoint are generated as they are, the resolving power of these images is k _x = [− π / (2a), + π / (2a)]. , K _y = [− π / (2a), + π / (2a)]. That is, these images do not have the resolving power in the ranges of the inherent Nyquist frequencies k _x = [− π / a, + π / a] and k _y = [− π / a, + π / a].

In the present embodiment, the image processing unit 205 performs processing for increasing the resolution in order to compensate for the amount of information reduced by dividing the functions. Specifically, the G _Lt pixel and G _Rt pixel is parallax pixels, by replacing a virtual parallax without pixel G _N, to produce a Bayer array of no parallax pixels only. Then, using the existing Bayer interpolation technique, an original Nyquist frequency k _x = [− π / a, + π / a], k _y = [− π / a, + π / a) as an intermediate image without parallax. ] Can be generated. After that, by superimposing the left viewpoint image having only a small resolving power in the frequency space and the intermediate image having no parallax, a color image of the left viewpoint having a resolving power within the range of the original Nyquist frequency is finally generated. be able to. The same can be said for the right-side color image.

  Each pixel in the pixel array shown in FIG. 11 includes various parallax pixels and non-parallax pixels when focusing on the opening 104, and various R pixels, G pixels, and B pixels when focusing on the color filter 102. Characterized in combination. Therefore, even if the outputs of the image sensor 100 are aligned with the pixel arrangement, they are not image data representing a specific image. That is, only when the pixel outputs of the image sensor 100 are separated and collected for each pixel group characterized by the same, image data representing one image corresponding to the feature is formed. For example, when the outputs of the parallax pixels are collected for each type of opening, a plurality of parallax image data having parallax can be obtained. In this way, each piece of image data separated and collected for each identically characterized pixel group is referred to as plane data.

The interpolation processing unit 231 of the image processing unit 205 receives mosaic image data M _mosaic (x, y), which is RAW original image data whose output values are arranged in the pixel arrangement order of the image sensor 100. Here, an image lacking at least one of R, G, and B information in each pixel is referred to as a mosaic image, and data forming the mosaic image is referred to as mosaic image data. However, even if at least one piece of information of R, G, and B is missing in each pixel, if it is not handled as an image in the first place, for example, if image data is composed of pixel values of monochrome pixels, it is treated as a mosaic image. Absent. Each output value is a linear gradation value proportional to the amount of light received by each photoelectric conversion element of the image sensor 100.

  In the present embodiment, the interpolation processing unit 231 of the image processing unit 205 performs gain correction for matching the overall brightness between the left and right at this stage. As the aperture stop is reduced, the illuminance of light incident on the left parallax pixel and the illuminance of light incident on the right parallax pixel have a large difference not only in the relative distribution between the left and right but also in the average signal level of the entire image. This is because it occurs. In the present embodiment, the gain correction for matching the overall brightness between the left and right is called global gain correction.

FIG. 12 is a diagram for explaining calculation of the gain value. For convenience, among the mosaic image data M _mosaic (x, y), the mosaic image data of the left parallax pixel of the G component is Lt _mosaic (x, y), and the mosaic image data of the right parallax pixel of the G component is Rt _mosaic ( x, y), and also in FIG. 12, only the left and right parallax pixels are extracted and illustrated. However, in the figure, the pixel types are described so that the types of pixels can be understood in accordance with the example of FIG. 11, but output values corresponding to the respective pixels are actually arranged.

FIG. 13 is a diagram for explaining the gain correction. When the interpolation processing unit 231 of the image processing unit 205 calculates the gain values for the left and right parallax pixels, as illustrated, the interpolation processing unit 231 calculates the Lt _mosaic (x, y) and Rt _mosaic (x, y) pixels. Gain correction is performed using the gain value obtained. Specifically, gain correction for the left parallax pixel is performed by the following (Equation 1), and gain correction for the right parallax pixel is performed by (Equation 2). For convenience, among the mosaic image data M ′ _mosaic (x, y), the mosaic image data of the G component left parallax pixel is the Lt ′ _mosaic (x, y), the mosaic image data of the G component right parallax pixel. _Is represented as Rt ′ _mosaic (x, y).
As a result, the interpolation processing unit 231 of the image processing unit 205 performs mosaic image data M ′ in which the left parallax pixel and the right parallax pixel in M _mosaic (x, y) are each corrected with one gain coefficient, as illustrated. _mosaic (x, y) can be generated. Next, the interpolation processing unit 231 of the image processing unit 205 generates a left parallax image and a right parallax image with low spatial frequency resolution as temporary parallax images.

FIG. 14 is a diagram for explaining the generation of a temporary parallax image. The interpolation processing unit 231 of the image processing unit 205 first separates the mosaic image data M ′ _mosaic (x, y) into a plurality of plane data. Each plane data at this point has an output value only at a pixel position where the output value exists in the RAW original image data. Therefore, the interpolation processing unit 231 of the image processing unit 205 performs interpolation processing based on each plane data, and generates plane data in which the empty grid is filled.

In FIG. 14, the left side of the upper diagram shows Lt ′ _mosaic (x, y), which is plane data obtained by extracting only the left parallax pixel from the mosaic image data M ′ _mosaic (x, y), and the right side shows the right parallax. It is a figure which shows _Rt'mosaic (x, y) which is the plane data which extracted only the pixel. In each figure, it describes so that the kind of pixel may be understood according to the example of FIG. 11, but actually the output value corresponding to each pixel is arranged.

In generating Lt ′ (x, y), which is left parallax image data having a low spatial frequency resolution, the interpolation processing unit 231 of the image processing unit 205 uses the pixel values that have become empty grids as pixels of the surrounding left parallax pixels. The value is calculated by interpolation processing. For example, the pixel value of the empty lattice P _L1 is calculated by averaging pixel values of four left parallax pixels adjacent in the diagonal direction. As shown in the lower left diagram of FIG. 14, the interpolation processing unit 231 of the image processing unit 205 performs an averaging process on the pixel values of the surrounding left parallax pixels for all the empty lattices, and performs interpolation processing. Lt ′ (x, y), which is plane data in which vacancies are filled, is generated. Note that the interpolation processing unit 231 of the image processing unit 205 may further perform interpolation processing using the pixel values calculated by the interpolation processing, or use only output values that exist at the stage of the RAW original image data. Interpolation processing may be performed.

Similarly, when generating Rt ′ (x, y), which is right parallax image data with a low spatial frequency resolution, the interpolation processing unit 231 of the image processing unit 205 converts the pixel value that has become an empty lattice into the peripheral right parallax. Calculation is performed by interpolation using the pixel value of the pixel. For example, the pixel value of the empty grid _PR1 is calculated by averaging pixel values of four right parallax pixels adjacent in the diagonal direction. As shown in the lower right diagram of FIG. 14, the interpolation processing unit 231 of the image processing unit 205 performs an interpolation process by averaging pixel values of surrounding right parallax pixels with respect to all the empty lattices. Rt ′ (x, y), which is plane data in which vacancies are filled, is generated.

  Next, the reference image data generation unit 232 of the image processing unit 205 performs gain correction for each pixel of Lt ′ (x, y) using the calculated gain value, and similarly, Rt ′ (x Gain correction is performed on each pixel x, y) using the calculated gain value. Thereby, the illuminances of the left parallax pixel and the right parallax pixel at the same pixel position are matched. In the present embodiment, gain correction using the gain value calculated for each pixel in this way is referred to as local gain correction in contrast to the above-described global gain correction.

FIG. 15 is a diagram for explaining the calculation of the gain value. As illustrated, the reference image data generation unit 232 of the image processing unit 205 calculates an average value for each pixel from Lt ′ (x, y) and Rt ′ (x, y). In performing the local gain correction, there are two methods for calculating the average value, namely, arithmetic mean and geometric mean. Here, an arithmetic average is employed so that the blur width of the subject image in the non-parallax pixel of the G component where the parallax disappears matches the blur width of the subject image of the pixel without parallax. In this case, specifically, the reference image data generation unit 232 of the image processing unit 205 calculates the average value by the following (Equation 3).

FIG. 16 is a diagram for explaining local gain correction. As described above, the average value is obtained by performing the local gain correction on each pixel. Therefore, the reference image data generation unit 232 of the image processing unit 205 includes m (x _m , y _n ) calculated by (Equation 3) for the pixel values of the left and right parallax pixels of the repetitive pattern 110, as shown in the figure. Local gain correction can be performed only by replacing with m (x _{m + 1} , y _{n + 1} ). In this sense, it can be said that the local gain correction is a modulation process for eliminating the parallax. Thereby, it is possible to obtain a Bayer array in which the pixel values of the parallax Gb pixel and the parallax Gr pixel are replaced with the pixel values of the non-parallax G pixel.

The reference image data generation unit 232 of the image processing unit 205 generates M _N (x, y) by replacing the pixel values of all the left and right parallax pixels with the corresponding average values calculated in (Equation 3). Note that local gain correction may be performed not on all the pixels of Lt ′ (x, y) and Rt ′ (x, y) but on the pixels corresponding to the positions of the left parallax pixel and the right parallax pixel in the Bayer array. Good.

Next, the reference image data generation unit 232 of the image processing unit 205 uses an existing color interpolation technique, and a color image without parallax having a resolution from M _N (x, y) to each Nyquist frequency. Data is generated as intermediate image data.

  FIG. 17 is a diagram for explaining interpolation of the G component. The G color interpolation is calculated with reference to the pixel in the drawing with respect to the position [i, j] of the center R pixel shown in the left diagram of FIG. The pixel position when G color interpolation is performed on the position of the B pixel is shown in the right diagram of FIG.

In the present embodiment, the filter processing unit 233 of the image processing unit 205 performs edge enhancement processing on the intermediate image data. The parallax image data generation unit 234 of the image processing unit 205 includes Lt ′ (x, y), Rt ′ (x, y), R _N (x, y), G _N (x, y), B _N (x, The left viewpoint color image data and the right viewpoint color image data are generated using the five plane data of y). Specifically, the parallax image data generation unit 234 of the image processing unit 205 generates left and right color image data by superimposing parallax components of the temporary parallax image on the non-parallax image. This generation process is called parallax modulation process.

The color image data of the left viewpoint includes R _Lt (x, y) that is red plane data corresponding to the left viewpoint, G _Lt (x, y) that is green plane data, and B _Lt (x, y) that is blue plane data. y) three color parallax plane data. Similarly, the color image data of the right viewpoint is red plane data R _Rt (x, y) corresponding to the right viewpoint, green plane data G _Rt (x, y), and blue plane data B _Rt. It is composed of three color parallax plane data of (x, y).

FIG. 18 is a diagram for explaining color parallax plane data generation processing. In particular, a process for generating R _Lt (x, y) and R _Rt (x, y), which are red parallax planes among the color parallax planes, will be described.

  As described above, according to the digital camera 10 of the present embodiment, moire components (aliasing) are enhanced in a stereoscopic image by performing edge enhancement on a 2D intermediate image without high parallax. Instead, edge enhancement can be performed with the high-frequency resolution obtained by actually resolving the high-frequency component of the parallax image. Furthermore, noise removal results with high edge-preserving performance can be obtained by removing noise from a high-resolution 2D intermediate image without parallax. Since the influence of moiré associated with the sampling density of the parallax pixels can be suppressed, it is possible to avoid a problem that inconsistency such as inconsistency of the same subject image appears between a plurality of parallax images.

<Embodiment 2>
--- Bayer type RGB sparse parallax pixel array, edge enhancement ---
The example using the image pick-up element arrange | positioned periodically by making the arrangement | sequence figure of the upper stage of FIG. 9 into a basic lattice is shown. The frequency resolution region of the reciprocal lattice space is also shown for each color and each parallax combination. This arrangement captures the property that parallax occurs only in the blurred subject area of the monocular pupil division method, imaging with a structure in which the density of parallax pixels is sparsely arranged and the remaining pixels are assigned to non-parallax pixels as much as possible It is an element. Both the non-parallax pixel and the parallax pixel have a Bayer arrangement as a basic structure, and have a structure in which R: G: B = 1: 2: 1 color filters are arranged in both the left parallax pixel and the right parallax pixel. That is, the resolution of an intermediate image without parallax captured by the original signal is more important than Embodiment 1, and a high-resolution three-dimensional image is obtained by superimposing the high-frequency component on the left parallax pixel and the right parallax pixel by parallax modulation. Try to get. Therefore, the color / parallax arrangement has the ability to obtain a high-resolution 2D image and 3D image even in the out-of-focus region.
The procedure of image processing is the same as in the first embodiment. Hereinafter, it demonstrates in order.

1) Color / parallax multiplexed mosaic image data input Single plate mosaic image with color and parallax multiplexed as shown in FIG. 9: M (x, y)
It is assumed that the gradation is a linear gradation output by A / D conversion.

For convenience, in the mosaic image M (x, y),
R _{N_mosaic} (x, y)
R _{Lt_mosaic} (x, y), the signal surface of the left parallax pixel of R component,
R _{Rt_mosaic} (x, y), the signal surface of the right parallax pixel of the R component,
G _{N_mosaic} (x, y), the signal plane of the left parallax pixel of G component,
G _{Lt_mosaic} (x, y)
G _{Rt_mosaic} (x, y), the signal plane of the right parallax pixel of G component,
B _{N_mosaic} (x, y)
_Let B _{Lt_mosaic} (x, y) be the signal surface of the left parallax pixel of the B component,
B _{Rt_mosaic} (x, y)
It will be expressed as

When all the non-parallax pixels have a full-opening mask, an arithmetic average method is adopted. When all non-parallax pixels have a half-aperture mask, the geometric average method is adopted. Therefore, the arithmetic average type is adopted in the present embodiment. In this way, a mosaic image in which the non-parallax pixel is corrected with one gain coefficient, the left parallax pixel with one gain coefficient, and the right parallax pixel with one gain coefficient is output as M ′ (x, y).

3) Generation of temporary parallax image A temporary left parallax image and a temporary right parallax image with low spatial frequency resolution are generated. Performs simple average interpolation in the G color plane, where only the left parallax pixels are collected. Linear interpolation is performed according to the distance ratio using pixel values that are close to each other. Similarly, simple average interpolation in the G color plane in which only the right parallax pixels are collected is performed. Similarly, simple average interpolation in the G color plane in which only non-parallax pixels are collected is performed. Similar processing is performed for each of R, G, and B. That is, R _{Lt_mosaic} (x, y) to R _Lt (x, y), R _{Rt_mosaic} (x, y) to R _Rt (x, y), R _{N_mosaic} (x, y) to R _N (x, y) ), G _{Lt_mosaic} (x, y) to G _Lt (x, y), G _{Rt_mosaic} (x, y) to G _Rt (x, y), G _{N_mosaic} (x, y) to G _N (x, y) y), B _{Lt_mosaic} (x, y) to B _Lt (x, y), B _{Rt_mosaic} (x, y) to G _Rt (x, y), B _{N_mosaic} (x, y) to G _N (x , y).
Temporary R component-free parallax image: R _N (x, y)
Temporary G component parallax-free image: G _N (x, y)
Temporary B component non-parallax image: B _N (x, y)
Temporary R component left parallax image: R _Lt (x, y)
Temporary G component left parallax image: G _Lt (x, y)
Temporary B component left parallax image: B _Lt (x, y)
Temporary R component right parallax image: R _Rt (x, y)
Temporary G component right parallax image: G _Rt (x, y)
Temporary B component right parallax image: B _Rt (x, y)
When creating temporary non-parallax images R _N (x, y), G _N (x, y), and B _N (x, y), high-definition is performed by introducing direction determination in the signal plane. May be.

4) Generation of parallax-free color mosaic image by correcting left and right illumination distribution (local gain balance correction)
Next, by performing local gain correction in units of pixels in the same manner as the global gain correction performed in step 1, first, the illuminances of the left parallax pixel in the screen and the right parallax pixel in the screen are matched. This operation eliminates the parallax between the left and right. Then, the illuminance is further adjusted between the signal plane obtained by taking the left-right average and the imaging signal plane of the non-parallax pixel. Then, a new Bayer surface with gain matching is created for all pixels. This is equivalent to replacing with an average value, and a Bayer surface in which the parallax disappears is created. This is written as M _N (x, y).

  In this case as well, there are two types of methods for setting the target value to be aligned as the reference point for each pixel: a method for selecting the arithmetic mean and a method for selecting the geometric mean as the method for eliminating the parallax between the left and right. When all non-parallax pixels have the mask area of the entire aperture, it is necessary to select an arithmetic average type in order to make the blur width of the subject image whose parallax disappeared between the left and right coincide with the blur width of the entire aperture. On the other hand, when all the non-parallax pixels have a half-aperture mask area, it is necessary to select a geometric average type in order to make the blur width of the subject image whose parallax disappeared between the left and right coincide with the blur width of the half aperture .

  In addition, the operation of averaging between the signal plane where the parallax disappeared between the left and right and the imaging signal plane of the non-parallax pixel is already aligned with the subject image of the same blur width, so the blur width is saved. There is a need to. Therefore, at this time, a common geometric average must be taken. At that time, a geometric average is taken in consideration of the density ratio between the non-parallax pixel and the parallax pixel in the image sensor array. That is, the ratio of the non-parallax pixel (N), the left parallax pixel (Lt), and the right parallax pixel (Rt) used in the second embodiment is N: L: R = 14: 1: 1, that is, N: (L Since + R) = 7: 1, a 7 / 8th power is given to non-parallax pixels and a 1 / 8th power is given to parallax pixels, giving a distribution that emphasizes high density non-parallax pixels. . That is, the reference image data generation unit 232 performs weighting according to the number of non-parallax pixels, left parallax pixels, and right parallax pixels. The specific formulas are given below.

In this way, the Bayer plane data is rewritten by using the pixel value obtained by averaging the average value of the left viewpoint image and the right viewpoint image with the reference viewpoint image having no parallax as a new non-parallax pixel value. None Outputs the Bayer plane image M _N (x, y).

5) Generation of reference image without parallax The same as in the first embodiment.
6) Edge enhancement processing for reference image without parallax The same as in the first embodiment.

7) Real parallax image generation Temporary left parallax color images R _Lt (x, y), G _Lt (x, y), B _Lt (x, y) generated in step 3 and in step 5 Using the high-resolution color images R _N (x, y), G _N (x, y), B _N (x, y) with high resolving power generated as intermediate processing, Color images R ′ _Lt (x, y), G ′ _Lt (x, y), and B ′ _Lt (x, y) are generated. Similarly, the provisional right parallax color images R _Rt (x, y), G _Rt (x, y), B _Rt (x, y) generated in step 3 and intermediate processing in step 5 are generated. color image R _N without high resolution parallax _{(x, y), G N} (x, y), using B _N (x, y), color image actually output to high right parallax resolving power R _'Rt (x, y), G ′ _Rt (x, y), and B ′ _Rt (x, y) are generated.

  There are two possible parallax modulation methods: a method using an arithmetic average as a reference point and a method using a geometric average as a reference point. Both can obtain a parallax modulation effect, but when the aperture mask of pixels without parallax of the image sensor is full aperture, a method using an arithmetic average as a reference point is adopted, and the aperture mask of pixels without parallax is a pixel with parallax. A method using the geometric mean as the reference point for the same half aperture is adopted. Therefore, in this embodiment, a method using an arithmetic mean as a reference point is used.

  When parallax modulation is performed, a geometric average is taken in consideration of the RGB density ratio between parallax pixels in the image sensor array. That is, R: G: B = 1: 2: 1 between the left parallax pixels and R: G: B = 1: 2: 1 between the right parallax pixels, so that the parallax modulation by the R component Giving a weight of 1/4 to the power, a weight of 1/2 to the parallax modulation by the G component, and a weight of 1/4 to the parallax modulation by the B component, emphasizing the parallax modulation by the dense G component Take distribution.

8) Conversion to output color space The same as in the first embodiment.

<Embodiment 3>
---- Monochromatic sparse parallax pixel array, edge enhancement ---
An example in which imaging elements arranged periodically are used with the upper arrangement diagram of FIG. 10 as a basic lattice. The frequency resolution area of the reciprocal lattice space is also shown for each parallax combination. This arrangement captures the property that parallax occurs only in the blurred subject area of the monocular pupil division method, and has a structure in which the density of parallax pixels is sparsely arranged and the remaining pixels are assigned to non-parallax pixels as much as possible. It is an image sensor.

The image processing procedure is approximately as follows.
1) Input of parallax multiplexed mosaic image data 2) Global gain balance correction of parallax mosaic image 3) Generation of temporary parallax image 4) Generation of parallax-free reference image by left and right local illuminance distribution correction (local gain balance correction)
5) Generation of reference image without parallax 6) Edge enhancement processing for reference image without parallax 7) Generation of actual parallax image 8) Conversion to output space

  When all the non-parallax pixels have a full-opening mask, an arithmetic average method is adopted. When all non-parallax pixels have a half-aperture mask, the geometric average method is adopted. Therefore, the arithmetic average type is adopted in the present embodiment. In this way, a mosaic image in which the non-parallax pixel is corrected with one gain coefficient, the left parallax pixel with one gain coefficient, and the right parallax pixel with one gain coefficient is output as M ′ (x, y).

3) Generation of temporary parallax image A temporary left parallax image and a temporary right parallax image with low spatial frequency resolution are generated. Simple average interpolation is performed in the signal plane that collects only the left parallax pixels. Linear interpolation is performed according to the distance ratio using pixel values that are close to each other. Similarly, simple average interpolation is performed in the signal plane in which only the right parallax pixels are collected. Similarly, simple average interpolation is performed in the signal plane in which only non-parallax pixels are collected. That is, generation Lt _mosaic (x, y) from Lt (x, y) and, Rt _mosaic (x, y) from the Rt (x, y) and, N _mosaic (x, y) from the N (x, y) and To do.
Temporary non-parallax image: N (x, y)
Temporary left parallax image: Lt (x, y)
Temporary right parallax image: Rt (x, y)
Note that when creating the temporary non-parallax image N (x, y), high-definition may be performed by introducing direction determination in the signal plane.

4) Generation of parallax-free reference images by correcting left and right illuminance distribution (local gain balance correction)
Next, by performing local gain correction in units of pixels in the same manner as the global gain correction performed in step 1, first, the illuminances of the left parallax pixel in the screen and the right parallax pixel in the screen are matched. This operation eliminates the parallax between the left and right. Then, the illuminance is further adjusted between the signal plane obtained by taking the left-right average and the imaging signal plane of the non-parallax pixel. In this way, a new parallax-free reference image plane in which gain matching is achieved in all pixels is created. This is equivalent to replacing with an average value, and an intermediate image plane in which the parallax disappears is completed. This is written as N (x, y).

  Also at this time, a geometric average is taken in consideration of the density ratio between the non-parallax pixels and the parallax pixels in the image sensor array. That is, the ratio of the non-parallax pixel (N), the left parallax pixel (Lt), and the right parallax pixel (Rt) used in the third embodiment is N: L: R = 14: 1: 1, that is, N: (L Since + R) = 7: 1, the 7 / 8th power is given to the parallax pixels, and the 1 / 8th power is given to the non-parallax pixels, giving a distribution that emphasizes the high density non-parallax pixels. .

  In this way, the pixel value obtained by taking the average value of the average value of the left viewpoint image and the right viewpoint image and the reference viewpoint image without parallax as a new pixel value without parallax is rewritten, and the parallax is rewritten. None Output monochrome image N (x, y).

5) Generation of reference image without parallax The same as in the first embodiment.

6) Edge enhancement processing for reference image without parallax It is only necessary to perform the calculation in which symbol Y in step 6-2) of Embodiment 1 is replaced with symbol N. That is,
N '(x, y) = N (x, y) + k * △ N (x, y)
In the following, the symbol N ′ is replaced with the symbol N.

7) Generation of actual parallax image Temporary left parallax image Lt (x, y) generated in step 3 and low-resolution monochrome image N (x, y) generated as intermediate processing in step 5 Is used to generate a left-parallax monochrome image Lt ′ (x, y) with high resolving power that is actually output. Similarly, using the temporary right parallax image Rt (x, y) generated in step 3 with low resolution and the monochrome image N (x, y) without parallax generated as intermediate processing in step 5 with actual resolution, To generate a right-parallax color image Rt ′ (x, y) with high resolving power.

  There are two possible parallax modulation methods: a method using an arithmetic average as a reference point and a method using a geometric average as a reference point. Both can obtain a parallax modulation effect, but when the aperture mask of pixels without parallax of the image sensor is full aperture, a method using an arithmetic average as a reference point is adopted, and the aperture mask of pixels without parallax is a pixel with parallax. A method using the geometric mean as the reference point for the same half aperture is adopted. Therefore, in this embodiment, a method using an arithmetic mean as a reference point is used.

7) Conversion to output color space The high-resolution non-parallax intermediate monochrome image N (x, y) and the high-resolution left-parallax monochrome image Lt '(x, y) obtained in this way Each right-parallax monochrome image Rt ′ (x, y) is subjected to appropriate gamma conversion and output as an output space image. As for these, all 2D and 3D images are generated with high-definition edge enhancement.

<Embodiment 4>
--- Bayer type RGB sparse parallax pixel array, noise removal ---
The image processing procedure is approximately as follows.
1) Color / parallax multiplexed mosaic image data input 2) Global gain balance correction of color / parallax mosaic image 3) Temporary parallax image generation 4) Generation of parallax-free color mosaic image by right and left local illuminance distribution correction (Local・ Gain balance correction
5) Generation of reference image without parallax 6) Noise removal processing for reference image without parallax 7) Generation of actual parallax image 8) Conversion to output color space Here, steps 1) to 6) and steps 7) to 8 are performed. ) Is the same as that of Embodiment 2, and the description thereof is omitted. Here, the noise removal process for the reference image without parallax will be described.

6) Noise removal processing for reference image without parallax 6-1) Color space conversion processing The same as in the first embodiment.
6-2) Noise removal processing Noise removal processing is performed on the luminance Y plane. A known high-performance noise removal process such as Japanese Patent Laid-Open No. 2006-309749 of the same inventor as the above-mentioned applicant may be used. Here, a two-argument product bilateral filter disclosed in WO2006 / 068025 of the same inventor as the present applicant is shown.
Here, σth is the noise fluctuation width. rth is a filter radius and can be set to an arbitrary size according to the spread width of the target noise. Note that the filter processing unit 233 may not perform noise removal filtering processing on the left parallax image data and the right parallax image data.

6-3) Inverse color space conversion processing Using the luminance component Y '(x, y) and the color difference components Cb' (x, y) and Cr '(x, y) from which noise has been removed, the YCbCr space is converted into the RGB space. return.
6-4) Transition to the original linear gradation space by inverse gradation conversion The same as in the first embodiment.

<Notes>
When the edge enhancement of the second embodiment and the noise removal of the fourth embodiment are used in combination, the edge enhancement processing of step 6 of the second embodiment may be performed after the noise removal processing of step 6 of the fourth embodiment is performed first. . When performing noise removal processing on the sparse monochrome parallax pixel array of the third embodiment, the same procedure as that performed for the luminance component of the noise removal processing shown in the fourth embodiment is applied to the monochrome surface. You can do this.

  In the above description, three primary colors, red, green and blue, are used as the primary colors constituting the color of the subject image. However, four or more colors including amber color may be used as primary colors. Instead of red, green, and blue, three primary colors that are complementary colors of yellow, magenta, and cyan can be used.

  The functions of the interpolation processing unit 231, the reference image data generation unit 232, the filter processing unit 233, and the parallax image data generation unit 234 described above include an interpolation processing step, a reference image data acquisition step, a filter processing step, and a parallax image data generation step. This can be realized by causing a computer to execute an image processing program including the image processing program. In the interpolation processing step, temporary left parallax image data for the left viewpoint and temporary right parallax image data for the right viewpoint are generated based on the output of the image sensor 100. The reference image data acquisition step generates reference image data having higher resolution than the temporary left parallax image data and the temporary right parallax image data using the pixel values of the temporary left parallax image data and the temporary right parallax image data. To do. In the filter processing step, at least one of edge adjustment and noise removal is applied to the reference image data. The parallax image data generation step is higher than the temporary left parallax image data and the temporary right parallax image data using the reference image data subjected to the filtering process, the temporary left parallax image data, and the temporary right parallax image data. Resolution left parallax image data and high resolution right parallax image data are generated.

  A device such as a personal computer may function as the image processing apparatus. The image processing apparatus may capture image data from another apparatus such as a camera. In this case, it plays a role as a parallax image data acquisition unit rather than an interpolation processing unit. It plays a role as a reference image data acquisition unit rather than a reference image data generation unit. In addition, even when the interpolation processing unit generates the parallax image data and the reference image data generation unit generates the reference image data, it can be said that the parallax image data and the reference image data are acquired by generating the parallax image data. Further, the reference image data generation unit 232 uses the pixel values of the temporary reference image data in which some pixels are missing in addition to the pixel values of the pixel values of the temporary left parallax image data and the temporary right parallax image data. Thus, reference image data that is a 2D intermediate image may be generated.

  In the above description, the filter processing unit 233 performs the edge enhancement process as the edge adjustment process on the reference image data. However, for example, when the image is reduced, an edge suppression process may be performed. In the above description, an image may refer to image data, or may refer to a subject image itself developed and visualized according to a format.

In the above description, the parallax image data generation unit 234 generates left parallax image data corresponding to the same viewpoint as the temporary left parallax image data. Similarly, the parallax image data generation unit 234 generates right parallax image data corresponding to the same viewpoint as the temporary right parallax image data. In this case, since the viewpoints of the left and right parallax pixels do not change, the parallax amounts of the temporary left parallax image data and the temporary right parallax image data, and the finally generated high-resolution left parallax image data and high The parallax amount of the resolved right parallax image data is the same. However, the amount of parallax can be controlled by parameterizing the amount of modulation when performing parallax modulation. When the amount of parallax changes, the viewpoint of the left parallax image data and the viewpoint of the left parallax image data also change. Therefore, it can also be said that the parallax image data generation unit 234 generates left parallax image data with a viewpoint different from the viewpoint of the temporary left parallax image data. Similarly, it can also be said that right parallax image data with a different viewpoint from the viewpoint of temporary right parallax image data is generated. As shown below, the stereoscopic effect can be made variable by the parameter C.

  FIG. 19 is a diagram illustrating an example of pixel value interpolation. The upper part of FIG. 19 is a diagram in which only pixel values of R non-parallax pixels in the pixel array shown in FIG. 7 are extracted. Using the pixel values of pixels having pixel values in the R0 image data, the pixel values of pixels having no pixel values are interpolated as shown in the lower part of FIG. That is, the parallax image data generation unit 234 generates 2D intermediate image data by interpolating the missing pixel values using the pixel values of the non-parallax pixels. For example, the pixel values for interpolation are averaged from the top and bottom, left and right, or the top and bottom and left and right adjacent pixels having pixel values.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention. For example, when the above-described image sensor is rotated by 90 °, it is a modified example of vertical parallax, and when it is rotated by 45 °, a modified example of oblique parallax is obtained. Further, left and right parallax may be set for a honeycomb arrangement in which the pixel arrangement is not arranged in a square lattice pattern.

  The execution order of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior”. It should be noted that they can be implemented in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for the sake of convenience, it means that it is essential to carry out in this order. is not.

DESCRIPTION OF SYMBOLS 10 Digital camera, 20 Shooting lens, 21 Optical axis, 100 Image sensor, 101 Micro lens, 102 Color filter, 103 Aperture mask, 104 Aperture, 105 Wiring layer, 106 Wiring, 107 Aperture, 108 Photoelectric conversion element, 109 Substrate 110 repetitive pattern, 201 control unit, 202 A / D conversion circuit, 203 memory, 204 drive unit, 205 image processing unit, 207 memory card IF, 208 operation unit, 209 display unit, 210 LCD drive circuit, 211 AF sensor, 220 Memory card, 231 interpolation processing unit, 232 reference image data generation unit, 233 filter processing unit, 234 parallax image data generation unit, 322 center line, 1801 distribution curve, 1802 distribution curve, 1803 distribution curve, 1804 distribution curve, 1805 distribution Curve, 1806 composite distribution curve, 1807 distribution curve, 1808 distribution curve, 1809 composite distribution curve

Claims (17)

Left parallax comprising a plurality of pixels having one aperture mask in one pixel and having a parallax-free pixel having an aperture mask that generates a viewpoint in the reference direction and an aperture mask that generates parallax in the left direction with respect to the reference direction A subject image is referenced through one optical system using an image sensor comprising a pixel array and at least three types of pixels, a right parallax pixel having an aperture mask that produces a parallax in the right direction with respect to a reference direction. Input a first image in which a directional viewpoint, a left viewpoint, and a right viewpoint are simultaneously captured on different pixels,
An image processing method for converting an image related to a left viewpoint and an image related to a right viewpoint,
Using the pixel value of the left parallax pixel of the first image to generate a temporary left viewpoint image for each pixel;
Using the pixel value of the right parallax pixel of the first image to generate a temporary right viewpoint image for each pixel;
Generating a reference viewpoint image for each pixel using a pixel value of at least a non-parallax pixel of the first image;
A step of performing edge enhancement processing on the reference viewpoint image to generate an edge-enhanced reference viewpoint image;
And a procedure for generating, for each pixel, an image relating to the left viewpoint and an image relating to the right viewpoint based on the edge-enhanced reference viewpoint image, the provisional left viewpoint image, and the provisional right viewpoint image. An image processing method characterized by the above. Left parallax comprising a plurality of pixels having one aperture mask in one pixel and having a parallax-free pixel having an aperture mask that generates a viewpoint in the reference direction and an aperture mask that generates parallax in the left direction with respect to the reference direction A subject image is referenced through one optical system using an image sensor comprising a pixel array and at least three types of pixels, a right parallax pixel having an aperture mask that produces a parallax in the right direction with respect to a reference direction. Input a first image in which a directional viewpoint, a left viewpoint, and a right viewpoint are simultaneously captured on different pixels,
An image processing method for converting an image related to a left viewpoint and an image related to a right viewpoint,
Using the pixel value of the left parallax pixel of the first image to generate a temporary left viewpoint image for each pixel;
Using the pixel value of the right parallax pixel of the first image to generate a temporary right viewpoint image for each pixel;
Using a pixel value of a non-parallax pixel of the first image to generate a reference viewpoint image for each pixel;
A procedure for performing noise removal processing on the reference viewpoint image to generate a noise-removed reference viewpoint image;
A procedure for generating an image related to the left viewpoint and an image related to the right viewpoint for each pixel based on the reference viewpoint image from which the noise has been removed, the temporary left viewpoint image, and the temporary right viewpoint image. An image processing method characterized by the above.   The procedure for generating the reference viewpoint image includes the pixel value of the reference viewpoint image using the pixel value of the left parallax pixel and the pixel value of the right parallax pixel in addition to the pixel value of the non-parallax pixel of the first image. The image processing method according to claim 1, wherein the image processing method is generated. When the first image is composed of a plurality of color components,
The procedure of generating the reference viewpoint image generates a reference viewpoint image for the plurality of color components, and the procedure of generating the edge-enhanced reference viewpoint image includes an edge for the luminance component of the reference viewpoint image. The image processing method according to claim 1, wherein enhancement processing is performed. When the first image is composed of a plurality of color components,
The procedure of generating the reference viewpoint image generates a reference viewpoint image for the plurality of color components, and the procedure of generating the noise-removed reference viewpoint image includes a luminance component and a color difference component of the reference viewpoint image. The image processing method according to claim 2, wherein noise removal processing is performed on the image processing.   3. The image according to claim 2, wherein the first image is an image picked up by an image pickup element having a pixel arrangement in which a density of pixels without parallax is higher than a density of pixels including a left parallax pixel and a right parallax pixel. Image processing method. First parallax image data corresponding to a viewpoint shifted in a first direction with respect to a reference direction, and a second parallax image corresponding to a viewpoint shifted in a second direction opposite to the first direction with respect to the reference direction Parallax image data acquisition unit for acquiring data;
A reference image data acquisition unit for acquiring reference image data corresponding to the reference direction and having higher resolution than the spatial frequency resolution of the first parallax image data and the second parallax image data;
A filter processing unit that performs at least one of edge adjustment and noise removal filtering on the reference image data;
Using the reference image data, the first parallax image data, and the second parallax image data on which the filtering process has been performed, third parallax image data corresponding to a viewpoint shifted in the first direction, and the second parallax image data An image processing apparatus comprising: a parallax image data generation unit that generates fourth parallax image data corresponding to a viewpoint shifted in a direction.   The reference image data acquisition unit acquires temporary reference image data in which some pixel values are missing, and interpolates the missing pixel values using pixel values of reference pixels corresponding to the reference direction, thereby The image processing apparatus according to claim 7, wherein the image processing apparatus generates reference image data.   The said reference image data acquisition part produces | generates the said reference image data by interpolating the missing pixel value using the pixel value of said 1st parallax image data and said 2nd parallax image data. The image processing apparatus described.   The reference image data acquisition unit weights according to the number of reference pixels corresponding to the reference image data, first pixels corresponding to the first parallax image data, and second pixels corresponding to the second parallax image data. The image processing apparatus according to claim 9.   The image processing apparatus according to claim 7, wherein the reference image data acquisition unit generates the reference image data for each color component of a color filter associated with each pixel.   The parallax image data generation unit generates the third parallax image data with a viewpoint different from the viewpoint of the first parallax image data, and the fourth parallax image data with a viewpoint different from the viewpoint of the second parallax image data. The image processing device according to claim 7, wherein the image processing device generates the image. An image sensor that outputs at least one of the first parallax image data and the second parallax image data;
An imaging device comprising: the image processing device according to claim 7.   The imaging apparatus according to claim 13, wherein the imaging element outputs the reference image data together.   The imaging device selects the pixel corresponding to the reference image data among the pixel corresponding to the reference image data, the pixel corresponding to the first parallax image data, and the pixel corresponding to the second parallax image data. The imaging device according to claim 14, which has a large amount.   The imaging device according to claim 15, wherein the filter processing unit does not perform a noise removal filtering process on the first parallax image data and the second parallax image data. First parallax image data corresponding to a viewpoint shifted in a first direction with respect to a reference direction, and a second parallax image corresponding to a viewpoint shifted in a second direction opposite to the first direction with respect to the reference direction Parallax image data acquisition step for acquiring data;
A reference image data acquisition step for acquiring reference image data corresponding to the reference direction and having higher resolution than the spatial frequency resolution of the first parallax image data and the second parallax image data;
A filtering process step of applying at least one of edge adjustment and noise removal filtering to the reference image data;
Using the reference image data, the first parallax image data, and the second parallax image data on which the filtering process has been performed, third parallax image data corresponding to a viewpoint shifted in the first direction, and the second parallax image data An image processing program for causing a computer to execute a parallax image data generation step for generating fourth parallax image data corresponding to a viewpoint shifted in a direction.