JP2014107594A - Image pick-up device and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pick-up device and an imaging apparatus, capable of resolving a problem of inability to produce natural blur because of asymmetric blur produced in horizontal and vertical directions in each of left and right parallax images.SOLUTION: An image pick-up device includes a pixel array having at least three types of pixels containing: a non-disparity pixel having an opening mask for achieving a viewpoint in a reference direction; a left disparity pixel having an opening mask for achieving a left direction viewpoint; and a right disparity pixel having an opening mask for achieving a right direction viewpoint, with regard to one optical system incoming light flux. The vertical open width of the opening mask of the non-disparity pixel is narrower than the horizontal open width.

Description

  The present invention relates to an imaging element and an imaging apparatus.

2. Description of the Related Art There is known an imaging apparatus that generates left and right parallax images having parallax with a single shooting using a single shooting optical system. In the imaging apparatus, the right half or the left half of all the pixels arranged in the imaging element is shielded from light by the light shielding member.
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP 2003-7994 A [Patent Document 2] JP 2011-197278 A

  The imaging device may include pixels (non-parallax pixels) that are not shielded by the light shielding member in addition to pixels (parallax pixels) that are shielded by the light shielding member. When parallax is generated on the left and right, in the parallax pixels, the horizontal width of the light receiving region that receives the incident light beam limited by the light blocking member is often shorter than the vertical width. On the other hand, in non-parallax pixels, the horizontal width of the light receiving region that receives an incident light beam that is not limited by the light shielding member is often the same as the vertical width. As described above, when the shape of the light receiving region is different between the parallax pixel and the non-parallax pixel, there is a problem that the blurring is different between the left and right parallax images and the non-parallax image.

  The imaging device according to the first aspect of the present invention includes a non-parallax pixel including an aperture mask that generates a viewpoint in the reference direction and an aperture mask that generates a viewpoint in the left direction with respect to an incident light beam of one optical system. An imaging device comprising a pixel array having at least three types of pixels, that is, a left parallax pixel and a right parallax pixel having an aperture mask that generates a right viewpoint, and a vertical aperture width of an aperture mask of a pixel with no parallax is a horizontal aperture Narrower than width.

  The imaging device according to the second aspect of the present invention includes a microlens that generates a viewpoint in a reference direction and a non-parallax pixel including an aperture mask, and a microlens that generates a viewpoint in the left direction with respect to an incident light beam of one optical system. And an imaging device having a pixel array having at least three types of pixels: a left parallax pixel having an aperture mask, a microlens that generates a right viewpoint, and a right parallax pixel having an aperture mask. The condensing characteristic of the microlens is deformed into an anisotropic shape so that the vertical direction is less than the horizontal direction.

  An imaging device according to a third aspect of the present invention includes the above-described imaging device and one optical system including a circular diaphragm in the middle of the optical path.

  The imaging device according to the fourth aspect of the present invention receives a deviated pixel in which a first light receiving region for receiving a subject luminous flux is set at a position displaced in the first axis direction with respect to the pixel center, and the subject luminous flux. The second light receiving area includes non-deviation pixels set at positions not deviated with respect to the pixel center, and the first area width in the first axis direction of the first light receiving area is the first axis direction. The first region width in the first axis direction in the second light receiving region is longer than the second region width in the second axis direction.

  In the imaging device according to the fifth aspect of the present invention, the first light receiving region for receiving the subject light beam condensed by the first microlens is set at a position displaced in the first axis direction with respect to the pixel center. A displacement pixel, and a second light-receiving region that receives the subject light beam collected by the second microlens is provided with a non-deviation pixel that is set at a position that is not displaced with respect to the pixel center. The first region width in the first axis direction in the light receiving region is shorter than the second region width in the second axis direction orthogonal to the first axis direction, and the incident light amount in the direction corresponding to the first axis direction of the second microlens. Is greater than the amount of incident light in the direction corresponding to the second axis direction orthogonal to the first axis direction.

  An image pickup apparatus according to a sixth aspect of the present invention includes the above-described image pickup element and one optical system having a circular diaphragm that adjusts a subject light beam guided to the image pickup element.

It is a figure explaining the structure of the digital camera which concerns on embodiment of this invention. It is a figure explaining the concept of defocus in a pixel without parallax. It is a figure explaining the concept of defocus in a parallax pixel. It is a figure which shows the light intensity distribution of a parallax pixel and a parallax pixel. It is a figure explaining the opening shape of the opening part in case there are two types of parallax pixels. It is a figure for demonstrating the asymmetry of a blur. It is a figure which shows the relationship between a parallax image and an image without parallax, and a depth of field. It is a figure explaining the structure of a pixel without parallax. 2 is a diagram for explaining a configuration of an image sensor 100. FIG. 2 is a diagram for explaining a configuration of an image sensor 100. FIG. It is a figure which shows the comparative example of a pixel arrangement | sequence. It is a figure which shows an example of a pixel arrangement | sequence. It is a figure which shows the variation of a pixel arrangement | sequence. It is a figure which shows the variation of a pixel arrangement | sequence. It is a figure for demonstrating the variation of an opening mask.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  The digital camera according to the present embodiment, which is one form of the imaging device, is configured to generate images with a plurality of viewpoints for one scene by one shooting. Each image having a different viewpoint is called a parallax image. In the present embodiment, a case where a right parallax image and a left parallax image from two viewpoints corresponding to the right eye and the left eye are generated will be described. The digital camera according to the present embodiment can generate a parallax-free image with no parallax from the central viewpoint as the viewpoint in the reference direction together with the parallax image.

  FIG. 1 is a diagram illustrating the configuration of a digital camera 10 according to an embodiment of the present invention. The digital camera 10 includes a photographic lens 20 as a photographic optical system, and guides a subject light beam incident along the optical axis 21 to the image sensor 100. The photographing lens 20 may be an interchangeable lens that can be attached to and detached from the digital camera 10. The digital camera 10 includes an image sensor 100, a control unit 201, an A / D conversion circuit 202, a memory 203, a drive unit 204, an image processing unit 205, a memory card IF 207, an operation unit 208, a display unit 209, and an LCD drive circuit 210. .

  As shown in the figure, the direction parallel to the optical axis 21 toward the image sensor 100 is defined as the Z-axis plus direction, the direction toward the front of the drawing on the plane orthogonal to the Z-axis is the X-axis plus direction, and the upward direction on the drawing is Y. The axis is defined as the plus direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

  The taking lens 20 is composed of a plurality of optical lens groups, and forms an image of a subject light flux from the scene in the vicinity of its focal plane. In FIG. 1, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens arranged in the vicinity of the pupil. The diaphragm 22 is a circular diaphragm, and is disposed in the vicinity of the pupil along the optical axis 21.

  The image sensor 100 is disposed near the focal plane of the photographic lens 20. The image sensor 100 is an image sensor such as a CCD or CMOS sensor in which a plurality of photoelectric conversion elements are two-dimensionally arranged. The image sensor 100 is controlled in timing by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202. The image sensor 100 includes parallax pixels and non-parallax pixels. The parallax pixel is a pixel that is set at a position where a light receiving region that receives a subject light flux is displaced in the X-axis direction with respect to the pixel center. Focusing on the fact that the light receiving area is deviated, the parallax pixel can also be referred to as a deviated pixel. On the other hand, the non-parallax pixel is a pixel set at a position where the light receiving region is not deviated from the pixel center. Focusing on the fact that the light receiving area is not displaced, pixels without parallax can also be referred to as non-deviation pixels. The parallax pixel at the left viewpoint may be referred to as the parallax Lt pixel, the parallax pixel at the right viewpoint as the parallax Rt pixel, and the non-parallax pixel as the N pixel. In some cases, the parallax image of the left viewpoint is referred to as a parallax Lt image, the parallax image of the right viewpoint is referred to as a parallax Rt image, and an image without parallax is referred to as an N image.

  The A / D conversion circuit 202 converts the image signal output from the image sensor 100 into a digital image signal and outputs the digital image signal to the memory 203. The image processing unit 205 performs various image processing using the memory 203 as a work space, and generates image data.

  The image processing unit 205 also has general image processing functions such as adjusting image data in accordance with the selected image format. The generated image data is converted into a display signal by the LCD drive circuit 210 and displayed on the display unit 209. The data is recorded on the memory card 220 attached to the memory card IF 207.

  The control unit 201 controls the digital camera 10 in an integrated manner. For example, the aperture of the diaphragm 22 is adjusted according to the set diaphragm value, and the photographing lens 20 is advanced and retracted in the optical axis direction according to the AF evaluation value. Further, the position of the photographing lens 20 is detected, and the focal length and the focus lens position of the photographing lens 20 are grasped. Further, a timing control signal is transmitted to the driving unit 204, and a series of imaging control until the image signal output from the imaging element 100 is processed into image data by the image processing unit 205 is managed.

  The operation unit 208 functions as a part of a reception unit that receives a user operation and transmits an instruction to the control unit 201. The operation unit 208 includes a plurality of operation members such as a shutter button that receives a shooting start instruction.

<Parallax pixel and blur characteristics>
Next, the concept of defocusing when the parallax Lt pixel and the parallax Rt pixel receive light will be described. First, it is a figure explaining simply the concept of defocus in a pixel without parallax. FIG. 2 is a diagram for explaining the concept of defocusing in a non-parallax pixel. As shown in FIG. 2A, when an object point that is a subject exists at the focal position, the subject luminous flux that reaches the image sensor light receiving surface through the lens pupil is steep with the pixel at the corresponding image point as the center. The light intensity distribution is shown. That is, if non-parallax pixels that receive the entire effective luminous flux that passes through the lens pupil are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity Drops rapidly.

  On the other hand, as shown in FIG. 2B, when the object point deviates from the focal position in the direction away from the light receiving surface of the image sensor, the subject luminous flux is received by the image sensor compared to the case where the object point exists at the focus position. It shows a gentle light intensity distribution on the surface. That is, the output value at the pixel of the corresponding image point is lowered, and a distribution having output values up to the peripheral pixels is shown.

  As shown in FIG. 2C, when the object point further deviates from the focal position, the subject luminous flux exhibits a gentler light intensity distribution on the image sensor light receiving surface. In other words, the output value at the pixel of the corresponding image point further decreases, and a distribution having output values up to the surrounding pixels is shown.

  As shown in FIG. 2D, when the object point is shifted from the focal position in the direction approaching the image sensor light receiving surface, the same light as when the object point is shifted in the direction away from the image sensor light receiving surface. The intensity distribution is shown.

  FIG. 3 is a diagram for explaining the concept of defocusing in the parallax pixels. The parallax Lt pixel and the parallax Rt pixel receive the subject luminous flux that arrives from one of the two parallax virtual pupils set as the optical axis target as a partial region of the lens pupil. In this specification, a method of capturing a parallax image by receiving subject light fluxes that arrive from different virtual pupils in a single lens pupil is referred to as a monocular pupil division imaging method.

  As shown in FIG. 3 (a), when an object point that is a subject exists at the focal position, steep light centering on the pixel of the corresponding image point, regardless of the parallax virtual pupil, regardless of the parallax virtual pupil. The intensity distribution is shown. If the parallax Lt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Further, even when the parallax Rt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. That is, even if the subject luminous flux passes through any parallax virtual pupil, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Match each other.

  On the other hand, as shown in FIG. 3B, when the object point deviates from the focal position in the direction away from the light receiving surface of the image sensor, the light intensity indicated by the parallax Lt pixel is greater than when the object point exists at the focal position. The distribution peak appears at a position away from the pixel corresponding to the image point in one direction, and its output value decreases. In addition, the width of the pixel having the output value is increased. That is, since the point image spreads in the horizontal direction of the light receiving surface of the image sensor, the amount of blur increases. The peak of the light intensity distribution indicated by the parallax Rt pixel appears at a position away from the pixel corresponding to the image point in the opposite direction to the one direction in the parallax Lt pixel and at an equal distance, and the output value similarly decreases. Similarly, the width of the pixel having the output value is increased. That is, the same light intensity distribution that is gentler than that in the case where the object point exists at the focal position appears at an equal distance from each other. The shift amount between the peaks of the light intensity distribution indicated by the parallax Lt pixel and the parallax Rt pixel corresponds to the parallax amount.

  Further, as shown in FIG. 3C, when the object point further deviates from the focal position, the same light intensity distribution that is more gentle than the state of FIG. 3B appears further apart. . Since the spread of the point image becomes larger, the amount of blur increases. In addition, since the deviation between the peaks of the light intensity distribution indicated by the parallax Lt pixel and the parallax Rt pixel is large, the parallax amount is also increased. In other words, it can be said that the more the object point deviates from the focal position, the more the amount of blur and the amount of parallax increase.

  As shown in FIG. 3D, when the object point deviates from the focal position in the direction approaching the light receiving surface of the image sensor, the light intensity indicated by the parallax Rt pixel is opposite to the state of FIG. The distribution peak appears at a position away from the pixel corresponding to the image point in the one direction. The peak of the light intensity distribution indicated by the parallax Lt pixel appears at a position away from the one direction in the parallax Rt pixel. That is, it is determined in which direction the peak of the light intensity distribution indicated by the parallax Lt pixel and the parallax Rt pixel appears in the direction away from the pixel corresponding to the image point according to the direction of deviation of the object point.

  When the change of the light intensity distribution explained in FIG. 2 and the change of the light intensity distribution explained in FIG. 3 are respectively graphed, they are expressed as shown in FIG. FIG. 4 is a diagram illustrating the light intensity distribution of the non-parallax pixel and the parallax pixel. In the figure, the horizontal axis represents the pixel position, and the center position is the pixel position corresponding to the image point. The vertical axis represents the output value of each pixel. Since this output value is substantially proportional to the light intensity, it is shown as the light intensity in the figure.

  As described above, when the object point deviates from the focal position in the direction approaching the image sensor light receiving surface, the same light intensity distribution as when the object point deviates in the direction away from the image sensor light receiving surface is shown. Therefore, in the figure, the change in the light intensity distribution when the image sensor is shifted in the direction approaching the light receiving surface of the image sensor is omitted. When the object point deviates in the direction away from the image sensor light receiving surface with respect to the peak of the light intensity distribution indicated by the parallax Lt pixel and the parallax Rt pixel when the object point deviates from the focal position in the direction approaching the image sensor light receiving surface. Since it is the same as the peak of the light intensity distribution indicated by the parallax Lt pixel and the parallax Rt pixel, it is omitted.

  FIG. 4A is a graph showing changes in the light intensity distribution described in FIG. A distribution curve 1801 represents the light intensity distribution corresponding to FIG. 2A and shows the steepest state. The distribution curve 1802 represents the light intensity distribution corresponding to FIG. 2B, and the distribution curve 1803 represents the light intensity distribution corresponding to FIG. Compared with the distribution curve 1801, it can be seen that the peak value gradually decreases and has a broadening.

  FIG. 4B is a graph showing changes in the light intensity distribution described in FIG. A distribution curve 1804 and a distribution curve 1805 represent the light intensity distribution of the parallax Lt pixel and the light intensity distribution of the parallax Rt pixel in FIG. As can be seen from the figure, these distributions have a line-symmetric shape with respect to the center position. Further, a combined distribution curve 1806 obtained by adding them shows a similar shape to the distribution curve 1802 in FIG. 2B, which is in the same defocused state as in FIG. 3B.

  A distribution curve 1807 and a distribution curve 1808 represent the light intensity distribution of the parallax Lt pixel and the light intensity distribution of the parallax Rt pixel in FIG. As can be seen from the figure, these distributions are also symmetrical with respect to the center position. Further, a combined distribution curve 1809 obtained by adding them shows a similar shape to the distribution curve 1803 in FIG. 2C which is in a defocus state equivalent to that in FIG. Note that the light intensity distribution of the parallax Lt pixel and the light intensity distribution of the parallax Rt pixel in FIG. 3D are interchanged with the positions of the light intensity distribution of the parallax Lt pixel and the light intensity distribution of the parallax Rt pixel in FIG. Therefore, they correspond to the distribution curve 1808 and the distribution curve 1807, respectively.

  FIG. 5 is a diagram illustrating the opening shape of the opening 104 when there are two types of parallax pixels. In FIG. 5A, the shape of the opening 104l of the parallax Lt pixel and the shape of the opening 104r of the parallax Rt pixel are obtained by dividing the shape of the opening 104n of the non-parallax pixel (N pixel) by the center line 322. The example which is the same as each shape is shown. That is, in FIG. 5A, the area of the opening 104n of the non-parallax pixel is the sum of the area of the opening 104l of the parallax Lt pixel and the area of the opening 104r of the parallax Rt pixel. In the present embodiment, the opening 104n of the non-parallax pixel is referred to as a full-opening opening, and the opening 104l and the opening 104r are referred to as half-opening openings. Here, the ratio of the horizontal direction (horizontal direction) and the vertical direction (vertical direction) of the paper surface is 1: 2 in the half opening. When the opening is located at the center of the photoelectric conversion element, the opening is said to be in the reference direction. The opening 104l of the parallax Lt pixel and the opening 104r of the parallax Rt pixel are offset in opposite directions with respect to a virtual center line 322 passing through the center (pixel center) of the corresponding photoelectric conversion element. . Accordingly, the opening 104l of the parallax Lt pixel and the opening 104r of the parallax Rt pixel each generate parallax in one direction with respect to the center line 322 and in the other direction opposite to the one direction.

  FIG. 5B shows the light intensity distribution when the object point is shifted from the focal position in the direction away from the light receiving surface of the image sensor in the pixel having each opening shown in FIG. In the figure, the horizontal axis represents the pixel position, and the center position is the pixel position corresponding to the image point. The curve Lt corresponds to the distribution curve 1804 in FIG. 5B, and the curve Rt corresponds to the distribution curve 1805 in FIG. A curve N corresponds to a pixel without parallax, and shows a similar shape to the combined distribution curve 1806 in FIG. Each of the openings 104n, 104l, and 104r functions as an aperture stop. Therefore, the blur width of the subject image captured by the non-parallax pixel having the opening 104n having an area double the opening 104l (opening 104r) is the parallax Lt pixel indicated by the composite distribution curve 1806 in FIG. And the blur width of the curve obtained by adding the parallax Rt pixels.

  FIG. 5C shows the light intensity distribution when the object point is deviated from the focal position in the direction approaching the image sensor light receiving surface in the pixel having each opening shown in FIG. In the figure, the horizontal axis represents the pixel position, and the center position is the pixel position corresponding to the image point. Curves Lt and Rt in FIG. 5C indicate that the blur width of the subject image captured by the non-parallax pixel having the opening 104n is approximately the same as the blur width of the curve obtained by adding the parallax Lt pixel and the parallax Rt pixel. While maintaining the relationship, the positional relationship is reversed with respect to the curve Lt and the curve Rt in FIG.

<Depth of field and asymmetric blur>
Next, the relationship between the depth of field and the blur asymmetry will be described. As is clear from FIGS. 5B and 5C, in the out-of-focus region, the blur width of the subject image captured by the parallax pixels is narrower than the blur width of the subject image captured by the non-parallax pixels. This means that the incident light flux of the lens is substantially reduced to the right half and the left half by the aperture mask of the parallax pixel in FIG. In other words, it can be said that two left and right virtual pupils are generated in a single lens pupil. That is, the aperture area in the aperture mask of the parallax pixel plays a role equivalent to the effect of the lens diaphragm.

  Generally, when the lens is squeezed, an image with a deep depth of field is captured. The opening of the opening mask in the parallax pixel is short in the horizontal direction and long in the vertical direction. Therefore, an image with a deep depth of field is captured for a subject having a frequency component in the horizontal direction such as a vertical line, while a shallow subject is captured for a subject having a frequency component in the vertical direction such as a horizontal line. An image with a depth of field is captured.

  FIG. 6 is a diagram for explaining blur asymmetry. For example, when a subject having a square patch as shown in FIG. 6A is imaged, a subject image as shown in FIG. 6A is obtained in the in-focus area. FIG. 6B shows the subject image captured by the left parallax pixel and the right parallax pixel together. In the out-of-focus area, a subject image in which the vertical line is less sharp and the horizontal line is sharper than the horizontal line as shown in FIG. 6B is captured. That is, since the opening of the opening mask in the parallax pixel is asymmetric in the horizontal direction and the vertical direction, the blur is asymmetric in the horizontal direction and the vertical direction of the subject image. This can also be referred to as blur anisotropy.

  If the subject image for the left eye and the subject image for the right eye in FIG. 6B are superimposed and displayed and a 2D image is obtained from the 3D image, the 2D image has a two-line blur caused by sharp blur in the horizontal direction. Such an unfavorable blur may occur (FIG. 6C).

  FIG. 7 is a diagram illustrating a relationship between the parallax image and the non-parallax image and the depth of field. Specifically, FIG. 7 shows a vertical stripe pattern chart when a stripe pattern chart of a subject image having a frequency of f [lines / mm] is captured with the pixel pitch of the image sensor 100 as a [mm]. FIG. 6 is a diagram illustrating subject distance dependency of MTF (Modulation Transfer Function) characteristics of a horizontal stripe pattern chart when the image is rotated 90 ° and imaged. The vertical axis represents MTF, and the horizontal axis represents the distance d from the digital camera 10. The MTF distribution represents how the striped chart is attenuated when moving back and forth from the in-focus position when the MTF near the optical axis at the in-focus position is 1. FIG. 7A shows an MTF distribution of a vertical stripe pattern chart and a horizontal stripe pattern chart related to the subject distance of a subject image having a constant frequency in an image without parallax (N image).

  As shown in FIG. 7A, in the non-parallax image, the MTF distributions of the vertical line stripe pattern chart and the horizontal line stripe pattern chart match. FIG. 7B shows the MTF distribution of the vertical line stripe pattern chart and the horizontal line stripe pattern chart relating to the object distance of the subject image having a constant frequency in the parallax images (the parallax Lt image and the parallax Rt image). The MTF distribution of the horizontal stripe pattern chart shown in FIG. 7B matches the MTF distribution of the horizontal stripe pattern chart shown in FIG. On the other hand, the MTF distribution of the vertical stripe pattern chart shown in FIG. 7 (b) is widely distributed in sections with higher contrast than the MTF distribution of the vertical stripe pattern chart shown in FIG. 7 (a), and the depth of field. It can be read that is deep. In other words, when the striped chart is moved back and forth from the in-focus position, in the parallax image, the horizontal line striped chart and the horizontal line striped chart have different contrasts. This produces the blur asymmetry shown in FIG. In FIG. 7, the difference in depth of field between the parallax image and the parallax image is shown, but it can also be regarded as a difference in depth of field between the vertical direction and the horizontal direction in the parallax image.

<Opening shape of parallax pixel and opening shape of non-parallax pixel>
As described above, in the parallax pixel, the horizontal opening width of the opening of the opening mask (that is, the horizontal width of the light receiving area) is the vertical opening width (that is, the vertical width of the light receiving area). It is 1/2 of. Accordingly, the depth of field in the horizontal direction is deep, and as a result, blurring is difficult to occur. On the other hand, the depth of field in the vertical direction is shallow, and as a result, blurring tends to occur. In the non-parallax pixel, the ratio of the horizontal direction to the vertical direction of the opening is 1: 1. There is no difference in depth of field between the horizontal direction and the vertical direction. From the above, it can be seen that the parallax pixel and the non-parallax pixel have different shapes of blur due to different shapes of the openings of the aperture mask, that is, different shapes of the light receiving regions. In the present embodiment, the blur asymmetry due to the depth of field asymmetry that occurs in the parallax image is reduced without changing the optical system. In the case of a pixel array in which non-parallax pixels are mixed in addition to parallax pixels, blur asymmetry can be reduced by devising the non-parallax pixels. That is, it aims at the effect of neutralizing the entire image by making the light condensing characteristics of the non-parallax pixels asymmetric between the horizontal direction and the vertical direction and asymmetric in the direction opposite to the asymmetry of the parallax pixels. The asymmetry of the luminous flux incident on the parallax pixel and the asymmetry of the luminous flux incident on the non-parallax pixel are neutralized by being discharged as an image signal captured by the image sensor and mixed by image processing. Therefore, blurring can be symmetrized only by changing the design of the image sensor alone.

Two methods are conceivable as a method for reducing the blur imbalance of the parallax image. The two methods will be described with reference to FIG. FIG. 8 is a diagram illustrating the configuration of a pixel without parallax. In the first method, as shown in FIG. 8A, the opening shape of the opening 104n of the non-parallax pixel is such that the horizontal direction is the long side and the vertical direction is the short side. Thereby, the horizontal region width in the light receiving region of the non-parallax pixel is longer than the vertical region width. Here, although the opening shape of the opening 104n is a horizontally long shape, the opening 104n is set at a position not deviated with respect to the pixel center. Therefore, the pixel having the opening 104n whose opening shape is a horizontally long shape is a pixel without parallax. The ratio of the long side a to the short side b is as follows in order to cancel the asymmetry of the opening at the opening of the opening mask of the parallax pixel.
a: b = 1: 1-v
Here, v is a parameter expressed as a function of the aperture shape of the parallax pixel. More specifically, it is a parameter indicating a reduction width from all openings in the vertical direction of the opening 104n. As will be described in detail later, v is also expressed as a function of the density of parallax pixels.

As shown in FIG. 8B, the second method is to change the light condensing characteristics of the microlens 101 while keeping the aperture shape of the aperture 104n of the non-parallax pixel in the full aperture. Specifically, the microlens 101 is deformed so that the vertical light collection characteristic is worse than the horizontal light collection characteristic, that is, the horizontal direction is the major axis and the vertical direction is the minor axis. The effect is that the opening is substantially narrower than in the horizontal direction. In other words, the microlens 101 is formed such that the incident light quantity in the horizontal direction is larger than the incident light quantity in the vertical direction. When the microlens 101 is viewed from above, it is deformed from a normal circle to an ellipse. Assuming that the major axis diameter is a and the minor axis diameter is b, an equation representing an elliptical shape is as follows.

  A description will be given of how to determine the ratio of the long side a to the short side b of the aperture when the aperture shape of the aperture mask of the non-parallax pixel is asymmetric. In the present embodiment, the aperture shape of the non-parallax pixel is set in consideration of the balance between the blur imbalance of the parallax image and the deterioration of the blur balance of the non-parallax image. Here, when a parallax image is generated by parallax modulation, which will be described later, a blur of a non-parallax image is reflected in a blur of a high-resolution parallax image that is finally generated. Therefore, in this embodiment, it is preferable to set the opening shape of the non-parallax pixel according to the density ratio of the parallax pixel and the non-parallax pixel. For example, if the non-parallax pixel is dominant over the parallax pixel, the blur of the parallax image has little influence on the overall high-resolution parallax image that is finally generated, so the non-parallax pixel dominates the parallax pixel. The closer the ratio of the major axis to the minor axis of the opening of the pixel without parallax is, the closer it is to 1. That is, it approaches the full opening. Accordingly, it is possible to improve the blur imbalance of the parallax image while keeping the deterioration of the blur of the parallax-free image small.

  On the other hand, if the parallax pixel is dominant over the non-parallax pixel, the blur of the parallax image has a great influence on the entire high-resolution parallax image that is finally generated. As it becomes more dominant, the ratio of the major axis to the minor axis of the aperture of the non-parallax pixel is brought closer to the constant limit value 2: 1. That is, the opening is closed within a meaningful range. That is, the limit point is a rectangular shape obtained by rotating a parallax pixel by 90 °. Thereby, it is possible to suppress the deterioration of the blur balance of the non-parallax image while greatly improving the blur imbalance of the parallax image.

  As described above, by setting the aperture shape of the non-parallax pixel based on the density ratio of the parallax pixel and the non-parallax pixel, the parallax image with improved blur imbalance when generating the parallax image by the parallax modulation processing Can be obtained. As a result, a more natural blurred parallax image can be obtained. Although the case where the aperture shape of the aperture mask of the non-parallax pixel is asymmetrical has been described here, the same can be said for the case where the condensing characteristic of the microlens is asymmetrical.

  Next, an example of the configuration of the image sensor 100 will be described. FIG. 9 is a diagram for explaining the configuration of the image sensor 100. FIG. 9 shows a configuration in which the aperture shape of the aperture mask of the non-parallax pixel is asymmetrical. FIG. 9A shows a state in which a plurality of pixels are arranged in a matrix in the pixel region. FIGS. 9B and 9C conceptually show a state in which a part of the image sensor 100 is enlarged. In particular, FIG. 9B shows a cross-sectional view (AA cross section) of three pixels adjacent in the X-axis direction. FIG. 9C shows a cross-sectional view (BB cross section) of three pixels adjacent in the Y-axis direction. In FIGS. 9B and 9C, description will be made assuming that the parallax Lt pixel, the non-parallax pixel, and the parallax Rt pixel are arranged adjacent to each other. Here, the opening corresponding to the parallax Lt pixel is referred to as the opening 104l, the opening corresponding to the parallax Rt pixel is referred to as the opening 104r, and the opening corresponding to the non-parallax pixel is referred to as the opening 104n. If not, it is simply referred to as the opening 104.

  As shown in FIGS. 9B and 9C, the image sensor 100 is configured by arranging a micro lens 101, a color filter 102, an aperture mask 103, a wiring layer 105, and a photoelectric conversion element 108 in this order from the subject side. ing. The photoelectric conversion element 108 is configured by a photodiode that converts incident light into an electrical signal. A plurality of photoelectric conversion elements 108 are two-dimensionally arranged on the surface of the substrate 109.

  An image signal converted by the photoelectric conversion element 108, a control signal for controlling the photoelectric conversion element 108, and the like are transmitted and received through the wiring 106 provided in the wiring layer 105. In addition, an opening mask 103 provided with one-to-one correspondence with each photoelectric conversion element 108 and having openings 104 arranged repeatedly two-dimensionally is provided in contact with the wiring layer 105.

  The opening 104l and the opening 104r are displaced in the X-axis direction with respect to the pixel center for each corresponding photoelectric conversion element 108, and their relative positions are strictly determined. The opening 104l and the opening 104r set a light receiving region in which the corresponding photoelectric conversion element 108 receives the subject light beam. Thereby, parallax occurs in the subject light flux received by the photoelectric conversion element 108.

  On the other hand, the opening 104n is not displaced with respect to the pixel center. A light receiving area where the corresponding photoelectric conversion element 108 receives the subject light beam is set by the opening 104n. Since the opening 104n is not deviated with respect to the pixel center, no parallax occurs in the subject light beam received by the photoelectric conversion element 108. The opening mask 103 may be arranged separately and independently corresponding to each photoelectric conversion element 108, or may be formed collectively for a plurality of photoelectric conversion elements 108 in the same manner as the manufacturing process of the color filter 102. .

  The color filter 102 is provided on the opening mask 103. The color filter 102 is a filter provided in a one-to-one correspondence with each photoelectric conversion element 108, which is colored so as to transmit a specific wavelength band to each photoelectric conversion element 108. In order to output a color image, at least two types of color filters that are different from each other may be arranged, but in order to obtain a color image with higher image quality, it is preferable to arrange three or more types of color filters. For example, a red filter (R filter) that transmits the red wavelength band, a green filter (G filter) that transmits the green wavelength band, and a blue filter (B filter) that transmits the blue wavelength band may be arranged in a grid pattern. The color filter may be not only a combination of primary colors RGB but also a combination of YCM complementary color filters.

  The microlens 101 is provided on the color filter 102. The microlens 101 is a condensing lens for guiding more incident subject light flux to the photoelectric conversion element 108. The microlenses 101 are provided in a one-to-one correspondence with the photoelectric conversion elements 108. The shape of the microlens 101 is the same in each of the parallax Lt pixel, the parallax Rt pixel, and the non-parallax pixel. In consideration of the relative positional relationship between the pupil center of the taking lens 20 and the photoelectric conversion element 108, the optical axis of the microlens 101 is shifted so that more subject light flux is guided to the photoelectric conversion element 108. It is preferable. Furthermore, the arrangement position may be adjusted so that more specific subject light beam, which will be described later, is incident along with the position of the opening 104 of the opening mask 103.

  Note that in the case of an image sensor with good light collection efficiency and photoelectric conversion efficiency, the microlens 101 may not be provided. In the case of a back-illuminated image sensor, the wiring layer 105 is provided on the side opposite to the photoelectric conversion element 108. Further, if the opening 104 of the opening mask 103 has a color component, the color filter 102 and the opening mask 103 can be formed integrally. Note that the color filter 102 is not provided when a monochrome image signal may be output.

  In the present embodiment, the opening mask 103 and the wiring 106 are provided separately, but the wiring 106 may serve the function of the opening mask 103. That is, a prescribed opening shape is formed by the wiring 106, and the incident light beam is limited by the opening shape to guide only a specific partial light beam to the photoelectric conversion element 108. In this case, the wiring 106 that forms the opening shape is preferably closest to the photoelectric conversion element 108 in the wiring layer 105.

Further, the opening mask 103 may be formed of a permeation blocking film that is provided over the photoelectric conversion element 108. In this case, the opening mask 103 is formed, for example, by sequentially laminating a SiN film and a SiO ₂ film to form a permeation blocking film and removing a region corresponding to the opening 104 by etching. Furthermore, by forming the photoelectric conversion element 108 so that the ratio of the horizontal area width and the vertical area width of the photoelectric conversion element 108 itself is different, the horizontal area width and the vertical area width are reduced. Different light receiving regions can also be formed.

  FIG. 10 is a diagram for explaining the configuration of the image sensor 100. FIG. 10 shows a configuration in which the light condensing characteristics of the microlens of the non-parallax pixel are asymmetrical. FIG. 10A shows a state in which a plurality of pixels are arranged in a matrix in the pixel region. FIGS. 10B and 10C conceptually show a state in which a part of the image sensor 100 is enlarged. FIG. 10B shows a cross-sectional view (AA cross section) of three pixels adjacent in the X-axis direction. FIG. 10C shows a cross-sectional view (BB cross section) of three pixels adjacent in the Y-axis direction. Here, a description will be given focusing on the configuration of pixels without parallax. The configuration of the parallax pixels is the same as that of the image sensor shown in FIG.

  As shown in FIGS. 10B and 10C, when the light condensing characteristics of the microlens 101n of the non-parallax pixel are asymmetrical, an aperture mask 103 having a full aperture is formed on the corresponding photoelectric conversion element 108. Exists. As shown in FIG. 10B, the opening width in the X-axis direction of the microlens 101n of the non-parallax pixel is wider than the opening width in the X-axis direction of the opening 104n. Therefore, the corresponding photoelectric conversion element 108 can receive the subject luminous flux over the entire X-axis direction. On the other hand, as shown in FIG. 10C, the opening width in the Y-axis direction of the microlens 101n of the non-parallax pixel is narrower than the width in the Y-axis direction of the opening 104n. Therefore, the corresponding photoelectric conversion element 108 cannot receive the subject light beam over the entire Y-axis direction, and can receive the subject light beam only in the region corresponding to the opening width of the microlens 101n in the Y-axis direction. By the action of the microlens 101n, a light receiving region in which the opening width in the Y-axis direction is narrower than the opening width in the X-axis direction is set in the photoelectric conversion element 108.

<Density ratio of parallax pixel and non-parallax pixel, and aperture shape of non-parallax pixel>
FIG. 11 is a diagram illustrating a comparative example of pixel arrangement. The image sensor 300 shown in FIG. 11 uses a 2 × 2 pixel pattern 310 indicated by a thick line in the drawing as a basic lattice. In the pattern 310, parallax Lt pixels are assigned to the upper left pixel and the lower right pixel. The parallax Rt pixel is assigned to the lower left pixel and the upper right pixel. Here, the image sensor 300 is a monochrome sensor. The arrangement of the image sensor 300 shown in FIG. 11 is N: Lt: Rt = 0: 1: 1. That is, the image sensor 300 illustrated in FIG. 11 is an image sensor in which only parallax pixels are arranged.

  Next, a pixel arrangement of an image sensor in which parallax pixels and non-parallax pixels are mixed will be described with reference to FIGS. 12 to 14 show a case where the opening shape of the opening of the pixel without parallax is adjusted. Note that, when the microlens is deformed, an opening mask with all openings may be used as an opening mask for pixels without parallax. FIG. 12 is a diagram illustrating an example of a pixel array of the present embodiment. The image sensor 100 shown in FIG. 12 uses a 2 × 2 pixel pattern 110 indicated by a bold line in the drawing as a basic lattice. In the pattern 110, pixels with no parallax are assigned to the upper left pixel and the lower right pixel. Further, the parallax Lt pixel is assigned to the lower left pixel, and the parallax Rt pixel is assigned to the upper right pixel. Here, the image sensor 100 is a monochrome sensor. The arrangement of the image sensor 100 shown in FIG. 12 is N: Lt: Rt = 2: 1: 1.

  FIG. 13 is a diagram illustrating a variation of the pixel arrangement of the present embodiment. The image sensor 100 shown in FIG. 13 uses a pattern 110 of adjacent 8 pixels × 8 pixels as a basic lattice. The pattern 110 includes four Bayer arrays each having 2 × 2 four pixels as a basic unit in the Y-axis direction and four in the X-axis direction. As shown in the figure, in the Bayer array, a green filter (G filter) is arranged for the upper left pixel and the lower right pixel, a blue filter (B filter) is arranged for the lower left pixel, and a red filter (R filter) is arranged for the upper right pixel.

Pixels in the pattern 110 are denoted by P _IJ . For example, the upper left pixel is _{P 11,} the upper right pixel is _{P 81.} As shown in the figure, the parallax pixels are arranged as follows.

P ₁₁ : Parallax Lt pixel + G filter (= G (Lt))
P ₅₁ ... Parallax Rt pixel + G filter (= G (Rt))
P ₃₂ ... Parallax Lt pixel + B filter (= B (Lt))
P ₇₂ ... Parallax Rt pixel + B filter (= B (Rt))
P ₂₃ ... Parallax Rt pixel + R filter (= R (Rt))
P ₆₃ ... Parallax Lt pixel + R filter (= R (Lt))
P ₄₄ ... Parallax Rt pixel + G filter (= G (Rt))
P _{84 ...} parallax Lt pixel + G filter (= G (Lt))
P ₁₅ ... Parallax Rt pixel + G filter (= G (Rt))
P ₅₅ ... Parallax Lt pixel + G filter (= G (Lt))
P ₃₆ ... Parallax Rt pixel + B filter (= B (Rt))
P ₇₆ ... Parallax Lt pixel + B filter (= B (Lt))
P ₂₇ ... Parallax Lt pixel + R filter (= R (Lt))
P ₆₇ ... Parallax Rt pixel + R filter (= R (Rt))
P ₄₈ ... Parallax Lt pixel + G filter (= G (Lt))
P ₈₈ ... Parallax Rt pixel + G filter (= G (Rt))
The other pixels are non-parallax pixels, and are any of the non-parallax pixel + R filter, the non-parallax pixel + G filter, and the non-parallax pixel + B filter. Here, the image sensor 100 is a color sensor. The arrangement of the image sensor 100 shown in FIG. 13 is N: Lt: Rt = 6: 1: 1.

  FIG. 14 is a diagram showing variations of the pixel arrangement of the present embodiment. The image sensor 100 shown in FIG. 14 uses a pattern 110 of adjacent 8 pixels × 8 pixels as a basic lattice. The pattern 110 includes four Bayer arrays each having 2 × 2 four pixels as a basic unit in the Y-axis direction and four in the X-axis direction. Moreover, as shown in the figure, the parallax pixels are arranged as follows.

P ₁₁ : Parallax Lt pixel + G filter (= G (Lt))
P ₅₁ ... Parallax Rt pixel + G filter (= G (Rt))
P ₃₂ ... Parallax Lt pixel + B filter (= B (Lt))
P ₆₃ ... Parallax Rt pixel + R filter (= R (Rt))
P ₁₅ ... Parallax Rt pixel + G filter (= G (Rt))
P ₅₅ ... Parallax Lt pixel + G filter (= G (Lt))
P ₇₆ ... Parallax Rt pixel + B filter (= B (Rt))
P ₂₇ ... Parallax Lt pixel + R filter (= R (Lt))
The other pixels are non-parallax pixels, and are any of the non-parallax pixel + R filter, the non-parallax pixel + G filter, and the non-parallax pixel + B filter. Here, the image sensor 100 is a color sensor. The arrangement of the image sensor 100 shown in FIG. 14 is N: Lt: Rt = 14: 1: 1.

  In FIG. 13, the parallax pixels are divided into one of a first group having a G filter, a second group having an R filter, and a third group having a B filter when viewed from the entire image sensor 100. The pattern 110 includes at least one parallax Lt pixel and parallax Rt pixel belonging to each group. As in the example of the figure, each of these parallax pixels and non-parallax pixels may be arranged as isotropic as possible in the pattern 110. In addition, since the parallax pixels and the non-parallax pixels are mixedly arranged, it appears to be randomly arranged at first glance. By being arranged with isotropic properties, RGB color information can be acquired as the output of the parallax pixels without causing a bias in the spatial resolution for each color component, so high-quality parallax image data Is obtained.

  In each of the above arrangements, a description will be given of what rectangular shape (rectangular shape) the opening shape of the opening portion of the non-parallax pixel should be. Since the density of parallax pixels is different in each array, it is assumed that the degree of influence of asymmetric blur is also different. It is assumed that the degree of influence of asymmetric blur is measured by blur in a 2D image generated intermediately through image processing described in each embodiment below. This is because a 3D image that is finally generated through image processing called parallax modulation, which will be described later, appears to be the same state as an intermediately generated 2D image when a right-eye image and a left-eye image are simply superimposed and displayed. It is.

  When the ratio of the total number of parallax pixels in the total number of pixels (density of parallax pixels) is represented by δ, the deformation ratio v in the vertical direction at the opening of the pixel without parallax is represented as a function of δ. However, the density of the parallax pixels is the sum of the same number of the left parallax pixels and the right parallax pixels, and the density ratio of the non-parallax pixels, the left parallax pixels, and the right parallax pixels is expressed by the following (Formula 1). .

N: Lt: Rt = 1−δ: δ / 2: δ / 2 (Formula 1)
In generating the above-described intermediately generated 2D image, in the image processing described in the following embodiment, a process of mixing the pixel value of the non-parallax pixel and the pixel value of the parallax pixel according to the density ratio is performed. Thereby, the spatial information sampled by all the pixels can be utilized to the maximum extent. In practice, the image processing unit 205 receives raw raw image data whose output values (pixel values) are arranged in the pixel arrangement order of the image sensor 100, and executes plane separation processing for separating the raw image data into a plurality of plane data. The plane data is image data gathered separately for each pixel group characterized in the same way. First, the image processing unit 205 removes the pixel values of the parallax pixels to form empty lattices. Then, the pixel value that has become an empty lattice is calculated by interpolation processing using the pixel values of the surrounding non-parallax pixels. As a result, an N image in which vacancies are filled is generated.

  Similarly, the image processing unit 205 removes pixel values other than the pixel value of the left parallax pixel from all the output values of the image sensor 100 to form a vacant lattice. Then, the pixel value that has become an empty lattice is calculated by interpolation processing using the pixel values of the surrounding left parallax pixels. As a result, an Lt image in which vacancies are filled is generated. Further, the image processing unit 205 removes pixel values other than the pixel value of the right parallax pixel from all output values of the image sensor 100 to form a vacant lattice. Then, the pixel value that becomes the empty grid is calculated by interpolation processing using the pixel values of the surrounding right parallax pixels. As a result, an Rt image in which vacancies are filled is generated. Thereafter, when an average image of the Lt image and the Rt image is generated, the average image also represents an image without parallax having different spatial information. Therefore, two types of non-parallax images of the N image and the average image of the Lt image and the Rt image are mixed to generate a new non-parallax image N ′. This is performed on each pixel position. Here, an example is shown in which a geometric average according to the parallax pixel density is taken. This is performed in a processing step called local gain balance correction described later.

（式２）
中間２Ｄ画像を視差画素の密度の関数として生成するので、δの極限値、すなわち、δ→１の場合には視差画素のみで作成した画像となり、δ→０の場合には視差なし画素のみで作成した画像となる。この極限状態で、視差なし画素の開口部の開口形状ａ：ｂが如何なる値を採るべきかを考察すると、δの関数として表す場合の出発点の境界条件が与えられる。 This can be schematically expressed as a function of the density ratio δ (Formula 2).

(Formula 2)
Since an intermediate 2D image is generated as a function of the density of parallax pixels, the limit value of δ, that is, an image created only with parallax pixels when δ → 1, and only with no parallax pixels when δ → 0. It will be the created image. Considering what value the aperture shape a: b of the aperture of the non-parallax pixel should take in this limit state, the boundary condition of the starting point when expressed as a function of δ is given.

  In the case of δ → 1, since it is composed only of parallax pixels, there is no non-parallax pixel. Accordingly, correction using non-parallax pixels is not possible. In the case of δ → 0, it is composed only of pixels without parallax, so it is equivalent to a normal 2D dedicated sensor, and there is no need to change the aperture shape of pixels without parallax. Therefore, v = 0. This must always be true no matter what the other conditions are. Therefore, v = 0 置 く δ can be set. That is, when δ is near 0, v is proportional to δ. When δ = 0, v = 0.

  Next, a case where the deformation ratio v of the aperture shape of the non-parallax pixel is expressed as a function of the aperture shape of the parallax pixel when the aperture is expanded in the horizontal direction from the state where the parallax pixel is a half aperture will be described. FIG. 15 is a diagram illustrating the opening shape of the opening mask of the parallax pixels. The opening 104l of the parallax Lt pixel extends from the center line 322 to the right by a width u. On the other hand, the opening 104r of the parallax Rt pixel extends from the center line 322 to the left by a width u. The opening of the parallax pixel can be expressed by (Equation 3).

Horizontal opening width: Vertical opening width = ((1/2) + u): 1 (Formula 3)
Even in this case, first, the boundary condition given by the extreme state is considered. Assume the limit u → 1/2 where the opening of the parallax pixel extends to the entire opening. In this case, since the parallax pixels can be handled as non-parallax pixels, all of them are non-parallax pixels. Therefore, it is not always necessary to transform a non-parallax pixel for an arbitrary δ regardless of the density of the parallax pixel. That is, v = 0. Since this must always hold regardless of other conditions, v = 0 置 く ((1/2) −u) can be set.

In summary, (Expression 4), (Expression 5), and (Expression 6) are obtained.
When δ = 1 Correction not possible (Formula 4)
When δ = 0, always v = 0∝δ (Formula 5)
For any δ, when u = 1/2, v = 0 ＝ ((1/2) −u) (Equation 6)
From the above, (Equation 7) can be derived.
v ∝δ ((1/2) −u) (Formula 7)

Since it was found that v is proportional to δ and ((1/2) -u), the proportionality coefficient is then determined to express the absolute quantity. In expressing the absolute quantity, it is better to consider the simple case where the parallax pixel is configured with a half-open state u = 0. Further, consider an array of N: Lt: Rt = 2: 1: 1 shown in FIG. 12 where the density of parallax pixels is δ = 1/2. Assuming that one N pixel in FIG. 12 corrects the asymmetric blur of the Lt pixel and the other N pixel plays a role of correcting the asymmetric blur of the Rt pixel, N: Lt: Rt = 1−δ: δ / 2 : Since δ / 2, the relationship shown in (Expression 8) is generally derived.
v = δ / 2 (Formula 8)
In the case of the pixel arrangement shown in FIGS. 12 to 14, it is shown below what value the deformation ratio v of the non-parallax pixel takes.

As shown in FIG. 12, when N: Lt: Rt = 2: 1: 1, that is, δ = 1/2, (Equation 9) is obtained.
V = 1/4 (Formula 9)
As shown in FIG. 13, when N: Lt: Rt = 6: 1: 1, that is, δ = 1/4, (Equation 10) is obtained.
V = 1/8 (Formula 10)
As shown in FIG. 14, when N: Lt: Rt = 14: 1: 1, that is, δ = 1/8, (Equation 11) is obtained.
V = 1/16 (Formula 11)

Since (Expression 8) is established when u = 0 in the above (Expression 7), it can be seen that the proportionality constant is 1. (Formula 12) can be derived as a general formula.
v = δ × ((1/2) −u) (Formula 12)
Therefore, the ratio a: b of the long side and the short side of the rectangular shape in the aperture mask of the pixel without parallax, or the ratio a: b of the major axis diameter to the minor axis diameter of the elliptical shape of the microlens can be expressed as follows. it can.
a: b = 1: 1−δ × ((1/2) −u) (Formula 13)
(Equation 13) also holds for a region where u is negative. That is, | u | <1/2. Also, 0 <δ <1.

In the case of u = 0, the ratio of a and b in the pixel array shown in FIGS.
As shown in FIG. 12, when N: Lt: Rt = 2: 1: 1, that is, δ = 1/2, (Equation 14) is obtained.
a: b = 1: 1-1 / 4 = 4: 3 (Formula 14)
As shown in FIG. 13, when N: Lt: Rt = 6: 1: 1, that is, δ = 1/4, (Equation 15) is obtained.
a: b = 1: 1-1 / 8 = 8: 7 (Formula 15)
As shown in FIG. 14, when N: Lt: Rt = 14: 1: 1, that is, δ = 1/8, (Equation 16) is obtained.
a: b = 1: 1-1 / 16 = 16: 15 (Formula 16)

<Embodiment 1>
Here, the arrangement shown in FIG. 12 is adopted as the arrangement of the image sensor 100. That is, the image sensor 100 is a monochrome sensor, and the arrangement of the image sensor 100 is N: Lt: Rt = 2: 1: 1. The configuration of the non-parallax pixel is one of the following. That is, an opening mask having a ratio of the long side to the short side of the opening is a: b = 4: 3, or an elliptical shape in which the ratio of the long axis diameter to the short axis diameter is a: b = 4: 3. Either a microlens with a modified light collection characteristic is used. Image processing for developing the image data thus captured will be described below. The image processing procedure is approximately as follows.

1) Input of parallax multiplexed mosaic image data 2) Global gain balance correction of parallax mosaic image 3) Generation of temporary parallax image 4) Generation of parallax-free reference image by left and right local illuminance distribution correction (local gain balance correction)
5) Generation of actual parallax image 6) Conversion to output space Hereinafter, description will be made in order.

1) Input of parallax multiplexed mosaic image data A single-panel monochrome mosaic image in which the parallax of FIG. 12 is multiplexed is represented by M (x, y). It is assumed that the gradation is a linear gradation output by A / D conversion.

For convenience, in the mosaic image M (x, y), the non-parallax pixel signal surface is N _mosaic (x, y), the left parallax pixel signal surface is L _mosaic (x, y), and the right parallax pixel signal surface. _Is expressed as Rt _mosaic (x, y).

  Thus, a mosaic image in which the non-parallax pixel is corrected with one gain coefficient, the left parallax pixel with one gain coefficient, and the right parallax pixel with one gain coefficient is output as M ′ (x, y).

3) Generation of temporary parallax image A temporary left parallax image having a low spatial frequency resolution and a temporary right parallax image having a low spatial frequency resolution are generated. Simple average interpolation is performed in the signal plane that collects only the left parallax pixels. Linear interpolation is performed according to the distance ratio using pixel values that are close to each other. Similarly, simple average interpolation is performed in the signal plane in which only the right parallax pixels are collected. Similarly, simple average interpolation is performed in the signal plane in which only non-parallax pixels are collected. That _is, generation _{Lt mosaic} (x, y) from Lt (x, y) _{and, Rt mosaic (x,} y) from the Rt (x, y) _{and, N mosaic (x,} y) from the N (x, y) and To do. A temporary non-parallax image is represented as N (x, y), a temporary left parallax image is represented as Lt (x, y), and a temporary right parallax image is represented as Rt (x, y). When generating a temporary non-parallax image N (x, y), it is preferable to introduce a direction determination in the signal plane and perform it with high definition.

4) Generation of parallax-free reference images by correcting left and right illuminance distribution (local gain balance correction)
Next, by performing local gain correction in units of pixels in the same manner as the global gain correction performed in step 1, first, the illuminances of the left parallax pixel in the screen and the right parallax pixel in the screen are matched. This operation eliminates the parallax between the left and right. Then, the illuminance is further adjusted between the signal plane obtained by taking the left-right average and the imaging signal plane of the non-parallax pixel. In this way, a new parallax-free reference image plane in which gain matching is achieved in all pixels is created. This is equivalent to replacing with an average value, and an intermediate image plane in which the parallax disappears is completed. This is written as N (x, y).

このように左視点の画像と右視点の画像の平均値をさらに視差のない基準視点の画像との平均値をとった画素値を新たな視差なし画素値として、モノクロ面のデータを書き換え、視差なしモノクロ面の画像Ｎ（ｘ，ｙ）を出力する。

As described above, the pixel value obtained by taking the average value of the left viewpoint image and the right viewpoint image and the average value of the reference viewpoint image without parallax as a new non-parallax pixel value is rewritten, and the parallax is rewritten. None Monochrome image N (x, y) is output.

5) Generation of actual parallax image Temporary left parallax image Lt (x, y) generated in step 3 and low-resolution monochrome image N (x, y) generated as intermediate processing in step 5 Is used to generate a left-parallax monochrome image Lt ′ (x, y) with high resolving power that is actually output. Similarly, using the temporary right parallax image Rt (x, y) generated in step 3 with low resolution and the monochrome image N (x, y) without parallax generated as intermediate processing in step 5 with actual resolution, A right parallax color image Rt ′ (x, y) with high resolving power is generated.

  A monochrome image without parallax forms a subject image that matches the blur width of the entire aperture. Therefore, in the denominator of the parallax modulation that keeps the ratio constant, an image with the blur width of the full aperture by the arithmetic mean of the left viewpoint image and the right viewpoint image is taken as a reference point, and the left and right images after the parallax modulation are again Modulation is applied so that the image has a half-aperture blur width.

6) Conversion to output color space The high-resolution non-parallax intermediate monochrome image N (x, y) and the high-resolution left-parallax monochrome image Lt ′ (x, y) obtained in this way Each right-parallax monochrome image Rt ′ (x, y) is subjected to appropriate gamma conversion and output as an output space image.

<Embodiment 2>
Here, the arrangement shown in FIG. 14 is adopted as the arrangement of the image sensor 100. That is, the image sensor 100 is a color sensor, and the arrangement of the image sensor 100 is N: Lt: Rt = 14: 1: 1. The configuration of the non-parallax pixel is one of the following. That is, an aperture mask in which the ratio of the long side to the short side of the opening is a: b = 16: 15 is used, or the light is condensed into an elliptical shape in which the ratio of the long axis to the short axis is a: b = 16: 15. Either a microlens with modified characteristics is used. Image processing for developing the image data thus captured will be described below. The image processing procedure is approximately as follows.

1) Color / parallax multiplexed mosaic image data input 2) Global gain balance correction of color / parallax mosaic image 3) Temporary parallax image generation 4) Generation of parallax-free color mosaic image by right and left local illuminance distribution correction (Local・ Gain balance correction
5) Generation of reference image without parallax 6) Generation of actual parallax image 7) Conversion to output color space

1) Color / Parallax Multiplexed Mosaic Image Data Input A single-panel mosaic image in which the color and parallax in FIG. 14 are multiplexed is represented by M (x, y). It is assumed that the gradation is a linear gradation output by A / D conversion.

Convenience of the mosaic image M (x, y), a signal surface of parallax without the R component pixel _{R N_mosaic} (x, y), a signal surface of the left parallax pixels of the R component _{R Lt_mosaic} (x, y), the signal surface of the right parallax pixels of the R component _{R Rt_mosaic} (x, y), a signal surface of the left parallax pixels of the G component _{G N_mosaic} (x, y), a signal surface of parallax without the G component pixel _{G Lt_mosaic} (x , Y), the signal surface of the right parallax pixel of G component is _{GRt_mosaic} (x, y), the signal surface of the _non- parallax pixel of B component is _{BN_mosaic} (x, y), and the signal surface of the left parallax pixel of B component is _Let B _{Lt_mosaic} (x, y) and the signal surface of the B component right parallax pixel be B _{Rt_mosaic} (x, y).

  Thus, a mosaic image in which the non-parallax pixel is corrected with one gain coefficient, the left parallax pixel with one gain coefficient, and the right parallax pixel with one gain coefficient is output as M ′ (x, y).

3) Generation of temporary parallax image A temporary left parallax image having a low spatial frequency resolution and a temporary right parallax image having a low spatial frequency resolution are generated. Simple average interpolation is performed in the G color plane where only the left parallax pixels are collected. Linear interpolation is performed according to the distance ratio using pixel values that are close to each other. Similarly, simple average interpolation in the G color plane in which only the right parallax pixels are collected is performed. Similarly, simple average interpolation in the G color plane in which only non-parallax pixels are collected is performed. Similar processing is performed for each of R, G, and B. That is, R _{Lt_mosaic} (x, y) to R _Lt (x, y), R _{Rt_mosaic} (x, y) to R _Rt (x, y), and R _{N_mosaic} (x, y) to R _N (x, y) ) _{and, G Lt_mosaic (x,} y) from the _G Lt (x, a _{y), G Rt_mosaic (x,} G Rt the (x, y) from _{y), G N_mosaic (x,} y) from _G N (x, y), B _{Lt_mosaic} (x, y) to B _Lt (x, y), B _{Rt_mosaic} (x, y) to B _Rt (x, y), B _{N_mosaic} (x, y) to B _N (x, y) , Y).

Here, the temporary R component non-parallax image is R _N (x, y), the temporary G component non-parallax image is G _N (x, y), and the temporary B component non-parallax image is B _N (x , Y), the left parallax image of the temporary R component is R _Lt (x, y), the left parallax image of the temporary G component is G _Lt (x, y), and the left parallax image of the temporary B component is B _Lt ( x, y). Similarly, the right parallax image of the temporary R component is R _Rt (x, y), the right parallax image of the temporary G component is G _Rt (x, y), and the right parallax image of the temporary B component is B _Rt (x , Y). In the case of generating temporary non-parallax images R _N (x, y), G _N (x, y), and B _N (x, y), high-definition is introduced by introducing direction determination in the signal plane. Good to do.

4) Generation of parallax-free color mosaic image by correcting left and right illumination distribution (local gain balance correction)
Next, by performing local gain correction in units of pixels in the same manner as the global gain correction performed in step 1, first, the illuminances of the left parallax pixel in the screen and the right parallax pixel in the screen are matched. This operation eliminates the parallax between the left and right. Then, the illuminance is further adjusted between the signal plane obtained by taking the left-right average and the imaging signal plane of the non-parallax pixel. As described above, a new Bayer surface in which gain matching is achieved in all pixels is created. This is equivalent to replacing with an average value, and a Bayer surface in which parallax disappears is completed. This is written as M _N (x, y).

  Note that the aperture mask of the non-parallax pixel is a full aperture. Therefore, an arithmetic average is used in order to make the blur width of the subject image whose parallax disappeared between the left and right coincide with the blur width of the entire aperture. This eliminates the parallax between the left and right.

  Furthermore, the operation to take the average between the signal plane where the parallax disappeared between the left and right and the imaging signal plane of the non-parallax pixel is already aligned with the subject image of the same blur width, so the blur width is saved. There is a need to. Therefore, in this case, a common geometric average must be taken. Here, the number of parallax pixels is smaller than the number of non-parallax pixels. In addition, the resolution of the parallax image is lower than the resolution of the image without parallax. Therefore, if the distribution of weights is evenly distributed between the pixel value of the non-parallax pixel and the average value of the left and right parallax pixels, the resolution of the obtained image decreases as a whole due to the influence of the resolution of the parallax image. Therefore, it is necessary to devise as close as possible to the resolution of an image without parallax. Therefore, it is preferable to take a geometric average in consideration of the density ratio between the non-parallax pixels and the parallax pixels in the pixel array on the image sensor. Specifically, the ratio of the non-parallax pixel (N), the left parallax pixel (Lt), and the right parallax pixel (Rt) used in the second embodiment is N: Lt: Rt = 14: 1: 1, that is, N : (Lt + Rt) = 7: 1, so that the 7 / 8th power is given to the non-parallax pixels and the 1 / 8th power is given to the parallax pixels, and the distribution is focused on the high density non-parallax pixels. To do. The specific formulas are given below.

このように左視点の画像と右視点の画像の平均値と、視差のない基準視点の画像との平均値をとった画素値を新たな視差なし画素値として、Ｂａｙｅｒ面のデータを書き換え、視差なしＢａｙｅｒ面の画像Ｍ_Ｎ（ｘ，ｙ）を出力する。

In this way, the data on the Bayer plane is rewritten using the average value of the left viewpoint image and the right viewpoint image and the average value of the reference viewpoint image without parallax as a new non-parallax pixel value. None Bayer plane image M _N (x, y) is output.

5) Generation of reference image without parallax A known Bayer interpolation technique is performed. As an example, there are interpolation algorithms shown in USP7957588 (WO2006 / 006373) and USP8259213 which are the same inventors as the present applicant.

6) Real parallax image generation Temporary left parallax color images R _Lt (x, y), G _Lt (x, y), B _Lt (x, y) generated in step 3 and in step 5 Using the color images R _N (x, y), G _N (x, y), and B _N (x, y) with high resolving power generated as intermediate processing, the left parallax with high resolving power that is actually output is output. Color images R ′ _Lt (x, y), G ′ _Lt (x, y), and B ′ _Lt (x, y) are generated. Similarly, the provisional right parallax color images R _Rt (x, y), G _Rt (x, y), B _Rt (x, y) generated in step 3 and intermediate processing in step 5 are generated. Using the color images R _N (x, y), G _N (x, y), and B _N (x, y) without parallax with high resolving power, the color image R ′ _Rt with high resolving power actually output is output. (X, y), G ′ _Rt (x, y), and B ′ _Rt (x, y) are generated.

  The aperture mask of the non-parallax pixel is a full aperture. Therefore, by adopting a method in which an arithmetic mean is used as a reference point as a method of parallax modulation, the blur width of the N image and (Lt image + Rt image) / 2 is reduced while transmitting the blur width of the parallax image. A parallax modulation effect to be corrected is obtained.

Here, for example, when generating the high-resolution left parallax image R ′ _Lt , even when parallax modulation is performed, the synergy considering the density ratio of RGB between the parallax pixels in the pixel array of the image sensor is taken into consideration. Take the average. That is, since R: G: B = 1: 2: 1 between the left parallax pixels and R: G: B = 1: 2: 1 between the right parallax pixels, the parallax modulation by the R component is performed. Giving a weight of ¼ power, a weight of ½ power to parallax modulation by G component, and a weight of ¼ power to parallax modulation by B component, giving importance to parallax modulation by high density G component Take distribution. Specifically, high-resolution left parallax images R ′ _Lt , G ′ _Lt , B ′ _Lt and high-resolution right parallax images R ′ _Rt , G ′ _Rt , B ′ _Rt are calculated using the following equations. To do.

7) Conversion to output color space The high-resolution intermediate color images R _N (x, y), G _N (x, y), and B _N (x, y) obtained in this way and high resolution Left parallax color images R _Lt (x, y), G _Lt (x, y), B _Lt (x, y), High resolution right parallax color images R _Rt (x, y), G _Rt (x , Y), B _Rt (x, y) are subjected to color matrix conversion and gamma conversion from the camera RGB having the spectral characteristics of the sensor to the standard sRGB color space, and output as an image in the output color space.

  Note that when the array shown in FIG. 13 is used as the array of the image sensor 100, that is, the array of N: Lt: Rt = 6: 1: 1, the configuration of the non-parallax pixels is one of the following: take. That is, an aperture mask in which the ratio of the long side to the short side of the opening is a: b = 8: 7 is used, or the light is condensed into an elliptical shape in which the ratio of the long axis to the short axis is a: b = 8: 7. Either a microlens with modified characteristics is used. Since the development process goes through the same procedure as in the second embodiment, a description thereof will be omitted. However, it is necessary to change the mixing ratio of the N image, the Lt image, and the Rt image in the case of local gain balance correction as the density of the parallax pixels changes.

  In the above embodiment, an example in which parallax is applied to the left and right has been described. However, if the image sensor and the optical system are simultaneously rotated by 90 degrees, an embodiment of an imaging system with vertical parallax is obtained. If it rotates 45 degrees, it will become embodiment of the imaging system of diagonal parallax.

  Even in a honeycomb structure in which pixels are not square as shown in Patent Document 2, the same concept holds in an array in which N pixels, Lt pixels, and Rt pixels are mixed. In other words, when the parallax pixels are configured so as to have a parallax on the left and right, the N pixels are deformed in the opposite direction to the parallax pixels between the horizontal direction and the vertical direction as in the embodiment of the left-right parallax.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Digital camera, 20 Shooting lens, 21 Optical axis, 22 Aperture, 100 Image sensor, 101 Micro lens, 102 Color filter, 103 Aperture mask, 104 Aperture, 104l Aperture, 104r Aperture, 104n Aperture, 105 Wiring layer , 106 wiring, 108 photoelectric conversion element, 109 substrate, 110 pattern, 201 control unit, 202 A / D conversion circuit, 203 memory, 204 drive unit, 205 image processing unit, 207 memory card IF, 208 operation unit, 209 display unit , 210 LCD drive circuit, 220 memory card, 300 image sensor, 310 pattern, 322 center line, 1801 distribution curve, 1802 distribution curve, 1803 distribution curve, 1804 distribution curve, 1805 distribution curve, 1806 composite distribution curve, 1807 distribution Line, 1808 distribution curve, 1809 synthetic distribution curve

Claims (17)

A non-parallax pixel with an aperture mask that generates a reference viewpoint, a left parallax pixel with an aperture mask that generates a left viewpoint, and an aperture that generates a right viewpoint with respect to an incident light beam of one optical system An imaging device having a pixel array having at least three types of pixels of right parallax pixels provided with a mask,
An imaging device in which a vertical aperture width of an aperture mask of the non-parallax pixel is narrower than a horizontal aperture width. The left parallax pixel and the right parallax pixel each include a half-opening mask in a region where they do not overlap each other, and the non-parallax pixel includes a full-opening mask in a region where the half-opening regions overlap each other,
When the density ratio between the non-parallax pixel, the left parallax pixel, and the right parallax pixel is represented by (1-δ): δ / 2: δ / 2,
The aperture ratio between the horizontal aperture width a and the vertical aperture width b of the non-parallax pixel is a: b = 1: (1− (δ / 2)), 0 <δ <1
The imaging device according to claim 1, which is set as follows.   The aperture ratio between horizontal and vertical of the non-parallax pixel is set according to the shape of the aperture mask of the parallax pixel in addition to the density ratio between the parallax pixel and the non-parallax pixel. The imaging device described. An opening mask (vertical opening width: horizontal opening width = 1: (1/2) + u) of a part of the left parallax pixel and the right parallax pixel that overlap each other with a half-opening area that does not overlap each other;
When the density ratio between the non-parallax pixel, the left parallax pixel, and the right parallax pixel is represented by (1-δ): δ / 2: δ / 2,
The aperture ratio between the horizontal aperture width a and the vertical aperture width b of the non-parallax pixel is a: b = 1: 1−δ × ((1/2) −u), 0 <δ <1, | u | < 1/2
The imaging device according to claim 3, wherein A parallax-free pixel having a microlens and an aperture mask for generating a reference viewpoint for an incident light beam of one optical system, a left-parallax pixel having a microlens and an aperture mask for generating a left viewpoint, and a right An imaging device comprising a pixel array having at least three types of pixels of a right parallax pixel having a microlens that generates a directional viewpoint and an aperture mask,
An imaging device that transforms the light condensing characteristic of the micro lens of the non-parallax pixel into an anisotropic shape so that a vertical direction is smaller than a horizontal direction. The left parallax pixel and the right parallax pixel each include a half-opening mask in a region where they do not overlap each other, and the non-parallax pixel includes a full-opening mask in a region where the half-opening regions overlap each other,
When the density ratio between the non-parallax pixel, the left parallax pixel, and the right parallax pixel is represented by (1-δ): δ / 2: δ / 2,
The ratio between the incident light amount a in the horizontal direction and the incident light amount b in the vertical direction of the micro lens of the non-parallax pixel is expressed as a: b = 1: (1− (δ / 2)), 0 <δ <1
The imaging device according to claim 5, wherein   The flatness ratio between horizontal and vertical of the microlens of the non-parallax pixel is set according to the shape of the opening mask of the parallax pixel in addition to the density ratio between the parallax pixel and the non-parallax pixel. Item 6. The imaging device according to Item 5. An opening mask (vertical opening width: horizontal opening width = 1: (1/2) + u) of a part of the left parallax pixel and the right parallax pixel that overlap each other with a half-opening area that does not overlap each other;
When the density ratio between the non-parallax pixel, the left parallax pixel, and the right parallax pixel is represented by (1-δ): δ / 2: δ / 2,
The ratio between the incident light quantity a in the horizontal direction and the incident light quantity b in the vertical direction of the microlens of the non-parallax pixel is expressed as a: b = 1: 1−δ × ((1/2) −u), 0 <δ < 1, | u | <1/2
The imaging device according to claim 7, wherein The imaging device according to any one of claims 1 to 8,
An image pickup apparatus including one optical system including a circular diaphragm in the middle of an optical path. A deviated pixel in which a first light receiving region for receiving a subject luminous flux is set at a position deviated in the first axis direction with respect to the pixel center;
A second light receiving region for receiving the subject luminous flux, and a non-deviation pixel set at a position not deviated with respect to the pixel center,
The first region width in the first axial direction in the first light receiving region is shorter than the second region width in the second axial direction orthogonal to the first axial direction, and the first axial direction in the second light receiving region. The first region width of the image sensor is longer than the second region width in the second axial direction. When the density ratio of the displacement pixel and the non-deviation pixel is represented by δ: 1-δ (where 0 <δ <1),
a: b = 1: (1− (δ / 2)) (a: first region width of the second light receiving region, b: second region width of the second light receiving region)
The imaging device according to claim 10, satisfying the relationship: The density ratio of the displacement pixel and the non-deviation pixel is represented by δ: 1-δ (where 0 <δ <1), and the second region width and the first region in the first light receiving region When the width ratio is expressed as 1: (1/2) + u (where | u | <1/2),
a: b = 1: (1−δ × ((1/2) −u)) (a: first region width of the second light receiving region, b: second region width of the second light receiving region)
The imaging device according to claim 10, satisfying the relationship:   13. The deviation pixel and the non-deviation pixel have an opening mask having different opening shapes, and the first light receiving region and the second light receiving region are set by the opening mask. The imaging device according to any one of the above. A deviated pixel in which a first light receiving region for receiving a subject light beam collected by the first microlens is deviated in the first axis direction with respect to the pixel center;
A second light receiving region for receiving a subject light beam collected by the second microlens, and a non-decentered pixel set at a position not deviated with respect to the pixel center,
The first region width in the first axial direction in the first light receiving region is shorter than the second region width in the second axial direction orthogonal to the first axial direction, and the first axial direction of the second microlens. The image pickup device has a larger incident light amount in a direction corresponding to 1 than an incident light amount in a direction corresponding to a second axis direction orthogonal to the first axis direction. When the density ratio of the displacement pixel and the non-deviation pixel is represented by δ: 1-δ (where 0 <δ <1),
a: b = 1: (1− (δ / 2)) (a: incident light amount in the first axis direction of the second microlens, b: incident light amount in the second axis direction of the second microlens)
The imaging device according to claim 14, wherein: The density ratio of the displacement pixel and the non-deviation pixel is represented by δ: 1-δ (where 0 <δ <1), and the second region width and the first region in the first light receiving region When the width ratio is expressed as 1: (1/2) + u (where | u | <1/2),
a: b = 1: (1−δ × ((1/2) −u)) (a: incident light quantity in the first axis direction of the second microlens, b: incidence in the second axis direction of the second microlens. Light intensity)
The imaging device according to claim 14, wherein: The image pickup device according to any one of claims 10 to 16,
An imaging apparatus comprising: an optical system having a circular diaphragm that adjusts a subject light flux guided to the imaging element.

Images