JP5907668B2 - Imaging device and imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging element.

For example, a configuration disclosed in Patent Document 1 is known as an imaging apparatus capable of deriving the defocus amount of the entire shooting screen and obtaining image data.
In Patent Literature 1, the defocus amount is calculated by switching the pupils in a time division manner. Then, image data is acquired without dividing the pupil.

JP 11-142723 A

  In the case of the configuration of Patent Document 1, there is a problem that a good image cannot be obtained, for example, when the subject is moving during shooting or when the amount of light changes during shooting.

  The present invention has been made in view of the above, and an object of the present invention is to provide an imaging device and an imaging element that can acquire defocus information of the entire screen and can easily generate a natural image.

In order to solve the above-described problems and achieve the object, the imaging apparatus of the present invention can be attached with a photographic lens or provided with a photographic lens.
It has an image sensor with pixels arranged in the horizontal and vertical directions,
It has a function to process signals from the image sensor,
The image sensor is composed of a blue pixel, a green pixel, and a red pixel that are projected onto an independent region at least on a conjugate plane.
Green pixels are arranged in a checkered pattern,
Blue pixels and red pixels are arranged not to be adjacent to each other,
The center-of-gravity incident angle of each green pixel is shifted in the first direction with respect to the center incident angle,
The gravity center incident angles of the four green pixels included in the eight pixels adjacent to each green pixel are all shifted in the second direction with respect to the central incident angle,
The second direction is the reverse direction of the first direction,
The second direction has both a horizontal component and a vertical component;
The image is output for image formation.
Image defocus information is acquired from the output of the green pixel.

In another aspect, an imaging device according to the present invention includes an imaging device in which pixels are arranged in a horizontal direction and a vertical direction,
Each pixel of the image sensor is composed of at least a blue pixel, a green pixel, and a red pixel projected onto an independent region on a paraxial conjugate plane when the photographing lens is mounted,
Green pixels are arranged in a checkered pattern,
Blue pixels and red pixels are arranged not to be adjacent to each other,
The center-of-gravity incident angle of each green pixel is shifted in the first direction with respect to the center incident angle,
The gravity center incident angles of the four green pixels included in the eight pixels adjacent to each green pixel are all shifted in the second direction with respect to the central incident angle,
Second direction, Ri reverse der in the first direction,
Second direction, and wherein Rukoto to have a both horizontal and vertical components.

  According to the present invention, it is possible to provide an imaging device and an imaging device that can acquire defocus information of the entire screen and can easily generate a natural image.

It is a figure which shows the internal structure of the digital camera which concerns on embodiment of this invention. It is a top view which shows the structure of an image pick-up element. It is a top view which shows the other structure of an image pick-up element. It is a figure explaining the gravity center of the incident angle of a pixel. It is a figure explaining the relationship between the amount of defocus of a horizontal direction, and signal strength. It is a figure explaining the relationship between the amount of defocus of a perpendicular direction, and signal strength. It is a figure explaining the relationship between the amount of defocus of a horizontal direction, and signal strength regarding a red pixel and a blue pixel. It is a figure explaining the method to produce | generate a parallax image. It is a flowchart which shows the rough flow of imaging | photography. It is a flowchart which shows the procedure of an imaging | photography subroutine. It is a flowchart which shows the procedure of a defocus amount calculation subroutine. It is a flowchart which shows the procedure of a focusing subroutine. It is a flowchart which shows the procedure of a parallax image generation subroutine. It is a figure which shows the spectral sensitivity characteristic curve of a blue pixel, a green pixel, and a red pixel. It is a figure explaining a blue pixel, a green pixel, and a red pixel from the short wavelength area | region side projected on the independent area | region on a conjugate plane. It is a figure explaining the inclination of an incident light beam, and the direction of a light-shielding part. It is a figure which shows the pixel arrangement | sequence which shields a part of opening of the conventional discrete or some G color pixel. It is a figure which shows the example which has arrange | positioned several pixel with respect to one on-chip lens.

  Hereinafter, the effect by the structure of the imaging device and imaging device of this embodiment is demonstrated. In addition, this invention is not limited by this embodiment. That is, in describing the embodiment, a lot of specific details are included for the purpose of illustration, but various variations and modifications may be added to these details without exceeding the scope of the present invention. Accordingly, the exemplary embodiments of the present invention described below are set forth without loss of generality or limitation to the claimed invention.

(First embodiment)
First, a camera provided with an imaging apparatus according to the first embodiment of the present invention will be described.
FIG. 1 shows the internal configuration of a digital still camera 11 (imaging device). The digital still camera 11 includes an interchangeable lens 11A and a camera body 11B. The interchangeable lens 11A is attached to the camera body 11B by the mount portion 14.

  The interchangeable lens 11 </ b> A (imaging optical system) includes a lens drive control unit 15, a zooming lens 16, a lens 17, a focusing lens 18, and a diaphragm 19. The lens drive control unit 15 is composed of peripheral components such as a microcomputer and a memory.

The lens drive control unit 15 controls the driving of the focusing lens 18 and the diaphragm 19, detects the state of the diaphragm 19, the zooming lens 16 and the focusing lens 18, transmits lens information to the body drive control unit 21, and transmits camera information. Receive and so on.
Further, the zooming lens 16, the lens 17, the focusing lens 18, and the diaphragm 19 constitute an imaging optical system 12.

  The camera body 11B includes an image sensor 20, a body drive control unit 21, an image sensor drive circuit 23, a liquid crystal display element drive circuit 25, a liquid crystal display element 26, a memory card 29, and a pixel signal acquisition unit 30 (calculation unit, image configuration unit). It has.

Pixels to be described later are arranged two-dimensionally on the image sensor 20. The imaging element 20 is disposed on the planned imaging plane of the interchangeable lens 11A, and captures a subject image formed by the interchangeable lens 11A.
The image sensor 20 is driven by the image sensor drive circuit 23. The operation of the image sensor drive circuit 23 is controlled by the body drive control unit 21. As the image pickup device 20 (imager), for example, a CCD (charge coupled device), a CMOS (Complementary Metal Oxide Semiconductor), or a back-illuminated CMOS can be used.

  The body drive control unit 21 includes peripheral components such as a microcomputer and a memory. In addition to the above-described control, the body drive control unit 21 reads the image signal from the image sensor 20, corrects the image signal, detects the defocused state of the image plane, receives lens information from the lens drive control unit 15, and the like. Transmits camera information (defocus amount) and controls the operation of the entire digital still camera.

  The body drive control unit 21 and the lens drive control unit 15 communicate with each other via the electrical contact unit 13 of the mount unit 14 to exchange (transmit / receive) various information. Further, the body drive control unit 21 includes a configuration for executing a defocus amount detection subroutine and an imaging subroutine described later.

  The liquid crystal display element driving circuit 25 drives a liquid crystal display element 26 disposed on the back of the camera body. The display device is not limited to liquid crystal but may be a so-called organic EL display element. Furthermore, the display device may be an electronic viewfinder in which the photographer observes the image displayed on the display device via the eyepiece 27. The memory card 29 is a portable storage medium that can be attached to and detached from the camera body 11B and that stores and stores image signals.

The subject image formed on the image sensor 20 through the interchangeable lens 11 </ b> A is photoelectrically converted by the image sensor 20. Then, the output is sent to the pixel signal acquisition unit 30. The pixel signal acquisition unit 30 (calculation unit, image configuration unit) acquires a pixel signal. Further, the image signal acquisition unit 30 serves as a calculation unit based on output data (first image signal and second image signal described later) of green (referred to as “G color” as appropriate) pixels on the image sensor 20. A defocus amount at each image position is calculated.
Further, hereinafter, blue is appropriately described as “B color” and red is appropriately described as “R color”.

  The calculated defocus amount is sent to the body drive control unit 21. Further, as the image construction unit, reference image data is constructed based on the acquired pixel signal. The reference image data is a so-called 2D image, and processing such as image compression or thinning may be added as necessary.

  The configured reference image data is output to the body drive control unit 21, the liquid crystal display element drive circuit 25, and the memory card 29. In addition, the body drive control unit 21 generates a plurality of parallax image data from the reference image data and the defocus amount at each image position. A method for generating a plurality of parallax image data will be described later.

  The body drive control unit 21 stores the generated image data in the memory card 29 and sends an image signal to the liquid crystal display element drive circuit 25 to cause the liquid crystal display element 26 to display an image. The displayed image may be an image based on reference image data, or an image based on parallax image data if the display device can display parallax images.

  The camera body 11B is provided with operation members (shutter buttons, focus detection position setting members, etc.) (not shown). The body drive control unit 21 detects operation state signals from these operation members. Then, the body drive control unit 21 controls operations (imaging operation, defocus amount detection operation, lens drive control unit communication operation, image processing operation, etc.) according to the detection result.

  The lens drive control unit 15 changes the lens information according to the lens focusing state, zooming state, aperture setting state, aperture opening F-number, and the like. Specifically, the lens drive control unit 15 monitors the positions of the zoom lens 16 and the focusing lens 18 and the stop position of the stop 19.

  Then, the lens drive control unit 15 calculates lens information according to the monitor information or selects lens information according to the monitor information from a lookup table prepared in advance. Further, the lens drive control unit 15 calculates a lens drive amount based on the received defocus amount, and drives the focusing lens 18 to a focal point by a drive source such as a motor (not shown) based on the lens drive amount. To do.

  The present embodiment is not limited to the interchangeable lens type imaging apparatus shown in FIG. 1 but can be applied to a fixed lens type imaging apparatus including what is called a compact camera. The photographing lens is not limited to a zoom lens, and may be a single focus lens.

FIG. 2 shows a pixel array of the image sensor 20 in the first embodiment.
FIG. 2 shows an example of a total of 256 pixels with 16 vertical pixels (L01 to L16) and 16 horizontal pixels (F01 to F16). However, the number of pixels is not limited to this. For example, the total number of pixels may exceed 10 million pixels.
In the following description, row numbers L01 to L16 and column numbers F01 to F16 are shown side by side when a specific pixel is shown. For example, the pixel corresponding to the column F01 in the L01 row is represented by “L01F01”.

Here, the spectral sensitivity characteristics of each pixel will be described.
FIG. 14 shows spectral sensitivity characteristic curves of blue (B color) pixels, green (G color) pixels, and red (R color) pixels. The horizontal axis represents wavelength (nm), and the vertical axis represents relative sensitivity (%). Blue pixels are highly sensitive in the short wavelength region. Red pixels are highly sensitive in the long wavelength region. The green pixel has high sensitivity in the middle region between the main sensitivity region of the blue image and the main sensitivity region of the red pixel.
In general, light in a wavelength region longer than about 650 nm to about 700 nm is cut or attenuated by an infrared cut filter disposed between the photographing lens and the image sensor and enters the image sensor.

Furthermore, the optical positional relationship between the image sensor and the photographic lens will be described.
FIG. 15 is a diagram for explaining blue pixels, green pixels, and red pixels from the short wavelength region side projected onto independent regions on the conjugate plane.

  In FIG. 15, consider the conjugate plane 204 of the image sensor 202 by the photographic lens 201. The image sensor 202 has each pixel 203 (A to P). For example, the conjugate image area corresponding to the pixel D is independent of the conjugate image areas of other pixels.

  This description assumes a paraxial layout. Actually, it overlaps with the projection area of another pixel on the conjugate plane by an aberration of the photographing lens 201, a diffraction phenomenon, or a low-pass filter (not shown) disposed in the optical path between the photographing lens 201 and the image sensor 202. In some cases. However, an image with a high resolving power can be basically obtained by using this paraxial layout.

  FIG. 16 is a diagram for explaining the inclination of the incident light beam and the direction of the light shielding portion. The configuration shown in FIG. 16 is an example in which the exit pupil is divided by decentering the opening of the pixel of the image sensor 20 (FIG. 1) with respect to the center of the photoelectric conversion element. FIG. 16 is a cross-sectional view illustrating the structure of two adjacent pixels of the image sensor 20.

  The pixel 41 has a microlens 42, a smooth layer 43 for forming a plane for forming the microlens 42, a light shielding film 44 for preventing color mixture of color pixels, and a color filter layer in order from the top. A smoothing layer 45 for flattening the surface and a photoelectric conversion element 46 are arranged. Similarly to the pixel 41, the pixel 51 also includes a micro lens 52, a smooth layer 53, a light shielding film 54, a smooth layer 55, and a photoelectric conversion element 56 in order from the top.

  Further, in these pixels 41 and 51, the light shielding films 44 and 54 have openings 48 and 58 that are eccentric to the outside from the central portions 47 and 57 of the photoelectric conversion elements 46 and 56, respectively.

  Here, it can be seen that the inclination direction of the light beam L41 and the inclination direction of the light beam L51 are directions in which a light shielding portion is formed. Further, depending on how the light shielding part is set, the inclination of the incident light beam and the direction of the light shielding part may be reversed. This is the case, for example, when the light shielding portion is arranged on the on-chip lenses 42 and 52.

Next, returning to FIG. For example, the pixel L05F07 is a G color pixel, and is shielded from light as indicated by the hatched portion in the lower right of the opening in the drawing. That is, the center of gravity of the incident angle of this pixel is inclined in the lower right direction , for example, the first direction . This means that the center of gravity of the incident angle is inclined to the right in the horizontal direction as shown in FIG. 4 (a), and is inclined downward in the vertical direction as shown in FIG. 4 (b). .

The G color pixels are arranged in a so-called checkered pattern.
Here, “arranged in a checkered pattern” means, for example, that pixels on a rectangle are alternately arranged with other pixels in both the horizontal direction and the vertical direction.

The four G color pixels included in the eight pixels adjacent to the pixel L05F07 are the four pixels L04F06, L04F08, L06F06, and L06F08. These four pixels are shielded from light so that the upper left of the opening is indicated by the hatched portion. That is, the center of gravity of the incident angle of this pixel is inclined in the upper left direction , for example, the second direction . Here, the second direction is opposite to the first direction.

  This means that the center of gravity of the incident angle is tilted to the left in the horizontal direction and tilted to the upper side in the vertical direction as shown in FIGS. 4 (c) and 4 (d). As described above, the G color pixel of the image sensor 20 is configured such that the direction of the incident angle is different in the horizontal direction and the vertical direction with respect to the adjacent G color pixel in the same direction.

  In other words, the centroid incidence angles of four adjacent G color pixels of each G color pixel are different in the same direction with respect to the center incidence angle with respect to the horizontal component of each G color pixel, and the vertical direction of each G color pixel. The components differ in the same direction with respect to the central incident angle.

  The “center of gravity incident angle” refers to, for example, the area centroid of the region surrounded by the curve in each of FIGS. 4A, 4B, 4C, and 4D.

  As a result, the blurred image in a state away from the focused state becomes natural. Also, the blurred image in the diffraction phenomenon becomes natural, and an excellent image can be obtained. By adopting such a configuration, the incident angle range is inevitably widened so that a large NA light beam can be received, and both defocus detection and image quality can be achieved.

  The relationship between the defocus amount in the horizontal direction of the G color pixel and the signal intensity will be described with reference to FIGS. 5 (a) and (b), FIG. 5 (c) and (d), FIG. 5 (e) and (f), FIG. 5 (g) and (h), and FIG. 5 (i) and (j) , Signal strengths of adjacent pixels in the horizontal direction are shown.

  Here, the horizontal output signals at the time of focusing are shown in FIGS. 5 (e) and 5 (f). When the defocus amount changes, FIGS. 5A and 5B, FIGS. 5C and 5D, FIGS. 5G and 5H, and FIGS. 5I and 5J. It has changed.

  As is clear from FIGS. 5A to 5J, the magnitude of the peak value of the signal intensity curve and its position change according to the defocus amount. As described above, for example, the defocus amount can be detected by comparing the output signals of L04, L06, and L08 in the horizontal direction with the output signals of L05, L07, and L09 by a method such as correlation calculation.

  The relationship between the defocus amount in the vertical direction of the G color pixel and the signal intensity will be described with reference to FIGS. 6A and 6B, FIGS. 6C and 6D, FIGS. 6E and 6F, FIGS. 6G and 6H, and FIGS. 6I and 6J. , Each shows the signal strength in the vertical direction.

  Here, the output signals in the vertical direction at the time of focusing are shown in FIGS. 6 (e) and 6 (f). When the defocus amount changes, FIGS. 6 (a) and 6 (b), FIGS. 6 (c) and 6 (d), FIGS. 6 (g) and 6 (h), and FIGS. 6 (i) and 6 (j). It has changed.

  As is clear from FIGS. 6A to 6J, the magnitude of the peak position of the signal intensity curve and its position change according to the defocus amount. Thus, for example, the defocus amount can be detected by comparing the output signals of F04, F06, and FL08 in the vertical direction with the output signals of F05, F07, and F09 using a technique such as correlation calculation.

  Next, regarding the R color pixel and the B color pixel, the relationship between the horizontal defocus amount and the signal intensity will be described with reference to FIGS. 7A and 7B, FIGS. 7C and 7D, FIGS. 7E and 7F, FIGS. 7G and 7H, and FIGS. 7I and 7J. , Signal strengths of adjacent pixels in the horizontal direction are shown.

  As shown in FIGS. 7A to 7J, even if the R color pixel and the B color pixel are defocused, the peak value of the signal intensity curve changes, but the position of the curve does not change. That is, the contrast of the image is lowered, but no image shift occurs. If an image is formed from a pixel that has not caused an image shift and a pixel that has caused an image shift, the image quality at an unfocused image position deteriorates.

  Thus, the center-of-gravity incidence angle of each B color pixel is substantially the same, and the center-of-gravity incidence angle of each R color pixel is substantially the same.

  In the present embodiment, for example, when an image is formed, the G color component is averaged with adjacent pixel values, so that a G color component without image displacement can be generated with little degradation in resolution. Further, a high-quality image can be obtained using the R color pixel, the B color pixel, and the G color pixel component.

Focusing on the image sensor itself in the present embodiment, the image sensor has the following configuration. That is, the imaging device includes an imaging device in which pixels are arranged in the horizontal direction and the vertical direction,
Each pixel of the image sensor is composed of at least a blue pixel, a green pixel, and a red pixel projected onto an independent region on a paraxial conjugate plane when the photographing lens is mounted,
Green pixels are arranged in a checkered pattern,
Blue pixels and red pixels are arranged not to be adjacent to each other,
The center-of-gravity incident angle of each green pixel is shifted in the first direction with respect to the center incident angle,
The gravity center incident angles of the four green pixels included in the eight pixels adjacent to each green pixel are all shifted in the second direction with respect to the central incident angle,
Second direction, Ri reverse der in the first direction,
Second direction, that have a both horizontal and vertical components.
Thereby, defocus information of the entire screen can be acquired, and a natural image can be easily generated.

  FIG. 17 shows a conventional pixel arrangement in which a part of the apertures of discrete or partial G color pixels are shielded. In this case, it is difficult for the G color component to generate a pixel value that does not cause image shift even when referring to adjacent pixel values.

  In such a case, a processing method is known in which a pixel in which a part of the opening is shielded is not used for image formation like a so-called pixel defect. However, when the defocus amount is calculated for the entire screen, the number of light-shielded pixels increases, and the image quality is greatly degraded. In addition, it is not preferable to dispose such pixel areas discretely because the areas in the screen where defocus values can be obtained are discrete.

  Further, FIG. 18 shows an example in which a plurality of pixels are arranged for one on-chip lens. This example is a reference example for comparison with the present embodiment. In this configuration example, the exit pupil is divided by dividing the photoelectric conversion unit of the image sensor 320.

The imaging device 320 transfers the photocharges accumulated in the n-type regions 332α and 332β and the n-type regions 332α and 332β that generate and accumulate photocharges together with the P-type well 331 and the P-type well 331 formed in the substrate. A floating diffusion portion (not shown) (hereinafter referred to as “FD portion”), a surface p + layer that collects the photocharges in order to efficiently transfer the photocharges accumulated in the n-type regions 332α and 332β to the FD portion. 333α, 333β, a transfer gate (not shown) for transferring photocharge to the FD portion, a SiO ₂ film 334 as a gate insulating film, a Bayer color filter 335, and a microlens 336 that collects light from a subject, Is provided.

  The microlens 336 is formed in a shape and position such that the pupil of the interchangeable lens (FIG. 1) and the surface p + layers 333α and 333β are substantially conjugate. Photoelectric charges are typically generated in the region 337.

  In the example illustrated in FIG. 18, the photoelectric conversion unit is divided into an n-type region 332α and a surface p + layer 333α, and an n-type region 32β and a surface p + layer 33β, whereby the exit pupil is divided. Light rays L31 and L32 enter the n-type region 32α and the surface p + layer 33α, and the n-type region 32β and the surface p + layer 33β, respectively.

  With this configuration, it is possible to easily generate pixel values with no image shift. However, the arrangement of R color pixels and B color pixels is also discrete. For this reason, the resolution for the number of pixels cannot be obtained, which is not preferable.

  Furthermore, the disadvantages of the conventional example when compared with this embodiment will be described below. Needless to say, this embodiment eliminates these problems.

The conventional imaging device has the following problems.
-Image data acquisition and image data acquisition for defocus amount calculation cannot be performed simultaneously.
-It is difficult to calculate the defocus amount for a moving subject.
-The pupil division mechanism is arranged on the photographing lens side.

With the configuration as described above, when the imaging apparatus uses an interchangeable lens, it is necessary to provide a pupil division mechanism for each interchangeable lens. Further, when at least one of the position of the diaphragm and the diameter of the diaphragm changes due to focusing or zooming, the pupil division mechanism also needs to be moved or adjusted in drive amount.
Conventionally, it has been proposed to use the obtained defocus amount for focusing. On the other hand, there is no disclosure of using or outputting image data and defocus amount data in combination.

  In addition to this, there is a method called so-called contrast AF (autofocus), in which a plurality of pieces of image data are acquired and a defocus amount is estimated from a change in contrast value.

  The contrast AF method has a problem that it is difficult to calculate a defocus amount for a moving subject. In addition, it is necessary to correspond to a driving system for the photographing lens.

  A method known as a so-called phase difference AF optical system captures the same object point as two images passing through different pupil positions of the photographing lens. Then, the defocus amount is calculated. In the phase difference AF, a plurality of pixels become conjugate positions of the same object point, and it becomes difficult to obtain high resolution and image quality.

  A method of calculating a defocus amount for only a partial area of the screen and performing focusing is also known. Naturally, with such a method, the defocus amount distribution of the entire screen cannot be obtained. Furthermore, it is considered difficult to use pixels for defocus calculation for image formation.

For example, a method of discretely arranging pixel regions that receive only a part of the pupil on the imaging surface and calculating the defocus amount has the following problems.
-Only defocus information at discrete positions can be obtained. Thereby, it is difficult to form a parallax image.
-Generally, the image will be different during defocusing. Thus, it is difficult to use the AF pixel for pixel formation.
In the configuration in which a microlens that divides the pupil is arranged near the imaging pixel and a plurality of pixels are arranged under the microlens, there is a problem that the resolution with respect to the number of imaging pixels is deteriorated. In addition, the incident NA (numerical aperture) may become small, and the image quality may deteriorate due to the influence of diffraction.

Furthermore, the problem of the conventional example of the imaging system capable of stereoscopic imaging will be described.
There is a conventional binocular stereoscopic imaging system generally called a stereo camera.
The problems of the binocular stereoscopic imaging system are listed below.
・ In the case of a short-distance subject, the parallax increases. This makes it impossible to shoot.
• If the optical axis of the two-lens optical system is directed inward so that a short-distance subject can be photographed, it becomes difficult to photograph a long-distance subject.
If the total number of pixels of the image sensor is the same as that of a general 2D (two-dimensional) imaging system, the number of pixels corresponding to one optical system in the stereo imaging apparatus is halved. Therefore, the image quality is degraded.
In order to obtain the same image quality as an image pickup system for general 2D shooting, an image pickup device having twice the number of pixels is required.
・ Furthermore, the shooting system becomes large in order to use two eyes.

There is also known a system that obtains a three-dimensional shape using information that is projected from an imaging device onto a target and reflected from the target. Such a system has the following problems.
・ It is difficult to obtain information on long-distance subjects.
-It is difficult to obtain a short-distance subject due to parallax problems.
-It is difficult to obtain a three-dimensional shape when the illuminance on the subject side is high.
-Due to the structure of the image sensor, it is difficult to obtain a color image when obtaining three-dimensional information of the entire screen.

Furthermore, a method of automatically converting a so-called single plane image into a stereoscopic image without defocus information is also known.
The problem with this method is that if the clue to convert to a stereoscopic image is in an unexpected state, it may be converted to an unnatural stereoscopic image. For example, specifically, the following problems may occur.
-Converting a flat poster photographed in a flat image into a three-dimensional image.
-When color information is converted into a three-dimensional image as a clue, a phenomenon occurs in which a signal in a subject or a light source such as neon is unnaturally converted.

  As described above, the present embodiment solves these problems.

(Modification)
Next, a pixel array according to a modification of the present embodiment will be described. FIG. 3 shows a pixel array according to the modification. By dividing the pixel into left and right, the defocus detection accuracy for the vertical component can be improved. A general subject often detects defocus in the vertical line direction. For this reason, a configuration with high defocus detection accuracy in the vertical line direction as in the present modification is advantageous.

  In the first embodiment, for example, the upper right, lower right, and lower left of the pixel are opened like the L02F02 pixel. For this reason, it is possible to obtain a sufficient amount of light as compared with the case where the left half is shielded as in this modification, and it is possible to reduce image quality deterioration due to the influence of diffraction.

Next, an example of a parallax image generation method will be described.
FIG. 8 is a diagram illustrating a method for generating a parallax image.
An arbitrary assumed amount of parallax (2 · P) and an assumed observation distance d are set. These numerical values may be fixed values with respect to the camera body, determined using the focal length information of the photographing lens, or input assuming the observation equipment.

The surface 101 is assumed to be the acquired reference image (2D), and has a horizontal cross section in the case shown in FIG.
The virtual line 103 is a perpendicular (virtual line) passing through the center 102 of the acquired reference image (2D), and may be considered to correspond to the optical axis of the photographing lens at the time of photographing. In addition, the coordinates of the center point 102 are (0, 0).

  When the amount of parallax to be given is 2 × P, a virtual line 104 parallel to a position shifted by −P in the horizontal direction with respect to the virtual line 103 is assumed. A point where the virtual line 104 and the reference image intersect is defined as a point 106.

  Similarly, a virtual line 105 parallel to the position shifted by P in the horizontal direction is assumed. A point where the virtual line 105 and the reference image intersect is defined as a point 107. At this time, the respective coordinates are (−P, 0) and (P, 0).

  An example of a method for obtaining the positions of the points 111 and L (x, y) of the left image corresponding to the point 108 at an arbitrary position (x, y) of the reference image will be described. A line parallel to the virtual line 104 passing through an arbitrary point 108 of the reference image is assumed. Assume a point 109 on this line that is deviated by the defocus amount Δd (x, y) of an arbitrary point 108 in the reference image.

  Assuming a virtual line 110 connecting the point 112 and the point 109 that are separated from the point 106 by the virtual observation distance d, the point where this line and the reference image intersect is obtained. The horizontal component at this point becomes the horizontal component at point 111. The vertical component is the same as the point 1008.

This is shown by conditional expressions (1) and (2). ΔLH (x, y) is a horizontal shift amount in the left image with respect to the image point (x, y) of the reference image. Similarly, ΔRH (x, y) is a horizontal shift amount in the right image with respect to the image point (x, y) of the reference image.
ΔLH (x, y) = (d / (d + Δd (x, y))) × (x + P) (1)
ΔRH (x, y) = (d / (d + Δd (x, y))) × (x−P) (2)

In this way, a left image and a right image that are parallax images can be generated from the reference image.
FIG. 8 shows one example of parallax image generation. Depending on the distribution of the defocus amount of the subject, the visibility during 3D observation, the emphasis of the stereoscopic effect, the suppression of the stereoscopic effect, etc., using the defocus amount of each image point, using other generation methods Also good. Further, an assumed cross point may be set as a virtual parameter.

  Depending on the camera posture at the time of shooting (in the case of vertical positioning), a vertical (vertical) parallax image may be generated. When displaying on the observation equipment, for example, the upper image may be displayed as the left image, and the lower image may be displayed as the right image.

  In the present embodiment, a parallax image is generated using defocus amount information. However, the present invention is not limited to this, and other image processing may be performed using the defocus amount information. Further, the defocus amount may be recorded together with the image data, and necessary image processing may be performed by an external device using this recording.

Next, a procedure for performing focusing and photographing using the above-described configuration will be described using a flowchart.
FIG. 9 is a flowchart showing a general flow of shooting.

  In step S101, shooting is started. In step S102, the focusing subroutine is called. In step S103, a shooting subroutine is executed. In step S105, a defocus amount calculation subroutine is executed. In step S104, a parallax image generation subroutine is executed.

Further explanation of the procedure of each subroutine will be continued.
FIG. 10 is a flowchart showing the procedure of the photographing subroutine.
In step S201, the image signal acquisition unit 30 reads a signal from the imaging pixel. In step S202, the body drive control unit 21 records the image signal on the memory card 29 or the like. In step S203, the liquid crystal display element 26 displays a recorded image.

FIG. 11 is a flowchart showing the procedure of the defocus amount calculation subroutine.
In step S301, the image signal acquisition unit 30 reads a signal from the G color pixel. In step S <b> 302, the image signal acquisition unit 30 performs a correlation operation between the odd pixel column and the even pixel column.

  In step S303, the image signal acquisition unit 30 performs a correlation operation between the odd-numbered pixel rows and the even-numbered pixel rows. In step S304, the image signal acquisition unit 30 calculates the defocus amount at each image point. Then, the body drive control unit 21 records the defocus amount of each image point on the memory card 29 or the like.

FIG. 12 is a flowchart showing the procedure of the focusing subroutine.
In step S401, the image signal acquisition unit 30 reads a signal of a G color pixel that is a distance measurement area. In step S <b> 402, the image signal acquisition unit 30 performs a correlation operation between the odd pixel column and the even pixel column.

  In parallel with step S402, in step S403, the image signal acquisition unit 30 performs correlation calculation between the odd-numbered pixel rows and the even-numbered pixel rows. In step S <b> 404, the image signal acquisition unit 30 functions as a calculation unit and outputs data (first image signal and second image signal described later) of green (referred to as “G” as appropriate) color pixels on the image sensor 20. ) To calculate the defocus amount at each image position.

  In step S404, it is determined whether or not the calculated defocus amount is within the depth of focus. If the determination result in step S405 is Yes (true), the subroutine is terminated.

  If the determination result of step S405 is No (false), the process proceeds to step S406. In step S <b> 406, the image signal acquisition unit 30 outputs a driving signal for the focusing lens 18 among the photographing lenses. The lens drive control unit 15 drives the focusing lens 18 of the photographic lens. Then, step S401 is repeated.

FIG. 13 is a flowchart showing the procedure of the parallax image generation subroutine. This subroutine executes the contents described in FIG.
In step S501, an assumed parallax amount, an assumed observation distance, and an assumed crosspoint position are set.
In step S502, the body drive control unit 21 records the image signal on the memory card 29 or the like and the image signal recording data and the defocus amount of each image point.

  In step S503, it is determined whether or not a parallax direction is set. If it is determined in step S503 that the parallax direction is set in the horizontal direction, the process proceeds to steps S504 and S505. In step S504, the right image is generated. In step S505, the left image is generated.

  If it is determined in step S503 that the parallax direction is set in the vertical direction, the process proceeds to step S506 and step S507. In step S506, an upper image is generated. In step S507, a lower image is generated.

In either case of the horizontal parallax setting and the vertical parallax setting, the parallax image is recorded in step S508.
Thus, a plurality of parallax images can be formed by the defocus information and the image output.

To summarize the above description, the digital camera 11 of the present embodiment and its modification example can be mounted with a photographic lens or provided with a photographic lens.
It includes an image sensor 20 in which pixels are arranged in a horizontal direction and a vertical direction,
An image signal acquisition unit 30 having a function of processing a signal from the image sensor 20 is provided.
Further, the image sensor 20 is configured by blue pixels, green pixels, and red pixels projected onto independent regions at least on the conjugate plane 204.
Thereby, the resolution of a picked-up image is securable by making it the structure which does not overlap on a paraxial conjugate plane. This is a countermeasure against a refocus type or a type in which one on-chip lens is arranged on a plurality of pixels that lower the resolution instead of acquiring defocus information as shown in FIG. Further, the B color pixel, the G color pixel, and the R color pixel enable acquisition of a color image, which is advantageous compared to a monochrome type imaging apparatus.
Green pixels are arranged in a checkered pattern.
Here, the blue pixels and the red pixels are arranged so as not to be adjacent to each other.
This defines a so-called Bayer array. The configuration in which the G color pixels are continuously arranged is characteristic.
In addition, as described above, the center-of-gravity incidence angle of each green pixel is shifted in the first direction with respect to the center incidence angle, and the center-of-gravity incidence of four green pixels included in the eight pixels adjacent to each green pixel. The angles are all shifted in the second direction with respect to the central incident angle, and the second direction is the opposite direction of the first direction .
As a result, the correlation calculation can be performed on the entire screen with either the horizontal component or the vertical component. Also, defocus information for the entire screen can be acquired. Here, it is only necessary to obtain the depth relationship with the peripheral image point, not the absolute amount.
Furthermore, by interpolating between adjacent pixels, information on the entire pupil of the photographing lens can be obtained. As a result, a natural image can be formed from the vicinity of the in-focus point to the non-in-focus point.
Each pixel outputs an image for image formation.
With the above-described pixel configuration, a signal that can easily generate a natural image is output.
Then, the image signal acquisition unit 20 calculates and acquires image defocus information from the output of the green pixel.
Thereby, defocus information and contour information can be acquired by phase difference detection or the like.
Further, a plurality of parallax images are formed by defocus information and image output. Thereby, a plurality of parallax images can be formed based on the defocus information.

  With the configuration of the embodiment described above, it is possible to obtain good color image information and defocus information of the entire screen from one shooting. In addition, by performing image processing that combines the image information and the defocus information, there is an effect that the expression of shooting can be enriched by generating a parallax image or the like.

  As described above, the above-described embodiment provides an imaging device that can derive the defocus amount of the entire screen and that can obtain good image quality, and an imaging device applied to the imaging device. Furthermore, the present invention can also be provided for application to an imaging system capable of high-quality stereoscopic imaging, for example. Here, “good quality” means, for example, that it is small and can be photographed simultaneously from a short distance to a long distance. As described above, a high-quality stereoscopic imaging system can be configured by providing a high-quality stereoscopic imaging system or by easily preparing a parallax image generated according to the conditions of the subject and the imaging lens.

  As described above, the present invention is suitable for a phase difference AF type imaging apparatus.

11 Digital Still Camera 11A Interchangeable Lens
DESCRIPTION OF SYMBOLS 11B Camera body 12 Imaging optical system 14 Mount part 15 Lens drive control part 16 Zooming lens 17 Lens 18 Focusing lens 19 Aperture 20 Imaging element 21 Body drive control part 23 Imaging element drive circuit 25 Liquid crystal display element drive circuit 26 Liquid crystal display element 29 Memory card 30 Pixel signal acquisition unit (calculation unit, image configuration unit)
41 Pixel 42 Microlens 43 Smoothing layer 44 Light-shielding film 45 Smoothing layer 46 Photoelectric conversion element 47 Center part, 57
48 Opening 52 Microlens 53 Smoothing Layer 54 Light-shielding Film 55 Smoothing Layer 56 Photoelectric Conversion Element 57 Center 58 Opening L41 Ray L51 Ray 331 P-type Well 332α, 332β n-type Region 333α, 333β Surface p + Layer SiO ₂ Film 334
335 Color filter 336 Micro lens

Claims (4)

A photographic lens can be attached or equipped with a photographic lens.
It has an image sensor with pixels arranged in the horizontal and vertical directions,
It has a function to process signals from the image sensor,
The image sensor is composed of a blue pixel, a green pixel, and a red pixel projected onto an independent region on at least a conjugate plane,
The green pixels are arranged in a checkered pattern,
The blue pixels and the red pixels are arranged not to be adjacent to each other,
The center-of-gravity incident angle of each green pixel is shifted in the first direction with respect to the center incident angle,
The centroid incident angles of the four green pixels included in the eight pixels adjacent to the green pixels are all shifted in the second direction with respect to the central incident angle.
The second direction is opposite to the first direction;
The second direction has both a horizontal component and a vertical component;
The image is output for image formation.
An image pickup apparatus that acquires image defocus information from an output of the green pixel. The imaging apparatus according to claim 1, wherein a plurality of parallax images are formed by the defocus information and the image output. The blue pixels have substantially the same center-of-gravity incidence angle,
  The imaging apparatus according to claim 1, wherein the red pixels have substantially the same center-of-gravity incidence angle. An image sensor in which pixels are arranged in a horizontal direction and a vertical direction,
Each pixel of the image sensor is composed of at least a blue pixel, a green pixel, and a red pixel projected onto an independent region on a paraxial conjugate plane when the photographing lens is mounted,
Green pixels are arranged in a checkered pattern,
The blue pixels and the red pixels are arranged not to be adjacent to each other,
The center-of-gravity incident angle of each green pixel is shifted in the first direction with respect to the center incident angle,
The centroid incident angles of the four green pixels included in the eight pixels adjacent to the green pixels are all shifted in the second direction with respect to the central incident angle.
The second direction is opposite to the first direction;
The image sensor according to claim 2, wherein the second direction has both a horizontal component and a vertical component.

Images