JP5887845B2 - Image processing method and image processing program - Google Patents

Description

  The present invention relates to an image processing method and an image processing program.

A stereo imaging device that captures a stereo image composed of a right-eye image and a left-eye image using two imaging optical systems is known. Such a stereo imaging device causes parallax to occur in two images obtained by imaging the same subject by arranging two imaging optical systems at regular intervals.
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP-A-8-47001

  However, in order to acquire an image that generates a plurality of parallaxes, it has been necessary to prepare a complicated photographing optical system corresponding to the number of images.

  In the image processing method according to the first aspect of the present invention, imaging is provided in which a plurality of types of aperture masks corresponding to a plurality of parallaxes are provided in a one-to-one correspondence with a plurality of two-dimensionally arranged photoelectric conversion elements. The first parallax image acquisition stage that acquires a plurality of first parallax images captured by the element, and the optical axis of the incident light more than the first parallax image acquisition stage by the diaphragm provided for the entire plurality of photoelectric conversion elements A second parallax image acquisition step of acquiring a plurality of second parallax images captured by an imaging device by shielding a peripheral luminous flux with respect to the first parallax image and a second parallax image for each of the plurality of parallaxes; And a parallax image generation step of generating a third parallax image corresponding to a new parallax different from a plurality of parallaxes corresponding to a plurality of types of aperture masks.

  In the image processing method according to the second aspect of the present invention, imaging is provided in which a plurality of types of aperture masks corresponding to a plurality of parallaxes are provided in a one-to-one correspondence with a plurality of photoelectric conversion elements arranged two-dimensionally. The first parallax image acquisition stage that acquires a plurality of first parallax images captured by the element, and the optical axis of the incident light more than the first parallax image acquisition stage by the diaphragm provided for the entire plurality of photoelectric conversion elements The second parallax image acquisition step of acquiring a plurality of second parallax images picked up by the imaging device by shielding the peripheral luminous flux with respect to the image, and associating the first parallax image and the second parallax image with different parallaxes And an image group generation stage for generating a parallax image group.

  The image processing program according to the third aspect of the present invention is an image pickup in which a plurality of types of aperture masks corresponding to a plurality of parallaxes are provided in a one-to-one correspondence with a plurality of photoelectric conversion elements arranged two-dimensionally. The first parallax image acquisition step for acquiring a plurality of first parallax images captured by the element, and the optical axis of the incident light more than the first parallax image acquisition stage by the diaphragm provided for the entire plurality of photoelectric conversion elements. A second parallax image acquisition step of acquiring a plurality of second parallax images captured by an imaging device by shielding a peripheral luminous flux with respect to the first parallax image and a second parallax image for each of the plurality of parallaxes; Based on the difference, the computer executes a parallax image generation step for generating a third parallax image corresponding to a new parallax different from the plurality of parallaxes corresponding to the plurality of types of aperture masks.

  The image processing program according to the fourth aspect of the present invention is an image pickup in which a plurality of types of aperture masks corresponding to a plurality of parallaxes are provided in a one-to-one correspondence with a plurality of photoelectric conversion elements arranged two-dimensionally. The first parallax image acquisition step for acquiring a plurality of first parallax images captured by the element, and the optical axis of the incident light more than the first parallax image acquisition stage by the diaphragm provided for the entire plurality of photoelectric conversion elements. The second parallax image acquisition step of acquiring a plurality of second parallax images captured by the imaging element by shielding the peripheral luminous flux with respect to the first parallax image and the second parallax image are associated with different parallaxes. Then, an image group generation step for generating a parallax image group is executed by a computer.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is a figure explaining the structure of the digital camera which concerns on embodiment of this invention. It is the schematic showing the cross section of the image pick-up element which concerns on embodiment of this invention. It is the schematic showing a mode that a part of imaging device concerning a 1st embodiment was expanded. It is a conceptual diagram explaining the relationship between a parallax pixel and a to-be-photographed object. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is an example of the synthetic aperture corresponding to the new parallax image formed based on the difference. It is a conceptual diagram explaining the process which produces | generates a parallax image. It is a figure explaining a Bayer arrangement. It is a figure which shows an example of a variation about allocation of the parallax pixel with respect to a Bayer arrangement | sequence. It is a figure explaining the shape of the opening mask which concerns on 2nd Embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is an example of the synthetic aperture corresponding to the new parallax image formed based on the difference. It is a figure explaining the shape of the opening mask which concerns on 3rd Embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is the schematic explaining the relationship between the aperture_diaphragm | restriction and synthetic aperture which concern on this embodiment. It is an example of the synthetic aperture corresponding to the new parallax image formed based on the difference. It is a figure which shows the method to colorize the parallax pixel which concerns on 4th Embodiment. FIG. 26 is a conceptual diagram illustrating a method for obtaining color parallax images having different parallaxes from the color parallax images obtained in FIG. 25. It is a figure which shows the processing flow from the acquisition of a 1st parallax image to the production | generation of the parallax image. It is a figure which shows the processing flow from the acquisition of a 1st parallax image to the production | generation of the parallax image group.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  The digital camera according to the present embodiment, which is one form of the imaging apparatus, generates images with a plurality of viewpoints for one scene by one shooting. Furthermore, the digital camera according to the present embodiment generates an image corresponding to a viewpoint different from the plurality of viewpoints generated at the time of shooting for the same scene by shooting with the aperture value changed from the same shooting position. . Each image having a different viewpoint is called a parallax image.

  FIG. 1 is a diagram illustrating the configuration of a digital camera 10 according to an embodiment of the present invention. The digital camera 10 includes a photographic lens 20 as a photographic optical system, and guides a subject light beam incident along the optical axis 21 to the image sensor 100. The photographing lens 20 may be an interchangeable lens that can be attached to and detached from the digital camera 10. The digital camera 10 includes an aperture 40, an image sensor 100, a drive unit 204, a control unit 201, an A / D conversion circuit 202, a memory 203, an image processing unit 205, a memory card IF 207, an operation unit 208, a display unit 209, and an LCD drive circuit. 210 and an AF sensor 211.

  As shown in the figure, the direction parallel to the optical axis 21 toward the image sensor 100 is defined as the z-axis plus direction, the direction toward the front of the drawing in the plane orthogonal to the z-axis is the x-axis plus direction, and the upward direction on the drawing is y. The axis is defined as the plus direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

  The taking lens 20 is composed of a plurality of optical lens groups, and forms an image of a subject light flux from the scene in the vicinity of its focal plane. In FIG. 1, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens arranged in the vicinity of the pupil.

  The diaphragm 40 is disposed along the optical axis 21 together with the photographing lens 20. In FIG. 1, the diaphragm 40 is disposed along the optical axis 21 closer to the image sensor 100 than the photographing lens 20. However, when the photographing lens 20 includes a plurality of lens groups, the optical axis 21 is arranged along the optical axis 21. Arranged between any of the lens groups. An example of the diaphragm 40 is an iris diaphragm having a plurality of diaphragm blades. When the diaphragm 40 is an iris diaphragm, a plurality of diaphragm blades are electromagnetically driven by the control of the control unit 201 to limit the amount of light passing through the subject light beam in a stepwise manner. One aperture 40 is provided for the entire image sensor 100.

  The image sensor 100 is disposed near the focal plane of the photographic lens 20. The image sensor 100 is an image sensor such as a CCD or CMOS sensor in which a plurality of photoelectric conversion elements are two-dimensionally arranged. The image sensor 100 is controlled in timing by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202.

  The A / D conversion circuit 202 converts the image signal output from the image sensor 100 into a digital image signal and outputs the digital image signal to the memory 203. The image processing unit 205 performs various image processing using the memory 203 as a work space, and generates image data. The image processing unit 205 generates new parallax image data from a plurality of parallax image data captured with different aperture values.

  In addition, the image processing unit 205 generates 2D image data and parallax image data as non-parallax image data from an input image signal in accordance with the pixel arrangement of the image sensor 100, or selects a selected image format. The function of adjusting the image data according to the above is also assumed. The generated image data is converted into a display signal by the LCD drive circuit 210 and displayed on the display unit 209. The data is recorded on the memory card 220 attached to the memory card IF 207.

  The AF sensor 211 is a phase difference sensor in which a plurality of distance measuring points are set for the subject space, and detects the defocus amount of the subject image at each distance measuring point. A series of shooting sequences is started when the operation unit 208 receives a user operation and outputs an operation signal to the control unit 201. Various operations such as AF and AE accompanying the imaging sequence are executed under the control of the control unit 201. For example, the control unit 201 analyzes the detection signal of the AF sensor 211 and executes focus control for moving a focus lens that constitutes a part of the photographing lens 20.

  Next, the configuration of the image sensor 100 will be described in detail. FIG. 2 is a schematic diagram showing a cross section of the image sensor according to the embodiment of the present invention. FIG. 2A is a schematic cross-sectional view of the image sensor 100 in which the color filter 102 and the aperture mask 103 are separately formed. FIG. 2B is a schematic cross-sectional view of an image pickup device 120 including a screen filter 121 in which a color filter portion 122 and an opening mask portion 123 are integrally formed as a modification of the image pickup device 100.

  As shown in FIG. 2A, the image sensor 100 is configured by arranging a micro lens 101, a color filter 102, an aperture mask 103, a wiring layer 105, and a photoelectric conversion element 108 in this order from the subject side. The photoelectric conversion element 108 is configured by a photodiode that converts incident light into an electrical signal. A plurality of photoelectric conversion elements 108 are two-dimensionally arranged on the surface of the substrate 109.

  An image signal converted by the photoelectric conversion element 108, a control signal for controlling the photoelectric conversion element 108, and the like are transmitted and received through the wiring 106 provided in the wiring layer 105. In addition, an opening mask 103 having openings 104 provided in one-to-one correspondence with each photoelectric conversion element 108 is provided in contact with the wiring layer. As will be described later, the opening 104 is shifted for each corresponding photoelectric conversion element 108 so that the relative position is precisely determined. As will be described in detail later, parallax occurs in the subject light beam received by the photoelectric conversion element 108 by the action of the opening mask 103 including the opening 104.

  On the other hand, the aperture mask 103 does not exist on the photoelectric conversion element 108 that does not generate parallax. In other words, it can be said that the aperture mask 103 having the aperture 104 that does not limit the subject luminous flux incident on the corresponding photoelectric conversion element 108, that is, allows the entire effective luminous flux to pass therethrough is provided. Although no parallax is generated, the aperture 107 formed by the wiring 106 defines the subject luminous flux that is incident, so the wiring 106 is regarded as an aperture mask that allows the entire effective luminous flux that does not cause parallax to pass. You can also. The opening mask 103 may be arranged separately and independently corresponding to each photoelectric conversion element 108, or may be formed collectively for a plurality of photoelectric conversion elements 108 in the same manner as the manufacturing process of the color filter 102. .

  The color filter 102 is provided on the opening mask 103. The color filter 102 is a filter provided in a one-to-one correspondence with each photoelectric conversion element 108, which is colored so as to transmit a specific wavelength band to each photoelectric conversion element 108. In order to output a color image, it is only necessary to arrange at least two types of color filters that are different from each other. However, in order to obtain a higher quality color image, it is preferable to arrange three or more types of color filters. For example, a red filter that transmits a red wavelength band, a green filter that transmits a green wavelength band, and a blue filter that transmits a blue wavelength band may be arranged in a lattice pattern. A specific arrangement will be described later.

  The microlens 101 is provided on the color filter 102. The microlens 101 is a condensing lens for guiding more incident subject light flux to the photoelectric conversion element 108. The microlenses 101 are provided in a one-to-one correspondence with the photoelectric conversion elements 108. In consideration of the relative positional relationship between the pupil center of the taking lens 20 and the photoelectric conversion element 108, the optical axis of the microlens 101 is shifted so that more subject light flux is guided to the photoelectric conversion element 108. It is preferable. Furthermore, the arrangement position may be adjusted so that more specific subject light beam, which will be described later, is incident along with the position of the opening 104 of the opening mask 103.

  As described above, one unit of the aperture mask 103, the color filter 102, and the microlens 101 provided in one-to-one correspondence with each photoelectric conversion element 108 is referred to as a pixel. In particular, a pixel provided with the opening mask 103 that generates parallax is referred to as a parallax pixel, and a pixel that is not provided with the opening mask 103 that generates parallax is referred to as a non-parallax pixel. For example, when the effective pixel area of the image sensor 100 is about 24 mm × 16 mm, the number of pixels reaches about 12 million.

  Note that in the case of an image sensor with good light collection efficiency and photoelectric conversion efficiency, the microlens 101 may not be provided. In the case of a back-illuminated image sensor, the wiring layer 105 is provided on the side opposite to the photoelectric conversion element 108.

  There are various variations in the combination of the color filter 102 and the aperture mask 103. In FIG. 2A, the color filter 102 and the opening mask 103 can be integrally formed if the opening 104 of the opening mask 103 has a color component. In addition, when a specific pixel is specialized as a pixel for obtaining luminance information of a subject, the corresponding color filter 102 does not need to be provided for the pixel. Or you may arrange | position the transparent filter which does not give coloring so that the substantially all wavelength band of visible light may be permeate | transmitted.

  When a pixel for obtaining luminance information is a parallax pixel, that is, when a parallax image is output as a monochrome image, the configuration of the image sensor 120 shown in FIG. 2B can be adopted. That is, the screen filter 121 in which the color filter part 122 that functions as a color filter and the opening mask part 123 having the opening 104 are integrally formed may be disposed between the microlens 101 and the wiring layer 105. it can.

  The screen filter 121 is formed by, for example, blue-green-red coloring in the color filter portion 122 and black in the opening mask portion 123 other than the opening portion 104. Since the image sensor 120 that employs the screen filter 121 has a shorter distance from the microlens 101 to the photoelectric conversion element 108 than the image sensor 100, the light collection efficiency of the subject light flux is high.

  Next, the relationship between the opening 104 of the opening mask 103 and the generated parallax will be described. FIG. 3 is a schematic diagram illustrating a state in which a part of the image sensor 100 according to the first embodiment is enlarged. Here, in order to simplify the explanation, the color arrangement of the color filter 102 is not considered until it is mentioned later. In the following description that does not refer to the color arrangement of the color filter 102, it can be considered that the image sensor is a collection of only parallax pixels having the color filter 102 of the same color. Therefore, the repetitive pattern described below may be considered as an adjacent pixel in the color filter 102 of the same color.

  As shown in FIG. 3, there are two types of opening masks 103, one having an opening 104l and the other having an opening 104r. The openings 104l and 104r are both rectangular. The openings 104l and 104r are provided so as to be relatively shifted with respect to the respective pixels. In the adjacent pixels, the openings 104l and 104r are provided at positions displaced from each other.

  In the example of FIG. 3, the opening 104l is shifted in the −x direction, and the opening 104r is shifted in the x direction. The entire imaging element 100 is a two-dimensional and periodic array of photoelectric conversion element groups each including a pair of two parallax pixels each having an opening mask 103 having an opening 104l and an opening mask 103 having an opening 104r. Has been. That is, in the image sensor 100, the repeating pattern 110 including the photoelectric conversion element group is periodically spread. Here, the x direction is, for example, the left-right direction when the digital camera 10 is imaged in the horizontal position.

  FIG. 4 is a conceptual diagram illustrating the relationship between the parallax pixels and the subject. 4A shows a photoelectric conversion element group of a repetitive pattern 110t arranged at the center orthogonal to the optical axis 21 in the image sensor 100, and FIG. 4B shows a repetitive pattern arranged in the peripheral portion. A 110 u photoelectric conversion element group is schematically shown. The subject 30 in FIGS. 4A and 4B is in the in-focus position with respect to the photographic lens 20. FIG. 4C schematically shows a relationship when the subject 30 existing at the out-of-focus position with respect to the photographing lens 20 is captured corresponding to FIG.

  First, the relationship between the parallax pixels and the subject when the photographing lens 20 captures the subject 30 that is in focus will be described. The subject light flux passes through the pupil of the photographic lens 20 and is guided to the image sensor 100. Two partial areas Pl and Pr are defined for the entire cross-sectional area through which the subject light flux passes. For example, in the pixel on the left side of the sheet of the photoelectric conversion element group constituting the repetitive patterns 110t and 110u, as can be seen from the enlarged view, only the subject luminous flux emitted from the partial region Pr reaches the photoelectric conversion element 108. The position of the opening 104r of the opening mask 103 is determined. Similarly, the position of the opening 104l is determined corresponding to the partial region Pl toward the right pixel.

  In other words, the position of the opening 104r is determined by the inclination of the principal ray Rr of the subject light beam emitted from the partial region Pr, which is defined by the relative positional relationship between the partial region Pr and the left pixel, for example. You can say. Then, when the photoelectric conversion element 108 receives the subject luminous flux from the subject 30 present at the in-focus position via the opening 104r, the subject luminous flux is coupled on the photoelectric conversion element 108 as shown by the dotted line. Image. Similarly, it can be said that the position of the opening 104l is determined by the inclination of the principal ray Rl toward the right pixel.

  As shown in FIG. 4A, the light beam emitted from the minute region Ot on the subject 30 that intersects the optical axis 21 among the subject 30 existing at the in-focus position passes through the pupil of the photographing lens 20. Then, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t is reached. That is, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t receives a light beam emitted from one minute region Ot through two partial regions Pr and Pl. Although the minute region Ot has an extent corresponding to the positional deviation of each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t, it can be approximated to substantially the same object point. Similarly, as shown in FIG. 4B, the light beam emitted from the minute region Ou on the subject 30 that is separated from the optical axis 21 among the subject 30 that exists at the in-focus position passes through the pupil of the photographing lens 20. It passes through and reaches each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u. That is, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u receives a light beam emitted from one minute region Ou through two partial regions Pr and Pl. Similarly to the micro area Ot, the micro area Ou has an extent corresponding to the positional deviation of each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u, but substantially the same object point. Can be approximated.

  In other words, as long as the subject 30 exists at the in-focus position, the minute area captured by the photoelectric conversion element group differs according to the position of the repetitive pattern 110 on the image sensor 100, and each pixel constituting the photoelectric conversion element group Captures the same minute region through different partial regions. In each repetitive pattern 110, corresponding pixels receive the subject luminous flux from the same partial area. In other words, in the drawing, for example, the left pixel of each of the repeated patterns 110t and 110u receives the subject luminous flux from the same partial region Pr.

  In the repetitive pattern 110t arranged in the center orthogonal to the optical axis 21, the left pixel in the repetitive pattern 110u arranged in the peripheral portion and the position of the opening 104r where the left pixel receives the subject light beam from the partial region Pr. The position of the opening 104r that receives the subject light flux from the partial region Pr is strictly different. However, from a functional point of view, these can be treated as the same type of aperture masks in terms of aperture masks for receiving the subject luminous flux from the partial region Pr. Therefore, in the example of FIG. 4, it can be said that each of the parallax pixels arranged on the image sensor 100 includes one of two types of opening masks corresponding to two parallaxes.

  Next, the relationship between the parallax pixels and the subject when the photographing lens 20 captures the subject 30 existing in the out-of-focus state will be described. Also in this case, the subject luminous flux from the subject 30 existing at the out-of-focus position passes through the two partial regions Pr and Pl of the pupil of the photographing lens 20 and reaches the image sensor 100. However, the subject light flux from the subject 30 existing at the out-of-focus position forms an image at another position, not on the photoelectric conversion element 108. For example, as shown in FIG. 4C, when the subject 30 exists at a position farther from the image sensor 100 than the subject 30, the subject luminous flux forms an image on the subject 30 side with respect to the photoelectric conversion element 108. On the other hand, when the subject is present at a position closer to the image sensor 100 than the subject 30 in FIGS. 4A and 4B, the subject luminous flux forms an image on the opposite side of the subject 30 from the photoelectric conversion element 108. To do.

Therefore, the subject luminous flux radiated from the minute region Ot ′ among the subjects 30 existing at the out-of-focus position depends on which of the two partial regions Pr and Pl corresponds to the corresponding pixels in the different sets of repetitive patterns 110. To reach. For example, as shown in the enlarged view of FIG. 4C, the subject luminous flux that has passed through the partial region P1 is incident on the photoelectric conversion element 108 having the opening 104l included in the repetitive pattern 110t ′ as the principal ray Rl ′. To do. Even if the subject light beam is emitted from the minute region Ot ′, the subject light beam that has passed through another partial region does not enter the photoelectric conversion element 108 included in the repetitive pattern 110t ′, and the repetitive pattern in the other repetitive pattern. The light enters the photoelectric conversion element 108 having a corresponding opening. In other words, the subject luminous flux that reaches each photoelectric conversion element 108 constituting the repetitive pattern 110 t ′ is a subject luminous flux emitted from a minute area different from the subject 30. That is, a subject luminous flux having a principal ray R1 ′ is incident on the photoelectric conversion element 108 corresponding to the left opening 104l, and the principal ray is Rr ⁺ to the photoelectric conversion element 108 corresponding to the right opening 104r. A subject luminous flux is incident, and this subject luminous flux is a subject luminous flux emitted from a minute region different from Ot ′ of the subject 30. Such a relationship is the same in the repeated pattern 110u arranged in the peripheral portion in FIG.

  Then, when viewed as a whole of the image sensor 100, for example, the left parallax image obtained by capturing the subject 30 with the photoelectric conversion element 108 corresponding to the opening 104l and the subject 30 with the photoelectric conversion element 108 corresponding to the opening 104r. If the right parallax image is an image with respect to a subject existing at the in-focus position, there is no deviation, and if it is an image with respect to a subject present in the out-of-focus position, a deviation occurs. The direction and amount of the shift are determined by how much the subject existing at the out-of-focus position is deviated from the in-focus position and by the distance between the partial area Pr and the partial area Pl. That is, the left parallax image and the right parallax image are parallax images.

  Therefore, when the outputs of the pixels corresponding to each other are collected in each of the repetitive patterns 110 configured as described above, a plurality of parallax images are obtained. That is, the output of the pixel that has received the subject luminous flux emitted from a specific partial area of the two partial areas Pl and Pr forms two parallax images.

  Although two parallax images are obtained using two types of aperture masks, three or more types of aperture masks can be provided. In this case, three or more parallax images can be obtained according to the number of types of aperture masks.

  FIG. 5A to FIG. 5D are schematic diagrams for explaining the relationship between the diaphragm and the synthetic aperture according to the present embodiment. The imaging element 100 is provided with an opening mask 103 having the rectangular openings 104l and 104r shown in FIG. The diaphragm 40a in FIG. 5A shows a state where the diaphragm 40 is fully opened, for example, a state where the F value is 1.8.

  As shown in FIG. 5B, the light beam emitted from the object point Ot passes through the photographing lens 20 and then enters the stop 40a. Out of the light beams incident on the diaphragm 40a, the light beams on the outer periphery are shielded by the diaphragm 40a. As a result, the light flux near the optical axis enters the image sensor 100.

  Of the light beam that has reached the image sensor 100, the light beam that has passed through the partial region Pl passes through the opening 104l and enters the photoelectric conversion element corresponding to the opening 104l. The light beam that has passed through the partial region Pr does not enter the photoelectric conversion element corresponding to the opening 104l. Therefore, light that passes through both the aperture 40a and the opening 104l and reaches the photoelectric conversion element is light that has passed through an aperture that optically overlaps the aperture 40a and the aperture 104l at the position of the aperture 40a. Optically equivalent. This “optically overlapped aperture” is referred to as a synthetic aperture.

  FIG. 5C shows a synthetic aperture 130la in which the diaphragm 40a and the aperture 104l are optically superimposed. The synthetic aperture 130la has a substantially semicircular shape corresponding to the left half of the substantially circular aperture 40a. The center of gravity of the transmission region through which light passes through the synthetic aperture 130la exists on the left side of the drawing with respect to the center of the circle of the stop 40a. In the figure, the center of gravity is indicated by X.

  FIG. 5D shows a synthetic aperture 130ra in which the aperture 40a and the aperture 104r are optically superimposed. The synthetic aperture 130ra has a substantially semicircular shape corresponding to the right half of the substantially circular diaphragm 40a. The center of gravity of the transmission region of the synthetic aperture 130ra exists on the right side of the drawing with respect to the center of the circle of the aperture 40a.

  The center of gravity of the transmission region of the synthetic opening 130la corresponds to the center of gravity of the partial region Pl. Therefore, the shift amount by which the center of gravity of the transmission region of the synthetic aperture 130la is shifted from the diaphragm 40a corresponds to the parallax amount of the parallax image obtained by the diaphragm 40a and the opening 104l. Similarly, the shift amount of the center of gravity of the transmission region of the synthetic aperture 130ra corresponds to the parallax amount of the parallax image obtained by the stop 40a and the aperture 104r.

  As described above, in this embodiment, two parallax images are obtained by the light fluxes passing through the synthetic aperture 130la and the plurality of partial regions Pl and Pr having a shape corresponding to the synthetic aperture 130ra.

  FIGS. 6A to 6D are schematic diagrams for explaining the relationship between other aperture values and the synthetic aperture. A diaphragm 40b in FIG. 6A shows a state in which the diaphragm 40 is made narrower than the diaphragm 40a in FIG. 5A, for example, a state in which the F value is set to 4.0. A part of the light beam radiated from the object point Ot and passed through the photographing lens 20 is shielded by the aperture 40 b and enters the image sensor 100.

  Since the aperture area of the diaphragm 40b is narrower than that of the diaphragm 40a, the luminous flux in the area closer to the optical axis than the diaphragm 40a among the luminous flux of incident light is also shielded. Therefore, the partial areas Pl and Pr that pass through the stop 40b are smaller than the partial areas Pl and Pr that pass through the stop 40a.

  As shown in FIG. 6C, the synthetic aperture 130lb corresponding to the partial region Pl has a semicircular radius shorter than that of the synthetic aperture 130la in FIG. It is closer to the center of the circle. Similarly, as shown in FIG. 6D, the substantially semicircular synthetic opening 130rb corresponding to the partial region Pr has a shorter semicircular radius than the synthetic opening 130ra in FIG. Thus, the position of the center of gravity of the transmission region is closer to the center of the circle.

  Therefore, the parallax images corresponding to the synthetic apertures 130lb and 130rb have a smaller parallax amount than the parallax images corresponding to the synthetic apertures 130la and 130ra. In this way, by changing the aperture value, parallax images with different amounts of parallax can be obtained for one scene without changing the configuration of the image sensor 100 or the shooting position. That is, by changing the aperture value, it is possible to obtain parallax images from different viewpoints for one scene without changing the configuration of the image sensor 100 or the shooting position and without using a plurality of optical systems. .

  FIG. 7A to FIG. 8D are schematic diagrams for explaining the relationship between still another diaphragm and the synthetic aperture. The diaphragm 40c in FIG. 7A shows a state in which the diaphragm 40 is narrowed more than the diaphragm 40b in FIG. 6, for example, a state where the F value is 8.0, and the diaphragm 40d in FIG. For example, the F value is 11.0.

  The diaphragm 40c in FIG. 7 (a) has a narrower transmission region than the diaphragm 40b in FIG. 6 (a), and therefore corresponds to the synthetic apertures 130lc and 130rc as shown in FIGS. 7 (c) and 7 (d). The parallax images to be processed have a smaller amount of parallax than the parallax images corresponding to the synthetic apertures 130lb and 130rb. Similarly, as shown in FIGS. 8C and 8D, the parallax images corresponding to the synthetic apertures 130ld and 130rd have a smaller parallax amount than the parallax images corresponding to the synthetic apertures 130lc and 130rc.

  As described above, according to the embodiments of FIGS. 5 to 8, by changing the aperture value of the aperture 40 and imaging a plurality of times, the number of types of aperture masks included in the image sensor 100 is obtained for one image sensor 100. It is possible to obtain parallax images corresponding to more viewpoints than corresponding viewpoints. In the above embodiment, when there are two types of aperture masks, it is possible to obtain 6 (= 2 × 3) different parallax images for one scene by imaging with three different aperture values. it can.

  FIG. 9 is an example of a synthetic aperture corresponding to a parallax image generated based on a difference between a synthetic aperture and another synthetic aperture. More specifically, the synthetic apertures 130le to lg in FIG. 9 correspond to the parallax images generated based on the difference between the parallax images corresponding to the synthetic apertures 130a to 130d in FIGS.

  By taking the difference between the first parallax image captured using the aperture 40a in FIG. 5 and the second parallax image captured using the aperture 40b in FIG. 6, a third parallax image that is a new parallax image is obtained. can get. For example, when the corresponding pixel value in the second parallax image corresponding to the synthetic aperture 130lb is subtracted from the pixel value of the first parallax image corresponding to the synthetic aperture 130la, the light beam passing through the synthetic aperture 130la passes through the synthetic aperture 130lb. A pixel value of the third parallax image generated from the light flux excluding the light flux is obtained. The third parallax image is optically equivalent to an image obtained by a light beam that has passed through the synthetic aperture 130le obtained by removing the transmissive region of the synthetic aperture 130lb from the transmissive region of the synthetic aperture 130la.

  The position of the center of gravity of the transmission area of the synthetic opening 130le is different from any of the center of gravity of the transmission area of the synthetic opening 130la and the synthetic opening 130lb. The position of the center of gravity of the transmission region of the synthetic aperture 130le is farther from the center of the circle than both the synthetic aperture 130la and the synthetic aperture 130lb. That is, the third parallax image corresponding to the synthetic aperture 130le has a larger amount of parallax than the first parallax image corresponding to the synthetic aperture 130la and the second parallax image corresponding to the synthetic aperture 130lb. Therefore, it is possible to obtain a parallax image from a viewpoint different from the viewpoint corresponding to the parallax image at the time of shooting obtained in FIGS.

  Similarly, a new third parallax image can be obtained from the difference between the first parallax image corresponding to the synthetic aperture 130ra and the second parallax image corresponding to the synthetic aperture 130rb.

  Here, when taking the difference, the overall brightness of each parallax image is such that, for each of a plurality of parallaxes, the exposure conditions other than the aperture correspond between the first parallax image and the second parallax image. Is preferably adjusted. The exposure conditions are, for example, sensitivity and exposure time.

  The exposure conditions other than the aperture are set by the control unit 201 when the parallax images shown in FIGS. 5 to 8 are captured. That is, the imaging of FIGS. 5 to 8 is performed with the sensitivity and the exposure time or the product of them being matched in advance.

  As another example, after the parallax images are acquired after the imaging in FIGS. 5 to 8, image processing may be performed so that the exposure conditions other than the aperture are equal. For example, in FIG. 5, when the first parallax image is captured at an ISO sensitivity of 100 and a shutter speed of 1/25 seconds, and the second parallax image is captured at an ISO sensitivity of 100 and a shutter speed of 1/50 seconds in FIG. You may match with the exposure conditions of a 1st parallax image by doubling the brightness of a parallax image.

  When such adjustment is performed, the light beam passing through the synthetic aperture 130lb can be removed from the light beam passing through the synthetic aperture 130la without excess or deficiency, and a more natural third parallax image can be obtained. Note that the exposure conditions including the aperture value, sensitivity, and exposure time are acquired at the time of imaging and written in the header portion of the parallax image.

  The brightness of the focused area in the newly generated third parallax image may be different from the brightness of the focused area of the first parallax image and the brightness of the focused area of the second parallax image. Therefore, the brightness of the in-focus area in the third parallax image is the same as one of the brightness of the in-focus area of the first parallax image, the brightness of the in-focus area of the second parallax image, and the average thereof. In this way, the overall brightness of the third parallax image may be adjusted. Thereby, the brightness of the 3rd parallax image produced | generated by image processing can be aligned with the other parallax image actually imaged, and the unnaturalness of those parallax images of the whole can be suppressed.

  A new parallax image can be generated using the differences between the parallax image obtained in FIG. 6, the parallax image obtained in FIG. 7, and the parallax image obtained in FIG. The synthetic aperture 130lf in FIG. 9 is an aperture corresponding to a parallax image generated by taking the difference between the parallax image corresponding to the synthetic aperture 130lc in FIG. 7 from the parallax image corresponding to the synthetic aperture 130lb in FIG. Similarly, the synthetic aperture 130lg in FIG. 9 is an aperture generated by taking the difference between the parallax image corresponding to the synthetic aperture 130lc in FIG. 7 and the left parallax image corresponding to the synthetic aperture 130ld in FIG. A new parallax image can be generated from the difference between the parallax images corresponding to the synthetic apertures 130rb to 130rd.

  As described above, according to the embodiment of FIG. 9, for each of a plurality of parallaxes, a difference between a plurality of parallax images obtained by changing the aperture value is obtained, thereby obtaining a plurality of parallaxes corresponding to the aperture mask at the time of imaging. Generates parallax images corresponding to different new parallaxes. Thereby, a parallax image having a parallax amount different from the parallax amount corresponding to the aperture mask at the time of imaging can be obtained for one scene without changing the configuration of the imaging element 100 or the shooting position. That is, a parallax image from a viewpoint different from the viewpoint corresponding to the aperture mask at the time of imaging can be obtained for one scene without changing the configuration of the imaging element 100 and the shooting position. In addition, since the synthetic aperture corresponding to the difference between the plurality of parallax images obtained by changing the aperture value has a smaller aperture area than the aperture mask at the time of imaging, the depth of field of the parallax image generated by the difference is Become deeper. Therefore, according to this method, it is possible to obtain a parallax image with less blur and with more enhanced parallax.

  FIG. 10 is a conceptual diagram illustrating processing for generating a parallax image. Starting from the left column, the generation of parallax image data generated by collecting the outputs of the parallax pixels corresponding to the synthetic aperture 130le, the generation of parallax image data corresponding to the synthetic aperture 130la, and the parallax corresponding to the synthetic aperture 130lb A state of generation of image data and a state of generation of parallax image data corresponding to the synthetic aperture 130rb will be described.

  As already described, a parallax image corresponding to the synthetic aperture 130lb and the synthetic aperture 130rb is obtained by photographing with the aperture 40b. The repetitive pattern 110 composed of a photoelectric conversion element group including a set of two left and right parallax pixels is arranged in a horizontal row, for example. Accordingly, the parallax pixels having the opening 104l exist on the image sensor 100, for example, every two pixels in the left-right direction and continuously in the vertical direction. Each of these pixels receives the subject luminous flux from different partial areas as described above. Therefore, when the outputs of these parallax pixels are gathered and arranged, a parallax image corresponding to the synthetic aperture 130lb and a parallax image corresponding to the synthetic aperture 130rb are obtained.

  However, since each pixel of the image sensor 100 in the present embodiment is a square pixel, simply gathering results in the number of pixels in the horizontal direction being decimated to ½, and vertically long image data is generated. End up. Therefore, by performing an interpolation process to make the number of pixels twice in the horizontal direction, parallax image data is generated as an image with an original aspect ratio. However, since the parallax image data before the interpolation process is an image that is thinned in the horizontal direction in the first place, the resolution in the horizontal direction is lower than the resolution in the vertical direction. That is, it can be said that the number of generated parallax image data and the improvement in resolution are in a conflicting relationship.

  When the aperture value is changed and a parallax image is acquired in the state of the aperture 40a, a left parallax image corresponding to the synthetic aperture 130la is obtained. Furthermore, a new parallax image corresponding to the synthetic aperture 130le is obtained from the difference between the left parallax image corresponding to the synthetic aperture 130la and the left parallax image corresponding to the synthetic aperture 130lb.

  The above interpolation process is also performed on the parallax image data corresponding to the synthetic aperture 130la, the synthetic aperture 130lb, and the synthetic aperture 130rb. Thereby, the digital camera 10 can generate a four-view parallax image having parallax in the horizontal direction. In the above example, the example in which one horizontal row is periodically arranged as the repeating pattern 110 has been described, but the repeating pattern 110 is not limited to this.

  Here, 14 parallax images corresponding to the synthetic apertures 130la to 130ld and 130ra to 130rd in FIGS. 5 to 8 and six parallax images such as the synthetic aperture 130le indicating the difference between them are used in a total of 14 types. A parallax image can be obtained. That is, parallax images from 14 different viewpoints can be obtained by the image sensor 100 having two types of openings, four aperture values, and image processing.

  Next, the color filter 102 and the parallax image will be described. FIG. 11 is a diagram illustrating the Bayer arrangement. As shown in the figure, the Bayer array is an array in which the green filter is assigned to the upper left and lower right pixels, the red filter is assigned to the lower left pixel, and the blue filter is assigned to the upper right pixel. Here, the upper left pixel to which the green filter is assigned is the Gb pixel, and the lower right pixel to which the green filter is assigned is the Gr pixel. In addition, a pixel to which a red filter is assigned is an R pixel, and a pixel to which blue is assigned is a B pixel. A horizontal direction in which Gb pixels and B pixels are arranged is defined as Gb row, and a horizontal direction in which R pixels and Gr pixels are arranged is defined as Gr row. A vertical direction in which Gb pixels and R pixels are arranged is a Gb column, and a vertical direction in which B pixels and Gr pixels are arranged is a Gr column.

  With respect to such an arrangement of the color filters 102, an enormous number of repetitive patterns 110 can be set depending on what color pixels the parallax pixels and non-parallax pixels are allocated to. If the outputs of pixels without parallax are collected, photographic image data having no parallax can be generated in the same way as normal photographic images. Therefore, if the ratio of pixels without parallax is relatively increased, a 2D image with high resolution can be output. In this case, since the number of parallax pixels is relatively small, the image quality is degraded as a 3D image including a plurality of parallax images. Conversely, if the ratio of the parallax pixels is increased, the image quality is improved as a 3D image, but the non-parallax pixels are relatively reduced, so that a 2D image with low resolution is output.

  FIG. 12 is a diagram illustrating an example of variations in the arrangement of color filters and parallax pixels. In the example of the figure, the same four pixels as the Bayer array are used as the repeated pattern 110. The R pixel and the B pixel are non-parallax pixels, and the Gb pixel is assigned to the parallax L pixel and the Gr pixel is assigned to the parallax R pixel. In this case, the opening 104 is determined so that the parallax L pixel and the parallax R pixel included in the same repetitive pattern 110 receive the light beam emitted from the same partial region when the subject is in the in-focus position. .

  In the example of the figure, Gb pixels and Gr pixels, which are green pixels with high visibility, are used as parallax pixels, so that it is expected to obtain a parallax image with high contrast. In addition, since the Gb pixel and the Gr pixel which are the same green pixels are used as the parallax pixels, it is easy to perform a conversion operation from these two outputs to an output having no parallax, and the output of the R pixel and the B pixel which are non-parallax pixels is high. High-quality 2D image data can be generated.

  FIG. 13 is a diagram for explaining the shape of the aperture mask according to the second embodiment. In the figure, the shape of the openings 204l and 204r of the opening mask 103 is a rectangle in which the length of the side perpendicular to the parallax direction is shorter than the length of the side parallel to the parallax direction.

  FIGS. 14A to 17D are schematic diagrams for explaining the relationship between the diaphragm and the synthetic aperture according to the second embodiment. This embodiment is the same as the first embodiment described in FIGS. 5 to 8 except that the shape of the opening 104 of the opening mask 103 is different.

  As shown in FIG. 14C, by using the opening mask 103 of FIG. 13, the transmissive region of the synthetic opening 230la is reduced in the upper and lower portions of the semicircle compared to the synthetic opening 130la in FIG. It has become a shape. Similarly, the transmission region of the synthetic opening 230la in FIG. 14D has a shape in which the upper and lower portions of the semicircle are omitted as compared with the synthetic opening 130la in FIG.

  Similarly, the transmission areas of the synthetic apertures 230lb to d and 230rb to d in FIGS. 15 to 17 captured with different aperture values compared to FIG. 14 are compared with the synthetic apertures 130lb to d and 130rb to d in FIGS. Thus, the upper and lower parts of the semicircle divided into left and right are omitted. Further, the center of gravity of the transmission area of the synthetic apertures 230la to 230d is the synthetic apertures 230la to 230d in the order away from the center. The center of gravity of the transmission region of the synthetic apertures 230ra-d is the synthetic apertures 230ra-d in order of increasing distance from the center.

  FIG. 18 is an example of a synthetic aperture corresponding to a new parallax image formed based on the difference between parallax images at the time of shooting in the present embodiment. The synthetic aperture 230le is an aperture corresponding to a new parallax image generated by subtracting the parallax image corresponding to the synthetic aperture 230lb from the parallax image corresponding to the synthetic aperture 230la. The synthetic aperture 230lf is an aperture corresponding to a new parallax image generated by subtracting the parallax image corresponding to the synthetic aperture 230lc from the parallax image corresponding to the synthetic aperture 230lb. The synthetic aperture 230lg is an aperture corresponding to a new parallax image generated by subtracting the parallax image corresponding to the synthetic aperture 230ld from the parallax image corresponding to the synthetic aperture 230lc.

  The synthetic apertures 230le to 230lg have fewer crescent-shaped curved portions than the synthetic apertures 130le to 130lg, so that the synthetic apertures 230le to 230lg are the center of gravity of the transmission region compared to the synthetic apertures 130le to 130lg. Can be separated from the center of the arc. For this reason, the parallax images corresponding to the synthetic apertures 230le to 230lg according to the present embodiment can have a larger amount of parallax than the parallax images corresponding to the synthetic apertures 130le to 130lg according to the first embodiment. .

  FIG. 19 is a view for explaining the shape of the opening mask according to the third embodiment. In this figure, the opening mask 103 has square openings 304l and 304r arranged so as to shift in an oblique direction. Specifically, the openings 304l and 304r are shifted in the upper left direction and the lower right direction. The image sensor 100 having such an opening mask 103 gives parallax in the vertical direction and the horizontal direction.

  FIGS. 20A to 23D are schematic diagrams for explaining the relationship between the diaphragm and the synthetic aperture according to the third embodiment. This embodiment is the same as the first embodiment described in FIGS. 5 to 8 except that the shapes of the openings 304l and 304r of the opening mask 103 are different.

  As shown in FIG. 20C, by using the opening mask 103 in FIG. 19, the lower half of the semicircle is saved in the transmission region of the synthetic opening 330la compared to the synthetic opening 130la in FIG. It has become a shape. Further, the transmission region of the synthetic aperture 330ra in FIG. 20D has a shape in which the upper half of the semicircle is omitted as compared with the synthetic aperture 130ra in FIG.

  Similarly, the apertures of synthetic apertures 330 lb to 330 lb and 330 rb to d in FIGS. 21 to 23 imaged with different aperture values from FIG. 20 are compared with the synthetic apertures 130 lb to 130 lb and 130 rb to d in FIGS. The upper half or the lower half of the semicircle is omitted. In addition, the center of gravity of the transmission area of the synthetic apertures 330la to 330d is the synthetic apertures 330la to 330d in the order away from the center. The centroids of the transmission regions of the synthetic apertures 330 ra to d are the synthetic apertures 330 ra to d in the order away from the center.

  FIG. 24 is an example of a synthetic aperture corresponding to a new parallax image formed based on the difference between parallax images at the time of shooting in the present embodiment. The synthetic aperture 330le is an aperture corresponding to a parallax image generated by subtracting the parallax image corresponding to the synthetic aperture 330lb from the parallax image corresponding to the synthetic aperture 330la. The synthetic aperture 330lf is an aperture corresponding to a new parallax image generated by subtracting the parallax image corresponding to the synthetic aperture 330lc from the parallax image corresponding to the synthetic aperture 330lb. The synthetic aperture 330lg is an aperture corresponding to a parallax image generated by subtracting the parallax image corresponding to the synthetic aperture 330ld from the parallax image corresponding to the synthetic aperture 330lc.

  The center of gravity of the transmission region of the synthetic openings 330le to 330lg is shifted in the left-right direction and the up-down direction. Therefore, the parallax images corresponding to the synthetic openings 330le to 330lg according to the present embodiment have parallax in the horizontal direction and the vertical direction.

  FIGS. 25A to 25H are diagrams illustrating a method for colorizing parallax pixels according to the fourth embodiment. FIGS. 25A and 25E show an example in which a plurality of types of apertures of an aperture mask are assigned to color filters arranged in a Bayer array on an image sensor for color images. A horizontally long opening is assigned to the upper right blue pixel B and lower left red pixel R. The upper left green pixel G is assigned an opening having a horizontally long left half, and the lower right green pixel G is assigned an opening having a horizontally long right half. In this way, when the left half opening and the right half opening of the green pixel G are added together, the opening becomes equal to the red pixel R and blue pixel B openings.

  Here, when the apertures arranged for the respective colors as shown in FIG. 25A and the maximum aperture 40a shown in FIG. 25B are synthesized, a synthetic aperture as shown in FIG. 25C is obtained. It is done. Similarly to (a), when the left half synthetic aperture and the right half synthetic aperture of the green pixel G are added together, they become equal to the red pixel R and blue pixel B synthetic apertures. For the green pixel G, a pair of parallax pixels having a parallax in the left-right direction is configured by combining a green pixel with a left half opening and a green pixel with a right half opening, but the blue pixel B and the red pixel R have no parallax. Pixel. Accordingly, a green single-color parallax image can be obtained as it is, but a full-color parallax image cannot be obtained.

Therefore, by applying the pixel values such as the light intensity of the left and right green pixels G to the blue pixel B and the red pixel R, the left and right pixel values are virtually obtained for blue and red. In particular,
R (lt) = R (n) × G (lt) / (G (lt) + G (rt)) Equation 1
R (rt) = R (n) × G (rt) / (G (lt) + G (rt)) Equation 2
Is obtained, the pixel values R (lt) and R (rt) of the left and right red pixels are obtained, and a red single-color parallax image is obtained. R (lt) and R (rt) indicate the pixel values of the calculated red left parallax pixel and right parallax pixel, R (n) indicates the pixel value of the detected non-parallax red pixel R, and G (Lt) and G (rt) indicate the pixel values of the detected left-parallax green pixel and right-parallax green pixel.

Similarly, for blue,
B (lt) = B (n) × G (lt) / (G (lt) + G (rt)) Equation 3
B (rt) = B (n) × G (rt) / (G (lt) + G (rt)) Equation 4
By calculating the pixel values B (lt) and B (rt) of the left and right blue pixels, and a blue single-color parallax image is obtained. B (lt) and B (rt) indicate the pixel values of the calculated blue pixel of the left parallax and the blue pixel of the right parallax, and B (n) indicates the pixel value of the detected blue pixel B without parallax. . FIG. 25D shows a synthetic aperture corresponding to red, green, and blue pixels having left and right parallax obtained in this way.

  Similarly, when the opening shown in FIG. 25 (e) is combined with the stop 40b shown in FIG. 25 (f) with a stop narrower than 40a, a combined opening as shown in FIG. 25 (g) is obtained. . FIG. 25 (h) shows a synthetic aperture corresponding to red, green, and blue pixels having left and right parallax, obtained by the same method as in FIG. 25 (d). Similarly to (c), when the left half synthetic aperture and the right half synthetic aperture of the green pixel G are added together, the relationship of being equal to the red pixel R and blue pixel B synthetic aperture is maintained. The color parallax image obtained from the synthetic aperture in FIG. 25D has a larger amount of parallax than the color parallax image obtained from the synthetic aperture in FIG. In this manner, color parallax images from different viewpoints can be obtained by changing the aperture value of the aperture.

  FIG. 26 is a conceptual diagram illustrating a method for obtaining color parallax images having different parallaxes from the color parallax images obtained in FIG. 25. FIG. 26A corresponds to the RGB synthetic aperture in FIG. 25D, and FIG. 26B corresponds to the RGB synthetic aperture in FIG. The parallax image corresponding to the synthetic aperture in FIG. 26C is obtained by subtracting the parallax image corresponding to the synthetic aperture in FIG. 26B from the parallax image corresponding to the synthetic aperture in FIG. Can do.

  Compared with the synthetic aperture in FIG. 26A, the synthetic aperture in FIG. 26C can move the position of the center of gravity away from the center of the pixel. For this reason, the parallax image corresponding to the synthetic aperture in FIG. 26C has a larger amount of parallax and emphasizes the stereoscopic effect than the parallax image corresponding to the synthetic aperture in FIG. Furthermore, since the parallax image corresponding to the synthetic aperture in FIG. 26C has a smaller aperture area than that in FIG. 26A, the depth of field is deep and blurring is less likely to occur. Therefore, according to this method, a clear color parallax image in which parallax is further enhanced can be obtained.

  The color arrangement of the color filter may not be a Bayer arrangement. For example, any pixel in the unit array may be a white pixel W having no color. Further, the left and right openings may be assigned not to the green pixel G but to the red pixel R, blue pixel B, or white pixel W. However, when left and right apertures are assigned only to pixels of some colors, it is necessary to calculate pixel values of other colors using Equations 1 to 4 in order to obtain a color parallax image. Therefore, in this case, it is desirable that the opening obtained by synthesizing the left and right openings provided in the parallax pixels is equal to the entire opening provided in the non-parallax pixels so that the formulas 1 to 4 can be applied.

  FIG. 27 is a diagram illustrating a processing flow from the acquisition of the first parallax image to the generation of the parallax image when the third parallax pixel is generated based on the first parallax pixel and the second parallax image. A series of processing in this flow starts when the control unit 201 receives an instruction for generating a parallax image from the user via the operation member, or when parallax image generation is incorporated in a predetermined control program, etc. Is done.

  In step S <b> 100, a photographing operation is performed by the control unit 201, and the image processing unit 205 acquires the number of first parallax images corresponding to the number of types of the opening 103 from the image sensor 100. In step S <b> 200, after the control unit 201 changes the aperture value of the aperture 40, a shooting operation is further performed, and the number of second parallax images corresponding to the number of types of the apertures 103 is acquired. The aperture value may be changed by so-called bracket photography, which is automatically performed by the control unit 201 as described above, or the aperture value may be switched manually. Further, the number of aperture value changes and subsequent imaging may be two or more. In step 300, the image processing unit 205 generates a third parallax image by taking the difference between the first parallax image and the second parallax image. The image processing unit 205 stores the first parallax image, the second parallax image, and the third parallax image in the memory card 220 in association with the respective parallaxes.

  FIG. 28 is a diagram illustrating a processing flow from acquisition of the first parallax image to generation of the parallax image in the case of generating a parallax image group based on the first parallax pixel and the second parallax image. A series of processing in this flow starts when the control unit 201 receives an instruction for generating a parallax image from the user via the operation member, or when parallax image generation is incorporated in a predetermined control program, etc. Is done.

  In step S <b> 201, a photographing operation is performed by the control unit 201, and the image processing unit 205 acquires the number of first parallax images corresponding to the number of types of the opening 103 from the imaging element 100. In step S <b> 201, after the control unit 201 changes the aperture value of the aperture 40, a shooting operation is further performed, and the number of second parallax images corresponding to the number of types of the apertures 103 is acquired. The aperture value may be changed by so-called bracket photography, which is automatically performed by the control unit 201 as described above, or the aperture value may be switched manually. Further, the number of aperture value changes and subsequent imaging may be two or more. In step 301, the image processing unit 205 stores the first parallax image and the second parallax image in the parallax image group memory card 220 in association with different parallaxes.

  In any of the above embodiments, the diaphragm 40 is gradually narrowed from the opened state. However, instead of this, the diaphragm 40 may be gradually opened from the narrowed state. Further, when generating a new parallax image based on the difference between the parallax images at the time of imaging, a new parallax image is obtained from the difference between the parallax image shown in FIG. 5 and the parallax image shown in FIG. It may be generated.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

  DESCRIPTION OF SYMBOLS 10 Digital camera, 20 Shooting lens, 21 Optical axis, 30 Subject, 40, 40a, 40b, 40c, 40d Aperture, 100 Image sensor, 101 Micro lens, 102 Color filter, 103 Aperture mask, 104, 104l, 104r 105 Wiring layer, 106 Wiring, 107 Opening, 108 Photoelectric conversion element, 109 Substrate, 110 Repeat pattern, 120 Image sensor, 121 Screen filter, 122 Color filter part, 123 Opening mask part, 130a, 130b, 130c, 130d, 130e, 130f, 130g synthetic aperture, 204 drive unit, 201 control unit, 202 A / D conversion circuit, 203 memory, 204l, 204r aperture, 205 image processing unit, 207 memory card IF, 208 operation unit, 209 display , 210 LCD drive circuit, 211 AF sensor, 220 memory card, 230a, 230b, 230c, 230d, 230e, 230f, 230g synthetic aperture, 304l, 304r aperture, 330a, 330b, 330c, 330d, 330e, 330f, 330g Opening

Claims (14)

A plurality of photoelectric conversion elements arranged two-dimensionally, each of said photoelectric conversion elements, to the entire cross-sectional area of the pupil subject light flux passes, from the respective partial areas corresponding to a plurality of parallax A first parallax image acquisition step of acquiring a plurality of first parallax images captured by an imaging device that receives the emitted subject light flux ;
The first parallax image acquisition stage is changed by a diaphragm provided for the whole of the plurality of photoelectric conversion elements, and the range in which the peripheral light flux with respect to the optical axis of the incident light is blocked is changed, and the image is picked up by the imaging element. A second parallax image acquisition step of acquiring a plurality of second parallax images;
For each of a plurality of parallaxes, a parallax image generation step of generating a third parallax image corresponding to a new parallax different from the plurality of parallaxes based on a difference between the first parallax image and the second parallax image for example Bei the door,
In the parallax image generation stage, for each of a plurality of parallaxes, the overall brightness of each parallax image is set so that an exposure condition other than the aperture corresponds between the first parallax image and the second parallax image. An image processing method that adjusts and takes the difference . In the parallax image generation step, the third parallax image is generated by using a difference between a pixel value of a pixel of the first parallax image and a pixel value of a corresponding pixel in the second parallax image as a pixel value. The image processing method according to claim 1 . The brightness of the focus area in the third parallax image is the same as any one of the brightness of the focus area of the first parallax image, the brightness of the focus area of the second parallax image, and the average thereof. The image processing method according to claim 1 , further comprising an image adjustment step of adjusting the overall brightness of the third parallax image. The image processing method according to claim 1 , wherein the diaphragm is an iris diaphragm. A plurality of photoelectric conversion elements arranged two-dimensionally, each of said photoelectric conversion elements, to the entire cross-sectional area of the pupil subject light flux passes, from the respective partial areas corresponding to a plurality of parallax A first parallax image acquisition step of acquiring a plurality of first parallax images captured by an imaging device that receives the emitted subject light flux ;
The first parallax image acquisition stage is changed by a diaphragm provided for the whole of the plurality of photoelectric conversion elements, and the range in which the peripheral light flux with respect to the optical axis of the incident light is blocked is changed, and the image is picked up by the imaging element. A second parallax image acquisition step of acquiring a plurality of second parallax images;
Each other and the second parallax image and the first parallax image, in association with different parallax, e Bei an image group generating step of generating a disparity image group,
In the image group generation stage, for each of a plurality of parallaxes, the brightness of the in-focus area in the first parallax image and the brightness of the in-focus area in the second parallax image are the same. An image processing method for adjusting the overall brightness to generate the parallax image group . The image processing method according to claim 5 , wherein the diaphragm is an iris diaphragm. The image processing method according to claim 1 , wherein the imaging element includes a color filter including a plurality of color filters provided in a one-to-one correspondence with each of the photoelectric conversion elements. A plurality of photoelectric conversion elements arranged two-dimensionally , each of the photoelectric conversion elements from each of the partial areas corresponding to a plurality of parallaxes with respect to the entire cross-sectional area of the pupil through which the subject luminous flux passes; A first parallax image acquisition step of acquiring a plurality of first parallax images captured by an image sensor that receives the emitted subject light flux ;
The first parallax image acquisition step is changed by the diaphragm provided for the whole of the plurality of photoelectric conversion elements, and the range in which the peripheral light flux with respect to the optical axis of the incident light is blocked is changed, and the image is captured by the imaging element. A second parallax image acquisition step of acquiring a plurality of second parallax images;
For each of a plurality of parallaxes, a parallax image generation step of generating a third parallax image corresponding to a new parallax different from the plurality of parallaxes based on a difference between the first parallax image and the second parallax image Are executed by a computer ,
In the parallax image generation step, for each of a plurality of parallaxes, the overall brightness of each parallax image is such that an exposure condition other than the aperture corresponds between the first parallax image and the second parallax image. An image processing program of the imaging apparatus that adjusts the difference and takes the difference . A plurality of photoelectric conversion elements arranged two-dimensionally, each of said photoelectric conversion elements, to the entire cross-sectional area of the pupil subject light flux passes, from the respective partial areas corresponding to a plurality of parallax A first parallax image acquisition step of acquiring a plurality of first parallax images captured by an image sensor that receives the emitted subject light flux ;
The first parallax image acquisition step is changed by the diaphragm provided for the whole of the plurality of photoelectric conversion elements, and the range in which the peripheral light flux with respect to the optical axis of the incident light is blocked is changed, and the image is captured by the imaging element. A second parallax image acquisition step of acquiring a plurality of second parallax images;
Causing the computer to execute an image group generation step of generating a parallax image group by associating the first parallax image and the second parallax image with different parallaxes ;
In the image group generation step, for each of a plurality of parallaxes, the brightness of the in-focus area in the first parallax image and the brightness of the in-focus area in the second parallax image are the same. An image processing program of an imaging apparatus that adjusts overall brightness to generate the parallax image group . The image processing program for an image pickup apparatus according to claim 8 or 9, wherein the image pickup device includes a color filter including a plurality of color filters provided in a one-to-one correspondence with each of the photoelectric conversion elements. A plurality of photoelectric conversion elements arranged two-dimensionally , each of the photoelectric conversion elements from each of the partial areas corresponding to a plurality of parallaxes with respect to the entire cross-sectional area of the pupil through which the subject luminous flux passes; An image sensor for receiving the emitted subject luminous flux ;
A diaphragm provided for the whole of the plurality of photoelectric conversion elements;
While acquiring a plurality of first parallax images picked up by the imaging device, and changing the range of shielding the luminous flux at the periphery of the optical axis of the incident light when the first parallax image is picked up by the diaphragm, A control unit that acquires a plurality of second parallax images captured by the imaging element;
For each of a plurality of parallaxes, an image processing unit that generates a third parallax image corresponding to a new parallax different from the plurality of parallaxes based on a difference between the first parallax image and the second parallax image; Bei to give a,
For each of a plurality of parallaxes, the image processing unit adjusts the overall brightness of each parallax image so that an exposure condition other than the aperture corresponds between the first parallax image and the second parallax image. An imaging device that adjusts and takes the difference . A plurality of photoelectric conversion elements arranged two-dimensionally , each of the photoelectric conversion elements from each of the partial areas corresponding to a plurality of parallaxes with respect to the entire cross-sectional area of the pupil through which the subject luminous flux passes; An image sensor for receiving the emitted subject luminous flux ;
A diaphragm provided for the whole of the plurality of photoelectric conversion elements;
While acquiring a plurality of first parallax images picked up by the imaging device, and changing the range of shielding the luminous flux at the periphery of the optical axis of the incident light when the first parallax image is picked up by the diaphragm, A control unit that acquires a plurality of second parallax images captured by the imaging element;
Wherein the first parallax image second parallax image and each other, in association with different parallax, e Bei an image processing unit that generates a parallax image group, and
The image processing unit, for each of a plurality of parallaxes, the entire parallax image so that the brightness of the in-focus area in the first parallax image and the brightness of the in-focus area in the second parallax image are the same. An imaging device that adjusts the brightness of the image to generate the parallax image group . The image processing method according to claim 1, wherein the plurality of photoelectric conversion elements are provided with a plurality of types of opening masks corresponding to the plurality of parallaxes in a one-to-one correspondence. The imaging device according to claim 11 or 12, wherein the imaging element is provided with a plurality of types of aperture masks corresponding to the plurality of parallaxes in a one-to-one correspondence with the photoelectric conversion element.

Images