JP6628617B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing device and an image processing method.

  Patent Document 1 discloses an imaging apparatus using a two-dimensional imaging element in which a plurality of photoelectric conversion units are formed for one microlens. The plurality of divided photoelectric conversion units are configured to receive light passing through different areas of the exit pupil of the photographing lens. A plurality of images can be generated from a plurality of signals generated by the plurality of divided photoelectric conversion units.

  Patent Document 2 discloses that an image signal is generated by adding signals received by a plurality of divided photoelectric conversion units.

  Patent Literature 3 discloses an image processing technique of combining images on a virtual imaging plane different from an imaging plane by shifting and adding a plurality of parallax images by a predetermined number of pixels.

U.S. Pat. No. 4,410,804 JP 2001-083407 A JP 2013-145979 A

  However, in the image processing technology of Patent Literature 3, no detailed disclosure is made of a method of setting the shift amount of the parallax image.

  An object of the present invention is to provide an image processing apparatus capable of setting a shift amount when adding a plurality of image signals by a more suitable method.

  An image processing apparatus according to an embodiment of the present invention is configured to perform reconstruction using a plurality of image signals generated by an image sensor based on each of a plurality of light beams that have passed through different portions of an exit pupil region of an imaging optical system. An image processing apparatus for generating an image, comprising: an image plane setting unit configured to set an image plane setting value indicating a position of a virtual image plane for generating the reconstructed image; Shift amount obtaining means for obtaining a shift amount when synthesizing the plurality of image signals based on a shift amount conversion coefficient set based on a pupil distance, an aperture value, and an image height, and the image plane setting value. And a reconstructed image generating means for generating the reconstructed image on the virtual imaging plane by relatively shifting the plurality of image signals by the shift amount and synthesizing the plurality of image signals. I do.

  An image processing method according to an embodiment of the present invention is configured to perform reconstruction using a plurality of image signals generated by an image sensor based on each of a plurality of light beams that have passed through different portions of an exit pupil region of an imaging optical system. An image processing method for generating an image, comprising setting an image plane setting value indicating a position of a virtual imaging plane for generating the reconstructed image, and setting an exit pupil distance and an aperture value in the imaging optical system and the image sensor. , And a shift amount conversion coefficient set based on an image height and the image plane setting value, and obtains a shift amount when combining the plurality of image signals, and shifts the plurality of image signals. The reconstructed image on the virtual imaging plane is generated by relatively shifting and synthesizing by an amount.

  According to the present invention, there is provided an image processing apparatus capable of setting a shift amount when adding a plurality of image signals by a more suitable method.

It is a block diagram of an imaging device concerning an embodiment of the present invention. FIG. 3 is a diagram illustrating a pixel array of the image sensor according to the embodiment. It is a top view (a) and a sectional view (b) of a pixel concerning an embodiment. FIG. 4 is a diagram illustrating a correspondence relationship between a pixel structure and a pupil region according to the embodiment. 5 is a graph illustrating a light intensity distribution in a pupil partial region according to the embodiment. FIG. 4 is a diagram illustrating a correspondence relationship between a pupil partial region and a photoelectric conversion unit of an image sensor according to the embodiment. FIG. 5 is a diagram illustrating a relationship between a defocus amount and an image shift amount of a first parallax image and a second parallax image according to the embodiment. FIG. 5 is a diagram illustrating a process of generating a reconstructed image according to the embodiment. FIG. 4 is a diagram illustrating an image plane movable range according to the embodiment. FIG. 4 is a diagram illustrating a relationship between a pupil region and an aperture as viewed from an image sensor according to the embodiment. FIG. 3 is a diagram illustrating line images near a center and a peripheral portion of the image sensor according to the embodiment. It is a figure explaining generation | occurrence | production of vignetting by the frame of a lens. 9 is a flowchart of acquisition of a shift amount conversion coefficient and recording on a captured image according to the embodiment. 5 is a flowchart of generating a reconstructed image using a shift amount conversion coefficient according to the embodiment. 5 is an example of a user interface for setting an image plane setting value according to the embodiment.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[Overall configuration of imaging device]
FIG. 1 is a block diagram illustrating a configuration of a camera that is an example of an imaging device according to an embodiment of the present invention. The imaging apparatus includes a first lens group 101, a diaphragm / shutter 102, a second lens group 103, a third lens group 105, and an optical low-pass filter 106 as an imaging optical system.

  The first lens group 101 is arranged on the side closest to the subject among a plurality of lens groups constituting the imaging optical system. The first lens group 101 is held by the lens barrel such that it can advance and retreat in the optical axis direction (the direction of the optical axis 303 in FIG. 1). The aperture / shutter 102 has a function of adjusting the light amount at the time of shooting by adjusting the aperture diameter thereof, and also has a function as an exposure time adjusting shutter at the time of shooting a still image. That is, the aperture value of the imaging optical system changes depending on the aperture diameter of the aperture shutter 102. The second lens group 103 advances and retreats in the optical axis direction integrally with the stop / shutter 102. The second lens group 103 implements a zoom function of performing a zooming operation in conjunction with the forward / backward movement of the first lens group 101. The third lens group 105 adjusts the focal position by moving back and forth in the optical axis direction. The optical low-pass filter 106 is an optical element for reducing false colors and moire that may occur in a captured image.

  The imaging device further includes an imaging device 107 including a two-dimensional CMOS sensor and peripheral circuits. The image sensor 107 is arranged on an image forming plane of the image forming optical system, and generates an image signal based on incident light from a subject.

  The imaging device further includes a zoom actuator 111, an aperture shutter actuator 112, and a focus actuator 114 as driving devices for the imaging optical system. The zoom actuator 111 moves the first lens group 101 and the second lens group 103 in the optical axis direction by rotating a cam cylinder (not shown). By this operation, a zooming operation is performed. The aperture shutter actuator 112 controls the aperture diameter of the aperture / shutter 102 to adjust the amount of photographing light, and also controls the exposure time when photographing a still image. The focus actuator 114 adjusts the focus by moving the third lens group 105 back and forth in the optical axis direction.

  The imaging device further includes an electronic flash 115 and an AF (autofocus) auxiliary light source 116. The electronic flash 115 is a lighting device for illuminating a subject during shooting. The electronic flash 115 includes, for example, a flash lighting device using a xenon tube and an LED that emits light continuously. The AF auxiliary light source 116 projects an image of a mask having a predetermined opening pattern onto a field via a light projecting lens. This makes it possible to improve the focus detection ability for a dark subject or a low-contrast subject.

  The imaging device further includes a CPU 121 that functions as a control unit that performs various controls of the imaging device. The CPU 121 includes an arithmetic unit, a ROM, a RAM, an A / D converter, a D / A converter, a communication interface circuit, and the like. The CPU 121 executes a predetermined program stored in the ROM. Accordingly, various circuits included in the imaging device are driven, and a series of operations such as focus detection, photographing, image processing, and recording are performed.

  The imaging device further includes an electronic flash control circuit 122 and an auxiliary light source driving circuit 123. The electronic flash control circuit 122 controls lighting of the electronic flash 115 in synchronization with the photographing operation. The auxiliary light source driving circuit 123 controls lighting of the AF auxiliary light source 116 in synchronization with the focus detection operation.

  The imaging device further includes an imaging device driving circuit 124 and an image processing circuit 125. The imaging device drive circuit 124 controls the imaging operation of the imaging device 107. The image processing circuit 125 performs A / D conversion on the image signal acquired by the image sensor 107 and transmits the signal to the CPU 121. The image processing circuit 125 performs processes such as γ conversion, color interpolation, and JPEG compression on the acquired image data.

  The imaging device further includes a focus drive circuit 126, an aperture shutter drive circuit 128, and a zoom drive circuit 129. The focus drive circuit 126 drives the focus actuator 114 based on the focus detection result to adjust the focus by moving the third lens group 105 back and forth in the optical axis direction. The aperture shutter drive circuit 128 drives the aperture shutter actuator 112 and controls the aperture diameter of the aperture / shutter 102. The zoom drive circuit 129 drives the zoom actuator 111 in accordance with the zoom operation of the photographer, and moves the first lens group 101 and the second lens group 103 in the optical axis direction.

  The imaging device further includes a display 131, an operation switch 132, and a flash memory 133. The display 131 is a display device such as a liquid crystal display. The display 131 displays information on the shooting mode of the camera, a preview image before shooting, a confirmation image after shooting, an in-focus state display image at the time of focus detection, and the like. The operation switches 132 are a group of switches including a power switch, a release (photographing trigger) switch, a zoom operation switch, a photographing mode selection switch, and the like. The flash memory 133 is a removable storage medium or a storage medium built in the imaging device, and stores data such as a captured image.

[Structure of imaging device]
FIG. 2 shows an arrangement of pixels and sub-pixels of the image sensor according to the present embodiment. FIG. 2 shows a part of a pixel array of a two-dimensional CMOS sensor which is the image sensor 107 of the present embodiment in a range of 4 columns × 4 rows.

  The pixel array of the image sensor 107 according to the present embodiment has a Bayer array in which the pixel groups 203 of 2 columns × 2 rows are repeatedly arranged. The pixel group 203 includes one pixel 200R having a spectral sensitivity (color) of R (red), two pixels 200G having a spectral sensitivity of G (green), and a pixel 200B having a spectral sensitivity of B (blue). Include one. The pixel 200 </ b> R is arranged at the upper left position in the pixel group 203. Pixels 200G are arranged at upper right and lower left positions in the pixel group 203. The pixel 200 </ b> B is arranged at a lower right position in the pixel group 203. In the following description, when it is not necessary to distinguish the colors of the pixels 200R, 200G, and 200B, these are collectively referred to as a pixel 200.

  Further, each of the plurality of pixels 200 included in the image sensor 107 includes the sub-pixels 201 and 202 arranged in 2 columns × 1 row. Each of the plurality of pixels 200 includes one microlens 305 for collecting incident light. That is, the sub-pixel 201 and the sub-pixel 202 share one microlens 305.

The image sensor 107 has a pixel array in which a number of pixels shown in FIG. 2 are arranged on a surface. With this configuration, the image sensor 107 can acquire an image signal and a focus detection signal. The period P in which the pixels 200 are arranged is, for example, 4 μm. The total number of pixels 200, that is, the number N of pixels is, for example, 5575 columns × 3725 rows = approximately 207.5 million. Lateral period _{P S} subpixels 201 and 202, for example, 2 [mu] m. The total number of sub-pixels 201 and 202, i.e., the number of subpixels _{N S} is, for example horizontal 11150 rows × vertical 3725 lines = about 41,500,000.

  FIG. 3A is a plan view of the pixel 200 of the image sensor 107 shown in FIG. 2 when viewed from the light receiving surface side (+ Z side) of the image sensor 107. The sub-pixel 201 of the pixel 200 has a photoelectric conversion unit 301. The sub-pixel 202 of the pixel 200 has a photoelectric conversion unit 302.

  FIG. 3B is a cross-sectional view of the A-A ′ cross section of FIG. 3A viewed from the −Y side. The pixel 200 includes a semiconductor substrate 300, an interlayer insulating layer 311, a wiring 307, a color filter 306, and a microlens 305. The interlayer insulating layer 311, the wiring 307, the color filter 306, and the microlens 305 are formed on the semiconductor substrate 300. The photoelectric conversion unit 301 includes a photodiode formed by an n-type semiconductor region 309 and a p-type semiconductor region 312 formed in the semiconductor substrate 300. The photoelectric conversion unit 302 includes a photodiode formed by an n-type semiconductor region 310 and a p-type semiconductor region 313 formed in the semiconductor substrate 300. These photodiodes may be pin structure photodiodes having an intrinsic layer sandwiched between the p-type semiconductor regions 312 and 313 and the n-type semiconductor regions 309 and 310, or may be pn junction photodiodes in which the intrinsic layer is omitted. There may be.

  The microlens 305 is a lens for condensing incident light on the surface of the semiconductor substrate 300, that is, on the light receiving surfaces 304 of the photoelectric conversion units 301 and 302. The direction of the optical axis 303 of the micro lens 305 is a direction perpendicular to the light receiving surface 304, that is, the Z direction.

  The color filter 306 is formed between the micro lens 305 and the photoelectric conversion units 301 and 302. For example, in the pixel 200G, the color filter 306 is made of a material having a high transmittance of green light and a spectral characteristic. Note that the transmittance of the color filter 306 may be changed for each of the sub-pixels 201 and 202. In some pixels 200, the color filter 306 may be omitted.

  The light incident on the pixel 200 is condensed by the microlens 305, separated by the color filter 306, and received by the photoelectric conversion unit 301 and the photoelectric conversion unit 302. In the photoelectric conversion unit 301 and the photoelectric conversion unit 302, pairs of electrons and holes are generated according to the amount of received light. After the electrons and holes are separated by a depletion layer, the electrons are accumulated in the n-type semiconductor regions 309 and 310, and the holes are imaged through a constant voltage source (not shown) connected to the p-type semiconductor regions 312 and 313. It is discharged outside the element 107.

  The electrons accumulated in the n-type semiconductor regions 309 and 310 are transferred to a floating diffusion region having capacitance via a transistor. The transferred electrons change the voltage of the floating diffusion region. Thereby, the charge generated by the incident light is converted into a voltage signal.

  The pixels 200R and 200B have the same configuration as the pixel 200G except for the color of the color filter 306.

  FIG. 4 is a schematic explanatory diagram showing the correspondence between the pixel structure shown in FIGS. 3A and 3B and the pupil region. 4 is a cross-sectional view of the pixel structure shown in FIG. 3A taken along the line AA ′ from the + Y side, and the exit pupil plane of the imaging optical system is shown in the upper part of FIG. It is shown. In FIG. 4, the direction of the X-axis in the sectional view is reversed with respect to FIG. 3 in order to correspond to the coordinate axes of the exit pupil plane.

  The pupil partial region 501 of the sub-pixel 201 is substantially conjugate with the light receiving surface of the photoelectric conversion unit 301 via the microlens 305. That is, the pupil partial region 501 represents a pupil region where the sub-pixel 201 can receive light. Since the center of gravity of the light receiving surface of the photoelectric conversion unit 301 is eccentric in the −X direction, the center of gravity of the pupil partial region 501 of the sub-pixel 201 is eccentric to the + X side on the pupil plane.

  Similarly, the pupil partial region 502 of the sub-pixel 202 is substantially conjugate with the light receiving surface of the photoelectric conversion unit 302 via the microlens 305. That is, the pupil partial region 501 represents a pupil region where the sub-pixel 202 can receive light. Since the center of gravity of the light receiving surface of the photoelectric conversion unit 302 is eccentric in the + X direction, the center of gravity of the pupil partial region 502 of the sub-pixel 202 is eccentric to the −X side on the pupil plane.

  The pupil region 500 is a pupil region corresponding to the entire pixel 200 including the photoelectric conversion unit 301 and the photoelectric conversion unit 302 (the sub-pixel 201 and the sub-pixel 202). The pupil region 400 is a pupil region in which vignetting due to a frame (a generic name of a lens frame and an aperture frame) is considered. The size, position, and shape of the pupil region 400 change depending on the combination of the aperture value of the aperture shutter 102, the exit pupil distance determined by the distance from the exit pupil to the light receiving surface 304, and the image height of the image sensor 107.

  FIG. 5 is a graph showing an example of the pupil intensity distribution. The horizontal axis indicates the position in the X direction, and the vertical axis indicates the light receiving rate. In the graph of FIG. 5, the pupil intensity distribution of the pupil partial region 501 is indicated by a solid line, and the pupil intensity distribution of the pupil partial region 502 is indicated by a broken line. The graph shows that the light intensity distributions of the pupil partial regions 501 and 502 change gently with respect to the position in the X direction.

  FIG. 6 is a diagram illustrating the correspondence between the pupil partial regions 501 and 502 and the photoelectric conversion units 301a, 302a, 301b, and 302b of the image sensor 107. Here, with respect to the branch numbers “a” and “b” added to the code of the photoelectric conversion unit, the photoelectric conversion units having the same branch number belong to the same pixel, and different branch numbers. It is assumed that the photoelectric conversion units marked with “” belong to different pixels. A plurality of light beams emitted from the subject 803 and having passed through different pupil partial regions 501 and 502 enter the imaging plane 800 of each pixel 200 of the imaging element 107 at different angles. These light beams are received by the sub-pixel 201 and the sub-pixel 202. The plurality of light beams emitted from the point a on the subject 803 pass through the pupil partial regions 501 and 502, and are received by the photoelectric conversion units 301a and 302a, respectively. The plurality of light beams emitted from the point b on the subject 803 pass through the pupil partial regions 501 and 502, and are received by the photoelectric conversion units 301b and 302b, respectively. The present embodiment shows an example in which the pupil region is divided into two pupils in the horizontal direction. However, the direction of pupil division is not limited to the horizontal direction. For example, pupil division may be performed in the vertical direction, and pupil division may be performed in a plurality of directions such as both the horizontal direction and the vertical direction. .

  The image sensor 107 according to the present embodiment is configured to be able to receive a plurality of light beams that have passed through different portions of the exit pupil region of the imaging optical system. That is, in the image sensor 107, the pixels 200 provided with the plurality of sub-pixels 201 and 202 each having the photoelectric conversion units 301 and 302 are arranged in a plurality of rows and a plurality of columns. In the present embodiment, a plurality of parallax images can be generated for each different pupil partial region by a light beam incident on each sub-pixel of the image sensor 107. That is, the first parallax image is generated by a signal based on the light beam received by the sub-pixel 201 of each pixel 200 of the image sensor 107. Further, a second parallax image is generated by a signal based on the light beam received by the sub-pixel 202 of each pixel 200 of the image sensor 107.

  In the present embodiment, the pixels 200R, 200G, and 200B have a Bayer arrangement. Therefore, each of the first parallax image and the second parallax image is a Bayer array image. If necessary, a demosaicing process may be performed on the first parallax image and the second parallax image.

  Further, by adding or averaging and reading out the signals obtained from the sub-pixels 201 and 202 of each pixel 200 of the image sensor 107, an image having a resolution corresponding to the number N of pixels can be generated. In the present embodiment, an image is generated by adding a plurality of parallax images (a first parallax image and a second parallax image).

[Relationship between defocus amount and image shift amount]
Hereinafter, the relationship between the defocus amount and the image shift amount of the first parallax image and the second parallax image acquired by the image sensor 107 of the present embodiment will be described.

  FIG. 7 is a diagram illustrating a relationship between a defocus amount of the first parallax image and the second parallax image and an image shift amount between the first parallax image and the second parallax image. An imaging element 107 (not shown) in FIG. 7 is arranged on the imaging surface 800. The exit pupil of the imaging optical system is divided into a pupil partial region 501 and a pupil partial region 502.

  The defocus amount d is defined as follows. The absolute value | d | of the defocus amount is the distance from the image forming position of the subject to the imaging surface. The sign of the defocus amount d is negative (d <0) when the imaging position of the subject is closer to the subject than the imaging surface (front focus state), and the imaging position of the subject is closer to the subject than the imaging surface. On the other hand, it is positive (d> 0) when it is on the opposite side (back focus state). D = 0 when the imaging position of the subject is on the imaging surface (focusing position) (focusing state). In FIG. 8, a subject 801 indicates the position of the subject in a focused state (d = 0). The subject 802 indicates the position of the subject in the front focus state (d <0). The front focus state (d <0) and the rear focus state (d> 0) are collectively referred to as a defocus state (| d |> 0).

  In the front focus state (d <0), of the light fluxes from the subject 802, the light fluxes that have passed through the pupil partial region 501 are once condensed, then spread to a width Γ1 around the center of gravity G1 of the light fluxes, and The image is blurred at 800. The blurred image is received by the sub-pixels 201 constituting each pixel 200 arranged in the image sensor, and a first parallax image is generated. Therefore, the first parallax image is recorded as a subject image in which the subject 802 is blurred over the blur width Γ1 near the center of gravity position G1 on the imaging surface 800. The blur width Γ1 of the subject image is substantially proportional to the absolute value | d | of the defocus amount d.

  Similarly, the light beam that has passed through the pupil partial region 502 spreads to a width of Γ2 around the center of gravity G2 of the light beam. The blurred image is received by the sub-pixel 202, and a second parallax image is generated. Therefore, the second parallax image is recorded as a subject image in which the subject 802 is blurred over the blur width # 2 in the vicinity of the center of gravity G2 on the imaging surface 800. The blur width Γ2 of the subject image is also substantially proportional to the magnitude | d | of the defocus amount d.

  Therefore, the absolute value | p | of the image shift amount p of the subject image between the first parallax image and the second parallax image (= the difference G1−G2 of the center of gravity of the light flux) is also the absolute value of the defocus amount d. d | The same applies to the back focus state (d> 0) except that the image shift direction of the subject image between the first parallax image and the second parallax image is opposite to the front focus state.

  Therefore, in the present embodiment, when the absolute value of the defocus amount of the first parallax image, the second parallax image, or the image obtained by adding the first parallax image and the second parallax image increases, the first parallax image and the second parallax image The absolute value of the image shift amount between images also increases.

[Image plane movement processing]
Hereinafter, a process of generating a reconstructed image by shift synthesis in the present embodiment will be described. FIG. 8 is a diagram illustrating a process of generating a reconstructed image in the pupil division direction (column direction, horizontal direction) from a plurality of parallax images (first and second parallax images) according to the present embodiment. The imaging plane 800 in FIG. 8 corresponds to the imaging plane 800 shown in FIGS. In FIG. 8, the first parallax image obtained by the pixel 200 in the i-th column of the image sensor 107 disposed on the imaging surface 800 is schematically represented as Ai, and the second parallax image is represented as Bi, where i is an integer. The first parallax image Ai is an image signal based on a light beam incident on the i-th pixel 200 at the principal ray angle θa corresponding to the pupil partial region 501 in FIG. The second parallax image Bi is an image signal based on the light beam incident on the i-th pixel 200 at the principal ray angle θb corresponding to the pupil partial region 502 in FIG. Note that the principal ray angles θa and θb are angles formed by a line perpendicular to the imaging surface 800 (that is, the optical axis 303) and a line corresponding to the center of gravity of the incident light beam.

  The first parallax image Ai and the second parallax image Bi have not only light intensity distribution information but also incident angle information. Therefore, it is understood that a reconstructed image on the virtual imaging plane 810 (an image obtained when the imaging plane 800 moves to the virtual imaging plane 810) can be generated by considering the following processing. . First, the first parallax image Ai is translated to the virtual imaging plane 810 along the direction of the light beam (the direction in which the optical axis 303 is inclined by the principal ray angle θa). Next, the second parallax image Bi is translated to the position of the virtual imaging plane 810 along the direction of the light beam (the direction in which the optical axis 303 is inclined by the principal ray angle θb). By combining these parallax images, a reconstructed image on the virtual imaging plane 810 can be generated.

  The amount of parallel movement at this time is assumed to be 0.5 pixels in the horizontal direction for both the first parallax image Ai and the second parallax image Bi. In this case, translating the first parallax image Ai along the light flux to the virtual imaging plane 810 is equivalent to shifting by +0.5 pixel in the horizontal direction. Translating the second parallax image Bi along the light flux to the virtual imaging plane 810 is equivalent to shifting the second parallax image Bi by −0.5 pixel in the horizontal direction. Therefore, the first parallax image Ai and the second parallax image Bi are relatively shifted by +1 pixel, that is, by combining the first parallax image Ai and the second parallax image Bi + 1 in correspondence with each other, the virtual imaging plane 810 is formed. Can be generated. The correspondence between the shift amount and the image plane movement amount when the imaging plane is moved by shifting and synthesizing the first parallax image Ai and the second parallax image Bi is represented by the principal ray angle θa and the principal ray angle θb. Is determined by the size of

The shift amount of the above-described shift synthesis is s, the row number is j, the column number is i, the first parallax image of j rows and i columns is A (j, i), and the second parallax image is B (j, i). write. By setting a reconstructed image obtained by shift-synthesizing these to I (j, i; s), the shift-synthesizing process is represented by the following equation (1).

  In the present embodiment, since the first parallax image A (j, i) and the second parallax image B (j, i) are in a Bayer array, parallax images of the same color are synthesized by setting the shift amount s to a multiple of two. can do. That is, it can be expressed as s = 2n (n is an integer). By setting the shift amount s in this way and performing the shift synthesis of Expression (1) so that signals of the same color are added, the reconstructed image I (j, i; s) can be obtained while maintaining the Bayer arrangement. Can be generated. Thereafter, the demosaicing process may be performed on the reconstructed image I (j, i; s) as necessary.

  Further, a reconstructed image may be generated after generating an interpolation signal between each pixel of the first parallax image A (j, i) and the second parallax image B (j, i). As a result, the shift amount s can be a non-integer, and the degree of freedom in setting the shift amount s is improved.

  As described above, in the present embodiment, a re-imaged image on the virtual imaging plane of the imaging optical system can be generated from a plurality of parallax images after shooting.

[Image plane movable range]
FIG. 9 is a diagram illustrating an image plane movable range in the present embodiment. When the allowable circle of confusion is δ and the aperture value of the imaging optical system is F, the depth of field at the aperture value F is ± Fδ. The division number of horizontal pupil division is N _H, the division number of the pupil division in the vertical direction and N _V. At this time, the effective aperture value F _{01 in} the horizontal direction of the pupil partial region 501 that has been narrowed by N _H × N _V division (2 × 1 division in this embodiment) is F ₀₁ = N _H F, which is smaller than F. growing. Therefore, the effective depth of field for the first parallax image is ± N _H F [delta] and N _H times deeper, focusing range spreads N _H times. With respect to the first parallax image, a focused subject image is obtained within the range of the effective depth of field ± N _H Fδ. Note that the same applies to the second parallax image.

In the reconstruction processing shown in FIG. 9, the image plane can be moved after photographing, and the focus position can be readjusted. However, since the range of the effective depth of field for the first and second parallax images is limited, the range of the defocus amount d in which the focus position can be readjusted is limited. The image plane movable range of the defocus amount d is approximately the range of the following equation (2).

  Note that the allowable circle of confusion diameter δ is defined by δ = 2ΔX (the reciprocal of the Nyquist frequency 1 / (2ΔX) in the pixel period ΔX) or the like.

[Acquisition of shift amount conversion coefficient]
FIG. 10 is a diagram illustrating a relationship between the pupil region and the aperture viewed from the image sensor 107 side. A pupil region 401 shown in FIG. 10 is viewed from the pixel 200 a arranged near the center of the image sensor 107 (ie, near the center of the opening of the aperture frame 600 of the aperture / shutter 102) through the opening of the aperture frame 600. Is an area where A pupil region 402 shown in FIG. 10 is a region that can be seen from the pixel 200b disposed around the image sensor 107 through the opening of the aperture frame 600. Solid lines and broken lines in the graph in FIG. 10 indicate pupil intensity distributions relating to pupil partial regions 501 and 502, respectively.

  FIG. 11A shows a line image near the center of the image sensor 107 in FIG. 10, and FIG. 11B shows a line image in a peripheral portion of the image sensor 107 in FIG. 11A and 11B are line images related to the pupil partial regions 501 and 502, respectively. G1 and G2 indicated by broken lines in the figure are the barycentric positions of the line images related to the pupil partial regions 501 and 502, respectively. Here, the meanings of the centroid positions G1 and G2 in FIGS. 11A and 11B are the same as those of the centroid positions G1 and G2 shown in FIG.

  According to FIGS. 7, 10, 11A and 11B, the principal ray angle θa described with reference to FIG. 8 is determined by the positional relationship between the center of gravity position G1 and the exit pupil, and the principal ray The angle θb is determined by the positional relationship between the position of the center of gravity G2 and the exit pupil. According to FIGS. 7, 10, 11A and 11B, the positions of the centers of gravity G1 and G2 are determined by the ranges of the pupil regions 401 and 402, respectively. In other words, each of the center-of-gravity positions G1 and G2 is the exit pupil distance determined by the position of the opening of the aperture frame 600, the aperture value that determines the size of the aperture of the aperture frame 600, and the light-receiving surface of the image sensor 107. It changes depending on the image height.

  As described with reference to FIG. 8, the image plane movement amount (shift amount conversion coefficient) per shift amount pixel in the shift synthesis is determined based on the principal ray angle θa and the principal ray angle θb. That is, the shift amount conversion coefficient changes depending on the exit pupil distance, the aperture value, and the image height in the imaging optical system and the image sensor 107. Therefore, if such information is obtained, the shift amount conversion coefficient can be calculated and obtained. The shift amount conversion coefficient is calculated in advance and stored in a storage medium or the like, so that the shift amount conversion coefficient can be read and used when the shift combining process is performed. At this time, the shift amount conversion coefficient may be recorded in the image data together with the image signal. Alternatively, every time the shift combining process is performed, the shift amount conversion coefficient may be obtained by performing calculations by the CPU 121 or the like of the imaging apparatus.

  As described above, according to the present embodiment, it is possible to obtain the shift amount conversion coefficient based on the exit pupil distance, the aperture value, and the image height, and use this to convert the image plane movement amount into the shift amount. Can be. For example, when the user sets conditions for generating a reconstructed image, it is easier to set the position of the image plane to be focused, that is, to set the image plane set value than to set the shift amount of the parallax image itself. It is. In addition, since the shift amount of the parallax image depends on the exit pupil distance, the aperture value, and the image height as described above, it is complicated for the user to set them in consideration. According to the present embodiment, the setting based on the image plane setting value becomes possible, so that it is possible to more suitably set the shift amount when adding a plurality of image signals.

[Acquisition of peripheral light amount information]
FIGS. 12A and 12B are diagrams illustrating the occurrence of vignetting due to a frame. Although the aperture value set at the time of photographing is one value, the effective aperture value (effective aperture value) may change according to the image height due to occurrence of vignetting. As shown in FIG. 12A, when there is one frame 601 that causes vignetting, the area of the pupil region 403 hardly changes due to the image height, so that the effective aperture value does not change much. However, as in frames 601 and 602 shown in FIG. 12B, there are a plurality of frames in a lens or the like of an actual imaging optical system. In this case, the area of the pupil region 405 which is the overlapping portion of the pupil regions 403 and 404 corresponding to the vignetting due to the frames 601 and 602 changes depending on the image height. Therefore, the effective aperture value may change according to the image height as a result of vignetting by the plurality of frames 601 and 602.

  In many cases, when vignetting due to a lens or the like occurs, the light amount decreases (peripheral light amount decreases) as the image sensor 107 approaches the edge of the imaging surface. This decrease in the peripheral light amount has a unique distribution for each lens. Therefore, the imaging device may store information (peripheral light amount information) on the peripheral light amount of the imaging optical system, and may have a function of correcting a decrease in the peripheral light amount after shooting. By referring to this peripheral light amount information, the amount of occurrence of vignetting due to the lens frame can be estimated. Therefore, an effective aperture value corresponding to the image height of the image sensor 107 can be acquired with reference to the peripheral light amount information.

The peripheral light amount information at the image height (x, y) of a certain image sensor is defined as V (x, y), and the function f (V (x, y)) is a function determined according to the shape of vignetting. The effective aperture value F 'at this time can be calculated by the following equation (3) using the aperture value F set at the time of shooting.

Since f (V (x, y)) has an optimal function form depending on the shape of vignetting caused by a lens or the like, it is necessary to use a function suitable for each lens. As an example of the function f (V (x, y)), a function that can be suitably used when the vignetting shape is close to a circle (Formula (4)) and when it is close to an ellipse (Formula (5)) is shown. .

  However, the function f (V (x, y)) is not limited to the equations (4) and (5), and can be set appropriately according to the shape of vignetting.

[Recording of shift amount conversion coefficient in captured image]
As described above, the shift amount conversion coefficient may be recorded in the image data together with the image signal. FIG. 13 is a flowchart showing a flow from the acquisition of the shift amount conversion coefficient to the recording of the shift amount conversion coefficient in the image data. Note that this processing can be mainly executed by the CPU 121 in the imaging device, for example. Further, it is assumed that the focus position adjustment at the time of shooting using the imaging device can be performed by AF (auto focus) or MF (manual focus).

  In step S1301, the CPU 121 determines the setting of the focus detection method of the imaging device used for capturing an image. Focusing methods may include AF and MF. If the focus detection method is set to AF, the process advances to step S1302. If the focus position setting is set to MF, the process advances to step S1303.

  In step S1302, the CPU 121 acquires position information including the image height (x, y) focused in the AF process from a storage medium or the like of the imaging device.

  In step S1303, the CPU 121 acquires position information including the image height (x, y) corresponding to the predetermined reference position from the storage medium of the imaging device or the like.

  Note that a fixed image height may be used in the following processing regardless of the focus detection method, and in that case, steps S1301 to S1303 may be omitted.

  In step S1304, the CPU 121 obtains at least the exit pupil distance and the aperture value from the shooting conditions in the image shooting from the storage medium of the imaging device, the memory provided in the lens unit, and the like.

  In step S1305, the CPU 121 acquires the peripheral light amount information from the storage medium of the imaging device, the memory provided in the lens unit, and the like. When the focus detection method is set to AF, the CPU 121 may acquire only the peripheral light amount information of the focused image height, or may acquire the peripheral light amount information of all image heights. When the focus detection method is set to MF, peripheral light amount information of only the reference image height may be acquired, or peripheral light amount information of all image heights may be acquired. Note that the aperture value acquired in step S1304 may be used as it is for acquiring the shift amount conversion coefficient, and in such a case, steps S1305 and S1306 may be omitted.

  In step S1306, the CPU 121 uses the peripheral light amount information acquired in step S1305 to calculate and acquire an effective aperture value for one or more image heights used for acquiring the shift amount conversion coefficient.

  In step S1307, the CPU 121 uses the exit pupil distance, the effective aperture value (or the aperture value acquired in step S1304), and the image height corresponding to these, and stores the shift amount stored in the storage medium of the imaging device. Get the conversion factor. Alternatively, the CPU 121 calculates and acquires a shift amount conversion coefficient using these pieces of information.

  In step S1308, the CPU 121 records the obtained shift amount conversion coefficient together with at least one of the image signals in the image data. Accordingly, when the image data is processed by an image processing device outside the imaging device, the shift amount conversion coefficient can be referred to without being acquired separately from the image data. Note that, for example, when the image plane movement processing can be performed in the imaging device, it is not essential to record the shift amount conversion coefficient in the image data. In such a case, step S1308 may be omitted.

[Generation method of reconstructed image]
FIG. 14 is a flowchart showing processing when performing reconstructed image processing by shift synthesis using a shift amount conversion coefficient. The processing of this flowchart can be performed by an image processing apparatus having an image plane setting unit, a shift amount obtaining unit, and a reconstructed image generating unit. The image processing device can be provided outside the imaging device. In this case, the imaging apparatus can supply the image processing apparatus with data used for the image plane setting unit, the shift amount acquisition unit, and the reconstructed image generation unit. In this case, the image processing apparatus may function as an image plane setting unit, a shift amount obtaining unit, and a reconstructed image generating unit, for example, when a computer executes a program.

  Further, the image processing device can be provided inside the imaging device. In this case, for example, the CPU 121 in the imaging apparatus may function as an image plane setting unit, a shift amount acquisition unit, and a reconstructed image generation unit by executing a program. Further, an image processing circuit may be further provided in the imaging apparatus, and the image processing circuit may function as an image plane setting unit, a shift amount obtaining unit, and a reconstructed image generating unit.

  The image plane setting unit is a unit that sets an image plane setting value indicating the position of the virtual imaging plane 810 that generates the reconstructed image. The shift amount determining unit converts a plurality of images based on a shift amount conversion coefficient set based on an exit pupil distance, an aperture value, and an image height in the imaging optical system and the image sensor 107, and an image plane setting value. This is a means for acquiring the shift amount at the time of combining. The reconstructed image generating unit shifts the plurality of parallax images by the shift amount obtained by the above-described shift amount determining unit and combines the plurality of parallax images according to Expression (1) to combine the reconstructed images on the virtual imaging plane 810. This is a means for performing shift synthesis.

  The specific contents of the flowchart in FIG. 14 will be described. In step S1401, the image processing apparatus acquires a shift amount conversion coefficient recorded in image data. The recording of the shift amount conversion coefficient in the image data is, for example, performed by the processing of the flowchart in FIG. It is not essential that the shift amount conversion coefficient is stored in the image data. In such a case, the shift amount conversion coefficient may be obtained by another method, such as obtaining the data from data separately recorded in a storage medium. You may.

  In step S1402, the image processing apparatus that functions as an image plane setting unit acquires an image plane setting value based on an operation from a user or the like. As for the order of step S1401 and step S1402, as shown in FIG. 14, step S1401 may be first, and conversely, step S1402 may be first.

  In step S1403, the image processing apparatus functioning as a shift amount obtaining unit uses the shift amount conversion coefficient obtained in step S1401 and the image plane setting value obtained in step S1402 to calculate the shift amount for synthesizing the parallax image. get.

  In step S1404, the image processing apparatus functioning as the reconstructed image generating unit shifts the plurality of parallax images relative to each other by the above-described shift amount using the shift amount acquired in step S1403, and synthesizes the virtual images. Generate a reconstructed image on the imaging plane.

  In step S1405, the display unit provided in the image processing device or the display unit of the device used integrally with the image processing device displays the reconstructed image generated in step S1404. Step S1405 is not essential.

  In the above-described step S1402, the setting of the image plane setting value by the image plane setting unit is performed by the user performing an operation (refocus operation) for determining, by the image processing apparatus, an amount by which the focus position of the captured image is moved. It is done by doing. FIGS. 15A and 15B are diagrams illustrating examples of a user interface screen for a user to specify a focus position. FIGS. 15A and 15B show operation images displayed on a display unit provided in the image processing apparatus or a display unit of an apparatus used integrally with the image processing apparatus.

  FIG. 15A is a diagram illustrating an example of a user interface. The operation screen 1500 displays one of the parallax images and an operation image. The parallax image displayed on operation screen 1500 includes first subject 1501 and second subject 1502 at different distances from the imaging device. An in-focus display frame 1503 is shown at the face of the first subject. A focus display frame 1503 indicates a reference position (a focused position) at which the image was focused at the time of shooting. Further, on the right side of the operation screen 1500, a slider bar 1504 as an operation unit is shown. The user can set the position of the virtual imaging plane 810 based on the position of the focus display frame 1503 by operating the knob of the slider bar 1504 displayed on the operation screen 1500 up and down.

  FIG. 15B is a diagram illustrating another example of the user interface. A pointer 1505 is displayed on the operation screen 1506 instead of the focus display frame 1503. The pointer 1505 also functions as an operation unit together with the slider bar 1504. The user can change the focus reference position by moving the pointer 1505 to a desired position. For example, when the user wants to focus on the second object 1502, the user moves the pointer 1505 to the position of the second object 1502, thereby changing the position of the virtual imaging plane 810 based on the position of the second object 1502. Can be set. Note that the position of the virtual imaging plane 810 may be automatically set on the operation screen 1506 in FIG. 15B so that a focused reconstructed image is generated at a location designated by the pointer 1505 by the user. . After the designation with the pointer 1505, the position of the virtual imaging plane 810 may be set by manually operating the slider bar 1504.

  As described above, the amount of movement of the image plane per one pixel shift differs depending on the exit pupil distance, the aperture value, and the image height in the imaging optical system and the image sensor 107. Therefore, for example, when the photographing condition is changed such as replacement of a lens, the image plane movement amount per one pixel shift changes. When the user interface for generating the reconstructed image is such as to allow the user to input the shift amount, it is necessary for the user to determine the shift amount in consideration of these parameters, and the operation becomes complicated. .

  However, in the present embodiment, the image processing apparatus acquires the shift amount using the input image plane setting value and the shift amount conversion coefficient. That is, the shift amount of the parallax image is automatically acquired only by the user inputting the image plane setting value indicating the position of the virtual imaging plane using the slider bar 1504 or the like. Therefore, the user can generate a reconstructed image by a certain operation that does not depend on a change in shooting conditions such as a lens.

(Other embodiments)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. It can also be realized by the following processing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Reference Signs List 101 First lens group 102 Aperture / shutter 103 Second lens group 105 Third lens group 106 Optical low-pass filter 107 Image sensors 501, 502 Pupil partial area 600 Aperture frame 810 Virtual image plane

Claims (10)

An image processing apparatus that generates a reconstructed image using a plurality of image signals generated by an imaging device based on each of a plurality of light beams that have passed through different portions of an exit pupil region of an imaging optical system,
Image plane setting means for setting an image plane setting value indicating the position of the virtual imaging plane to generate the reconstructed image,
The plurality of image signals are synthesized based on the shift amount conversion coefficient set based on the exit pupil distance, the aperture value, and the image height in the imaging optical system and the image sensor, and the image plane setting value. Shift amount obtaining means for obtaining the shift amount at the time of,
Reconstructed image generating means for generating the reconstructed image on the virtual image plane by relatively shifting the plurality of image signals by the shift amount and synthesizing the plurality of image signals;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the shift amount conversion coefficient is set based on information on a peripheral light amount of the imaging optical system.   3. The image processing apparatus according to claim 2, wherein the information on the peripheral light amount of the imaging optical system includes information on vignetting of incident light by a lens frame included in the imaging optical system.   4. The image processing apparatus according to claim 1, wherein the shift amount conversion coefficient is recorded in image data together with at least one of the plurality of image signals. 5.   The plurality of image signals are images obtained by performing autofocus focusing by an imaging device including the imaging element, and the image height is an image height focused in the autofocus. The image processing apparatus according to claim 1, wherein:   The plurality of image signals are images obtained by performing focus adjustment with manual focus by an imaging device including the imaging device, and the image height is an image height corresponding to a predetermined reference position. The image processing apparatus according to claim 1, wherein: A display unit that displays at least one of the plurality of images based on the plurality of image signals and an operation unit;
The image processing apparatus according to claim 1, wherein the image plane setting unit sets the image plane set value based on an operation of the operation unit by a user. An image processing method for generating a reconstructed image using a plurality of image signals generated by the imaging device based on each of a plurality of light beams that have passed through different portions of the exit pupil region of the imaging optical system,
Setting an image plane setting value indicating the position of the virtual imaging plane to generate the reconstructed image,
The plurality of image signals are synthesized based on the shift amount conversion coefficient set based on the exit pupil distance, the aperture value, and the image height in the imaging optical system and the image sensor, and the image plane setting value. Get the shift amount when
An image processing method, wherein the reconstructed image on the virtual imaging plane is generated by relatively shifting and synthesizing the plurality of image signals by the shift amount.   A program for causing a computer to execute the image processing method according to claim 8.   A storage medium storing a program for causing a computer to execute the image processing method according to claim 8.

Images