JP2015015629A - Image processing device and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which since image light is blurred with respect to an image sensor due to camera shake, respective left and right images themselves become blurred images.SOLUTION: An image processing device comprises: a parallax image acquisition unit that acquires a parallax image including a parallax amount indicating parallax with respect to the same object; a correction amount acquisition unit that acquires a camera shake correction amount for correcting camera shake in imaging; and a corrected image generation unit that generates a corrected image obtained by correcting the parallax amount of the parallax image acquired by the parallax image acquisition unit on the basis of the camera shake correction amount acquired by the correction amount acquisition unit.

Description

  The present invention relates to an image processing device and an imaging device.

Some imaging devices that can shoot 3D video images adjust the parallax between the left and right images when a large camera shake occurs during shooting (see, for example, Patent Document 1).
Japanese Patent Application Laid-Open No. 2012-90259

  However, the image pickup apparatus has a problem that the left and right images themselves become blurred images because image light blurs with respect to the image sensor due to camera shake.

  In the first aspect of the present invention, the image processing apparatus includes a parallax image acquisition unit that acquires a parallax image including a parallax amount indicating parallax with respect to the same target, and a camera shake correction amount that corrects camera shake during imaging. A correction amount acquisition unit to be acquired, and a correction image generation unit that generates a correction image in which the parallax amount of the parallax image acquired by the parallax image acquisition unit is corrected based on the camera shake correction amount acquired by the correction amount acquisition unit. Prepare.

  According to a second aspect of the present invention, there is provided an imaging apparatus, the image processing apparatus, a plurality of two-dimensionally arranged photoelectric conversion elements that photoelectrically convert incident light into an electric signal, and incident light A light-shielding unit provided with an opening through which light from a partial region in the cross-sectional region is incident on a corresponding plurality of photoelectric conversion elements, and a photographing unit that photographs a target and generates a parallax image.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is a figure explaining the structure of a digital camera. It is a conceptual diagram which represents notably the mode that a part of imaging device was expanded. It is a figure explaining the example of a production | generation process of 2D image data and parallax image data. It is a figure explaining the concept of defocusing. It is a figure which shows the light intensity distribution which a parallax pixel outputs. It is a figure explaining the concept of defocusing at the time of camera shake and camera shake correction. It is the elements on larger scale of FIG.6 (c). It is a functional block diagram of an image processing part. It is a conceptual diagram explaining camera-shake correction amount. An example of the flowchart of operation | movement of a digital camera is shown. An example of the method for correcting the parallax amount in step S116 of the flowchart of FIG. 10 will be described. It is explanatory drawing which shows the change of the pixel value by correction | amendment of the parallax amount of FIG.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 is a diagram illustrating the configuration of the digital camera 10. The digital camera 10 generates images of a plurality of viewpoints for one scene by shooting once. Each image having a different viewpoint is called a parallax image.

  The digital camera 10 includes a photographic lens 20 as a photographic optical system and a correction lens 30 for correcting camera shake, and guides a subject luminous flux incident along the optical axis 21 to the image sensor 100. The digital camera 10 further includes an aperture 22 that shields the periphery of the subject light flux. The taking lens 20, the correction lens 30 and the aperture 22 may be interchangeable lenses that can be attached to and detached from the digital camera 10. The digital camera 10 includes an image sensor 100, a control unit 201, an A / D conversion circuit 202, a memory 203, a drive unit 204, a memory card IF 207, an operation unit 208, a display unit 209, an LCD drive circuit 210, an AF sensor 211, and a drive unit. 250, a camera shake sensor 252 and an image processing unit 260.

  As shown in the figure, the direction parallel to the optical axis 21 toward the image sensor 100 is defined as the Z-axis plus direction, the direction toward the front of the drawing on the plane orthogonal to the Z-axis is the X-axis plus direction, and the upward direction on the drawing is Y. The axis is defined as the plus direction. The coordinate axis is a coordinate axis fixed to the digital camera 10, and the relationship with the composition in photographing in FIG. 1 is that the X axis is the horizontal direction and the Y axis is the vertical direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

  The taking lens 20 is composed of a plurality of optical lens groups, and together with the correction lens 30, forms a subject light flux from the scene in the vicinity of its focal plane. In FIG. 1, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens disposed in the vicinity of the pupil.

  The correction lens 30 is movable in the XY plane, and optically corrects an image blur on the image sensor 100 due to a user's camera shake. The taking lens 20 and the correction lens 30 may be collectively referred to as a taking optical system.

  The diaphragm 22 is disposed in the vicinity of the pupil of the photographing optical system, and shields the peripheral region of the subject light beam incident on the image sensor 100. The aperture amount of the aperture 22 is variable and is set by control from the control unit 201. An example of the diaphragm 22 is an iris diaphragm, but is not limited thereto.

  The image sensor 100 is disposed in the vicinity of the focal plane of the photographing optical system. The image sensor 100 is an image sensor such as a CCD or CMOS sensor in which a plurality of photoelectric conversion elements are two-dimensionally arranged. The image sensor 100 is controlled in timing by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202.

  The A / D conversion circuit 202 converts the image signal output from the image sensor 100 into a digital image signal and outputs the digital image signal to the memory 203. The image processing unit 260 performs various image processing using the memory 203 as a work space, and generates image data.

  The AF sensor 211 is, for example, a phase difference sensor in which a plurality of distance measuring points are set for the subject space. The AF sensor 211 detects the defocus amount of the subject image at each distance measuring point. Note that the image sensor 100 may also serve as an AF sensor, and in that case, a separate AF sensor 211 may not be provided.

  The camera shake sensor 252 includes a gyro sensor or the like, detects a change in the posture of the digital camera 10 mainly due to a user's camera shake, and outputs the change to the control unit 201. The change in posture includes pitching around the X axis, yawing around the Y axis, shifting in the X axis direction and the Y axis direction, and the like.

  The driving unit 250 corrects blurring of an image captured by the image sensor 100 by driving the correction lens 30 in response to a change in posture of the digital camera 10 detected by the camera shake sensor 252. In this case, the control unit 201 calculates the direction and amount of movement of the correction lens 30 in accordance with the direction and amount of change in posture of the digital camera 10 detected by the camera shake sensor 252. The drive unit 250 drives the correction lens 30 with the direction and amount of movement calculated by the control unit 201. The direction and amount of movement of the correction lens 30 may be referred to as a camera shake correction amount.

  The control unit 201 controls the digital camera 10 in an integrated manner. For example, the aperture of the aperture 22 is adjusted according to the set aperture value, and the photographing lens 20 is advanced and retracted in the optical axis direction according to the defocus amount of the AF sensor 211. Further, the position of the photographing lens 20 is detected, and the focal length and the focus lens position of the photographing lens 20 are grasped. Furthermore, a timing control signal is transmitted to the drive unit 204, and a series of sequences until the image signal output from the image sensor 100 is processed into image data by the image processing unit 260 is managed.

  The image processing unit 260 also performs general image processing functions such as adjusting image data according to the selected image format. The generated image data is converted into a display signal by the LCD drive circuit 210 and displayed on the display unit 209. The data is recorded on the memory card 220 attached to the memory card IF 207.

  A series of shooting sequences is started when the operation unit 208 receives a user operation and outputs an operation signal to the control unit 201. Various operations such as AF and AE accompanying the imaging sequence are executed under the control of the control unit 201. For example, the control unit 201 analyzes the detection signal of the AF sensor 211 and executes focus control for moving a focus lens that constitutes a part of the photographing lens 20.

  FIG. 2 is a conceptual diagram conceptually showing a state in which a part of the image sensor 100 is enlarged. In the pixel area, 20 million or more pixels are arranged in a matrix. In the present embodiment, 64 pixels of adjacent 8 pixels × 8 pixels form one basic lattice 110. The basic grid 110 includes four Bayer arrays having 4 × 2 × 2 basic units in the Y-axis direction and four in the X-axis direction. As shown in the figure, in the Bayer array, a green filter (G filter) is arranged for the upper left pixel and the lower right pixel, a blue filter (B filter) is arranged for the lower left pixel, and a red filter (R filter) is arranged for the upper right pixel.

  The basic grid 110 includes parallax pixels and non-parallax pixels. The parallax pixel is a pixel that receives a partial light beam that is deviated from the optical axis among incident light beams that pass through the photographing lens 20. The parallax pixel is provided with an aperture mask having a deviated opening that is deviated from the center of the pixel so as to transmit only the partial light flux. For example, the opening mask is provided so as to overlap the color filter. In this embodiment, the aperture mask defines the parallax Lt pixel that is defined so that the partial light beam reaches the −X side with respect to the pixel center, and the partial light beam that reaches the + X side with respect to the pixel center. There are two types of parallax Rt pixels. On the other hand, the non-parallax pixel is a pixel that is not provided with an aperture mask, and is a pixel that receives the entire incident light beam that passes through the photographing lens 20.

  Note that the parallax pixel is not limited to the aperture mask when receiving the partial light beam that is deviated from the optical axis, but has various configurations such as a selective reflection film in which the light receiving region and the reflective region are separated, and a deviated photodiode region. Can be adopted. In other words, the parallax pixel only needs to be configured to receive a partial light beam that is deviated from the optical axis, among incident light beams that pass through the photographing lens 20.

Pixels in the basic grid 110 are denoted by _PIJ . For example, the end of the pixel of -X and + Y are _{P 11,} the end of the pixel of + X and -Y is _{P 81.} As shown in the figure, the parallax pixels are arranged as follows.
P ₁₁ : Parallax Lt pixel + G filter (= G (Lt))
P ₅₁ ... Parallax Rt pixel + G filter (= G (Rt))
P ₃₂ ... Parallax Lt pixel + B filter (= B (Lt))
P ₆₃ ... Parallax Rt pixel + R filter (= R (Rt))
P ₁₅ ... Parallax Rt pixel + G filter (= G (Rt))
P ₅₅ ... Parallax Lt pixel + G filter (= G (Lt))
P ₇₆ ... Parallax Rt pixel + B filter (= B (Rt))
P ₂₇ ... Parallax Lt pixel + R filter (= R (Lt))
The other pixels are non-parallax pixels, and are any of the non-parallax pixel + R filter, the non-parallax pixel + G filter, and the non-parallax pixel + B filter.

  When viewed as a whole of the image sensor 100, the parallax pixels are classified into one of a first group having a G filter, a second group having an R filter, and a third group having a B filter. Includes at least one parallax Lt pixel and parallax Rt pixel belonging to each group. As in the example in the figure, these parallax pixels and non-parallax pixels may be arranged with randomness in the basic lattice 110. By arranging with randomness, RGB color information can be acquired as the output of the parallax pixels without causing bias in the spatial resolution for each color component, so that high-quality parallax image data can be obtained. can get.

  FIG. 3 is a diagram illustrating an example of processing for generating 2D image data and parallax image data. The concept of processing for generating a RAW image data set including 2D image data and parallax image data from RAW original image data output from the image sensor 100 will be described with reference to FIG.

  As can be seen from the arrangement of the parallax pixels and the non-parallax pixels in the basic grid 110, even if the digital image signals of the image sensor 100 are aligned with the pixel arrangement, they are not image data representing a specific image. Only when the digital image signals of the image sensor 100 are separated and collected for each identically characterized pixel group, image data representing one image corresponding to the feature is formed. For example, when the parallax Rt pixel and the parallax Lt pixel are gathered together, parallax image data indicating left and right parallax images having parallax can be obtained. In this way, each piece of image data separated and collected for each identically characterized pixel group is referred to as plane data.

  The image processing unit 260 receives RAW original image data in which the digital image signals are arranged in the pixel arrangement order of the image sensor 100, and executes plane separation processing for separating the digital image signal into a plurality of plane data as pixel values. The left column of the figure shows an example of processing for generating 2D-RGB plane data indicating a 2D image.

In generating the 2D-RGB plane data, the image processing unit 260 first removes the pixel values of the parallax pixels to form a vacant lattice. Then, the pixel value that becomes the empty grid is calculated by interpolation processing using the pixel values of the surrounding pixels. For example, the pixel values of the vacancy _{P 11} is the pixel value of the G filter pixels adjacent in an oblique _{direction, P -1-1, P 2-1, P} -12, averages calculates the pixel values of _{P 22} To calculate. Further, for example, the pixel value of the empty lattice P ₆₃ is obtained by averaging pixel values of P ₄₃ , P ₆₁ , P ₈₃ , and P ₆₅ which are pixel values of the adjacent R filter by skipping one pixel in the X direction and the Y direction. To calculate. Similarly, for example, the pixel value of the empty lattice P ₇₆ is obtained by averaging the pixel values of P ₅₆ , P ₇₄ , P ₉₆ , and P ₇₈ that are the pixel values of the adjacent B filter by skipping one pixel in the X direction and the Y direction. Calculate by calculation.

  Since the 2D-RGB plane data interpolated in this way is the same as the output of a normal imaging device having a Bayer array, various processes can be performed as 2D image data thereafter. That is, known Bayer interpolation is performed to generate color image data in which RGB data is aligned for each pixel. The image processing unit 260 performs image processing as a general 2D image according to a predetermined format such as JPEG when generating still image data and MPEG when generating moving image data.

  In the present embodiment, the image processing unit 260 further separates the 2D-RGB plane data for each color, performs the interpolation process as described above, and generates each plane data as reference image data. That is, three types of data are generated: Gn plane data as green reference image plane data, Rn plane data as red reference image plane data, and Bn plane data as blue reference image plane data.

  The right column of the figure shows an example of generation processing of two G plane data, two R plane data, and two B plane data as parallax pixel data. The two G plane data are GLt plane data as left parallax image data and GRt plane data as right parallax image data. The two R plane data are RLt plane data and right parallax image data as left parallax image data. The two B plane data are the BLt plane data as the left parallax image data and the BRt plane data as the right parallax image data.

In generating the GLt plane data, the image processing unit 260 removes pixel values other than the pixel value of the G (Lt) pixel from all output values of the image sensor 100 to form a vacant lattice. As a result, two pixel values P ₁₁ and P ₅₅ remain in the basic grid 110. Therefore, the basic grid 110 is equally divided into four in the XY plane, the 16 pixels on the −X and + Y sides are represented by the output value of P ₁₁ , and the 16 pixels on the + X and −Y sides are represented by the output value of P _55. Let me represent. Then, 16 pixels on the + X and + Y sides and 16 pixels on the -X and -Y sides are interpolated by averaging the peripheral representative values adjacent in the X direction and the Y direction. That is, the GLt plane data has one value in units of 16 pixels.

Similarly, when generating the GRt plane data, the image processing unit 260 removes pixel values other than the pixel value of the G (Rt) pixel from all the output values of the image sensor 100 to form an empty grid. Then, two pixel values P ₅₁ and P ₁₅ remain in the basic grid 110. Therefore, the basic grid 110 is divided into four equal parts in the XY plane, the 16 pixels on the + X and + Y sides are represented by the output value of P ₅₁ , and the 16 pixels on the −X and −Y sides are represented by the output value of P _15. Let me represent. Then, 16 pixels on the -X and + Y sides and 16 pixels on the + X and -Y sides are interpolated by averaging the peripheral representative values adjacent in the X and Y directions. That is, the GRt plane data has one value in units of 16 pixels. In this way, it is possible to generate GLt plane data and GRt plane data having a resolution lower than that of 2D-RGB plane data.

In generating the RLt plane data, the image processing unit 260 removes pixel values other than the pixel values of the R (Lt) pixels from all output values of the image sensor 100 to form a vacant lattice. Then, the primitive lattice _110, the pixel values of _{P 27} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. Similarly, when generating the RRt plane data, the image processing unit 260 removes pixel values other than the pixel value of the R (Rt) pixel from all the output values of the image sensor 100 to form an empty grid. Then, the pixel value P ₆₃ remains in the basic grid 110. This pixel value is set as a representative value for 64 pixels of the basic grid 110. In this way, RLt plane data and RRt plane data having a resolution lower than that of 2D-RGB plane data are generated. In this case, the resolution of the RLt plane data and the RRt plane data is lower than the resolution of the GLt plane data and the GRt plane data.

In generating the BLt plane data, the image processing unit 260 removes pixel values other than the pixel value of the B (Lt) pixel from all the output values of the image sensor 100 to form a vacant lattice. Then, the primitive lattice _110, the pixel values of _{P 32} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. Similarly, when generating the BRt plane data, the image processing unit 260 removes pixel values other than the pixel value of the B (Rt) pixel from all the output values of the image sensor 100 to obtain an empty grid. Then, the primitive lattice _110, the pixel values of _{P 76} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. In this way, BLt plane data and BRt plane data having a resolution lower than that of 2D-RGB plane data are generated. In this case, the resolution of the BLt plane data and the BRt plane data is lower than the resolution of the GLt plane data and the GRt plane data, and is equal to the resolution of the RLt plane data and the RRt plane data.

  In the present embodiment, the image processing unit 260 generates color image data of the left viewpoint and color image data of the right viewpoint using these plane data. Furthermore, the moving image may be generated by arranging the plane data in time series.

  FIG. 4 is a diagram for explaining the concept of defocusing. The parallax Lt pixel and the parallax Rt pixel receive the subject luminous flux that arrives from one of the two parallax virtual pupils set as the optical axis target as a partial region of the lens pupil. In the optical system of the present embodiment, since the actual subject light flux passes through the entire lens pupil, the light intensity distributions corresponding to the parallax virtual pupil are not distinguished from each other until the parallax pixel is reached. However, the parallax pixel outputs an image signal obtained by photoelectrically converting only the partial light flux that has passed through the parallax virtual pupil by the action of the aperture mask that each has. Therefore, the pixel value distribution indicated by the output of the parallax pixel may be considered to be proportional to the light intensity distribution of the partial light flux that has passed through the corresponding parallax virtual pupil.

  As shown in FIG. 4A, when an object point that is a subject exists at the focal position, the output of each parallax pixel is the corresponding image point regardless of the subject luminous flux that has passed through any parallax virtual pupil. This shows a steep pixel value distribution centering on this pixel. If the parallax Lt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Further, even when the parallax Rt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. That is, even if the subject luminous flux passes through any parallax virtual pupil, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Match each other.

  On the other hand, as shown in FIG. 4B, when the object point deviates from the focal position, the peak of the pixel value distribution indicated by the parallax Lt pixel corresponds to the image point, compared to the case where the object point exists at the focal position. Appearing at a position away from the pixel in one direction, and its output value decreases. In addition, the width of the pixel having the output value is increased. The peak of the pixel value distribution indicated by the parallax Rt pixel appears at a position away from the pixel corresponding to the image point in the opposite direction to the one direction in the parallax Lt pixel and at an equal distance, and the output value similarly decreases. Similarly, the width of the pixel having the output value is increased. That is, the same pixel value distribution that is gentler than that in the case where the object point exists at the focal position appears at an equal distance from each other. Further, as shown in FIG. 4C, when the object point further deviates from the focal position, the same pixel value distribution that becomes more gentle as compared with the state of FIG. 4B appears further apart. . That is, it can be said that the amount of blur and the amount of parallax increase as the object point deviates from the focal position. In other words, the amount of blur and the amount of parallax change in conjunction with defocus. That is, the amount of blur and the amount of parallax have a one-to-one relationship.

  FIGS. 4B and 4C show the case where the object point shifts away from the focal position, but when the object point moves away from the focal position, as shown in FIG. Compared to FIGS. 4B and 4C, the relative positional relationship between the pixel value distribution indicated by the parallax Lt pixel and the pixel value distribution indicated by the parallax Rt pixel is reversed. Due to such a defocus relationship, when viewing a parallax image, the viewer visually recognizes a subject existing far behind the focal position and visually recognizes a subject present in front.

  When the change of the pixel value distribution described in FIGS. 4B and 4C is graphed, it is expressed as shown in FIG. In the figure, the horizontal axis represents the pixel position, and the center position is the pixel position corresponding to the image point. The vertical axis represents the output value (pixel value) of each pixel. As described above, this output value is substantially proportional to the light intensity.

  The distribution curve 804 and the distribution curve 805 represent the pixel value distribution of the parallax Lt pixel and the pixel value distribution of the parallax Rt pixel in FIG. 4B, respectively. As can be seen from the figure, these distributions have a line-symmetric shape with respect to the center position. Further, a combined distribution curve 806 obtained by adding them shows a pixel value distribution of pixels without parallax with respect to the situation of FIG. 4B, that is, a pixel value distribution when the entire subject light beam is received, and a substantially similar shape.

  A distribution curve 807 and a distribution curve 808 represent the pixel value distribution of the parallax Lt pixel and the pixel value distribution of the parallax Rt pixel in FIG. 4C, respectively. As can be seen from the figure, these distributions are also symmetrical with respect to the center position. Further, a combined distribution curve 809 obtained by adding them shows a substantially similar shape to the pixel value distribution of the non-parallax pixels for the situation of FIG.

  FIG. 6 is a diagram for explaining the concept of defocusing during camera shake and camera shake correction. FIG. 6 (a) is a reprint of FIG. 4 (b) for explanation. FIG. 6B shows a state in which the digital camera 10 is rotated around the Y axis, that is, yawed due to camera shake. FIG. 3C shows a state in which camera shake correction is performed on a state yawing due to camera shake.

  Consider a case where image light of an object point is input in the vicinity of the position X0 of the center of the image sensor 100 in the X direction as shown in FIG. Assume that the digital camera 10 yaws counterclockwise in the figure due to camera shake as shown in FIG. 6B from the state of FIG. Due to the camera shake, the image light of the object point moves from the position X0 to the vicinity of the position X1 in the −X direction on the image sensor 100.

  When the shake sensor 252 detects the yawing, the driving unit 250 moves the correction lens 30 in the + X direction parallel to the imaging surface of the imaging element 100. As a result, the lens pupil also moves in the direction of the arrow in FIG. 6C, and the image light of the object point moves to a position X2 close to the position X0 in FIG. Therefore, image light blur due to yawing is corrected. Although it is preferable to move the correction lens 30 so that the position X2 coincides with the position X0, it may not be completely coincident by giving priority to other conditions.

  FIG. 7 is an enlarged view showing the inside of a broken circle in FIG. In FIG. 7, the image pickup surface of the image pickup device 100 is drawn in the vertical direction in the drawing.

  As shown in FIG. 7, when the correction lens 30 moves from the center position, the input position of the image light to the image sensor 100 changes although the input position of the image light substantially coincides with that before camera shake. As the correction lens 30 is moved in the + X direction parallel to the imaging surface of the image sensor 100, the chief ray angle θR input to the parallax Rt pixel and the chief ray angle θL input to the parallax Lt pixel are the principal rays before camera shake. It changes from the angle of the ray. Further, in the example shown in FIG. 7, the chief ray angle θR input to the parallax Rt pixel is larger than the chief ray angle θL input to the parallax Lt pixel.

  Here, since the sensitivity of the parallax pixel has a dependency on the angle at which the image light is input, the parallax amount of the parallax image output from the parallax pixel is changed by the change in the angle before and after the camera shake correction. Therefore, the amount of parallax differs between the plurality of parallax images associated with each other, such as continuous shooting and moving images, due to camera shake correction for each parallax image. Therefore, there is a possibility that an observer who views the plurality of parallax images may feel uncomfortable or uncomfortable.

  Therefore, in the present embodiment, a corrected image in which the parallax amount of the parallax image is corrected based on the camera shake correction amount is generated. In this case, it is preferable to perform correction so that the parallax amount of the corrected image is smaller than the parallax amount of the original parallax image. Furthermore, it is preferable that the parallax amounts of the plurality of parallax images are corrected so that the parallax amounts are within a predetermined range between the parallax images associated with each other. Instead of this or in addition to this, it is preferable to generate a corrected image in which the amount of parallax is reduced as the camera shake correction amount is larger.

  FIG. 8 is a functional block diagram of the image processing unit 260. The image processing unit 260 includes a parallax image acquisition unit 261, a correction amount acquisition unit 262, a correction image generation unit 264, and a table storage unit 266.

  The parallax image acquisition unit 261 acquires a RAW image data set from the memory 203 or the memory card 220. The correction amount acquisition unit 262 acquires the camera shake correction amount written in the header portion of the RAW image data set. The corrected image generation unit 264 generates a corrected image data set in which the parallax amount of the RAW image data set is corrected based on the camera shake correction amount. The table storage unit 266 stores a correspondence table in which the relationship between the camera shake correction amount and the parallax amount to be corrected is associated.

  FIG. 9 is a conceptual diagram illustrating the camera shake correction amount. In FIG. 9, the relationship between the center position of the correction lens 30 alone and the center position of the imaging surface of the image sensor 100 is depicted.

  In FIG. 9, the center position V0 of the correction lens 30 is at the center position of the image sensor 100 before the camera shake correction. After that, it is assumed that the camera has been moved to the position V1 at the end of exposure by drawing a solid locus in the drawing by camera shake correction during exposure.

  In this case, as the camera shake correction amount used to correct the parallax amount, for example, the maximum value Vmax in the X-axis direction and the information in the ± direction are used. Instead of or in addition to this, the value in the X-axis direction of the position V2 obtained by time-integrating the trajectory, in other words, the time average value Vavg and the information in the ± direction may be used. Further, the locus itself may be used as a camera shake correction amount.

  The amount of parallax correction with respect to the amount of camera shake correction is set by grasping the relationship between the amount of camera shake correction and the amount of change in parallax experimentally or analytically. In this case, a correction amount for correcting the parallax amount uniformly for the entire screen of the parallax image may be set for the camera shake correction amount, or a different correction amount may be set for each region of the screen. . When a different correction amount is set for each area of the screen, the area of the screen may be set according to the direction of camera shake correction.

  FIG. 10 shows an example of a flowchart of the operation of the digital camera 10. The operation shown in FIG. 10 is started when the release button is input to the operation unit 208.

  The camera shake sensor 252 detects the posture of the digital camera 10 and outputs it to the control unit 201 (S102). The control unit 201 calculates a camera shake correction amount of the correction lens 30 that corrects camera shake based on the output from the camera shake sensor 252, and the drive unit 250 drives the correction lens 30 to move it by the camera shake correction amount (S104). The control unit 201 records the camera shake correction amount in the memory 203 (S105). Steps S102 to S105 are repeated until the exposure set according to the imaging conditions is completed (S106: No).

  When the exposure is completed (S106: Yes), the image processing unit 260 generates the RAW image data set shown in FIG. 3 from the RAW original image data output from the image sensor 100 (S108). The RAW image data set is written into the memory 203 or the memory card 220. In this case, the amount of camera shake correction and, if the RAW image data set is one frame of continuous shooting or moving image, are written in the header part of the RAW image data set.

  The parallax image acquisition unit 261 acquires the RAW image data set written in the memory 203 or the memory card 220 (S109). Furthermore, the parallax image acquisition unit 261 determines whether or not the RAW image data set is associated with another RAW image data set (S110). In this case, the parallax image acquisition unit 261 refers to the header part of the RAW image data set, and when it is written that it is one frame of continuous shooting or moving image, It is determined that they are associated (S110: Yes), and when it is not written that it is one frame of continuous shooting or moving image, it is determined that they are not associated (S110: No).

  When it is determined that the parallax image acquisition unit 261 is associated with another RAW image data set (S110: Yes), the correction amount acquisition unit 262 acquires the camera shake correction amount of the RAW image data set (S112). In this case, the correction amount acquisition unit 262 reads out the camera shake correction amount from the header portion of the RAW image data set, and calculates the camera shake correction amount corresponding to the correspondence table. For example, when a camera shake correction locus is recorded in the header portion of the RAW image data, and the correspondence table associates the maximum value Vmax or the average value Vavg of the camera shake correction amount with the parallax correction amount, the correction amount The acquisition unit 262 calculates the maximum value Vmax or the average value Vavg as shown in FIG. 8 from the read camera-shake correction locus, and passes it to the corrected image generation unit 264.

  The corrected image generation unit 264 identifies the parallax correction amount corresponding to the camera shake correction amount by referring to the correspondence table stored in the table storage unit 266 (S114). The corrected image generation unit 264 generates a corrected image data set obtained by correcting the RAW image data set with the parallax correction amount (S116).

  The corrected image generation unit 264 stores the corrected image data set generated in step S116 or the RAW image data set acquired by the parallax image acquisition unit 261 when it is determined that there is no association in step S110. (S118). Thus, the operation according to this flowchart is completed.

  FIG. 11 shows an example of a method for correcting the amount of parallax in step S116 of the flowchart of FIG. FIG. 12 is an explanatory diagram showing changes in pixel values due to correction of the parallax amount in FIG.

  In particular, in the examples of FIGS. 11 and 12, by introducing the stereo adjustment parameter, a corrected image data set in which the parallax amount is arbitrarily corrected within a range in which the main subject is not blurred is generated.

  In the correspondence table, the amount of camera shake correction is associated with a three-dimensional adjustment parameter C that is uniformly applied to the entire parallax image. The three-dimensional adjustment parameter C is preferably set in a range of 0.5 <C <1. In the correspondence table, it is preferable that the three-dimensional adjustment parameter C is smaller as the camera shake correction amount is larger.

  FIG. 11 shows an example of processing for generating red right parallax image data and red left parallax image data in the corrected image data set.

The red parallax image data is generated using the pixel values of the Rn plane data of the RAW image data set acquired by the parallax image acquisition unit 261, and the luminance values of the RLt plane data and the RRt plane data. Specifically, for example, when calculating the pixel value RLt _mn of the target pixel position (i _m , j _n ) of the left parallax image data, first, the corrected image generation unit 264 first uses the same pixel position (i _{m) of the} Rn plane data. , J _n ) extract the pixel value Rn _mn . Next, the correction image generation unit 264, the same pixel position of RLt plane data _(i m, _{j n)} from the luminance value _{RLt mn,} the same pixel position of the RRT plane data _(i m, _{j n)} the luminance value _{RRT mn} from To extract. Then, the image processing unit 260, the pixel values _{Rn mn,} by multiplying the value obtained by distributing the luminance value _{RLt mn} and _{RRT mn} stereoscopic adjustment parameter C, and calculates the pixel value RLct _mn. Specifically, it is calculated by the following equation (1).
RLct _mn = 2Rn _mn x {C · RLt _mn + (1-C) · RRt _mn } / (RLt _mn + RRt _mn ) (1)

Similarly, the target pixel position in the right parallax image data _(i m, _{j n)} may be calculated pixel value RRCt _mn of the correction image generation unit 264, the luminance value _{RLt mn} and the luminance value _{RRT mn} to the pixel values _{Rn mn} Is multiplied by the value distributed by the three-dimensional adjustment parameter C. Specifically, it is calculated by the following equation (2).
RRct _mn = 2Rn _mn × {C · RRt _mn + (1-C) · RLt _mn } / (RLt _mn + RRt _mn ) (2)

Correction image generation unit 264, such processing is sequentially executed from the pixel at the left end and the upper end (1,1) to a right end and the lower end of the coordinate (i _{0, j 0).}

  When the generation processing of the red parallax image data and the left parallax image data is completed, the green right parallax image data and the left parallax image data, and the blue right parallax image data and the left parallax image data are similarly generated. . With the above processing, a corrected image data set including parallax information is completed.

  FIG. 12A shows a G (Lt) pixel, a G (Rt) pixel, and an R (Lt) pixel when a white subject light beam from an object point located at a position shifted by a certain amount from the focal position is received. , R (Rt) pixels, B (Lt) pixels, and B (Rt) pixels.

  FIG. 12B shows the R (N) pixel, G (N) pixel, and B (N) pixel that are non-parallax pixels when a white subject light beam from the object point in FIG. 12A is received. It is the graph which arranged the output value. It can be said that this graph also represents the pixel value distribution of each color.

  When the above processing is performed for each corresponding pixel with C = 0.8, the light intensity distribution represented by the graph of FIG. As can be seen from FIG. 12 and the above formulas (1) and (2), the parallax amount is larger as the three-dimensional adjustment parameter C is closer to 1 in the range of 0.5 <C <1, and the parallax amount is smaller as it is closer to 0.5. Become. That is, the three-dimensional adjustment parameter C includes the parallax amount indicated by the parallax Lt pixel and the parallax Rt pixel when the parallax amount based on the parallax amount indicated by the parallax Lt pixel and the parallax amount indicated by the parallax Rt pixel is assigned to the non-parallax pixel. The weight with the amount of parallax shown is adjusted. One of the weights of the parallax amount indicated by the parallax Lt pixel and the parallax amount indicated by the parallax Rt pixel is larger as the stereoscopic adjustment parameter C is closer to 1, and the parallax amount closer to 0.5 is closer to 0.5. Assigned to a pixel.

  As described above, according to the present embodiment, it is possible to reduce discomfort and discomfort due to a change in the amount of parallax due to camera shake correction while correcting blur of image light on the image sensor 100 due to camera shake.

  In the correspondence table, a camera shake correction amount and a parallax correction amount are associated with each other. Instead, the shake amount of the digital camera 10 and the parallax correction amount may be associated with each other in the correspondence table. In this case, the blur amount of the digital camera 10 is written in the header part of the RAW image data set. Also, a parallax correction amount based on a parallax change when camera shake correction is performed on the blur amount of the digital camera 10 is calculated in advance, and the blur amount of the digital camera 10 and the parallax correction amount are calculated in the correspondence table. Are associated.

  In step S116 of the flowchart, when the camera shake correction amount is in a predetermined range, for example, when it is larger than the threshold, a 2D image with no parallax may be used as a corrected image. In this case, Rn plane data, Gn plane data, and Bn plane data in the RAW image data set may be used as a corrected image data set. Instead, RGB image data in which parallax is eliminated by setting the stereoscopic adjustment parameter C to 0.5 may be used as a corrected image data set. On the other hand, when the camera shake correction amount is in another predetermined range, for example, when it is smaller than the second threshold value smaller than the threshold value, the parallax image may not be corrected.

  The parallax amount may be corrected using a parameter other than the three-dimensional adjustment parameter C. For example, when the RAW source image data set has a parallax amount map obtained by separating the parallax amount from the image, the parallax amount of the map may be directly corrected to be within a predetermined range.

  In the above-described embodiment, the correction of the parallax amount when the correction lens 30 moves in the X-axis direction has been described. Instead of or in addition to this, the parallax amount may be corrected when the correction lens 30 moves in the Y-axis direction. In this case, the relationship between the movement of the correction lens 30 in the Y-axis direction and the change in the amount of parallax is specified experimentally or analytically, and the camera shake correction amount, which is the amount of movement in the Y direction, is determined based on the relationship. A correspondence table with the parallax correction amount may be stored.

  In the above embodiment, the correction lens 30 has been described as moving in the XY plane. Instead of this, the above embodiment may be applied when the correction lens 30 moves while tilting with respect to the XY plane. Also in this case, the relationship between the movement of the correction lens 30 while tilting and the change in the amount of parallax is obtained experimentally or analytically, and based on the relationship, the amount of camera shake correction that is the amount of movement in the Y direction and the amount of parallax are determined. It is only necessary to store a correspondence table with the correction amount. In place of or in addition to providing the correction lens 30, the above-described embodiment may be applied to the case where camera shake is corrected by moving the image sensor 100.

  In the above embodiment, the parallax amount is corrected when the RAW image data set is associated with another RAW image data set. Instead of this, the amount of parallax may be corrected regardless of whether it is associated with another RAW image data set.

  In the above embodiment, the parallax amount of the RAW image data set generated by the image signal output from the image sensor 100 is corrected. Instead of or in addition to this, a RAW image data set generated by a device other than the image sensor 100 may be acquired from the outside, and the amount of parallax may be corrected. The parallax amount may be corrected for a data set of another image format without being limited to the RAW image data set.

  Further, in the above embodiment, when the parallax image acquisition unit 261 refers to the header portion of the RAW image data set and writes that it is one frame of continuous shooting or moving image, another RAW Determined to be associated with the image dataset. Instead of this, or in addition to this, the parallax image acquisition unit 261 is a RAW image data set that is associated with each other in which the shooting times written in the header portion of the RAW image data set are within a certain period of time. You may judge. In this case, if the camera shake correction amount at the time of any imaging is greater than or equal to the threshold value, any of the RAW image data sets generated by imaging within that time may be corrected to a 2D image. Good. On the other hand, when the camera shake correction amount at any time of imaging is within the threshold value, the amount of change in the parallax of the next RAW image data set is less than a certain amount with respect to the previous RAW image data set. In addition, the parallax amount of the next RAW image data set may be corrected.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 10 Digital camera, 20 Shooting lens, 21 Optical axis, 22 Aperture, 30 Correction lens, 100 Image sensor, 110 Basic lattice, 201 Control part, 202 A / D conversion circuit, 203 Memory, 204 Drive part, 207 Memory card IF, 208 operation unit, 209 display unit, 210 LCD drive circuit, 211 AF sensor, 220 memory card, 250 drive unit, 252 camera shake sensor, 260 image processing unit, 261 parallax image acquisition unit, 262 correction amount acquisition unit, 264 correction image generation , 266 table storage unit, 804 distribution curve, 805 distribution curve, 806 composite distribution curve, 807 distribution curve, 808 distribution curve, 809 composite distribution curve

Claims (6)

A parallax image acquisition unit that acquires a parallax image including a parallax amount indicating parallax with respect to the same object;
A correction amount acquisition unit for acquiring a camera shake correction amount for correcting camera shake during imaging;
An image processing comprising: a correction image generation unit configured to generate a correction image in which the parallax amount of the parallax image acquired by the parallax image acquisition unit is corrected based on the camera shake correction amount acquired by the correction amount acquisition unit. apparatus. A storage unit that stores a correspondence table in which a relationship between the camera shake correction amount and the parallax amount to be corrected is associated;
The image processing apparatus according to claim 1, wherein the corrected image generation unit generates the corrected image with reference to the correspondence table.   The image processing apparatus according to claim 1, wherein the corrected image is generated by reducing the parallax amount as the camera shake correction amount increases.   The image processing apparatus according to claim 1, wherein the correction image generation unit generates a correction image with no parallax when the camera shake correction amount is within a predetermined range. The parallax image acquisition unit determines whether or not the parallax images are associated when acquiring a plurality of the parallax images;
The correction image generation unit generates the plurality of correction images by associating the plurality of parallax images when the parallax image acquisition unit determines that the plurality of parallax images are associated with each other. 5. The image processing apparatus according to any one of 4 above. An image processing apparatus according to any one of claims 1 to 5,
A plurality of two-dimensionally arranged photoelectric conversion elements that photoelectrically convert incident light into an electrical signal, and light from a partial region in a cross-sectional area of the incident light, respectively, to the corresponding plurality of photoelectric conversion elements An imaging apparatus comprising: a light-shielding unit provided with an opening for incidence, and a photographing unit that photographs the target and generates the parallax image.

Images