JP6439285B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

There is known a stereo imaging device that acquires a stereo image composed of a right-eye image and a left-eye image using two photographing optical systems.
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP-A-8-47001

  When creating a three-dimensional image using stereo image data captured by a stereo imaging device, the depth in stereoscopic vision can be adjusted by adjusting the amount of parallax. In order to generate a more natural three-dimensional image, in order to set an adjustment amount of the parallax amount for each subject (or for each pixel constituting the image), adjustment is performed by a plurality of parameters such as a focal length of the lens and an aperture value. However, it is difficult to automatically adjust these parameters to set an optimal adjustment amount for each subject (for each pixel).

  An image processing apparatus according to a first aspect of the present invention includes an acquisition unit that acquires image data, and an evaluation value of a subject area included in the image of the image data is two-dimensionally arranged in correspondence with the position of the subject area. An evaluation map generation unit that generates the evaluation map, and a determination unit that determines an adjustment parameter value for adjusting the amount of parallax, which is applied when generating parallax image data from the image data, corresponding to the evaluation value of the evaluation map And a generation unit that applies the adjustment parameter value to the image data and generates parallax image data that is an image having a parallax amount adjusted to each other.

  The image processing program according to the second aspect of the present invention includes an acquisition step of acquiring image data, and an evaluation value of a subject area included in the image of the image data is two-dimensionally arranged in correspondence with the position of the subject area. An evaluation map generation step for generating the evaluation map, and a determination step for determining an adjustment parameter value for adjusting the amount of parallax, which is applied when generating the parallax image data from the image data, corresponding to the evaluation value of the evaluation map And a generation step of generating parallax image data that is an image having parallax amounts adjusted to each other by applying the adjustment parameter value to the image data.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is a figure explaining the structure of the digital camera which concerns on embodiment of this invention. It is a conceptual diagram which represents notably the mode that a part of imaging device was expanded. It is a figure explaining the example of a production | generation process of 2D image data and parallax image data. It is a figure explaining the concept of defocusing. It is a figure which shows the light intensity distribution which a parallax pixel outputs. It is a figure which shows pixel value distribution for demonstrating the concept of adjustment parallax amount. It is a figure explaining the production | generation process of color parallax plane data. It is a figure explaining the change of RGB pixel value distribution. It is a figure which shows the input image and the image of a saliency map. It is a figure which illustrates the parameter value calculation function of a three-dimensional adjustment parameter. It is a figure which shows the relationship between a viewer's convergence angle and the amount of parallax. It is a rear view of the digital camera which displays the menu screen of parallax amount restriction | limiting. It is a processing flow in moving image shooting. It is a processing flow until it produces | generates parallax color image data. It is a figure explaining a preferable opening shape. It is a figure explaining cooperation with a digital camera and TV monitor. It is a processing flow in the moving image photography of the digital camera as a modification. It is a processing flow in the TV monitor as a modification.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  The digital camera according to the present embodiment, which is one form of the imaging device, is configured to generate images with a plurality of viewpoints for one scene by one shooting. Each image having a different viewpoint is called a parallax image. In the present embodiment, a case where a right parallax image and a left parallax image from two viewpoints corresponding to the right eye and the left eye are generated will be described. The digital camera in the present embodiment can generate a parallax-free image without parallax from the central viewpoint together with the parallax image.

  FIG. 1 is a diagram illustrating the configuration of a digital camera 10 according to an embodiment of the present invention. The digital camera 10 includes a photographic lens 20 as a photographic optical system, and guides a subject light beam incident along the optical axis 21 to the image sensor 100. The photographing lens 20 may be an interchangeable lens that can be attached to and detached from the digital camera 10. The digital camera 10 includes an image sensor 100, a control unit 201, an A / D conversion circuit 202, a memory 203, a drive unit 204, an image processing unit 205, a memory card IF 207, an operation unit 208, a display unit 209, and an LCD drive circuit 210. .

  As shown in the figure, the direction parallel to the optical axis 21 toward the image sensor 100 is defined as the Z-axis plus direction, the direction toward the front of the drawing on the plane orthogonal to the Z-axis is the X-axis plus direction, and the upward direction on the drawing is Y. The axis is defined as the plus direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

  The taking lens 20 is composed of a plurality of optical lens groups, and forms an image of a subject light flux from the scene in the vicinity of its focal plane. In FIG. 1, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens arranged in the vicinity of the pupil. Further, in the vicinity of the pupil, a diaphragm 22 that restricts the incident light beam concentrically with the optical axis 21 as the center is disposed.

  The image sensor 100 is disposed near the focal plane of the photographic lens 20. The image sensor 100 is an image sensor such as a CCD or CMOS sensor in which a plurality of photoelectric conversion elements are two-dimensionally arranged. The image sensor 100 is controlled in timing by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202.

  The A / D conversion circuit 202 converts the image signal output from the image sensor 100 into a digital image signal and outputs the digital image signal to the memory 203. The image processing unit 205 performs various image processing on the digital image signal using the memory 203 as a work space, and generates captured image data. The captured image data includes reference image data generated from the output of the non-parallax pixels of the image sensor 100 and parallax image data generated from the output of the parallax pixels of the image sensor 100, as will be described later. In the case where the processing unit until the captured image data is generated is an imaging unit, the imaging unit is configured to include the imaging element 100, an A / D conversion circuit 202, a memory 203, a control unit 201, and an image processing unit 205.

  The control unit 201 controls the digital camera 10 in an integrated manner. For example, the aperture of the diaphragm 22 is adjusted according to the set diaphragm value, and the photographing lens 20 is advanced and retracted in the optical axis direction according to the AF evaluation value. Further, the position of the photographing lens 20 is detected, and the focal length and the focus lens position of the photographing lens 20 are grasped. Further, a timing control signal is transmitted to the drive unit 204, and a series of sequences from when the image signal output from the image sensor 100 is processed into captured image data by the image processing unit 205 is managed, and the captured image data To get.

  The image processing unit 205 includes an evaluation map generation unit 231, an adjustment parameter value determination unit 232, a parallax image data generation unit 233, and a moving image generation unit 234. The evaluation map generation unit 231 generates an evaluation map for evaluating the captured image data.

  The adjustment parameter value determination unit 232 determines a stereoscopic adjustment parameter value for adjusting the amount of parallax, which is applied when generating parallax image data from captured image data. The parallax image data generation unit 233 applies the stereoscopic adjustment parameter value to the captured image data, and generates parallax image data that becomes images having parallax amounts adjusted to each other. The moving image generation unit 234 generates a 3D moving image file by joining the new parallax image data generated by the parallax image data generation unit.

  The image processing unit 205 also has general image processing functions such as adjusting image data according to the selected image format. The generated captured image data is converted into a display signal by the LCD drive circuit 210 and displayed on the display unit 209. The data is recorded on the memory card 220 attached to the memory card IF 207.

  FIG. 2 is a conceptual diagram conceptually showing a state in which a part of the image sensor 100 is enlarged. In the pixel area, 20 million or more pixels are arranged in a matrix. In the present embodiment, 64 pixels of adjacent 8 pixels × 8 pixels form one basic lattice 110. The basic grid 110 includes four Bayer arrays having 4 × 2 × 2 basic units in the Y-axis direction and four in the X-axis direction. As shown in the figure, in the Bayer array, a green filter (G filter) is arranged for the upper left pixel and the lower right pixel, a blue filter (B filter) is arranged for the lower left pixel, and a red filter (R filter) is arranged for the upper right pixel.

  The basic grid 110 includes parallax pixels and non-parallax pixels. The parallax pixel is a pixel that receives a partial light beam that is deviated from the optical axis among incident light beams that pass through the photographing lens 20. The parallax pixel is provided with an aperture mask having a deviated opening that is deviated from the center of the pixel so as to transmit only the partial light flux.

  For example, the opening mask is provided so as to overlap the color filter. In the present embodiment, the parallax Lt pixel defined so that the partial light beam reaches the left side with respect to the pixel center and the parallax specified so that the partial light beam reaches the right side with respect to the pixel center by the aperture mask. There are two types of Rt pixels. On the other hand, the non-parallax pixel is a pixel that is not provided with an aperture mask, and is a pixel that receives the entire incident light beam that passes through the photographing lens 20.

  Note that the parallax pixel is not limited to the aperture mask when receiving the partial light beam that is deviated from the optical axis, but has various configurations such as a selective reflection film in which the light receiving region and the reflective region are separated, and a deviated photodiode region. Can be adopted. In other words, the parallax pixel only needs to be configured to receive a partial light beam that is deviated from the optical axis, among incident light beams that pass through the photographing lens 20.

Pixels in the basic grid 110 are denoted by _PIJ . For example, the upper left pixel is _{P 11,} the upper right pixel is _{P 81.} As shown in the figure, the parallax pixels are arranged as follows.
P ₁₁ : Parallax Lt pixel + G filter (= G (Lt))
P ₅₁ ... Parallax Rt pixel + G filter (= G (Rt))
P ₃₂ ... Parallax Lt pixel + B filter (= B (Lt))
P ₆₃ ... Parallax Rt pixel + R filter (= R (Rt))
P ₁₅ ... Parallax Rt pixel + G filter (= G (Rt))
P ₅₅ ... Parallax Lt pixel + G filter (= G (Lt))
P ₇₆ ... Parallax Rt pixel + B filter (= B (Rt))
P ₂₇ ... Parallax Lt pixel + R filter (= R (Lt))
The other pixels are non-parallax pixels, and are any of the non-parallax pixel + R filter, the non-parallax pixel + G filter, and the non-parallax pixel + B filter.

  When viewed as a whole of the image sensor 100, the parallax pixels are classified into one of a first group having a G filter, a second group having an R filter, and a third group having a B filter. Includes at least one parallax Lt pixel and parallax Rt pixel belonging to each group. As in the example in the figure, these parallax pixels and non-parallax pixels may be arranged with randomness in the basic lattice 110. By arranging with randomness, RGB color information can be acquired as the output of the parallax pixels without causing bias in the spatial resolution for each color component, so that high-quality parallax image data can be obtained. can get.

  Next, the concept of processing for generating 2D image data and parallax image data from captured image data output from the image sensor 100 will be described. FIG. 3 is a diagram illustrating an example of processing for generating 2D image data and parallax image data.

  As can be seen from the arrangement of parallax pixels and non-parallax pixels in the basic grid 110, image data representing a specific image is not obtained even if the output of the image sensor 100 is aligned with the pixel arrangement. Only when the pixel outputs of the image sensor 100 are separated and collected for each pixel group characterized in the same manner, image data representing one image in accordance with the characteristics is formed. For example, when the left and right parallax pixels are gathered together, left and right parallax image data having parallax can be obtained. In this way, each piece of image data separated and collected for each identically characterized pixel group is referred to as plane data.

  The image processing unit 205 receives raw raw image data in which output values (pixel values) are arranged in the order of pixel arrangement of the image sensor 100, and executes plane separation processing for separating the raw image data into a plurality of plane data. The left column of the figure shows an example of processing for generating 2D-RGB plane data as 2D image data.

  In generating the 2D-RGB plane data, the image processing unit 205 first removes the pixel values of the parallax pixels to form an empty grid. Then, the pixel value that becomes the empty grid is calculated by interpolation processing using the pixel values of the surrounding pixels.

For example, the pixel values of the vacancy _{P 11} is the pixel value of the G filter pixels adjacent in an oblique _{direction, P -1-1, P 2-1, P} -12, averages calculates the pixel values of _{P 22} To calculate. Further, for example, the pixel value of the empty lattice P ₆₃ is calculated by averaging the pixel values of P ₄₃ , P ₆₁ , P ₈₃ , and P ₆₅ that are adjacent R filter pixel values by skipping one pixel vertically and horizontally. To do. Similarly, for example, the pixel value of the air grating _{P 76} is the pixel value of the adjacent B filter skipping one pixel vertically and _{horizontally,} and averaging operation of the pixel values of _{P 56, P 74, P 96} , P 78 calculate.

  Since the 2D-RGB plane data interpolated in this way is the same as the output of a normal imaging device having a Bayer array, various processes can be performed as 2D image data thereafter. That is, known Bayer interpolation is performed to generate color image data in which RGB data is aligned for each pixel. The image processing unit 205 performs image processing as a general 2D image according to a predetermined format such as JPEG when generating still image data and MPEG when generating moving image data.

  In the present embodiment, the image processing unit 205 further separates the 2D-RGB plane data for each color, performs the above-described interpolation processing, and generates each plane data as reference image data. That is, three types of data are generated: Gn plane data as green reference image plane data, Rn plane data as red reference image plane data, and Bn plane data as blue reference image plane data.

  The right column of the figure shows an example of generation processing of two G plane data, two R plane data, and two B plane data as parallax pixel data. The two G plane data are GLt plane data as left parallax image data and GRt plane data as right parallax image data. The two R plane data are RLt plane data and right parallax image data as left parallax image data. The two B plane data are the BLt plane data as the left parallax image data and the BRt plane data as the right parallax image data.

In generating the GLt plane data, the image processing unit 205 removes pixel values other than the pixel values of the G (Lt) pixels from all output values of the image sensor 100 to form a vacant lattice. As a result, two pixel values P ₁₁ and P ₅₅ remain in the basic grid 110. Therefore, it divided into four equal basic grid 110 vertically and horizontally, the 16 pixels of the top left is represented by an output value of the P _11, is representative of the 16 pixels in the lower right in the output value of the P _55. Then, for the upper right 16 pixels and the lower left 16 pixels, average values of neighboring representative values adjacent in the vertical and horizontal directions are averaged and interpolated. That is, the GLt plane data has one value in units of 16 pixels.

Similarly, when generating the GRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the G (Rt) pixel from all the output values of the image sensor 100 to obtain an empty grid. Then, two pixel values P ₅₁ and P ₁₅ remain in the basic grid 110. Therefore, the basic grid 110 is divided into four equal parts vertically and horizontally, the upper right 16 pixels are represented by the output value of P ₅₁ , and the lower left 16 pixels are represented by the output value of P ₁₅ . The upper left 16 pixels and the lower right 16 pixels are interpolated by averaging the peripheral representative values adjacent vertically and horizontally. That is, the GRt plane data has one value in units of 16 pixels. In this way, it is possible to generate GLt plane data and GRt plane data having a resolution lower than that of 2D-RGB plane data.

In generating the RLt plane data, the image processing unit 205 removes pixel values other than the pixel value of the R (Lt) pixel from all output values of the image sensor 100 to form a vacant lattice. Then, the primitive lattice _110, the pixel values of _{P 27} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. Similarly, when generating the RRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the R (Rt) pixel from all output values of the image sensor 100 to form a vacant lattice. Then, the pixel value P ₆₃ remains in the basic grid 110. This pixel value is set as a representative value for 64 pixels of the basic grid 110. In this way, RLt plane data and RRt plane data having a resolution lower than that of 2D-RGB plane data are generated. In this case, the resolution of the RLt plane data and the RRt plane data is lower than the resolution of the GLt plane data and the GRt plane data.

In generating the BLt plane data, the image processing unit 205 removes pixel values other than the pixel values of the B (Lt) pixels from all output values of the image sensor 100 to form a vacant lattice. Then, the primitive lattice _110, the pixel values of _{P 32} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. Similarly, when generating the BRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the B (Rt) pixel from all the output values of the image sensor 100 to obtain an empty grid. Then, the primitive lattice _110, the pixel values of _{P 76} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. In this way, BLt plane data and BRt plane data having a resolution lower than that of 2D-RGB plane data are generated. In this case, the resolution of the BLt plane data and the BRt plane data is lower than the resolution of the GLt plane data and the GRt plane data, and is equal to the resolution of the RLt plane data and the RRt plane data.

  In the present embodiment, the image processing unit 205 uses these plane data to generate left-viewpoint color image data and right-viewpoint color image data. In particular, by introducing a stereo adjustment parameter, color image data in which the parallax amount is adjusted while the blur amount is maintained is generated. Hereinafter, prior to specific processing in the present embodiment, the generation principle of color image data in which the amount of parallax is adjusted and the processing of a comparative example with respect to the processing of the present embodiment will be described with reference to FIGS. To do.

  FIG. 4 is a diagram for explaining the concept of defocusing. The parallax Lt pixel and the parallax Rt pixel receive a subject light flux that arrives from one of two parallax virtual pupils that are set symmetrically with respect to the optical axis as a partial region of the lens pupil.

  In the optical system of the present embodiment, since the actual subject light flux passes through the entire lens pupil, the light intensity distributions corresponding to the parallax virtual pupil are not distinguished from each other until the parallax pixel is reached. However, the parallax pixel outputs an image signal obtained by photoelectrically converting only the partial light flux that has passed through the parallax virtual pupil by the action of the aperture mask that each has. Therefore, the pixel value distribution indicated by the output of the parallax pixel may be considered to be proportional to the pixel value distribution of the partial light flux that has passed through the corresponding parallax virtual pupil.

  As shown in FIG. 4A, when an object point that is a subject exists at the focal position, the output of each parallax pixel is the corresponding image point regardless of the subject luminous flux that has passed through any parallax virtual pupil. This shows a steep pixel value distribution centering on this pixel. If the parallax Lt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Further, even when the parallax Rt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. That is, even if the subject luminous flux passes through any parallax virtual pupil, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Match each other.

  On the other hand, as shown in FIG. 4B, when the object point deviates from the focal position, the peak of the pixel value distribution indicated by the parallax Lt pixel corresponds to the image point, compared to the case where the object point exists at the focal position. Appearing at a position away from the pixel in one direction, and its output value decreases. In addition, the width of the pixel having the output value is increased. The peak of the pixel value distribution indicated by the parallax Rt pixel appears at a position away from the pixel corresponding to the image point in the opposite direction to the one direction in the parallax Lt pixel and at an equal distance, and the output value similarly decreases. Similarly, the width of the pixel having the output value is increased. That is, the same pixel value distribution that is gentler than that in the case where the object point exists at the focal position appears at an equal distance from each other. Further, as shown in FIG. 4C, when the object point further deviates from the focal position, the same pixel value distribution that becomes more gentle as compared with the state of FIG. 4B appears further apart. . That is, it can be said that the amount of blur and the amount of parallax increase as the object point deviates from the focal position. In other words, the amount of blur and the amount of parallax change in conjunction with defocus. That is, the amount of blur and the amount of parallax have a one-to-one relationship.

  FIGS. 4B and 4C show the case where the object point shifts away from the focal position, but when the object point moves away from the focal position, as shown in FIG. Compared to FIGS. 4B and 4C, the relative positional relationship between the pixel value distribution indicated by the parallax Lt pixel and the pixel value distribution indicated by the parallax Rt pixel is reversed. Due to such a defocus relationship, when viewing a parallax image, the viewer visually recognizes a subject existing far behind the focal position and visually recognizes a subject present in front.

  When the change of the pixel value distribution described in FIGS. 4B and 4C is graphed, it is expressed as shown in FIG. In the figure, the horizontal axis represents the pixel position, and the center position is the pixel position corresponding to the image point. The vertical axis represents the output value (pixel value) of each pixel. As described above, this output value is substantially proportional to the light intensity.

  The distribution curve 1804 and the distribution curve 1805 represent the pixel value distribution of the parallax Lt pixel and the pixel value distribution of the parallax Rt pixel in FIG. 4B, respectively. As can be seen from the figure, these distributions have a line-symmetric shape with respect to the center position. Further, a combined distribution curve 1806 obtained by adding them shows a pixel value distribution of pixels without parallax with respect to the situation of FIG. 4B, that is, a pixel value distribution when the entire subject luminous flux is received, and a substantially similar shape.

  A distribution curve 1807 and a distribution curve 1808 represent the pixel value distribution of the parallax Lt pixel and the pixel value distribution of the parallax Rt pixel in FIG. As can be seen from the figure, these distributions are also symmetrical with respect to the center position. Also, a combined distribution curve 1809 obtained by adding them shows a shape substantially similar to the pixel value distribution of the non-parallax pixels for the situation of FIG.

  As can be seen from the above, if the pixel value of the parallax Lt pixel and the pixel value of the parallax Rt pixel of the pixel value distribution obtained by actually obtaining the output value of the image sensor 100 and interpolating the sky lattice are used, It is possible to create a simple pixel value distribution. At this time, the amount of parallax expressed as an interval between peaks can be adjusted while maintaining the amount of blur expressed by the spread of the pixel value distribution approximately. That is, it is possible to generate an image having a parallax amount adjusted between a 2D image generated from non-parallax pixels and a 3D image generated from parallax pixels while maintaining the blur amount of the 2D image almost as it is. it can.

  FIG. 6 is a diagram illustrating a pixel value distribution for explaining the concept of the adjusted parallax amount. Lt distribution curves 1901 and Rt distribution curves 1902 indicated by solid lines in the figure are distribution curves in which actual pixel values of Lt plane data and Rt plane data are plotted. For example, it corresponds to the distribution curves 1804 and 1805 in FIG. The distance between the peaks of the Lt distribution curve 1901 and the Rt distribution curve 1902 represents the 3D parallax amount, and the greater the distance, the stronger the stereoscopic effect during image reproduction.

  A 2D distribution curve 1903 obtained by adding 50% each of the Lt distribution curve 1901 and the Rt distribution curve 1902 has a convex shape with no left-right bias. The 2D distribution curve 1903 corresponds to a shape in which the height of the combined distribution curve 1806 in FIG. That is, an image based on this distribution is a 2D image with a parallax amount “0”.

  The adjusted Lt distribution curve 1905 is a curve obtained by adding 80% of the Lt distribution curve 1901 and 20% of the Rt distribution curve 1902. The peak of the adjusted Lt distribution curve 1905 is displaced closer to the center than the peak of the Lt distribution curve 1901 as much as the component of the Rt distribution curve 1902 is added. Similarly, the adjusted Rt distribution curve 1906 is a curve obtained by adding 20% of the Lt distribution curve 1901 and 80% of the Rt distribution curve 1902. The peak of the adjusted Rt distribution curve 1906 is displaced closer to the center than the peak of the Rt distribution curve 1902 by the amount to which the component of the Lt distribution curve 1901 is added.

  Therefore, the adjusted parallax amount represented by the distance between the peaks of the adjusted Lt distribution curve 1905 and the adjusted Rt distribution curve 1906 is smaller than the 3D parallax amount. Therefore, the stereoscopic effect during image reproduction is alleviated. On the other hand, since the spread of each of the adjusted Lt distribution curve 1905 and the adjusted Rt distribution curve 1906 is equivalent to the spread of the 2D distribution curve 1903, it can be said that the amount of blur is equal to that of the 2D image.

  That is, the amount of adjustment parallax can be controlled depending on the ratio of the Lt distribution curve 1901 and the Rt distribution curve 1902 to be added. Then, by applying this adjusted pixel value distribution to each plane of color image data generated from pixels without parallax, the color of the left viewpoint that gives a stereoscopic effect different from that of parallax image data generated from parallax pixels Image data and right-view color image data can be generated.

Specifically, left-viewpoint color image data and right-viewpoint color image data can be generated from the nine plane data described with reference to FIG. Color image data of the left viewpoint, RLt _c plane data is red plane data corresponding to the left viewpoint, a green plane data GLt _c plane data, and three color parallax plane data BLt _c plane data is blue plane data Consists of. Similarly, the color image data of the right-side perspective is, RRT _c plane data is red plane data corresponding to the right viewpoint, a green plane data GRT _c plane data, and three of BRt _c plane data is blue plane Datacolor Consists of parallax plane data.

FIG. 7 is a diagram illustrating color parallax plane data generation processing as a comparative example. In particular, a generation process of RLt _c plane data and RRt _c plane data, which are red parallax planes among color parallax planes, will be described.

The red parallax plane is generated using the pixel value of the Rn plane data described with reference to FIG. 3, and the pixel value of the RLt plane data and the RRt plane data. Specifically, for example, when calculating the pixel value RLt _mn of the target pixel position (i _m , j _n ) of the RLt _c plane data, first, the pixel value is calculated from the same pixel position (i _m , j _n ) of the Rn plane data. Rn _mn is extracted. Then, the same pixel position of RLt plane data _(i m, _{j n)} pixel values _{RLt mn} from extracts the pixel values _{RRT mn} from the same pixel position of the RRT plane data _(i m, _{j n).} Then, the pixel value _{Rn mn,} by multiplying the value obtained by distributing the pixel values _{RLt mn} and _{RRT mn} stereoscopic adjustment parameter C, and calculates the pixel value _{RLt cmn.} Specifically, it is calculated by the following equation (1). However, in the color parallax plane generation process of the comparative example, the stereoscopic adjustment parameter C is set to a single value in the range of 0.5 ≦ C ≦ 1.0 regardless of the pixels.
Similarly, when calculating the pixel value RRt _cmd of the target pixel position (i _m , j _n ) of the RRt _c plane data, the three-dimensional adjustment parameter is obtained by adding the pixel value RLt _mn and the pixel value RRt _mn to the extracted pixel value Rn _mn. Calculated by multiplying the value distributed in C. Specifically, it is calculated by the following equation (2).
Such processing is sequentially executed from (1, 1) which is the pixel at the left end and the upper end to (i ₀ , j ₀ ) which is the coordinates at the right end and the lower end.

When the generation processing of RLt _c plane data and RRt _c plane data that are red parallax planes is completed, generation processing of GLt _c plane data and GRt _c plane data that are green parallax planes is then executed. Specifically, the pixel same pixel position _(i m, _{j n)} of Rn plane data in the above description, instead of extracting the pixel values _{Rn mn} from the same pixel position of Gn plane data _(i m, _{j n)} from The value Gn _mn is extracted. Moreover, the same pixel position of RLt plane data _(i m, _{j n)} from instead of extracting the pixel value _{RLt mn,} extracts the pixel value _{GLt mn} from the same pixel position of GLt plane data _(i m, _{j n).} Similarly, the same pixel position of the RRT plane data _(i m, _{j n)} from instead of extracting the pixel value _{RRT mn,} extracts the pixel value _{GRT mn} from the same pixel position of GRT plane data _(i m, _{j n)} . Then, the parameters of Equation (1) and Equation (2) are appropriately changed and processed in the same manner.

Further, when the generation processing of the GLt _c plane data and the GRt _c plane data which are green parallax planes is completed, the generation processing of the BLt _c plane data and BRt _c plane data which are blue parallax planes is executed next. Specifically, the pixel same pixel position _(i m, _{j n)} of Rn plane data in the above description, instead of extracting the pixel values _{Rn mn} from the same pixel position of Bn plane data _(i m, _{j n)} from Extract the value Bn _mn . Moreover, the same pixel position of RLt plane data _(i m, _{j n)} from instead of extracting the pixel value _{RLt mn,} extracts the pixel value _{BLt mn} from the same pixel position of BLt plane data _(i m, _{j n).} Similarly, the same pixel position of the RRT plane data _(i m, _{j n)} from instead of extracting the pixel value _{RRT mn,} extracts the pixel value _{BRt mn} from the same pixel position of BRt plane data _(i m, _{j n)} . Then, the parameters of Equation (1) and Equation (2) are appropriately changed and processed in the same manner.

Through the above processing, left-view color image data (RLt _c- plane data, GLt _c- plane data, BLt _c- plane data) and right-view color image data (RRt _c- plane data, GRt _c- plane data, BRt _c- plane data) Is generated. That is, the color image data of the left viewpoint and the right viewpoint can be acquired by a relatively simple process as a virtual output that does not actually exist as a pixel of the image sensor 100.

  In addition, since the value of the stereoscopic adjustment parameter C can be changed within the range of 0.5 ≦ C ≦ 1.0, the amount of parallax as a 3D image can be increased while maintaining the amount of blur of the 2D color image due to pixels without parallax. Can be adjusted. Therefore, if these image data are reproduced by a 3D image compatible reproduction device, the viewer of the stereoscopic video display panel can appreciate the 3D video in which the stereoscopic effect is appropriately adjusted as a color image. In particular, since the processing is simple, it is possible to generate image data at high speed and to deal with moving images.

  Next, the above processing will be described from the viewpoint of pixel value distribution and color. FIG. 8 is a diagram for explaining a change in RGB pixel value distribution. FIG. 8A shows a G (Lt) pixel, a G (Rt) pixel, and an R (Lt) pixel when a white subject light beam from an object point located at a position deviated by a certain amount from the focal position is received. , R (Rt) pixels, B (Lt) pixels, and B (Rt) pixels.

  FIG. 8B shows the R (N) pixel, G (N) pixel, and B (N) pixel that are non-parallax pixels when a white subject light beam from the object point in FIG. 8A is received. It is the graph which arranged the output value. It can be said that this graph also represents the pixel value distribution of each color.

  When the above processing is performed for each corresponding pixel with C = 0.8, the pixel value distribution represented by the graph of FIG. 8C is obtained. As can be seen from the figure, a distribution corresponding to the pixel values of RGB is obtained.

  Next, adjustment of the parallax amount in the present embodiment will be described. In the adjustment of the parallax amount as the comparative example described above, the pixel value of the corresponding pixel in the parallax plane data is obtained by applying a single stereoscopic adjustment parameter C to the pixel value of each pixel that displays the image of the plane data. Calculated.

  On the other hand, in the adjustment of the amount of parallax in the present embodiment, by applying a different value of the three-dimensional adjustment parameter C for each pixel using an evaluation map for evaluating the plane data, the corresponding pixel in the parallax plane data is adjusted. Pixel value is calculated.

  The evaluation map generation unit 231 of the image processing unit 205 in this embodiment generates an evaluation map in which the evaluation values of the display pixels of the subject included in the image of the captured image data are two-dimensionally arranged corresponding to the pixel positions. To do. Here, the evaluation map generation unit 231 generates an evaluation map from the captured image data.

  Preferably, the evaluation map generation unit 231 generates a preliminary evaluation map from the Lt plane data and the Rt plane data in the captured image data, and generates an evaluation map by combining these preliminary evaluation maps. More preferably, the evaluation map generation unit 231 distributes preliminary evaluation value (notice level described later) distribution data obtained by quantifying the difference between a pixel displaying a subject and surrounding pixels in association with the accuracy of the viewer's gaze. In this case, a so-called saliency map is used as the preliminary evaluation map.

  In this way, when the evaluation map is generated by combining the preliminary evaluation maps as the saliency maps, the evaluation value of the display pixel of the subject in the evaluation map is the pixel-by-pixel preliminary evaluation value of each pixel in the preliminary evaluation map. It becomes the value synthesized. For this reason, the evaluation value of the display pixel of the subject in the evaluation map in this case is a numerical value that relates the difference between the pixel displaying the subject and the surrounding pixels in the same manner as the preliminary evaluation value of this pixel, in association with the accuracy with which the viewer gazes. It can be said that it has become a value.

  Here, the accuracy is a degree of certainty and certainty. In addition, the accuracy of the viewer's gaze is a numerical value representing the probability of the viewer's gaze.

  Further, in this embodiment, synthesizing the preliminary evaluation maps means taking a geometric average of the preliminary evaluation values for each display pixel of the subject in the preliminary evaluation map of the Lt plane data and the Rt plane data.

  When the evaluation map is generated by combining the preliminary evaluation values for the display pixels of the subject in the Lt and Rt plane data for each pixel, a sense of incongruity and a feeling of fatigue during viewing can be suppressed. In order to explain this effect, let us consider a case in which parallax image data is generated such that the near-side subject and the far-side subject are adjacent vertically or horizontally on the image in the defocus area of the digital camera 10.

  In this case, the preliminary evaluation value for each pixel in the preliminary evaluation map is unlikely to cause a visual field conflict because the preliminary evaluation value of a pixel that has a different degree of saliency for each viewpoint and easily causes a visual field conflict and the saliency for each viewpoint are common. Pixel preliminary evaluation values are mixed. Then, when such preliminary evaluation values are synthesized for each pixel, the evaluation values of pixels that have different saliency for each viewpoint and are likely to cause a visual field conflict, and pixels that are less likely to cause a visual field conflict due to the common saliency for each viewpoint. The difference from the evaluation value becomes clear.

  This makes it easy to make the value of the stereoscopic adjustment parameter C for a pixel that is likely to cause a visual field conflict different from the value of the stereoscopic adjustment parameter C for a pixel that is less likely to cause a visual field conflict. Therefore, by making the amount of parallax given to a pixel that is likely to cause a visual field conflict to be smaller than the amount of parallax given to a pixel that is less likely to cause a visual field conflict, a sense of incongruity or fatigue during viewing can be suppressed.

  Note that the saliency map is image data obtained by imitating the function of the human visual receptive field and extracting the part that calls for visual attention. In other words, the saliency map includes the pixels that display the subject and the peripheral pixels. This is distribution data of the saliency that is obtained by quantifying the difference in association with the accuracy with which the viewer gazes. The saliency map is generated by calculating the saliency value of each pixel according to the degree of difference from the surroundings using three indicators of color, luminance, and edge direction.

  Such a saliency map is, for example, the paper “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis” by Itti et al. (Publication name: The IEEE Transactions on Pattern Analysis and Machine Intelligence, publication country: USA, publication place: IEEE Computer Society, date of issue: Nov 1998, volume: Volume 20, pages: pp. 1254-1259). In this method, first, a color image is decomposed into three features of color, luminance, and edge direction, each Gaussian pyramid is generated, and a feature map is generated by performing center-surround calculation using these three Gaussian pyramids. Then, the saliency map is generated by normalizing and integrating all the feature maps.

  FIG. 9 is a diagram showing an example of the saliency map generated by this method. Here, the right image in the figure is an image showing a saliency map, and the left image is an original image used to generate the saliency map, that is, an input image.

  This input image is a color abstract picture illustrating a plurality of dots 300, the background color is white, and the color of the dots 300 is red. More specifically, some of the dots 300 are faded, and a green line 301 is drawn on the second dot 300 from the lower right side so as to cross the center.

  Further, over the entire surface of the image, the spots are shaded by small green dots. When a human appreciates such an input image, the second dot 300 from the lower right is most likely to be noticed due to human sensitivity.

  On the other hand, the image of the saliency map is, for example, a grayscale image of 256 gradations, and indicates that the saliency is higher as the pixel is lighter and the saliency is lower as the pixel is darker. In this figure, it can be seen that the second dot 300 from the right side has the highest saliency, and the saliency is evaluated according to human sensitivity.

  The saliency of a subject in such a saliency map tends to be low when the subject is blurred. Therefore, when the saliency map of the captured image is generated, the saliency is likely to be high because the contour is clear in the subject at the focus position. In addition, since the outline is blurred in a subject away from the focus position, the saliency is likely to be low.

  As a method for generating the saliency map, another conventionally known method may be used. Examples of such methods include the paper “Graph Based Visual Saliency” by Harel et al. (Publication name: Advances in Neural Information Processing Systems (NIPS), Publication year: 2007, pages: pp. 545-552), Jiang et al. Can be used as described in the paper "Automatic salient object segmentation based on context and shape prior" (publication name: British Machine Vision Conference, publication year: 2011, pages: pp. 1-12).

  The adjustment parameter value determination unit 232 of the image processing unit 205 in the present embodiment determines the value of the stereoscopic adjustment parameter C for each pixel in association with the evaluation value of the evaluation map. In other words, the adjustment parameter value determination unit 232 sets the value of the stereoscopic adjustment parameter C for the corresponding pixels of the Lt plane data and the Rt plane data based on the data for each pixel of the evaluation map, that is, the evaluation value for each pixel. decide.

  More specifically, the adjustment parameter value determination unit 232 stores a parameter value calculation function for calculating the value of the three-dimensional adjustment parameter C from each value that can be taken by the evaluation value for each pixel in the evaluation map. Then, the adjustment parameter value determination unit 232 calculates the value of the three-dimensional adjustment parameter C for each pixel using this parameter value calculation function, and determines the corresponding pixel.

  In this parameter value calculation function, the larger the evaluation value, the closer the stereoscopic adjustment parameter C value to 1 so that the amount of parallax does not change, and the smaller the evaluation value, the closer the stereoscopic adjustment parameter C value to 0.5 and the parallax. It is preferable to reduce the amount. In addition, it is preferable that the amount of change in the three-dimensional adjustment parameter C when the evaluation value is large is larger than the amount of change in the three-dimensional adjustment parameter C when the evaluation value is small. Specifically, it is preferable that the increment of the value of the stereo adjustment parameter C with respect to the unit increment of the evaluation value when the evaluation value is small is larger than the increment of the value of the stereo adjustment parameter C with respect to the unit increment when the evaluation value is large. .

  Further, as the parameter value calculation function, a non-linear function is preferable, and for example, a function representing gamma correction such that the gamma value γ satisfies 0 <γ <1 can be used. FIG. 10 is a diagram illustrating a specific example of the parameter value calculation function.

  Here, the horizontal axis (x-axis) in the figure represents an evaluation value (here, a geometric mean of saliency expressed in 256 levels) as an input value, and 0 ≦ x ≦ 255. The vertical axis (y-axis) indicates the value of the three-dimensional adjustment parameter C as an output value, and 0.50 ≦ y ≦ 1.00.

Among the functions shown in this figure, as a function of gamma correction such that the gamma value γ satisfies 0 <γ <1, “y = x ^{1 / 4.4} ” or “y = x ^{1 / 2.2} ” Can be used. The parameter value calculation function is not limited to the function shown in FIG. 10, and a parameter value calculation function that is normalized so that the range of the value of the stereo adjustment parameter C is 0.75 to 1.00 may be used.

The parallax image data generation unit 233 of the image processing unit 205 in the present embodiment generates parallax image data from plane data. The parallax image data includes Lt _c parallax plane data and Rt _c parallax plane data having parallax with each other.

More specifically, the parallax image data generation unit 233 applies the value of the corresponding stereoscopic adjustment parameter C to each pixel of the Lt plane data and the Rt plane data using, for example, the above-described equation (1). . Thus, Lt _c parallax plane data and Rt _c parallax plane data having a parallax amount different from the parallax amount between images of the Lt plane data and the Rt plane data are generated.

  Next, the relationship between the viewer and the video when the 3D image data as described above is played back by a playback device will be described. FIG. 11 is a diagram illustrating the relationship between the vergence angle of the viewer and the amount of parallax. The eyeball 50 represents the eyeball of the viewer, and the figure shows the right eye 51 and the left eye 52 being separated.

The display unit 40 reproduces non-adjusted image data whose parallax amount is not adjusted, and displays a subject 61 for the right-eye image and a subject 62 for the left-eye image. Object 61 and the object 62 are the same object, so were present at a position shifted from the focal position at the time of shooting, the display unit 40 is displayed at a distance with a disparity amount D _1.

  Since the eyeball 50 is intended to be viewed with the eyeballs coincident with each other, the viewer views the position of the lifting distance L1 (in the drawing) where the straight line connecting the right eye 51 and the subject 61 and the straight line connecting the left eye 52 and the subject 62 intersect. (Represented by a square).

The convergence angle at this time is θ ₁ as shown in the figure. In general, when the angle of convergence is increased, the video is uncomfortable and causes eye strain. Therefore, in the present embodiment, as described above, adjusted image data in which the parallax amount is adjusted with the stereoscopic adjustment parameter is generated so that the smaller the evaluation value in the evaluation map, the smaller the parallax amount is. The figure shows a state in which the adjusted image data is reproduced over the non-adjusted image data.

The display unit 40 displays a subject 71 of the right-eye image and a subject 72 of the left-eye image of the adjustment image data. The subject 71 and the subject 72 are the same subject, and the subjects 61 and 62 are also the same subject. These subjects are displayed with pixels having small evaluation values in the evaluation map. Object 71 and the object 72, the display unit 40 is displayed at a distance with a disparity amount D _2. The viewer recognizes that the subject exists at the position of the lifting distance L2 (represented by a triangle in the figure) where the straight line connecting the right eye 51 and the subject 71 intersects with the straight line connecting the left eye 52 and the subject 72.

The convergence angle at this time is θ ₂ smaller than θ ₁ . Therefore, the viewer can feel the extreme feeling of lifting and can reduce the accumulation of eye strain. Note that the amount of parallax is appropriately adjusted as will be described later, so that the viewer can appreciate the video with a comfortable floating feeling (a three-dimensional effect with a feeling of depression when the defocus relationship is reversed).

  Note that the amount of parallax used as the description of FIG. 11 is represented by the separation distance in the display unit 40, but the amount of parallax can be defined in various forms. You may define by the pixel unit in picked-up image data, and you may define by the shift | offset | difference width with respect to the horizontal width of an image.

  According to the adjustment of the parallax amount in the present embodiment, the value of the stereoscopic adjustment parameter C is maintained such that the parallax amount is maintained as a result of the evaluation value becoming large (the degree of saliency increases) in the pixel displaying the subject at the focus position. Is determined. Then, as the subject moves away from the focus position, the value of the three-dimensional adjustment parameter C is determined so that the evaluation value becomes smaller (the saliency becomes lower) in the pixel displaying the subject, and as a result, the parallax amount becomes smaller.

  However, even if the value of the three-dimensional adjustment parameter C is determined in this way, if the parallax amount of the subject away from the focus position still remains large, the viewer will feel uncomfortable and tired. Therefore, in this embodiment, the image processing unit 205 adjusts the parallax amount so that the amount of parallax falls between the set lower limit value and the upper limit value in order to surely prevent the viewer from feeling uncomfortable and tired. Data may be generated.

  First, the limitation on the amount of parallax will be described. FIG. 12 is a rear view of the digital camera 10 that displays a menu screen for limiting the amount of parallax.

  The limit of the parallax amount is set, for example, as a lower limit value −m and an upper limit value + m. The absolute value of the lower limit value and the upper limit value may be different. Here, the parallax amount is expressed in units of pixels of the parallax image in the adjusted parallax image data.

  The amount of parallax that the viewer feels uncomfortable and tired varies from viewer to viewer. Therefore, it is preferable to configure the digital camera 10 so that a photographer who is a user of the digital camera 10 can change the setting of the parallax amount restriction at the time of shooting.

  The digital camera 10 has, for example, four options as shown in the figure as a menu for limiting the amount of parallax. Specifically, “Standard” presets the range that a standard viewer feels comfortable with, presets a range wider than the standard, “strong” allows a larger amount of parallax, and presets a range narrower than the standard In addition, “weak” that allows only a smaller amount of parallax, and “manual” in which the photographer inputs an upper limit value and a lower limit value are provided. When “Manual” is selected, the photographer can sequentially specify “maximum lifting amount” as an upper limit value and “maximum sinking amount” as a lower limit value in units of pixels. The photographer selects and designates these by operating a dial button 2081 that is a part of the operation unit 208.

  When the allowable parallax amount range is set as described above, the image processing unit 205 generates adjusted parallax image data in which the parallax amount is adjusted to fall within this range. At this time, the adjustment parameter value determination unit 232 determines the value of the three-dimensional adjustment parameter C so that the parallax amount of each subject is within the set range.

  Specifically, the adjustment parameter value determination unit 232 determines the parameter value calculation function corresponding to the set parallax amount range from the parameter value calculation functions stored in advance in association with the range of parallax amounts that can be set. Is detected. In these parameter calculation functions, the maximum value of the three-dimensional adjustment parameter C as an output value is different. Then, by using such a parameter value calculation function, the value of the stereoscopic adjustment parameter C is determined so that the parallax amount of the subject is within the range.

  For example, as a parameter calculation function corresponding to “standard” in the parallax amount restriction menu, a function having a maximum value of the three-dimensional adjustment parameter C of 1 can be used. In addition, as a parameter calculation function corresponding to “higher” in the parallax amount restriction menu, a function in which the maximum value of the stereoscopic adjustment parameter C is larger than 1 can be used.

  As the parameter calculation function corresponding to “weak” in the parallax amount restriction menu, a function in which the maximum value of the stereoscopic adjustment parameter C is smaller than 1 can be used. Further, in the parameter calculation function corresponding to “Manual” in the parallax amount restriction menu, the maximum value of the three-dimensional adjustment parameter C is arbitrarily changed.

  The parallax amount adjustment processing according to the present embodiment does not require complicated processing such as cutting out for each subject object and moving the object in the horizontal direction using the depth map information as in the prior art. Accordingly, since the calculation can be performed at a higher speed than in the prior art, the image processing unit 205 can easily cope with the output of the adjusted parallax image data in real time even for moving image shooting in which the state of the subject changes every moment.

  The digital camera 10 according to the present embodiment includes an auto 3D moving image mode in which a parallax image data adjusted to a comfortable parallax amount is continuously generated and a moving image file is generated by connecting the parallax image data as one of the moving image shooting modes. . Prior to shooting, the photographer operates the mode button 2082 that is part of the operation unit 208 to select the auto 3D moving image mode.

  Next, a series of processing flows of the digital camera 10 will be described. FIG. 13 is a processing flow in moving image shooting.

  The flow in the figure starts when the mode button 2082 is operated by the photographer and the automatic 3D moving image mode is started. When the auto 3D moving image mode set by the photographer in advance is started for the parallax amount range, the control unit 201 acquires the parallax amount range set by the photographer from the system memory in step S11.

  In step S12, the control unit 201 waits for a recording start instruction for the photographer to press the recording start button. When a recording start instruction is detected (YES in step S12), the control unit 201 proceeds to step S13 and executes AF and AE. In step S <b> 14, the control unit 201 performs charge accumulation and readout of the image sensor 100 via the drive unit 204, and acquires captured image data as one frame. During this period, the control unit 201 may continue driving the focus lens and the diaphragm 22 in accordance with the detection result in step S13.

In step S17, the image processing unit 205 determines the value of the stereoscopic adjustment parameter C for each corresponding pixel of the captured image data based on the data for each pixel in the evaluation map, and the color image data (RLt _c plane data, GLt for the left viewpoint). _c plane data, BLt _c plane data) and right-view color image data (RRt _c plane data, GRt _c plane data, BRt _c plane data). Specific processing will be described later.

  If the control unit 201 determines in step S18 that a recording stop instruction has not been received from the photographer, the control unit 201 returns to step S13 to execute the next frame processing. If it is determined that a recording stop instruction has been received, the process proceeds to step S19.

  In step S19, the moving image generation unit 234 connects the color image data of the left viewpoint and the color image data of the right viewpoint generated continuously, executes format processing according to a 3D-compatible moving image format such as Blu-ray 3D, and the moving image Generate a file. Then, the control unit 201 records the generated moving image file on the memory card 220 via the memory card IF 207, and ends a series of flows. Note that the recording to the memory card 220 may be sequentially executed in synchronization with the generation of the left-viewpoint color image data and the right-viewpoint color image data, and the file end process may be executed in synchronization with the recording stop instruction. Further, the control unit 201 is not limited to recording on the memory card 220, but may be configured to output to an external device via a LAN.

  Next, the process of step S17 in FIG. 13 will be described in detail. FIG. 14 is a processing flow of step S17 until the color image data of the left viewpoint and the parallax color image data that is the color image data of the right viewpoint are generated.

  In step S101, the parallax image data generation unit 233 acquires captured image data. In step S102, as described with reference to FIG. 3, the captured image data is plane-separated into image data without parallax and parallax image data. In step S103, the parallax image data generation unit 233 executes an interpolation process for interpolating vacancies existing in the separated plane data as described with reference to FIG. Then, based on the Lt plane data and the Rt plane data, temporary color parallax image data of the left viewpoint and the right viewpoint is generated.

  The evaluation map generation unit 231 generates an evaluation map in step S104. For example, the evaluation map generation unit 231 generates a saliency map (provisional evaluation map) of temporary color parallax image data generated from Lt plane data and a saliency map of temporary color parallax image data generated from Rt plane data. Generate each. Then, the evaluation values of the display pixels of the subject are two-dimensionally arranged by combining the saliency (preliminary evaluation value) for each pixel in the Lt plane data and the saliency for each pixel in the Rt plane data for each pixel. Generated evaluation map.

  In step S105, the parallax image data generation unit 233 initializes each variable. Specifically, first, 1 is substituted into the color variable Cset. The color variable Cset represents 1 = red, 2 = green, 3 = blue. Also, 1 is substituted into the coordinate variables i and j. Further, 1 is substituted into the parallax variable S. The parallax variable S represents 1 = left, 2 = right.

  In step S106, the parallax image data generation unit 233 extracts the evaluation value of the pixel position (i, j) in the evaluation map. In step S107, the adjustment parameter value determination unit 232 determines the value of the three-dimensional adjustment parameter C in association with the extracted evaluation value.

  At this time, the parallax image data generation unit 233 uses the parameter value calculation function corresponding to the parallax amount range set by the photographer to determine the stereoscopic adjustment parameter so that the parallax amount of the subject falls within the range. Note that the parallax image data generation unit 233 further stores a plurality of parameter value calculation functions according to the type of pixel, in other words, the type of plane data, and sets a parameter value calculation function corresponding to the type of pixel to be processed. It is good also as determining the value of a three-dimensional adjustment parameter using.

In step S108, the parallax image data generation unit 233 extracts a pixel value from the target pixel position (i, j) of the Cset plane. For example, Cset = 1, when a target pixel position (1,1), pixel values to be extracted is _{Rn 11.} Further, in step S109, the parallax image data generation unit 233 extracts a pixel value from the target pixel position (i, j) of the Lt _Cset plane data and the Rt _Cset plane data. For example, when the target pixel position is (1, 1), the pixel values to be extracted are Lt _Cset11 and Rt _Cset11 .

In step S110, the parallax image data generation unit 233 calculates a pixel value of the target pixel position (i, j) corresponding to the parallax variable S. For example, when Cset = 1 and S = 1 and the target pixel position is (1, 1), RRLt _C11 is calculated. Specifically, for example, it is calculated by the above-described equation (1). Here, the value of the three-dimensional adjustment parameter C is the value set in step S107.

  The parallax image data generation unit 233 increments the parallax variable S in step S111. In step S112, it is determined whether or not the parallax variable S exceeds 2. If not, the process returns to step S110. If it exceeds, the process proceeds to step S113.

In step S113, the parallax image data generation unit 233 assigns 1 to the parallax variable S and increments the coordinate variable i. Then, in step S114, it is determined whether coordinate variable i exceeds _{i 0.} If not, the process returns to step S106. If it exceeds, the process proceeds to step S115.

In step S115, the parallax image data generation unit 233 assigns 1 to the coordinate variable i and increments the coordinate variable j. Then, in step S116, it is determined whether coordinate variable j exceeds _{j 0.} If not, the process returns to step S106. If it exceeds, the process proceeds to step S117.

When the process proceeds to step S117, since the pixel values of all the left and right pixels for Cset are aligned, the parallax image data generation unit 233 arranges these pixel values to generate the parallax-adjusted plane data. For example, when Cset = 1, RLt _c plane data and RRt _c plane data are generated.

In step S118, the parallax image data generation unit 233 assigns 1 to the coordinate variable j and increments the color variable Cset. In step S119, it is determined whether or not the color variable Cset exceeds 3. If not, the process returns to step S106. If so, the parallax image data generation unit 233 collects the left viewpoint plane data (RLt _c plane data, GLt _c plane data, and BLt _c plane data) into the left viewpoint color image data in step S120, and Then, the right viewpoint plane data (RRt _c plane data, GRt _c plane data, and BRt _c plane data) are collected as right viewpoint color image data, and the process returns to the flow of FIG.

  Next, a preferable opening shape of the opening mask described with reference to FIG. 2 will be described. FIG. 15 is a diagram illustrating a preferred opening shape.

  It is preferable that the opening part 105 of the parallax Lt pixel and the opening part 106 of the parallax Rt pixel are deviated in directions opposite to each other including the center with respect to the corresponding pixel. Specifically, it is preferable that each of the openings 105 and 106 has a shape in contact with a virtual center line 322 passing through the center of the pixel or a shape straddling the center line 322.

  In particular, as illustrated, the shape of the opening 105 and the shape of the opening 106 are preferably the same as the respective shapes obtained by dividing the shape of the opening 104 of the non-parallax pixel by the center line 322. In other words, the shape of the opening 104 is preferably equal to the shape in which the shape of the opening 105 and the shape of the opening 106 are adjacent to each other.

  According to the above embodiment, an evaluation map in which the evaluation values of the display pixels of the subject are two-dimensionally arranged is generated, and the value of the three-dimensional adjustment parameter C for adjusting the parallax amount is made to correspond to the evaluation value of the evaluation map. Determination is performed to generate parallax image data. Accordingly, it is possible to generate parallax image data by making the value of the stereoscopic adjustment parameter C different between a pixel having a large evaluation value and a pixel having a small evaluation value.

  Thereby, it is possible to prevent the parallax amount from changing in any one of the pixel having a large evaluation value and the pixel having a small evaluation value, and to reduce the parallax amount in the other pixel. As a result, it is possible to reduce the amount of parallax in the display pixels of the subject not noticed by the viewer while maintaining the stereoscopic effect without changing the amount of parallax in the display pixels of the subject focused by the viewer. Therefore, unlike the conventional case where the adjustment amount of the parallax amount is uniformly determined, a sharp three-dimensional image can be generated.

  In addition, since it is possible to generate parallax image data in which the parallax amount is adjusted without using data related to the subject distribution in the depth direction of the scene, there is no need to generate data related to the subject distribution in the depth direction when generating captured image data. It can be omitted. In addition, the process for generating the parallax image data can be simplified by the amount of time and effort required for analyzing the data regarding the subject distribution in the depth direction when generating the parallax image data.

  In addition, the difference between the subject area and the surrounding area is used as a preliminary evaluation value or evaluation value of the pixel in association with the accuracy with which the viewer gazes. Can be adjusted. Therefore, it is possible to reliably reduce a sense of incongruity when the viewer views a stereoscopic image based on the parallax image data.

  Also, the small (or high) probability that the viewer gazes at a certain pixel generally means that the subject displayed on the pixel is far from the focus area (or close to the focus area) at the time of image capturing. Therefore, by adjusting the value of the three-dimensional adjustment parameter C according to the accuracy, the amount of parallax given to the subject that is likely to cause discomfort far from the focus area is changed to the amount of parallax given to the subject that is close to the focus area and hardly causes discomfort. Can be made smaller. Accordingly, it is possible to further reduce the uncomfortable feeling when the viewer views a stereoscopic image based on the parallax image data.

In the above description, the calculation formula used by the parallax image data generation unit 233 employs the above formulas (1) and (2) using the weighted arithmetic mean. can do. For example, if the weighted geometric mean is used, it can be expressed in the same manner as the above formulas (1) and (2),
Can be adopted as a calculation formula. In this case, the amount of blur maintained is not the amount of blur due to the output of the non-parallax pixel but the amount of blur due to the output of the parallax pixel.

Moreover, as another calculation formula, it represents like the said Formula (1) (2),
May be adopted. In this case, when calculating GLt _cmn , GRt _cmn , BLt _cmn , and BRt _cmn , the cubic root term does not change.

Moreover,
May be adopted. Also in this case, when calculating GLt _cmn , GRt _cmn , BLt _cmn , and BRt _cmn , the cubic root term does not change.

  Next, cooperation with the display device will be described. FIG. 16 is a diagram for explaining the cooperation between the digital camera 10 and the TV monitor 80. The TV monitor 80 includes, for example, a display unit 40 made of liquid crystal, a memory card IF 81 that receives the memory card 220 taken out from the digital camera 10 and acquires photographed image data from the memory card 220, and a viewer operates at hand. The remote control 82 is used. The TV monitor 80 is compatible with 3D image display. The display format of the 3D image is not particularly limited. The right-eye image and the left-eye image may be displayed in a time-sharing manner, or may be an interlace in which strips are arranged in a horizontal or vertical direction. Further, it may be a side-by-side format arranged on one side and the other side of the screen.

  The TV monitor 80 decodes the moving image file that is formatted including the color image data of the left viewpoint and the color image data of the right viewpoint, and displays a 3D image on the display unit 40. In this case, the TV monitor 80 serves as a general display device that displays a standardized moving image file. However, the TV monitor 80 can also function as an image processing apparatus that bears at least part of the function of the control unit 201 and at least part of the function of the image processing unit 205 described with reference to FIG. Specifically, an image processing unit including the evaluation map generation unit 231, the adjustment parameter value determination unit 232, the parallax image data generation unit 233, and the moving image generation unit 234 described in FIG. 1 is incorporated in the TV monitor 80. With this configuration, it is possible to realize function sharing different from the function sharing by the combination of the digital camera 10 and the TV monitor 80 in the above-described embodiment. The modification is demonstrated below.

  In the modified example, the adjustment image data generation process in which the parallax amount is adjusted by the stereoscopic adjustment parameter is performed not on the digital camera 10 side but on the TV monitor 80 side. Therefore, the digital camera 10 does not have to include the evaluation map generation unit 231, the adjustment parameter value determination unit 232, and the parallax image data generation unit 233 with respect to the configuration of FIG. 1.

  The processing operation of the digital camera 10 in the modification will be specifically described. FIG. 17 is a processing flow in moving image shooting of the digital camera 10 as a modification. The processes related to the respective processes in the process flow of FIG. 13 are denoted by the same step numbers, and the description thereof is omitted except for the description of different processes and additional processes.

  In step S20, continuously generated photographic image data are connected to create a moving image file. Note that the moving image file may be data at any stage described with reference to FIG. 3 as long as it includes reference image data and left-right viewpoint parallax image data as captured image data of successive frames. That is, separation processing, interpolation processing, and plane data processing may be performed in step S14 as processing of the digital camera 10, or part or all may be performed in the TV monitor 80 as an image processing apparatus. And the control part 201 outputs the produced | generated moving image file to the memory card 220, and complete | finishes a series of flows.

  Next, the processing operation of the TV monitor 80 in the modification will be described. FIG. 18 is a processing flow in the moving image reproduction of the TV monitor 80 as a modified example. The processes related to the respective processes in the process flow of FIG. 13 are denoted by the same step numbers, and the description thereof is omitted except for the description of different processes and additional processes. However, the TV monitor 80 includes an image processing unit including an evaluation map generation unit 231, an adjustment parameter value determination unit 232, a parallax image data generation unit 233, and a moving image generation unit 234. The image processing unit corresponds to the image processing unit 205 described with reference to FIG.

  When the control unit detects an instruction to reproduce a 3D image, the control unit generates color image data of the left viewpoint and the right viewpoint adjusted in parallax in step S17 illustrated in FIG. However, in this process, the control unit acquires captured image data by decoding a moving image file in the memory card IF81.

  Proceeding to step S <b> 33, the control unit displays a 3D image based on color image data of the left viewpoint and the right viewpoint adjusted in parallax amount on the display unit 40. Then, in step S34, the control unit determines whether there is an instruction to stop playback from the viewer or whether the image data to be played back has ended. If none of the data is applicable, the control unit returns to step S31, The playback process for the next frame is started. On the other hand, in the case of any of the above, a series of reproduction processing is ended.

  In the above modification, it may be configured such that the viewer can input the parallax amount range by operating the remote controller 82 during reproduction on the TV monitor 80. The adjustment parameter value determination unit 232 of the TV monitor 80 acquires the input parallax amount range, and determines the value of the stereoscopic adjustment parameter C according to the parallax amount range. If comprised in this way, the TV monitor 80 can display the 3D image according to the preference for every viewer.

  In the above embodiment, the description has been made on the premise of moving image shooting. However, the configuration for outputting the parallax image data in which the parallax amount is adjusted can be applied to still image shooting as well. The still image shot in this way does not cause extreme parallax between the left and right images, and does not give the viewer a sense of incongruity.

  In the above embodiment, the TV monitor 80 has been described as an example of an image processing apparatus. However, the image processing apparatus can take various forms. For example, a device such as a PC, a mobile phone, or a game device that includes or is connected to the display unit can be an image processing apparatus.

  In the above embodiments, the image sensor 100 has been described as having a parallax-free pixel and a parallax pixel. However, the image sensor 100 may have only parallax pixels. In this case, for example, instead of the pixel value of the Rn plane data in the above equation (1), an average of the pixel value of the RLt plane data and the pixel value of the RRt plane data is used.

In the above embodiment it has been described as using the same captured image data to generate an evaluation map and the disparity image data (Lt _c disparity plane data and Rt _c disparity plane data). However, as long as image data for the same scene is used, the same image data (CG image or the like) that is not a captured image may be used, or different image data may be used.

  As a specific example using different image data, there is a case where captured image data of frames before and after continuously captured in moving image capturing is used for generating parallax image data and an evaluation map. In some cases, image data captured by a digital camera is used to generate parallax image data, and preliminary image data having a smaller number of pixels than the captured image data acquired in conjunction with the captured image data is used to generate an evaluation map. Can be mentioned.

  As such preliminary image data, image data of a live view image displayed on the display unit 209 when the captured image data is acquired by the digital camera can be used. Even when image data having a different number of pixels is used, if the correspondence between the pixels is taken into consideration, the corresponding pixel of the other image data is based on the data for each pixel of the evaluation map generated from the one image data. The value of the stereoscopic adjustment parameter C can be determined respectively to generate parallax image data.

  In the above embodiment, the evaluation value of the display pixel of the subject is used as the evaluation value arranged in the evaluation map. However, the evaluation value of the display area of the subject may be used. In this case, the adjustment parameter value determination unit 232 determines the value of the three-dimensional adjustment parameter C for each display area of the subject.

  In the above embodiment, the preliminary evaluation maps are synthesized by taking the geometric mean of the preliminary evaluation values for each display pixel of the subject in the preliminary evaluation map. However, the preliminary evaluation maps may be synthesized by simply multiplying the preliminary evaluation values for each display pixel of the subject.

  Further, the preliminary evaluation maps may be synthesized by simply adding the preliminary evaluation values for each display pixel of the subject or by taking an arithmetic mean. Alternatively, the preliminary evaluation maps may be synthesized by performing an AND operation so that only the preliminary evaluation value common between the Lt plane data and the Rt plane data is left for each display pixel of the subject.

  In the above embodiment, the preliminary evaluation map (preferably the saliency map) of the Lt plane data and the preliminary evaluation map of the Rt plane data are synthesized to generate the evaluation map. However, a saliency map for an image of 2D image data may be generated as an evaluation map. Alternatively, the evaluation map may be generated by combining the preliminary evaluation map of Lt plane data (preferably the saliency map), the preliminary evaluation map of Rt plane data, and the preliminary evaluation map of 2D image data. When combining three preliminary evaluation maps, weighting may be performed for each type of preliminary evaluation map.

  In the above embodiment, the adjustment parameter value determination unit 232 stores the parameter value calculation function, and the parameter value calculation function is used to determine the value of the three-dimensional adjustment parameter C. However, the adjustment parameter value determination unit 232 stores a look-up table in which values that can be taken by the evaluation values for each pixel in the evaluation map and the value of the solid adjustment parameter C are associated with each other, and the three-dimensional adjustment is performed using this lookup table. The value of the adjustment parameter C may be determined.

  Each processing flow described in this embodiment is executed by a control program that controls the control unit. The control program is recorded in a built-in nonvolatile memory, and is appropriately expanded in the work memory to execute each process. Alternatively, the control program recorded in the server is transmitted to each device via the network, and is expanded in the work memory to execute each process. Alternatively, a control program recorded on the server is executed on the server, and each device executes processing in accordance with a control signal transmitted via the network.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 10 Digital camera, 20 Shooting lens, 21 Optical axis, 22 Aperture, 50 Eyeball, 51 Right eye, 52 Left eye, 40 Display part, 61, 62, 71, 72 Subject, 80 TV monitor, 81 Memory card IF, 82 Remote control, 100 Image sensor, 104, 105, 106 aperture, 110 basic grid, 201 control unit, 202 A / D conversion circuit, 203 memory, 204 drive unit, 205 image processing unit, 207 memory card IF, 208 operation unit, 209 display unit , 210 LCD drive circuit, 231 evaluation map generation unit, 232 adjustment parameter value determination unit, 233 parallax image data generation unit, 234 moving image generation unit, 322 center line, 1804, 1805, 1807, 1808 distribution curve, 1806, 1809 composite distribution Curve, 1901 Lt distribution curve, 1902 t distribution curve, 1903 2D distribution curve, 1905 adjustment Lt distribution curve, 1906 adjustment Rt distribution curve, 2081 dial button, 2082 mode button

Claims (10)

An acquisition unit for acquiring image data;
An evaluation map generating unit that generates an evaluation map in which the evaluation value of the subject area included in the image of the image data is two-dimensionally arranged in correspondence with the position of the subject area;
A determination unit configured to determine an adjustment parameter value for adjusting a parallax amount, which is applied when generating parallax image data from the image data, in correspondence with the evaluation value of the evaluation map;
A generation unit that applies the adjustment parameter value to the image data and generates parallax image data that becomes images having parallax amounts adjusted to each other ;
The image data includes first parallax image data and second parallax image data having parallax with each other,
The image data further includes reference image data different from the first parallax image data and the second parallax image data,
The evaluation map generation unit is an image processing device that generates the evaluation map from the reference image data . An acquisition unit for acquiring image data;
An evaluation map generating unit that generates an evaluation map in which the evaluation value of the subject area included in the image of the image data is two-dimensionally arranged in correspondence with the position of the subject area;
A determination unit configured to determine an adjustment parameter value for adjusting a parallax amount, which is applied when generating parallax image data from the image data, in correspondence with the evaluation value of the evaluation map;
A generation unit configured to apply the adjustment parameter value to the image data and generate parallax image data to be images having parallax amounts adjusted to each other;
With
The image data includes first parallax image data and second parallax image data having parallax with each other,
The evaluation map generation unit generates a preliminary evaluation map from each of the first parallax image data and the second parallax image data, and synthesizes the preliminary evaluation map to generate the evaluation map. Before Symbol generation section, wherein the first parallax image data by applying the adjustment parameter value in the second parallax image data, the disparity between the respective images of the second parallax image data and the first parallax image data with each other the amount image processing apparatus according to claim 1 or 2 to generate a third parallax image data and the fourth parallax image data so that images with different parallax amount from. The evaluation map generation unit according to any one of the three values a difference a viewer is quantified in association with accuracy to look between the subject region and the peripheral region of claim 1, wherein the evaluation value Image processing device. The determination unit is configured to increase an increment of the adjustment parameter value with respect to a unit increment of the evaluation value when the evaluation value is small larger than an increment of the adjustment parameter value with respect to a unit increment of the evaluation value when the evaluation value is large. The image processing apparatus according to claim 4 . An acquisition unit for acquiring image data;
An evaluation map generating unit that generates an evaluation map in which the evaluation value of the subject area included in the image of the image data is two-dimensionally arranged in correspondence with the position of the subject area;
A determination unit configured to determine an adjustment parameter value for adjusting a parallax amount, which is applied when generating parallax image data from the image data, in correspondence with the evaluation value of the evaluation map;
A generation unit configured to apply the adjustment parameter value to the image data and generate parallax image data to be images having parallax amounts adjusted to each other;
With
The evaluation map generation unit sets the evaluation value to a value obtained by associating the difference between the area of the subject and the surrounding area with the probability that the viewer gazes,
The determination unit is configured to increase an increment of the adjustment parameter value with respect to a unit increment of the evaluation value when the evaluation value is small larger than an increment of the adjustment parameter value with respect to a unit increment of the evaluation value when the evaluation value is large. An image processing apparatus. The image processing apparatus according to claim 6 , wherein the evaluation map generation unit generates the evaluation map from preliminary image data having a smaller number of pixels than the image data acquired by the acquisition unit in conjunction with the image data. . The determination section, the image processing apparatus according to any one of claims 1 to 7 for determining the adjustment parameter values for each pixel of the image data. An image sensor;
An imaging apparatus and an image processing apparatus according to any one of claims 1 to 8, wherein the acquisition unit acquires the image data by processing the output signal of the imaging element. An acquisition step of acquiring image data;
An evaluation map generating step for generating an evaluation map in which the evaluation value of the subject area included in the image of the image data is two-dimensionally arranged in correspondence with the position of the subject area;
A determination step for determining an adjustment parameter value for adjusting a parallax amount, which is applied when generating parallax image data from the image data, in association with the evaluation value of the evaluation map;
An image processing program for causing a computer to execute a generation step of generating parallax image data to be images having parallax amounts adjusted to each other by applying the adjustment parameter value to the image data ,
The image data includes first parallax image data and second parallax image data having parallax with each other,
The image data further includes reference image data different from the first parallax image data and the second parallax image data,
The evaluation map generating step is an image processing program for generating the evaluation map from the reference image data.