WO2015163350A1

WO2015163350A1 - Image processing device, imaging device and image processing program

Info

Publication number: WO2015163350A1
Application number: PCT/JP2015/062202
Authority: WO
Inventors: 潤弥萩原; 清茂芝崎; 石賀　健一; 文樹中村; 祐介 ▲高▼梨
Original assignee: 株式会社ニコン
Priority date: 2014-04-22
Filing date: 2015-04-22
Publication date: 2015-10-29
Also published as: JP2017108194A

Abstract

Stereo image data as imaged by a stereo imaging device could have parallax occurring in the extreme between left and right images due to the disposition of a subject in a scene, and a viewer could feel a sense of discomfort or fatigue in such a case. Accordingly, provided is an image processing device that is equipped with: an acquisition unit that acquires image data; a calculation unit that calculates the amount of parallax for each of a plurality of subjects that are included in an image of the image data; a determination unit that determines, for each of the plurality of subjects, in accordance with the calculated parallax amounts, an adjustment parameter value that is applied when parallax image data is generated from the image data and that adjusts the amount of parallax; and a generation unit that applies the adjustment parameter value to the image data and generates parallax image data which becomes an image having the mutually adjusted parallax amounts.

Description

Image processing apparatus, imaging apparatus, and image processing program

The present invention relates to an image processing device, an imaging device, and an image processing program.

There is known a stereo imaging device that acquires a stereo image composed of a right-eye image and a left-eye image using two photographing optical systems.
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP-A-8-47001

Stereo image data captured by a stereo imaging device may cause extreme parallax between the left and right images due to the placement of the subject in the scene, and viewers will feel uncomfortable and tired during viewing. was there. On the other hand, even when the parallax amount is adjusted to suppress such extremely generated parallax, there is a demand for the main subject to have a stereoscopic effect.

An image processing apparatus according to a first aspect of the present invention includes an acquisition unit that acquires image data, a calculation unit that calculates a parallax amount of each of a plurality of subjects included in an image of the image data, and parallax image data from the image data. A determination unit that determines an adjustment parameter value for adjusting the amount of parallax applied to each of a plurality of subjects in accordance with the calculated amount of parallax, and applies the adjustment parameter value to the image data. And a generation unit that generates parallax image data that is an image having parallax amounts adjusted to each other.

An image processing program according to a second aspect of the present invention includes an acquisition step of acquiring image data, a calculation step of calculating the amount of parallax for each of a plurality of subjects included in the image of the image data, and parallax image data from the image data. A determination step for determining an adjustment parameter value for adjusting the amount of parallax applied to each of a plurality of subjects in accordance with the calculated amount of parallax, and applying the adjustment parameter value to the image data And causing the computer to execute a generation step of generating parallax image data to be images having parallax amounts adjusted to each other.

Note that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is a figure explaining the structure of the digital camera which concerns on embodiment of this invention. It is a conceptual diagram which represents notably the mode that a part of imaging device was expanded. It is a figure explaining the example of a production | generation process of 2D image data and parallax image data. It is a figure explaining the concept of defocusing. It is a figure which shows the light intensity distribution which a parallax pixel outputs. It is a figure which shows the light intensity distribution for demonstrating the concept of adjustment parallax amount. It is a figure explaining the production | generation process of color parallax plane data. It is a figure which shows typically the relationship between the amount of parallax and the value of a three-dimensional adjustment parameter. It is a figure which shows a look-up table. It is a figure which shows the relationship between the amount of parallax in the case where the shape of a parallax amount curve differs with the imaging | photography conditions, and the target value of a parallax amount. It is a figure explaining the change of the light intensity distribution of RGB. It is a figure which shows the relationship between a viewer's convergence angle and the amount of parallax. It is a figure which shows typically the relationship between the contrast which shows the sharpness of an image, and the amount of parallax. It is a figure which shows typically the relationship between a subject distribution and the amount of parallax. It is a figure which shows typically the relationship between an aperture value and the amount of parallax. It is a figure which shows the concept of a focus shift typically. It is a figure which shows typically the relationship between the amount of parallax, the value of a three-dimensional adjustment parameter, an aperture value, etc. It is a figure explaining subject specification. It is a processing flow in the moving image photography which concerns on 1st Example. It is a processing flow until it produces | generates parallax color image data. It is a processing flow in the moving image photography which concerns on 2nd Example. It is a figure explaining a preferable opening shape. It is a figure explaining cooperation with a digital camera and TV monitor. It is a figure which shows typically the relationship between the amount of parallax and the value of a three-dimensional adjustment parameter.

Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

The digital camera according to the present embodiment, which is a form of the imaging device, is configured to generate a plurality of viewpoint images for one scene by one shooting. Each image having a different viewpoint is called a parallax image. In the present embodiment, a case where a right parallax image and a left parallax image from two viewpoints corresponding to the right eye and the left eye are generated will be described. The digital camera in the present embodiment can generate a parallax-free image without parallax from the central viewpoint together with the parallax image.

FIG. 1 is a diagram illustrating a configuration of a digital camera 10 according to an embodiment of the present invention. The digital camera 10 includes a photographic lens 20 as a photographic optical system, and guides a subject light beam incident along the optical axis 21 to the image sensor 100. The photographing lens 20 may be an interchangeable lens that can be attached to and detached from the digital camera 10. The digital camera 10 includes an image sensor 100, a control unit 201, an A / D conversion circuit 202, a memory 203, a drive unit 204, an image processing unit 205, a memory card IF 207, an operation unit 208, a display unit 209, and an LCD drive circuit 210. .

As shown in the figure, the direction parallel to the optical axis 21 toward the image sensor 100 is defined as the Z-axis plus direction, the direction toward the front of the drawing on the plane orthogonal to the Z-axis is the X-axis plus direction, and the upward direction on the drawing is Y. The axis is defined as the plus direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

The photographing lens 20 is composed of a plurality of optical lens groups, and forms an image of a subject light flux from the scene in the vicinity of its focal plane. In FIG. 1, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens arranged in the vicinity of the pupil. Further, in the vicinity of the pupil, a diaphragm 22 that restricts the incident light beam concentrically with the optical axis 21 as the center is disposed.

The image sensor 100 is disposed in the vicinity of the focal plane of the photographing lens 20. The image sensor 100 is an image sensor such as a CCD or CMOS sensor in which a plurality of photoelectric conversion elements are two-dimensionally arranged. The image sensor 100 is controlled in timing by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202.

The A / D conversion circuit 202 converts the image signal output from the image sensor 100 into a digital image signal and outputs the digital image signal to the memory 203. The image processing unit 205 performs various image processing on the digital image signal using the memory 203 as a work space, and generates captured image data. The captured image data includes reference image data generated from the output of the non-parallax pixels of the image sensor 100 and parallax image data generated from the output of the parallax pixels of the image sensor 100, as will be described later.

The control unit 201 controls the digital camera 10 in an integrated manner. For example, the aperture of the diaphragm 22 is adjusted according to the set diaphragm value, and the photographing lens 20 is advanced and retracted in the optical axis direction according to the AF evaluation value. Further, the position of the photographing lens 20 is detected, and the focal length and the focus lens position of the photographing lens 20 are grasped. Further, a timing control signal is transmitted to the drive unit 204, and a series of imaging control until the image signal output from the imaging element 100 is processed into captured image data by the image processing unit 205 is managed, and the captured image Get the data.

Also, the control unit 201 includes a depth information detection unit 230. The depth information detection unit 230 detects the subject distribution in the depth direction with respect to the scene. Specifically, the control unit 201 detects the subject distribution from the defocus amount for each subdivided area using defocus information used for autofocus. Note that the defocus information may use the output of a phase difference sensor provided exclusively, or may use the output of parallax pixels of the image sensor 100. When the output of the parallax pixel is used, the parallax image data processed by the image processing unit 205 can also be used. Alternatively, the subject distribution can be detected even when the focus lens is advanced and retracted without using the defocus information and the AF evaluation value by the contrast AF method is calculated for each subdivided region.

As described above, the image processing unit 205 processes the image signal output from the image sensor 100 to generate captured image data. The image processing unit 205 includes a calculation unit 231, a determination unit 232, a parallax image data generation unit 233, and a moving image generation unit 234.

The calculation unit 231 calculates the parallax amount of each subject included in the image of the captured image data, that is, the unadjusted parallax amount from the left and right parallax image data described later. The determination unit 232 determines a change condition for changing the parallax amount according to the parallax amount of each subject calculated by the calculation unit 231. More specifically, the determination unit 232 determines the value of the stereoscopic adjustment parameter so that the amount of parallax between output parallax images falls within the target amount of parallax. This stereoscopic adjustment parameter is a parameter that is applied to adjust the amount of parallax of parallax image data when generating parallax image data from captured image data.

The parallax image data generation unit 233 applies parallax adjustment parameters to the captured image data to generate parallax image data that becomes images having parallax amounts adjusted to each other. Details of the calculation unit 231, the determination unit 232, and the parallax image data generation unit 233 will be described later. The moving image generation unit 234 connects the parallax image data and generates a 3D moving image file.

The image processing unit 205 also has general image processing functions such as adjusting image data according to the selected image format. The generated captured image data is converted into a display signal by the LCD drive circuit 210 and displayed on the display unit 209. The data is recorded on the memory card 220 attached to the memory card IF 207.

The operation unit 208 functions as a part of a reception unit that receives a user operation and transmits an instruction to the control unit 201. The operation unit 208 includes a plurality of operation members such as a shutter button that receives a shooting start instruction.

FIG. 2 is a conceptual diagram conceptually showing a state in which a part of the image sensor 100 is enlarged. In the pixel area, 20 million or more pixels are arranged in a matrix. In the present embodiment, 64 pixels of adjacent 8 pixels × 8 pixels form one basic lattice 110. The basic grid 110 includes four Bayer arrays having 4 × 2 × 2 basic units in the Y-axis direction and four in the X-axis direction. As shown in the figure, in the Bayer array, a green filter (G filter) is arranged for the upper left pixel and the lower right pixel, a blue filter (B filter) is arranged for the lower left pixel, and a red filter (R filter) is arranged for the upper right pixel.

The basic grid 110 includes parallax pixels and non-parallax pixels. The parallax pixel is a pixel that receives a partial light beam that is deviated from the optical axis of the photographing lens 20 out of the incident light beam that is transmitted through the photographing lens 20. The parallax pixel is provided with an aperture mask having a deviated opening that is deviated from the center of the pixel so as to transmit only the partial light flux. For example, the opening mask is provided so as to overlap the color filter. In the present embodiment, the parallax Lt pixel defined so that the partial light beam reaches the left side with respect to the pixel center and the parallax specified so that the partial light beam reaches the right side with respect to the pixel center by the aperture mask. There are two types of Rt pixels. On the other hand, the non-parallax pixel is a pixel that is not provided with an aperture mask, and is a pixel that receives the entire incident light beam that passes through the photographing lens 20.

Note that the parallax pixel is not limited to the aperture mask when receiving the partial light beam that is deviated from the optical axis, but has various configurations such as a selective reflection film in which the light receiving region and the reflective region are separated, and a deviated photodiode region. Can be adopted. In other words, the parallax pixel only needs to be configured to receive a partial light beam that is deviated from the optical axis, among incident light beams that pass through the photographing lens 20.

Pixels in the basic grid 110 are denoted by _PIJ . For example, the upper left pixel is _{P 11,} the upper right pixel is _{P 81.} As shown in the figure, the parallax pixels are arranged as follows.

P ₁₁ : Parallax Lt pixel + G filter (= G (Lt))
P ₅₁ ... Parallax Rt pixel + G filter (= G (Rt))
P ₃₂ ... Parallax Lt pixel + B filter (= B (Lt))
P ₆₃ ... Parallax Rt pixel + R filter (= R (Rt))
P ₁₅ ... Parallax Rt pixel + G filter (= G (Rt))
P ₅₅ ... Parallax Lt pixel + G filter (= G (Lt))
P ₇₆ ... Parallax Rt pixel + B filter (= B (Rt))
P ₂₇ ... Parallax Lt pixel + R filter (= R (Lt))
The other pixels are non-parallax pixels, and are any of the non-parallax pixel + R filter, the non-parallax pixel + G filter, and the non-parallax pixel + B filter.

When viewed as a whole of the image sensor 100, the parallax pixels are classified into one of a first group having a G filter, a second group having an R filter, and a third group having a B filter. Includes at least one parallax Lt pixel and parallax Rt pixel belonging to each group. As in the example in the figure, these parallax pixels and non-parallax pixels may be arranged with randomness in the basic lattice 110. By arranging with randomness, RGB color information can be acquired as the output of the parallax pixels without causing bias in the spatial resolution for each color component, so that high-quality parallax image data can be obtained. can get.

Next, the concept of processing for generating 2D image data and parallax image data from captured image data output from the image sensor 100 will be described. FIG. 3 is a diagram illustrating an example of processing for generating 2D image data and parallax image data.

As can be seen from the arrangement of parallax pixels and non-parallax pixels in the basic grid 110, even if the output of the image sensor 100 is aligned with the pixel arrangement and arranged as it is, it does not become image data representing a specific image. Only when the pixel outputs of the image sensor 100 are separated and collected for each pixel group characterized in the same manner, image data representing one image in accordance with the characteristics is formed. For example, when the left and right parallax pixels are gathered together, left and right parallax image data having parallax can be obtained. In this way, each piece of image data separated and collected for each identically characterized pixel group is referred to as plane data.

The image processing unit 205 receives raw raw image data in which output values (pixel values) are arranged in the order of pixel arrangement of the image sensor 100, and executes plane separation processing for separating the raw image data into a plurality of plane data. The left column of the figure shows an example of processing for generating 2D-RGB plane data as 2D image data.

In generating 2D-RGB plane data, the image processing unit 205 first removes the pixel values of the parallax pixels to form a vacant lattice. Then, the pixel value that becomes the empty grid is calculated by interpolation processing using the pixel values of the surrounding pixels. For example, the pixel value of the empty lattice P ₁₁ is obtained by averaging the pixel values of P ₋₁ ₋₁ , P ₂₋₁ , P ₋₁₂ , and P ₂₂ which are the pixel values of the G filter pixels adjacent in the diagonal direction. To calculate. Further, for example, the pixel value of the empty lattice P ₆₃ is calculated by averaging the pixel values of P ₄₃ , P ₆₁ , P ₈₃ , and P ₆₅ that are adjacent R filter pixel values by skipping one pixel vertically and horizontally. To do. Similarly, for example, the pixel value of the air grating _{P 76} is the pixel value of the adjacent B filter skipping one pixel vertically and _{horizontally,} and averaging operation of the pixel values of _{_{P 56, P 74, P 96}} , P 78 calculate.

Since the 2D-RGB plane data interpolated in this way is the same as the output of a normal image sensor having a Bayer array, various processes can be performed as 2D image data thereafter. That is, known Bayer interpolation is performed to generate color image data in which RGB data is aligned for each pixel. The image processing unit 205 performs image processing as a general 2D image according to a predetermined format such as JPEG when generating still image data and MPEG when generating moving image data.

In the present embodiment, the image processing unit 205 further separates the 2D-RGB plane data for each color and performs the interpolation processing as described above to generate each plane data as reference image data. That is, three types of data are generated: Gn plane data as green reference image plane data, Rn plane data as red reference image plane data, and Bn plane data as blue reference image plane data.

The right column of the figure shows an example of processing for generating two G plane data, two R plane data, and two B plane data as parallax pixel data. The two G plane data are GLt plane data as left parallax image data and GRt plane data as right parallax image data. The two R plane data are RLt plane data and right parallax image data as left parallax image data. The two B plane data are the BLt plane data as the left parallax image data and the BRt plane data as the right parallax image data.

In generating the GLt plane data, the image processing unit 205 removes pixel values other than the pixel values of the G (Lt) pixels from all output values of the image sensor 100 to form a vacant lattice. As a result, two pixel values P ₁₁ and P ₅₅ remain in the basic grid 110. Therefore, we divided into four equal basic grid 110 vertically and horizontally, the 16 pixels of the top left is represented by an output value of the P _11, is representative of the 16 pixels in the lower right in the output value of the P _55. Then, for the upper right 16 pixels and the lower left 16 pixels, average values of neighboring representative values adjacent in the vertical and horizontal directions are averaged and interpolated. That is, the GLt plane data has one value in units of 16 pixels.

Similarly, when generating the GRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the G (Rt) pixel from all the output values of the image sensor 100 to obtain an empty grid. Then, two pixel values P ₅₁ and P ₁₅ remain in the basic grid 110. Therefore, the basic grid 110 is divided into four equal parts vertically and horizontally, the upper right 16 pixels are represented by the output value of P ₅₁ , and the lower left 16 pixels are represented by the output value of P ₁₅ . The upper left 16 pixels and the lower right 16 pixels are interpolated by averaging the peripheral representative values adjacent vertically and horizontally. That is, the GRt plane data has one value in units of 16 pixels. In this way, GLt plane data and GRt plane data having a resolution lower than that of 2D-RGB plane data can be generated.

In generating the RLt plane data, the image processing unit 205 removes pixel values other than the pixel value of the R (Lt) pixel from all output values of the image sensor 100 to form a vacant lattice. Then, the primitive lattice _110, the pixel values of _{P 27} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. Similarly, when generating the RRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the R (Rt) pixel from all output values of the image sensor 100 to form a vacant lattice. Then, the pixel value P ₆₃ remains in the basic grid 110. This pixel value is set as a representative value for 64 pixels of the basic grid 110. In this way, RLt plane data and RRt plane data having a lower resolution than 2D-RGB plane data are generated. In this case, the resolution of the RLt plane data and the RRt plane data is lower than the resolution of the GLt plane data and the GRt plane data.

In generating the BLt plane data, the image processing unit 205 removes pixel values other than the pixel values of the B (Lt) pixels from all output values of the image sensor 100 to form a vacant lattice. Then, the primitive lattice _110, the pixel values of _{P 32} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. Similarly, when generating the BRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the B (Rt) pixel from all the output values of the image sensor 100 to obtain an empty grid. Then, the primitive lattice _110, the pixel values of _{P 76} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. In this way, BLt plane data and BRt plane data having a resolution lower than that of 2D-RGB plane data are generated. In this case, the resolution of the BLt plane data and the BRt plane data is lower than the resolution of the GLt plane data and the GRt plane data, and is equal to the resolution of the RLt plane data and the RRt plane data.

In the present embodiment, image processing may be performed on the output image data so that the amount of parallax between generated images is within the target amount of parallax. In this case, the image processing unit 205 generates left-view color image data and right-view color image data using these plane data. In particular, by introducing the stereo adjustment parameter, color image data in which the parallax amount as a 3D image is adjusted while maintaining the blur amount of the 2D color image is generated. Prior to specific processing, the generation principle will be described first.

FIG. 4 is a diagram for explaining the concept of defocusing. The parallax Lt pixel and the parallax Rt pixel receive a subject light flux that arrives from one of two parallax virtual pupils that are set symmetrically with respect to the optical axis as a partial region of the lens pupil. In the optical system of the present embodiment, since the actual subject light flux passes through the entire lens pupil, the light intensity distributions corresponding to the parallax virtual pupil are not distinguished from each other until the parallax pixel is reached. However, the parallax pixel outputs an image signal obtained by photoelectrically converting only the partial light flux that has passed through the parallax virtual pupil by the action of the aperture mask that each has. Therefore, the pixel value distribution indicated by the output of the parallax pixel may be considered to be proportional to the light intensity distribution of the partial light flux that has passed through the corresponding parallax virtual pupil.

As shown in FIG. 4A, when an object point that is a subject exists at the focal position, the output of each parallax pixel is the corresponding image point regardless of the subject luminous flux that has passed through any parallax virtual pupil. This shows a steep pixel value distribution centering on this pixel. If the parallax Lt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Further, even when the parallax Rt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. That is, even if the subject luminous flux passes through any parallax virtual pupil, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Match each other.

On the other hand, as shown in FIG. 4B, when the object point deviates from the focal position, the peak of the pixel value distribution indicated by the parallax Lt pixel corresponds to the image point, compared to the case where the object point exists at the focal position. Appearing at a position away from the pixel in one direction, and its output value decreases. In addition, the width of the pixel having the output value is increased. The peak of the pixel value distribution indicated by the parallax Rt pixel appears at a position away from the pixel corresponding to the image point in the opposite direction to the one direction in the parallax Lt pixel and at an equal distance, and the output value similarly decreases. Similarly, the width of the pixel having the output value is increased. That is, the same pixel value distribution that is gentler than that in the case where the object point exists at the focal position appears at an equal distance from each other. Further, as shown in FIG. 4C, when the object point further deviates from the focal position, the same pixel value distribution that becomes more gentle as compared with the state of FIG. 4B appears further apart. . That is, it can be said that the amount of blur and the amount of parallax increase as the object point deviates from the focal position. In other words, the amount of blur and the amount of parallax change in conjunction with defocus. That is, the amount of blur and the amount of parallax have a one-to-one relationship.

FIGS. 4B and 4C show the case where the object point shifts away from the focal position, but when the object point moves away from the focal position, as shown in FIG. Compared to FIGS. 4B and 4C, the relative positional relationship between the pixel value distribution indicated by the parallax Lt pixel and the pixel value distribution indicated by the parallax Rt pixel is reversed. Due to such a defocus relationship, when viewing a parallax image, the viewer visually recognizes a subject existing far behind the focal position and visually recognizes a subject present in front.

FIG. 5 is a graph showing changes in the pixel value distribution described in FIGS. 4B and 4C. In the figure, the horizontal axis represents the pixel position, and the center position is the pixel position corresponding to the image point. The vertical axis represents the output value (pixel value) of each pixel. As described above, this output value is substantially proportional to the light intensity.

The distribution curve 1804 and the distribution curve 1805 represent the pixel value distribution of the parallax Lt pixel and the pixel value distribution of the parallax Rt pixel in FIG. 4B, respectively. As can be seen from the figure, these distributions have a line-symmetric shape with respect to the center position. Further, a combined distribution curve 1806 obtained by adding them shows a pixel value distribution of pixels without parallax with respect to the situation of FIG. 4B, that is, a pixel value distribution when the entire subject luminous flux is received, and a substantially similar shape.

The distribution curve 1807 and the distribution curve 1808 represent the pixel value distribution of the parallax Lt pixel and the pixel value distribution of the parallax Rt pixel in FIG. 4C, respectively. As can be seen from the figure, these distributions are also symmetrical with respect to the center position. Also, a combined distribution curve 1809 obtained by adding them shows a shape substantially similar to the pixel value distribution of the non-parallax pixels for the situation of FIG.

In the present embodiment, when image processing is performed using the stereoscopic adjustment parameter, the pixel value of the parallax Lt pixel of such a pixel value distribution obtained as an output value of the image sensor 100 and subjected to interpolation processing on the empty grid. And a pixel value of the parallax Rt pixel are used to create a virtual pixel value distribution. At this time, the amount of parallax expressed as an interval between peaks is adjusted while approximately maintaining the amount of blur expressed by the spread of the pixel value distribution. That is, in this embodiment, the image processing unit 205 is adjusted between the 2D image generated from the non-parallax pixel and the 3D image generated from the parallax pixel while maintaining the blur amount of the 2D image almost as it is. An image having a parallax amount is generated. FIG. 6 is a diagram illustrating a pixel value distribution for explaining the concept of the adjusted parallax amount.

Lt distribution curve 1901 and Rt distribution curve 1902 indicated by solid lines in the figure are distribution curves in which actual pixel values of Lt plane data and Rt plane data are plotted. For example, it corresponds to the distribution curves 1804 and 1805 in FIG. The distance between the peaks of the Lt distribution curve 1901 and the Rt distribution curve 1902 represents the 3D parallax amount, and the greater the distance, the stronger the stereoscopic effect during image reproduction.

The 2D distribution curve 1903 obtained by adding 50% each of the Lt distribution curve 1901 and the Rt distribution curve 1902 has a convex shape with no left-right bias. The 2D distribution curve 1903 corresponds to a shape in which the height of the combined distribution curve 1806 in FIG. That is, an image based on this distribution is a 2D image with a parallax amount of zero.

The adjusted Lt distribution curve 1905 is a curve obtained by adding 80% of the

Lt distribution curve

1901 and 20% of the Rt distribution curve 1902. The peak of the adjusted Lt distribution curve 1905 is displaced closer to the center than the peak of the Lt distribution curve 1901 as much as the component of the Rt distribution curve 1902 is added. Similarly, the adjusted Rt distribution curve 1906 is a curve obtained by adding 20% of the

Lt distribution curve

1901 and 80% of the Rt distribution curve 1902. The peak of the adjusted Rt distribution curve 1906 is displaced closer to the center than the peak of the Rt distribution curve 1902 by the amount to which the component of the Lt distribution curve 1901 is added.

Therefore, the adjusted parallax amount represented by the distance between the peaks of the adjusted Lt distribution curve 1905 and the adjusted Rt distribution curve 1906 is smaller than the 3D parallax amount. Therefore, the stereoscopic effect during image reproduction is alleviated. On the other hand, since the spread of each of the adjusted Lt distribution curve 1905 and the adjusted Rt distribution curve 1906 is equivalent to the spread of the 2D distribution curve 1903, it can be said that the amount of blur is equal to that of the 2D image.

That is, the amount of adjustment parallax can be controlled by how much the Lt distribution curve 1901 and the Rt distribution curve 1902 are added. Then, by applying this adjusted pixel value distribution to each plane of color image data generated from pixels without parallax, the color of the left viewpoint that gives a stereoscopic effect different from that of parallax image data generated from parallax pixels Image data and right-view color image data can be generated.

In the present embodiment, left-view color image data and right-view color image data are generated from the nine plane data described with reference to FIG. Color image data of the left viewpoint, RLt _c plane data is red plane data corresponding to the left viewpoint, a green plane data GLt _c plane data, and three color parallax plane data BLt _c plane data is blue plane data Consists of. Similarly, the color image data of the right-side perspective is, RRT _c plane data is red plane data corresponding to the right viewpoint, a green plane data GRT _c plane data, and three of BRt _c plane data is blue plane Datacolor Consists of parallax plane data.

FIG. 7 is a diagram for explaining color parallax plane data generation processing. In particular, a generation process of RLt _c plane data and RRt _c plane data, which are red parallax planes among color parallax planes, will be described.

The red parallax plane is generated using the pixel value of the Rn plane data described with reference to FIG. 3, and the pixel value of the RLt plane data and the RRt plane data. Specifically, for example, when calculating the pixel value RLt _mn of the target pixel position (i _m , j _n ) of the RLt _c plane data, first, the parallax image data generation unit 233 of the image processing unit 205 stores the Rn plane data A pixel value Rn _mn is extracted from the same pixel position (i _m , j _n ). Next, the parallax image data generating unit 233, the same pixel position of RLt plane data _(i m, _{j n)} pixel values _{RLt mn} from the same pixel position of RRt plane data _(i m, _{j n)} pixel values from RRt Extract _mn . Then, the parallax image data generation unit 233 multiplies the pixel value Rn _mn by the value obtained by distributing the pixel values RLt _mn and RRt _mn by the value of the stereoscopic adjustment parameter C, thereby calculating the pixel value RLt _cmn . Specifically, it is calculated by the following equation (1).

Similarly, when calculating the pixel value RRt _cmd of the target pixel position (i _m , j _n ) of the RRt _c plane data, the parallax image data generation unit 233 uses the extracted pixel value Rn _mn and the pixel value RLt _{mn as} well. The pixel value RRt _mn is calculated by multiplying the value obtained by distributing the three-dimensional adjustment parameter C by the value. Specifically, it is calculated by the following equation (2).

The parallax image data generation unit 233 sequentially executes such processing from (1, 1) which is the pixel at the left end and the upper end to (i ₀ , j ₀ ) which is the coordinates at the right end and the lower end.

When the generation processing of RLt _c plane data and RRt _c plane data that are red parallax planes is completed, generation processing of GLt _c plane data and GRt _c plane data that are green parallax planes is then executed. Specifically, the pixel same pixel position _(i m, _{j n)} of Rn plane data in the above description, instead of extracting the pixel values _{Rn mn} from the same pixel position of Gn plane data _(i m, _{j n)} from The value Gn _mn is extracted. Moreover, the same pixel position of RLt plane data _(i m, _{j n)} from instead of extracting the pixel value _{RLt mn,} extracts the pixel value _{GLt mn} from the same pixel position of GLt plane data _(i m, _{j n).} Similarly, the same pixel position of the RRT plane data _(i m, _{j n)} from instead of extracting the pixel value _{RRT mn,} extracts the pixel value _{GRT mn} from the same pixel position of GRT plane data _(i m, _{j n)} . And the value of each parameter of Formula (1) and Formula (2) is changed suitably, and it processes similarly.

Further, when the generation processing of the GLt _c plane data and the GRt _c plane data which are green parallax planes is completed, the generation processing of the BLt _c plane data and BRt _c plane data which are blue parallax planes is executed next. Specifically, the pixel same pixel position _(i m, _{j n)} of Rn plane data in the above description, instead of extracting the pixel values _{Rn mn} from the same pixel position of Bn plane data _(i m, _{j n)} from Extract the value Bn _mn . Moreover, the same pixel position of RLt plane data _(i m, _{j n)} from instead of extracting the pixel value _{RLt mn,} extracts the pixel value _{BLt mn} from the same pixel position of BLt plane data _(i m, _{j n).} Similarly, the same pixel position of the RRT plane data _(i m, _{j n)} from instead of extracting the pixel value _{RRT mn,} extracts the pixel value _{BRt mn} from the same pixel position of BRt plane data _(i m, _{j n)} . And the value of each parameter of Formula (1) and Formula (2) is changed suitably, and it processes similarly.

Through the above processing, left-view color image data (RLt _c- plane data, GLt _c- plane data, BLt _c- plane data) and right-view color image data (RRt _c- plane data, GRt _c- plane data, BRt _c- plane data) Is generated. That is, the color image data of the left viewpoint and the right viewpoint can be acquired by a relatively simple process as a virtual output that does not actually exist as a pixel of the image sensor 100.

In addition, since the value of the stereoscopic adjustment parameter C can be changed within the range of 0.5 <C <1, the amount of parallax as a 3D image is adjusted while maintaining the amount of blur of the 2D color image due to pixels without parallax. be able to. Therefore, if these image data are reproduced by a 3D image compatible reproduction device, the viewer of the stereoscopic video display panel can appreciate the 3D video in which the stereoscopic effect is appropriately adjusted as a color image. In particular, since the processing is simple, it is possible to generate image data at high speed and to deal with moving images.

However, in the present embodiment, in the color parallax plane data generation process as described above, the value of the stereoscopic adjustment parameter C is determined and used in a range of 0.5 <C <1 for each subject. In this regard, prior to specific description regarding determination of the value of the stereoscopic adjustment parameter C, first, the advantage of generating color parallax plane data using the stereoscopic adjustment parameter C determined for each subject will be described.

FIG. 8 is a diagram schematically showing the relationship between the parallax amount and the value of the three-dimensional adjustment parameter. In this figure, the horizontal axis represents the distance from the digital camera 10, that is, the depth with respect to the scene, and the vertical axis represents the amount of parallax.

The digital camera 10, the object located at a distance L ₁₀ (see focus object in the drawing) are focused. In the present embodiment, the threshold value of the allowable amount of parallax is predetermined as ± m.

The plurality of subjects included in the captured image data have unadjusted parallax amounts in the left and right parallax image data. For example, in the photographic image of a scene represented in Figure 8, in addition to the subject located at a distance L _10, 2 two subject located at a distance L ₂₀ and the distance L ₃₀ (see a near point object and far-point object in FIG. )It is included. The unadjusted parallax amounts for these three subjects are the parallax amount m ₁₀ (m ₁₀ = 0), the parallax amount m ₂₀ (m ₂₀ > m), and the parallax amount m ₃₀ (m ₃₀ <−m).

In this embodiment, the value of the three-dimensional adjustment parameter C is determined for each subject according to such an unadjusted parallax amount, and is used for adjusting the parallax amount. For example, for focusing the object value of the three-dimensional adjustment parameter C is determined as C _10, the parallax amount m ₁₀ unadjusted this value C ₁₀ is adjusted to the target value m ₁₀₀ of the parallax amount. However, since the amount of parallax m ₁₀ is a parallax amount of the focusing object, the target value m ₁₀₀ of the parallax amount m ₁₀ and the parallax amount of unadjusted are both 0.

Further, the value of the stereoscopic adjustment parameter C is determined as C ₂₀ for the near-point subject, and the unadjusted parallax amount m ₂₀ is adjusted to the parallax amount target value m ₂₀₀ (m ₂₀₀ ≦ m) by this value C ₂₀ . Has been. Further, the value of the stereoscopic adjustment parameter C is determined as C ₃₀ for the far-point subject, and the unadjusted parallax amount m ₃₀ is set to the parallax amount target value m ₃₀₀ (m ₃₀₀ ≧ −m) by this value C ₃₀ . It has been adjusted.

Similarly, when another subject exists in the scene shown in FIG. 8, the unadjusted parallax amount for this subject is a value indicated by the parallax amount curve 1622 in the figure. The target value of the parallax amount is a value indicated by the adjusted parallax amount curve 1623 in the drawing.

Here, the parallax amount curve 1622 in the figure represents the relationship between the distance from the digital camera 10 and the unadjusted parallax amount generated in each subject when it is assumed that the subject is located at each distance. The parallax amount curve 1622 includes a region outside the range of −m to + m. From the viewpoint of facilitating understanding, in the figure, the region outside the range of −m to + m in the parallax amount curve 1622 is surrounded by the frame W1, and the region within the range of −m to + m is surrounded by the frame W2. Yes.

The adjusted parallax amount curve 1623 represents the relationship between the distance from the digital camera 10 and the target value of the parallax amount generated in each subject when it is assumed that the subject is located at each distance. The target value of the parallax amount indicated by the adjusted parallax amount curve 1623 is included in the range of −m to + m within the distance range in which the parallax amount curve 1622 is set.

The adjustment parallax amount curve 1623 as described above has a shape in which different values of the three-dimensional adjustment parameter C are set for a subject at least in the range of −m to + m and a subject outside the range of −m to + m. It has become. The specific shape will be described later.

In the functions of the parallax amount curve 1622 and the adjusted parallax amount curve 1623, the depth relationship between subjects, that is, the relationship between the distance from the digital camera 10 and the parallax amount is not broken. Specifically, the greater the distance from the digital camera 10, the smaller the parallax amount. When the parallax amount exceeds 0, the negative value increases. In other words, the parallax amount m _a distance greater object from the digital camera 10, the distance and the parallax amount m _b a small object meets m _b ≧ m _a.
In the adjusted parallax amount curve 1623, the parallax amount larger than + m is adjusted to be + m which is the upper limit of the allowable parallax amount, and the parallax amount smaller than −m is the lower limit of the allowable parallax amount −m It is adjusted to become. In this way, the parallax of the subject that is not the main subject is not suppressed more than necessary, and the depth relationship of the subject is not disrupted.

According to the adjusted parallax amount curve 1623 as described above, the unadjusted parallax amount along the parallax amount curve 1622 is generally maintained for the subject in the range where the unadjusted parallax amount is in the range of −m to + m. Then, the value of the three-dimensional adjustment parameter C is determined and the amount of parallax is adjusted. As a result, the stereoscopic effect is maintained in the subject within the range of −m to + m. Here, adjusting the parallax amount so that the unadjusted parallax amount is generally maintained means that the parallax amount is adjusted so that the unadjusted parallax amount is maintained or becomes a value close to the unadjusted parallax amount. Means to be adjusted.

On the other hand, in a subject whose unadjusted parallax amount is not included in the range of −m to + m, the stereoscopic adjustment parameter C different from that of the subject in the range of −m to + m is included so as to be included in the range of −m to + m. Is determined and the amount of parallax is adjusted. As a result, the amount of parallax is suppressed in subjects outside the range of −m to + m, and fatigue and discomfort during viewing are reduced.

As described above, when the color parallax plane data is generated using the value of the stereoscopic adjustment parameter C determined for each subject, the amount of parallax between a subject with a large unadjusted amount of parallax and a subject with a small amount of parallax. The adjustment amount can be intentionally varied. For this reason, it is possible to reduce the amount of parallax for a non-main subject while maintaining a stereoscopic effect for the main subject, and to reduce the sense of discomfort and fatigue during viewing.

When there are a plurality of subjects having the same distance from the digital camera 10, the unadjusted parallax amounts are the same for the plurality of subjects, and therefore the values of the three-dimensional adjustment parameter C for adjusting the parallax amount are the same. Are also equal. From this, for each subject included in the image, the value of the stereoscopic adjustment parameter C is determined according to the unadjusted parallax amount. The unadjusted parallax amounts for a plurality of subjects included in the image Is the same as determining the value of the three-dimensional adjustment parameter C.

Subsequently, the principle of determining the three-dimensional adjustment parameter C for each subject will be described. The principle described below is for facilitating understanding of a lookup table 2310 described later. Therefore, the value of the stereoscopic adjustment parameter C does not necessarily have to be determined in the digital camera 10 according to this principle.

As described above, in the present embodiment, the three-dimensional adjustment parameter C is such that the unadjusted parallax amount is generally maintained for subjects in the range of −m to + m among a plurality of subjects included in the image. A value is calculated. For a subject outside the range of −m to + m, the value of the stereoscopic adjustment parameter C is determined so that the parallax amount is adjusted within the range of −m to + m.

To determine the value of the three-dimensional adjustment parameter C in this way, first, the parallax amount of the subject is calculated from the left and right parallax image data. Thereby, for example, the unadjusted parallax amounts m ₁₀ , m ₂₀ , and m ₃₀ for the focused subject, the near point subject, and the far point subject are detected, respectively.

Further, the distance from the digital camera 10 to the subject is calculated based on the defocus amount from the focal position. Thereby, for example, distances L ₁₀ , L ₂₀ , and L ₃₀ from the digital camera 10 to the focused subject, the near point subject, and the far point subject are calculated.

Next, for each subject located outside the range of −m to + m, the target value of the parallax amount corresponding to the calculated distance is determined from the adjusted parallax amount curve 1623. Specifically, a point corresponding to the calculated distance is determined from the adjusted parallax amount curve 1623, and the vertical coordinate of this point is determined as the target value of the parallax amount. Thereby, for example, the target value m ₁₀₀ , the target value m ₂₀₀ , and the target value m ₃₀₀ of the parallax amount for the focused subject, the near point subject, and the far point subject are determined.

Then, the value of the stereoscopic adjustment parameter C is calculated so that the unadjusted parallax amount is adjusted to the target value of the parallax amount for each subject. Thus, for example, for a focused subject with an unadjusted parallax amount m ₁₀ (m ₁₀ = 0), the value of the stereoscopic adjustment parameter C is calculated so that the parallax amount m ₁₀ is substantially maintained. The value C ₂₀ as parallax amount m ₂₀ unadjusted for near-point object is adjusted to the target value m ₂₀₀ of the parallax amount is calculated. Further, a value C ₃₀ is calculated such that the unadjusted parallax amount m ₃₀ for the far point subject is adjusted to the target value m ₃₀₀ for the parallax amount.

In order to calculate the value of the stereoscopic adjustment parameter C that adjusts the unadjusted parallax amount for a certain subject to the target value of the parallax amount, first, a 3D image is generated using the certain stereoscopic adjustment parameter C. After that, processing for calculating the parallax amount of the subject is performed. Then, by repeating this process while feeding back the calculation result, the value of the stereo adjustment parameter C when the unadjusted parallax amount is adjusted to the target value of the parallax amount is calculated. Preferably, a lookup table is generated in advance from the correspondence between the unadjusted parallax amount obtained in this way, the target value of the parallax amount, and the value of the stereoscopic adjustment parameter C, and this lookup table is used. Then, the value of the three-dimensional adjustment parameter C is calculated.

Here, a specific shape of the adjustment parallax amount curve 1623 used for determining the value of the three-dimensional adjustment parameter C as described above will be described. The adjusted parallax amount curve 1623 is composed of a part in the area surrounded by the frame W1 and a part in the area surrounded by the frame W2. Accordingly, the adjustment parallax amount curve 1623 can set different values of the stereoscopic adjustment parameter C for at least a subject within the range of −m to + m and a subject outside the range of −m to + m. .

Further, a portion of the adjusted parallax amount curve 1623 in the region surrounded by the frame W1 is formed so that the target value of the parallax amount is included in the range of −m to + m. In addition, the portion of the adjusted parallax amount curve 1623 in the region surrounded by the frame W <b> 2 is formed to approximate the parallax amount curve 1622.

Here, consider the case where the subject is continuous in the depth direction of the scene, in other words, the case where the subject is aligned in the depth direction. In this case, it is preferable that the parallax amount smoothly transition between these subjects in the depth direction.

Therefore, the adjusted parallax amount curve 1623 is preferably continuous in the depth direction, that is, the distance direction from the digital camera 10, and more preferably the differential value is continuous. In this case, the value of the stereoscopic adjustment parameter C is determined so that the adjusted parallax amount is continuous in the depth direction. Further, the value of the stereoscopic adjustment parameter C is determined so that the differential value of the adjusted parallax amount is continuous in the depth direction.

In order to make the adjusted parallax amount curve 1623 continuous in the distance direction, the shape of the portion of the adjusted parallax amount curve 1623 in the region surrounded by the frame W2 may be adjusted. Specifically, among the parts in the region surrounded by the frame W2, the part near the boundary with the frame W1 may be deformed so as to be continuous with the part surrounded by the frame W1.

Further, in order to make the differential value of the adjusted parallax amount curve 1623 continuous in the distance direction, the shape of the portion of the adjusted parallax amount curve 1623 in the region surrounded by the frame W2 may be adjusted. Specifically, among the portions in the region surrounded by the frame W2, a portion near the boundary with the frame W1 may be deformed so that the differential value is continuous with respect to the portion surrounded by the frame W1.

In order to select the adjusted parallax amount curve 1623, for example, a curve that overlaps with the parallax amount curve 1622 in at least a part of the parallax amount section in the range of −m to + m and has ± m as an asymptotic line is used. By generating, the portion surrounded by the frame W1 in the adjusted parallax amount curve 1623 is generated. Such a curve is derived from a hyperbolic tangent curve.

Further, a portion surrounded by the frame W2 in the adjusted parallax amount curve 1623 is generated from the parallax amount curve 1622. Then, the portion in the region surrounded by the frame W2 is deformed so that the portion near the boundary with the frame W1 is continuous with the portion surrounded by the frame W1. At this time, the differential value of the adjusted parallax amount curve 1623 is made continuous at the boundary between the frame W1 and the frame W2. Thereby, the adjusted parallax amount curve 1623 is selected.

Note that the functions of the parallax amount curve 1622 and the adjusted parallax amount curve 1623 described above depend on the imaging conditions that affect the parallax amount (for example, the aperture value, the focus position, the focal length when the photographing lens 20 is a zoom lens, etc.), respectively. Can change. Therefore, when the value of the stereoscopic adjustment parameter C is determined using the adjustment parallax amount curve 1623 according to the principle described above, the function of the adjustment parallax amount curve 1623 can be stored in the digital camera 10 for each imaging condition. That's fine.

However, there are an infinite number of imaging conditions depending on combinations of values such as aperture value, focus position, and focal length. Therefore, if the function of the adjusted parallax amount curve 1623 is stored for each imaging condition, the amount of data increases.

Therefore, in this embodiment, the value of the stereoscopic adjustment parameter C is obtained in advance by an experiment in accordance with the above-described principle without causing the digital camera 10 to store a plurality of functions regarding the adjusted parallax amount curve 1623. Hereinafter, a specific configuration will be described. FIG. 9 is a diagram showing a lookup table 2310 stored by the determining unit 232 in the present embodiment.

The lookup table 2310 is a table that is referred to when the determination unit 232 determines the value of the stereoscopic adjustment parameter C. This lookup table 2310 is stored in advance in the storage unit of the digital camera 10. As shown in FIG. 9, in the look-up table 2310, each value (m ₁₀ , m ₂₀ , m ₃₀ ,...) That the parallax amount m can take, and the value of the stereo adjustment parameter C corresponding to each value (C ₁₀ , C ₂₀ , C ₃₀ ,...) Are described in pairs.

Here, of the parallax amount m in the lookup table 2310, the value of the stereoscopic adjustment parameter C corresponding to each value within the range of −m to + m is determined so that the unadjusted parallax amount is generally maintained. Yes. Further, the value of the stereo adjustment parameter C corresponding to each value outside the range of −m to + m is determined so that the parallax amount is adjusted within the range of −m to + m.

Such a lookup table 2310 is generated through an experiment using a prototype, for example. Specifically, in the prototype of the digital camera 10, an imaging condition that affects the amount of parallax is set to any arbitrary condition.

At this time, an image is taken while moving the position of the subject in the depth direction with respect to the scene while keeping the shooting conditions constant. Each captured image data is associated with a distance to the subject.

Next, the amount of parallax of the subject is calculated from the left and right parallax image data in each captured image data, and the correspondence data between the amount of parallax and the distance from the digital camera 10 to the subject at the time of shooting is plotted on the coordinate plane. In this coordinate plane, as shown in FIG. 8 described above, the horizontal axis is the distance from the digital camera 10, that is, the depth with respect to the scene, and the vertical axis is the amount of parallax.

Then, a parallax amount curve 1622 is generated by generating approximate curves of these plots. Once the parallax amount curve 1622 is generated, the adjusted parallax amount curve 1623 is then selected as described above.

When the parallax amount curve 1622 and the adjusted parallax amount curve 1623 are obtained, one point on the horizontal axis is selected, and an unadjusted parallax amount corresponding to this point is detected from the parallax amount curve 1622. That is, the distance from the digital camera 10 is selected, and the unadjusted parallax amount when the subject is located at this distance is detected.

Then, the value of the stereoscopic adjustment parameter C is determined such that the detected unadjusted parallax amount is adjusted to the target value of the parallax amount. Thereby, when the unadjusted parallax amount (for example, the parallax amount m ₁₀ ) is in the range of −m to + m, the value of the stereoscopic adjustment parameter C (for example, the value C) such that the unadjusted parallax amount is generally maintained. ₁₀ ) is determined.

On the other hand, when the detected unadjusted parallax amount (for example, the parallax amount m ₂₀ ) is outside the range of −m to + m, the stereoscopic image is adjusted so that the unadjusted parallax amount is within the range of −m to + m. The value of the adjustment parameter C (for example, the value C ₂₀ ) is determined. Therefore, since the parallax amount can be suppressed in the subject that is not the main subject while the stereoscopic subject is left as the main subject, it is possible to reduce a sense of discomfort and fatigue during viewing.

In order to calculate the value of the stereoscopic adjustment parameter C that adjusts the unadjusted parallax amount for a certain subject to the target value of the parallax amount, first, as described above, a certain stereoscopic adjustment parameter C is used. After the 3D image is generated, a process for calculating the parallax amount of the subject is performed. Then, by repeating this process while feeding back the calculation result, the value of the stereo adjustment parameter C when the unadjusted parallax amount is adjusted to the target value of the parallax amount is calculated. Preferably, a lookup table is generated in advance from the correspondence between the unadjusted parallax amount obtained in this way, the target value of the parallax amount, and the value of the stereoscopic adjustment parameter C, and this lookup table is used. Then, the value of the three-dimensional adjustment parameter C is calculated.

Then, the unadjusted parallax amount for this point and the value of the stereoscopic adjustment parameter C are stored in the lookup table 2310 in association with each other. Thereafter, similarly, another point on the horizontal axis is selected, and the unadjusted parallax amount corresponding to that point and the value of the three-dimensional adjustment parameter C are stored in the lookup table 2310 in association with each other. As a result, a lookup table 2310 is generated.

According to the lookup table 2310 as described above, the value of the stereoscopic adjustment parameter C can be determined from the unadjusted parallax amount without detecting the subject distance. In other words, the value of the stereoscopic adjustment parameter C can be determined by the lookup table 2310 as long as an unadjusted parallax amount can be detected regardless of the arrangement of the subject in the scene.

Here, the target value of the parallax amount is determined using the adjusted parallax amount curve 1623, and the depth relationship between the subjects (the relationship between the distance from the digital camera 10 and the parallax amount) is not broken in this curve. Therefore, as long as the depth relationship of the subject is not broken in the captured image, the depth relationship of the subject is not broken even if the parallax amount is adjusted.

Therefore, the same lookup table 2310 can be used regardless of the imaging conditions that affect the amount of parallax (aperture value, focus position, focal length when the taking lens 20 is a zoom lens). . Hereinafter, this point will be described with a specific example.

FIG. 10 is a diagram illustrating the relationship between the amount of parallax and the target value of the amount of parallax when the shape of the parallax amount curve varies depending on the shooting conditions. In this figure, as in FIG. 8, the horizontal axis represents the distance from the digital camera 10, and the vertical axis represents the amount of parallax.

The digital camera 10 focuses on a subject (see the focused subject in the figure) located at the distances L ₁₁ and L ₁₂ . Also, between the two diagrams shown in FIGS. 10A and 10B, the focal position and the aperture value as the shooting conditions that affect the parallax amount are changed. As a result, the parallax amount curve 1626 is changed. 1628 are different from each other.

In the scenes shown in FIGS. 10A and 10B, the distance from the digital camera 10 to each subject is different from each other. Specifically, the distance from the digital camera 10 to the in-focus subject, in a scene represented by FIG. 10 (a) has a distance L _11, the distance L ₁₂ in the scene represented by FIG. 10 (b) It has become.

Further, the distance from the digital camera 10 to the near-point subject is the distance L _{21 in the} scene shown in FIG. 10A and the distance L _{22 in the} scene shown in FIG. Yes. The distance from the digital camera 10 to the far point subject is the distance L _{31 in the} scene shown in FIG. 10A and the distance L _{32 in the} scene shown in FIG. Yes.

On the other hand, the unadjusted parallax amount for the focused subject is the parallax amount m ₁₀ (= 0). Moreover, any parallax amount unadjusted for near-point object is a parallax amount m _20, the parallax amount unadjusted for far point object has a both parallax amount m _30.

In such a case, when the stereoscopic adjustment parameter C is determined using the look-up table 2310, the unadjusted parallax amount m ₁₀ for the focused subject, the near point subject, and the far point subject in FIG. , M ₂₀ , m ₃₀ are respectively adjusted so that the parallax amounts m ₁₀₀ , m ₂₀₀ , m ₃₀₀ are adjusted. Similarly, the unadjusted parallax amounts m ₁₀ , m ₂₀ , and m ₃₀ become parallax amounts m ₁₀₀ , m ₂₀₀ , and m ₃₀₀ for the focused subject, the near point subject, and the far point subject in FIG. The value of the three-dimensional adjustment parameter C is determined so as to be adjusted.

Similarly, assuming that the subject is located at each distance from the digital camera 10, the relationship between the distance and the target value of the parallax amount generated by the stereoscopic adjustment parameter C for each subject is shown in FIG. a) and

curves

1627 and 1629 represented by bold broken lines in FIG. Therefore, regardless of the imaging conditions that affect the amount of parallax (aperture value, focus position, focal length when the taking lens 20 is a zoom lens), the same lookup table 2310 is used to adjust the stereoscopic adjustment parameter C. It can be seen that the value can be determined.

Then, according to the lookup table 2310 described above, the same effect as when the function of the adjusted parallax amount curve 1623 is stored in the digital camera 10 for each imaging condition can be obtained. That is, since the main subject can have a stereoscopic effect and the amount of parallax can be suppressed in a subject that is not the main subject, it is possible to reduce a sense of discomfort and fatigue during viewing.

Next, the above processing will be described from the viewpoint of pixel value distribution and color. FIG. 11 is a diagram for explaining changes in RGB pixel value distribution. FIG. 11A shows a G (Lt) pixel, a G (Rt) pixel, and an R (Lt) pixel when a white subject light beam from an object point located at a position deviated by a certain amount from the focal position is received. , R (Rt) pixels, B (Lt) pixels, and B (Rt) pixels.

FIG. 11B shows R (N) pixels, G (N) pixels, and B (N) pixels that are non-parallax pixels when a white subject light beam from the object point in FIG. 11A is received. It is the graph which arranged the output value. It can be said that this graph also represents the pixel value distribution of each color.

When the above processing is performed for each corresponding pixel with C = 0.8, the pixel value distribution represented by the graph in FIG. As can be seen from the figure, a distribution corresponding to the pixel values of RGB is obtained.

Next, the relationship between the viewer and the video when 3D image data is played back by a playback device will be described. FIG. 12 is a diagram illustrating the relationship between the vergence angle of the viewer and the amount of parallax. The eyeball 50 represents the eyeball of the viewer, and the figure shows the right eye 51 and the left eye 52 being separated.

The display unit 40 reproduces non-adjusted image data whose parallax amount is not adjusted, and displays a subject 61 for the right-eye image and a subject 62 for the left-eye image. Object 61 and the object 62 are the same object, so were present at a position shifted from the focal position at the time of shooting, the display unit 40 is displayed at a distance with a disparity amount D _1.

Since the eyeball 50 is intended to be viewed with the eyeballs coincident with each other, the viewer views the position of the lifting distance L1 (in the drawing) where the straight line connecting the right eye 51 and the subject 61 and the straight line connecting the left eye 52 and the subject 62 intersect. (Represented by a square).

The convergence angle at this time is θ ₁ as shown in the figure. In general, when the angle of convergence is increased, the video is uncomfortable and causes eye strain. Therefore, when image processing is performed using the stereoscopic adjustment parameter in the present embodiment, adjusted image data in which the parallax amount is adjusted by the stereoscopic adjustment parameter as described above is generated. The figure shows a state where the adjusted image data is reproduced over the non-adjusted image data.

The display unit 40 displays a subject 71 of the right-eye image and a subject 72 of the left-eye image of the adjustment image data. The subject 71 and the subject 72 are the same subject, and the

subjects

61 and 62 are also the same subject. Object 71 and the object 72, the display unit 40 is displayed at a distance with a disparity amount D _2. The viewer recognizes that the subject exists at the position of the lifting distance L2 (represented by a triangle in the figure) where the straight line connecting the right eye 51 and the subject 71 intersects with the straight line connecting the left eye 52 and the subject 72.

The convergence angle at this time is θ ₂ smaller than θ ₁ . Therefore, the viewer can feel the extreme feeling of lifting and can reduce the accumulation of eye strain. Note that the amount of parallax is appropriately adjusted as will be described later, so that the viewer can appreciate the video with a comfortable floating feeling (a three-dimensional effect with a feeling of depression when the defocus relationship is reversed).

In addition, although the parallax amount used as description of FIG. 12 was represented by the separation distance in the display part 40, a parallax amount can be defined in various formats. You may define by the pixel unit in picked-up image data, and you may define by the shift | offset | difference width with respect to the horizontal width of an image.

Further, the adjustment of the parallax amount can be executed by various methods without using the method of changing the value of the three-dimensional adjustment parameter C. Hereinafter, a method of adjusting the parallax amount without changing the value of the three-dimensional adjustment parameter C will be described.

FIG. 13 is a diagram schematically illustrating the relationship between the contrast indicating the sharpness of an image and the amount of parallax. The horizontal axis represents the distance from the digital camera 10, and the vertical axis represents the amount of parallax and the height of contrast. The digital camera 10 is focused on the main subject is located at a distance L _p.

The contrast curve 1610 forms the highest convex curve at the distance L _p that is the distance to the focal position. That illustrates how gradually blurred with increasing distance from the distance L _p back and forth.

Parallax amount curve 1620, at a distance L _p indicates parallax amount 0, than the distance L _p approaches the digital camera 10 side, shows the curve slope increases. That is, the parallax amount curve 1620 shows a positive value on the near side of the distance L _p , and indicates that the closer the subject is, the higher the image is visually recognized.

On the other hand, the parallax amount curve 1620 than the distance L _p As the distance from the digital camera 10 side, shows the curve slope becomes smaller. That is, the parallax amount curve 1620 distance L _p from indicate a negative value in the back side, it represents that it is visible sinks slowly as more distant object.

If the viewer does not feel discomfort and fatigue when the amount of parallax is in the range of −m to + m, the subject composing the scene moves from the distance L _f (the amount of parallax at this time is + m) to the distance L _r ( The amount of parallax at this time may be distributed between -m). That is, if the closest near subject from the digital camera 10 exists at the distance L _f and the farthest far subject exists at the distance L _r , the viewer can adjust the amount of parallax without adjusting the amount of parallax in the subsequent image processing. You can enjoy 3D video comfortably. On the other hand, when the near-point object is in front of the distance L _f than the distance L _f _'(parallax amount at this time is + m') are present, since exceeds the parallax amount allowed, viewer discomfort, fatigue Learn.

Furthermore, we will continue to explain the relationship between the subject distribution and the amount of parallax. FIG. 14 is a diagram schematically illustrating the relationship between the subject distribution and the amount of parallax.

Each diagram in FIG. 14 corresponds to the diagram in FIG. 11 excluding the contrast curve 1610. In addition, it is assumed that there are a near point subject and a far point subject in addition to the focused subject as the main subject to be focused. In FIG. 14 (a), the in-focus object distance _{L 10,} near point subject to _{L 20,} far point object is present in _{L 30.}

When the parallax amount range set as the allowable range is from −m to + m, the value of the parallax amount curve 1620 with respect to the distance L ₃₀ of the far-point subject is within this range. However, the value of the parallax amount curve 1620 with respect to the distance _{L 20} of near-point object is over + m.

14 (b) is a diagram of the subject situation, showing the concept of parallax amount when focus object is moved from the distance L ₁₀ to the back side of the distance L ₁₁ in FIG. 14 (a). In this case, the distance L ₁₁ is the focal position, the parallax amount relative to the image of the near point object has not moved (the distance L _20), as indicated by the parallax amount curve 1620, in comparison with FIG. 14 (a) It becomes considerably large. That is, the excess amount from the allowable range increases.

FIG. 14 (c), shows the object status of FIG. 14 (b), the near point object from the distance _{L 20} to the back side of the distance _{L 21,} the concept of parallax amount when further moved to the distance _{L 22} It is. Although focus position parallax amount curve 1620 remain such because of the distance L ₁₁ draw the same curve as FIG. 14 (b), the by near point object is shifted to the rear side, the parallax amount at the time of the distance L ₂₁ is acceptable Although exceeding the range, the excess amount is smaller than the excess amount of FIG. If further moved until the distance L _22, the parallax amount is within an allowable range.

That is, it can be said that the subject distribution in the depth direction with respect to the scene and the position of the subject to be focused are parameters that determine whether or not the parallax amount falls within the set allowable range.

Next, the relationship between the aperture value and the parallax amount will be described. FIG. 15 is a diagram schematically illustrating the relationship between the aperture value and the amount of parallax. Similarly to FIG. 13, the horizontal axis represents the distance from the digital camera 10, and the vertical axis represents the amount of parallax and the height of contrast. 15A shows a state where the aperture value is F1.4, FIG. 15B shows a state where the aperture value is F4, and FIG. 15C shows a state where the aperture value is F8. To express. Incidentally, the focal length of the taking lens 20 are the same in both states, also, the digital camera 10 is focused on the main subject is located at a distance L _10.

Contrast curve 1610, the highest in the distance L ₁₀ is a distance to be focal positions in any state. On the other hand, as the aperture 22 is reduced, that is, as the aperture value is increased, a relatively high value is obtained even before and after the focal length. That is, it shows that the depth of field becomes deeper as the image is taken with the aperture 22 being reduced. Parallax amount curve 1620, at a distance _{L 10} shows the parallax amount 0, approaches the digital camera 10 side than the distance _{L 10,} shows the curve slope increases. On the other hand, the parallax amount curve 1620, as the distance from the digital camera 10 side than the distance _{L 10,} shows the curve slope becomes smaller.

The parallax amount curve 1620 becomes gentler as the aperture value increases. That is, as compared with the case where the aperture value is F1.4, the amount of parallax in front of the focal position and the amount of parallax in the back become smaller as F4 and F8 change. If the viewer does not feel discomfort and fatigue when the amount of parallax falls within the range of −m to + m, the entire parallax amount curve 1620 falls within this range when the aperture value is F8. Even if the subject is present at any distance, the viewer can comfortably appreciate the 3D video.

On the other hand, when the aperture values are F1.4 and F4, the parallax amount exceeds + m on the short distance side of the parallax amount curve 1620. Specifically, in the case of F1.4, than the distance _{L 24} exceeds the + m in front of the region, in the case of F4, exceeds + m in front of the area than the distance _{L 25.} In this case, since the slope of the parallax amount curve 1620 at F4 is gentler than the slope of the parallax amount curve 1620 at F1.8, the relationship of L ₂₅ <L ₂₄ is established. At these aperture values, if the subject is present at a distance closer than the distance L ₂₄ or the distance L ₂₅ , the viewer will feel uncomfortable and tired when viewing the captured 3D video.

Therefore, in the present embodiment, the change condition for changing the parallax amount is changed so that the parallax amount between the generated images falls within the target parallax amount (allowable parallax amount: for example, a range of ± m). . Specifically, an imaging condition that affects the amount of parallax is changed, or a value of a stereoscopic adjustment parameter used for image processing is changed.

First, the change of imaging conditions will be described. As described with reference to FIG. 15, the aperture value affects the amount of parallax. Therefore, the aperture value may be changed according to the detected subject distribution so that the amount of parallax between output parallax images is within the allowable amount of parallax. For example, in the situation of FIG. 15A (the initial aperture value is F1.4, the focused subject is the distance L ₁₀ ), and the near-point subject is present at the distance L ₂₅ , the amount of parallax exceeds + m. Therefore, the determination unit 232 changes the aperture value from F1.4, at a distance _{L 25} in the F4 is a aperture value parallax amount is + m with respect to the subject.

The aperture value is changed to a large value not only when the near-point subject exceeds the allowable parallax amount range but also when the far-point subject exceeds the allowable parallax amount range. When the parallax amount of the near point subject and the far point subject has a margin with respect to the allowable parallax amount, the aperture value may be changed to a small value, that is, the direction in which the aperture 22 is opened. In this case, the shutter speed can be changed to the high speed side, and the ISO sensitivity can be changed to the low sensitivity side.

The relationship between the in-focus subject distance and the parallax amount curve 1620 for each aperture value is prepared in advance as a lookup table. The determination unit 232 can extract and determine the aperture value to be changed by referring to the lookup table with the subject distribution and the allowable parallax amount as input values.

In addition to changing the aperture value, there is a focus shift method for changing the focus position as a change in the imaging condition. FIG. 16 is a diagram schematically showing the concept of focus shift. The vertical axis and the horizontal axis are the same as those in FIG.

Contrast curves 1610 and the parallax amount curve 1620, focusing subject exists at a distance L _10, represents the contrast curve and a parallax amount curve when focused on the subject by moving the focus lens. In this case, the peak value of the contrast curve 1610, is higher than the focus threshold E _s is evaluated with focusing.

When the near-point subject exists at the position of the distance L ₂₇ , the parallax amount is + m _{0 when} referring to the parallax amount curve 1620, which exceeds the allowable parallax amount + m. Therefore, in the focus shift to correct the focus lens position in a range above the focus threshold E _s, it falls within the acceptable range parallax amount at the distance L _27.

In the case of the example in the figure, a parallax amount curve 1621 where the parallax amount with respect to the near-point subject is + m is selected, and a distance L _p where the parallax amount is 0 in the parallax amount curve 1621 is extracted. Then, by changing the focus lens position, the distance L _p as the focusing position. The contrast curve 1611 is a contrast curve at this time. Since the object is present in the distance L ₁₀ in fact, the contrast value for the subject is reduced by Δe as shown. Contrast value at this time has only to above the focus threshold E _s. In this way, an image shot with the focus lens position changed can be evaluated as in-focus as the image, although the contrast value for the main subject is slightly reduced, and the parallax amount for the near-point subject is within an allowable range. Yes.

If the contrast value for the distance L _p does not exceed the focus threshold E _s, the correction of the focusing lens position is not allowed. That is, when the parallax amount relative to the near-point object in parallax amount curve 1620 largely exceeds the allowable amount of parallax, changing the focus lens position in a range above the focus threshold E _s, fit the parallax amount within the allowable range I can't. In this case, it may be used in combination with other methods such as changing the aperture value to a large value.

In the case of adjusting the parallax amount by focus shift, a lookup table prepared in advance as a relationship between the focused subject distance and the parallax amount curve for each aperture value may be used. The determination unit 232 can extract and determine the distance L _p by referring to the lookup table using the subject distribution and the allowable parallax amount as input values. The control unit 201 corresponds to the distance L _p to change the position of the focus lens. Control unit 201, the contrast value obtained as a result of determining whether above the focus threshold E _s. If it is determined that it exceeds, the photographing sequence is continued as it is. If it is determined that the value does not exceed, the focus lens position is returned and the control shifts to another method. Alternatively, actual control unit 201 without moving the focus lens, whether the focal position is the attenuation of contrast when shifted to the L _p determination unit 232 calculates the L _10, above the focus threshold E _s It may be judged. In this case, for example, if the contrast AF method, when the focus adjustment with respect to the distance L _10, may also refer to the actual evaluation value has already been obtained.

Here, as described above, the lookup table 2310 for determining the value of the three-dimensional adjustment parameter C is an imaging condition that affects the amount of parallax (a diaphragm value, a focus position, and a case where the photographing lens 20 is a zoom lens). The same can be used regardless of the focal length. Therefore, the parallax amount adjustment method as described above can be used in combination with a method of adjusting the parallax amount by changing the value of the three-dimensional adjustment parameter C.

FIG. 17 is a diagram schematically illustrating the relationship between the parallax amount, the value of the three-dimensional adjustment parameter, the aperture value, and the like. In this figure, as in FIG. 8, the horizontal axis represents the distance from the digital camera 10, and the vertical axis represents the amount of parallax.

The digital camera 10, the object located at a distance L ₁₀ (see focus object in the drawing) are focused. However, the image captured by the digital camera 10, in addition to the subject located at a distance L _10, the distance L ₂₀ and the distance L 2 one object located ₃₀ (see a near point object and far-point object in the drawing) It is included. The unadjusted parallax amounts for these three subjects are the parallax amount m ₁₀ , the parallax amount m _20, and the parallax amount m ₃₀ .

When the parallax amount is adjusted by using the method by changing the value of the three-dimensional adjustment parameter C and another method, first, the determination unit 232 adjusts the parallax amount by changing the aperture value. Thus, the parallax amount curve 1622 is transformed into the parallax amount curve 1624, and the parallax amounts given to the focused subject, the near point subject, and the far point subject become the parallax amount m ₁₀₁ , the parallax amount m _201, and the parallax amount m ₃₀₁ . Note that the focus position may be changed instead of the aperture value.

Next, the calculation unit 231 does not adjust the parallax amount m ₁₀₁ , the parallax amount m _201, and the parallax amount m ₃₀₁ adjusted by changing the aperture value, the focus position, and the like, depending on the value of the stereoscopic adjustment parameter C. The parallax amount for adjustment is calculated from the left and right parallax image data. Then, as described with reference to FIG. 10, the determination unit 232 uses the lookup table 2310 and uses the look-up table 2310 to create a solid corresponding to the unadjusted parallax amount m ₁₀₁ , parallax amount m _201, and parallax amount m _301. The value of the adjustment parameter C is determined for each subject.

The value of the stereo adjustment parameter C determined in this way is equal to the value determined by the following method. That is, first, the adjusted parallax amount curve 1625 is generated with respect to the parallax amount curve 1624 instead of the parallax amount curve 1622. Then, the value of the stereoscopic adjustment parameter C is calculated such that the unadjusted parallax amount is adjusted to the target value of the parallax amount. In this case, for a subject whose parallax amount exceeds the allowable parallax amount, the parallax is suppressed so that the viewer does not feel discomfort or fatigue, and for a subject whose parallax amount falls within the allowable parallax amount, the parallax is more emphasized. be able to.

Therefore, if the adjustment of the parallax amount by changing the imaging condition and the adjustment of the parallax amount by changing the value of the three-dimensional adjustment parameter C are performed together as described above, an adjusted parallax amount curve having a different shape can be obtained. The same effect as that used when determining the value can be obtained. Therefore, variations in the adjustment amount of the parallax amount can be increased.
It should be noted that a function used to determine the target value of the parallax amount can be set in advance so that the adjusted parallax amount curve 1625 is obtained for each imaging condition. In this case, the amount of data increases, but the amount of parallax can be adjusted without changing the shooting conditions.

FIG. 18 is a diagram for explaining subject designation. In particular, FIG. 18A shows a subject distribution in the depth direction from the digital camera 10 in a certain scene, and FIG. 18B is a rear view of the digital camera 10 displaying the scene in a live view.

As shown in FIG. 18A, the scene is composed of an adult 301 (distance L _f ), a boy 302 (distance L _p ), and a girl 303 (distance L _r ) in order from the digital camera 10. Then, as shown in FIG. 18B, a live view image that captures this scene is displayed on the display unit 209. Here, the boy 302 is the focused subject.

In the description so far, as described with reference to FIG. 8, for the subject whose unadjusted parallax amount is outside the range of −m to + m, the three-dimensional adjustment that increases the parallax amount adjustment amount. The value of parameter C was calculated. In addition, for a subject whose unadjusted parallax amount is within the range of −m to + m, the value of the stereoscopic adjustment parameter C was calculated so that the unadjusted parallax amount is generally maintained.

However, regardless of whether or not the amount of parallax is within the range of −m to + m, the amount of parallax can be suppressed by increasing the amount of adjustment of the amount of parallax for subjects that are not the main subject for the photographer. Is preferred. Accordingly, it is not always necessary to maintain the amount of parallax for a subject whose amount of parallax is in the range of −m to + m.

For example, if the boy 302 and the girl 303 are the main subjects in the scene of FIG. 18A, it is assumed that the viewer gazes at these two people at the time of viewing. Therefore, the parallax amount may be suppressed so that the viewer does not feel discomfort and fatigue with respect to the subject image of the adult 301. Therefore, the control unit 201 receives a photographer's instruction as to which subject should be the main subject.

The display unit 209 displays a title 320 (for example, “Please set the main subject area”) indicating that the user instruction is accepted. In this state, the user adjusts the position and size of the frame 310 to specify a range of the area including the subject image that is desired to be the main subject. The display unit 209 is provided with a touch panel 2083 as a part of the operation unit 208, and the control unit 201 acquires the output of the touch panel 2083 and determines which subject is the main subject.

Thereby, the subject of the adult 301 not included in the frame 310 is designated as the subject whose parallax amount is desired to be reduced. For the subject of the adult 301, the determination unit 232 determines that the unadjusted parallax amount is 0.5 from the value calculated from the lookup table 2310 regardless of whether or not the unadjusted parallax amount is within the range of −m to + m. A value close to is determined as the value of the three-dimensional adjustment parameter C. However, in this case, it is preferable to determine the value of the three-dimensional adjustment parameter C so that the depth relationship between the subject of the adult 301 and another subject does not collapse.

Thereby, in the subject of the adult 301, the adjustment amount of the parallax amount becomes larger and the parallax amount is suppressed as compared with the case where the value of the stereoscopic adjustment parameter C calculated from the lookup table 2310 is used as it is. As a result, the amount of adjustment of the parallax amount for each subject is, for example, as shown superimposed on each subject in FIG.

Here, these adjustment amounts mean that the degree of adjustment of the parallax amount is larger as the absolute value is larger. A positive value (for example, +2) means that the amount of parallax is adjusted in a direction in which the distance from the digital camera 10 increases, and a negative value (for example, −1) decreases the distance from the digital camera 10. This means that the amount of parallax is adjusted in the direction.

As can be seen from these adjustment amounts, according to the processing as described above, the adjustment amount of the parallax amount is reduced and the stereoscopic effect is maintained in the boy 302 and the girl 303 as the main subjects. On the other hand, for the adult 301 who is not the main subject, the amount of parallax adjustment is increased and the amount of parallax is reduced.

Next, a processing flow in moving image shooting of the digital camera 10 will be described. As described above, in order for the amount of parallax between generated images to fall within the allowable amount of parallax, the case where the value of the stereoscopic adjustment parameter C used for image processing is changed, the imaging conditions that affect the amount of parallax, and The value of the three-dimensional adjustment parameter C may be changed.

<First embodiment>
As a first embodiment, a processing flow of moving image shooting for adjusting the amount of parallax by changing the value of the stereoscopic adjustment parameter C used for image processing will be described. FIG. 19 is a processing flow in moving image shooting according to the first embodiment.

Note that the flow in the figure starts when the mode button is operated by the photographer and the auto 3D video mode is started. The parallax amount range is set in advance by the photographer.

When the auto 3D moving image mode is started, the determination unit 232 acquires the parallax amount range set by the photographer from the system memory in step S11. In step S12, the control unit 201 executes AF and AE.

Proceeding to step S13, the control unit 201 receives from the user, via the touch panel 2083, designation of a range of an area including a subject image that is desired to be a main subject, as described with reference to FIG. In step S15, the control unit 201 waits for a recording start instruction for the photographer to press the recording start button.

When the recording start instruction is detected (YES in step S15), the control unit 201 proceeds to step S17. If no instruction is detected, the process returns to step S12. Note that after returning to step S12, the specified subject may be tracked and the process of step S13 may be skipped.

In step S17, the control unit 201 executes AF and AE again according to the changed imaging condition. In step S18, the control unit 201 performs charge accumulation and readout of the image sensor 100 via the drive unit 204, and acquires captured image data as one frame. The amount of parallax between the parallax images in the captured image data acquired here does not fall within the set amount of parallax depending on the subject distribution and the shooting conditions.

In step S33, the parallax image data generation unit 233 receives the value of the stereoscopic adjustment parameter C determined by the determination unit 232 and the captured image data, and receives color image data (RLt _c plane data, GLt _c plane data, left viewpoint) BLt _c plane data) and right viewpoint color image data (RRt _c plane data, GRt _c plane data, BRt _c plane data) are generated. Specific processing will be described later.

In step S19, for the case where the photographer wants to change the main subject during moving image shooting, the control unit 201 accepts from the user the range specification of the area including the subject image that he wants to be the main subject. If the control unit 201 determines in step S21 that it has not received a recording stop instruction from the photographer, it returns to step S17 and executes the next frame process. If it is determined that a recording stop instruction has been received, the process proceeds to step S22.

In step S22, the moving image generating unit 234 connects the continuously generated left-viewpoint color image data and right-viewpoint color image data, and executes format processing according to a 3D-compatible moving image format such as Blu-ray 3D. Generate a file. Then, the control unit 201 records the generated moving image file on the memory card 220 via the memory card IF 207, and ends a series of flows.

Note that the recording to the memory card 220 may be sequentially executed in synchronization with the generation of the color image data of the left viewpoint and the color image data of the right viewpoint, and the file end process may be executed in synchronization with the recording stop instruction. Further, the control unit 201 is not limited to recording on the memory card 220, and may be configured to output to an external device via a LAN, for example.

FIG. 20 is a processing flow of step S33 until the color image data of the left viewpoint and the parallax color image data which is the color image data of the right viewpoint are generated.

The parallax image data generation unit 233 acquires captured image data in step S101. In step S102, as described with reference to FIG. 3, the captured image data is plane-separated into image data without parallax and parallax image data. In step S103, the parallax image data generation unit 233 executes an interpolation process for interpolating vacancies existing in the separated plane data as described with reference to FIG.

The parallax image data generation unit 233 initializes each variable in step S104. Specifically, first, 1 is substituted into the color variable Cset. The color variable Cset represents 1 = red, 2 = green, 3 = blue. Also, 1 is substituted into the coordinate variables i and j. Further, 1 is substituted into the parallax variable S. The parallax variable S represents 1 = left, 2 = right.

In step S105, the parallax image data generation unit 233 extracts a pixel value from the target pixel position (i, j) of the Cset plane. For example, Cset = 1, when a target pixel position (1,1), pixel values to be extracted is _{Rn 11.} Further, in step S106, the parallax image data generation unit 233 extracts a pixel value from the target pixel position (i, j) of the Lt _Cset plane data and the Rt _Cset plane data. For example, when the target pixel position is (1, 1), the pixel values to be extracted are Lt _Cset11 and Rt _Cset11 .

The calculation unit 231 calculates an unadjusted parallax amount for the subject displayed at the pixel at the target pixel position (i, j) in step S107 from the left and right parallax image data. In step S <b> 108, the determination unit 232 acquires the value of the stereoscopic adjustment parameter C corresponding to the parallax amount calculated by the calculation unit 231 from the lookup table 2310.

At this time, as described with reference to FIG. 18, the determination unit 232 applies to a subject that is determined not to be a main subject by the photographer even if the amount of parallax is within a range of −m to + m. A value closer to 1 than the value calculated from the lookup table 2310 may be determined as the value of the stereoscopic adjustment parameter C. Further, the determination unit 232 may calculate the luminance value of the subject included in the image, and further adjust the value of the stereoscopic adjustment parameter C so that the parallax amount decreases as the luminance value decreases.

In step S109, the parallax image data generation unit 233 calculates the pixel value of the target pixel position (i, j) corresponding to the parallax variable S. For example, when Cset = 1 and S = 1 and the target pixel position is (1, 1), RRLt _C11 is calculated. Specifically, for example, it is calculated by the above-described equation (1). Here, the value of the stereoscopic adjustment parameter C is the value determined in step S108 for the subject displayed by the target pixel.

The parallax image data generation unit 233 increments the parallax variable S in step S110. In step S111, it is determined whether or not the parallax variable S exceeds 2. If not, the process returns to step S109. If it exceeds, the process proceeds to step S112.

In step S112, the parallax image data generation unit 233 assigns 1 to the parallax variable S and increments the coordinate variable i. Then, in step S113, it is determined whether coordinate variable i exceeds _{i 0.} If not, the process returns to step S105. If it exceeds, the process proceeds to step S114.

In step S114, the parallax image data generation unit 233 assigns 1 to the coordinate variable i and increments the coordinate variable j. Then, in step S115, it is determined whether coordinate variable j exceeds _{j 0.} If not, the process returns to step S105. If it exceeds, the process proceeds to step S116.

When the process proceeds to step S116, since the pixel values of all the left and right pixels for Cset are aligned, the parallax image data generation unit 233 arranges these pixel values and generates plain image data. For example, when Cset = 1, RLt _c plane data and RRt _c plane data are generated.

In step S117, the parallax image data generation unit 233 assigns 1 to the coordinate variable j and increments the color variable Cset. In step S118, it is determined whether or not the color variable Cset exceeds 3. If not, the process returns to step S105. If it exceeds, color image data of the left viewpoint (RLt _c plane data, GLt _c plane data, BLt _c plane data) and color image data of the right viewpoint (RRt _c plane data, GRt _c plane data, BRt _c plane data) If all of the above are complete, the flow returns to the flow of FIG.

<Second embodiment>
As a second embodiment, a processing flow of moving image shooting for adjusting the parallax amount by changing the imaging condition that affects the parallax amount and the value of the three-dimensional adjustment parameter C will be described. FIG. 21 is a processing flow in moving image shooting according to the second embodiment. The processing related to each processing in the processing flow of FIG. 19 is denoted by the same step number, and the description thereof is omitted except for the description of different processing and additional processing.

In this flow, when the control unit 201 detects a recording start instruction in step S15 (YES in step S15), the process proceeds to step S16. In step S16, the determination unit 232 changes the shooting condition, and proceeds to step S17.

Specifically, in step S16, the determination unit 232 changes the aperture value as described with reference to FIG. 15 or changes the focus lens position as described with reference to FIG. In addition to this, an imaging condition that affects the amount of parallax may be changed so as to be within the set range of the amount of parallax. For example, if the photographing lens 20 is a zoom lens, the focal length can be changed.

In the process of step S107 in the process of step S33, the calculation unit 231 calculates an unadjusted parallax amount for the subject displayed by the pixel at the target pixel position (i, j) from the left and right parallax image data. This parallax amount is the parallax amount adjusted by changing the aperture value, the focus position, and the like. Also, in this flow, if the control unit 201 determines in step S21 that a recording stop instruction has not been received from the photographer, the control unit 201 returns to step S16 and executes the next frame processing.

Next, a preferable opening shape of the opening mask described with reference to FIG. 2 will be described. FIG. 22 is a diagram illustrating a preferred opening shape.

It is preferable that the opening part 105 of the parallax Lt pixel and the opening part 106 of the parallax Rt pixel are deviated in directions opposite to each other including the center with respect to the corresponding pixel. Specifically, it is preferable that each of the

openings

105 and 106 has a shape in contact with a virtual center line 322 passing through the center of the pixel or a shape straddling the center line 322.

In particular, as illustrated, the shape of the opening 105 and the shape of the opening 106 are preferably the same as the respective shapes obtained by dividing the shape of the opening 104 of the non-parallax pixel by the center line 322. In other words, the shape of the opening 104 is preferably equal to the shape in which the shape of the opening 105 and the shape of the opening 106 are adjacent to each other.

In the above description, the calculation formula used by the parallax image data generation unit 233 employs the above formulas (1) and (2) using the weighted arithmetic mean, but not limited to this, various calculation formulas are adopted. Can do. For example, if the weighted geometric mean is used, it can be expressed in the same manner as the above formulas (1) and (2)

Can be adopted as a calculation formula. In this case, the amount of blur maintained is not the amount of blur due to the output of the non-parallax pixel but the amount of blur due to the output of the parallax pixel.

Moreover, as another calculation formula, it represents like the said Formula (1) (2),

May be adopted. In this case, when calculating GLt _cmn , GRt _cmn , BLt _cmn , and BRt _cmn , the cubic root term does not change.

Moreover,

May be adopted. Also in this case, when calculating GLt _cmn , GRt _cmn , BLt _cmn , and BRt _cmn , the cubic root term does not change.

Next, cooperation with the display device will be described. FIG. 23 is a diagram for explaining cooperation between the digital camera 10 and the TV monitor 80. The TV monitor 80 includes a display unit 40 made of, for example, liquid crystal, a memory card IF 81 that receives the memory card 220 taken out from the digital camera 10, a remote controller 82 that is operated by a viewer at hand, and the like. The TV monitor 80 is compatible with 3D image display. The display format of the 3D image is not particularly limited. For example, the right-eye image and the left-eye image may be displayed in a time-division manner, or may be an interlace in which strips are arranged in a horizontal or vertical direction. Further, it may be a side-by-side format arranged on one side and the other side of the screen.

The TV monitor 80 decodes a moving image file that includes the color image data of the left viewpoint and the color image data of the right viewpoint, and displays a 3D image on the display unit 40. In this case, the TV monitor 80 serves as a general display device that displays a standardized moving image file.

However, the TV monitor 80 can also function as an image processing apparatus that bears at least part of the function of the control unit 201 and at least part of the function of the image processing unit 205 described with reference to FIG. Specifically, an image processing unit including the calculation unit 231, the determination unit 232, the parallax image data generation unit 233, and the moving image generation unit 234 described in FIG. 1 is incorporated in the TV monitor 80. By configuring in this way, it is possible to realize function sharing different from the function sharing by the combination of the digital camera 10 and the TV monitor 80 in the second embodiment.

Specifically, the digital camera 10 does not perform image processing using the stereoscopic adjustment parameter, and associates the depth information detected by the depth information detection unit 230 with the generated captured image data. As an image processing apparatus, the TV monitor 80 determines the value of the stereoscopic adjustment parameter C for each subject with reference to the associated depth information, and performs image processing using the stereoscopic adjustment parameter C for the acquired image data. Execute. The TV monitor 80 displays the 3D image with the parallax amount adjusted in this way on the display unit 40.

In the above modification, the viewer may be configured to be able to input some adjustment information during playback on the TV monitor 80. For example, the viewer can input the parallax amount range by operating the remote controller 82. The TV monitor 80 acquires the input parallax amount range as adjustment information, and the determination unit 232 determines the value of the stereoscopic adjustment parameter C according to the parallax amount range. If comprised in this way, the TV monitor 80 can display the 3D image according to the preference for every viewer.

According to the above embodiment, the parallax amount of each of the plurality of subjects included in the image of the captured image data is calculated, and the value of the stereoscopic adjustment parameter C is set for each of the plurality of subjects according to the calculated amount of parallax. Then, the value of the stereoscopic adjustment parameter C is applied to the captured image data to generate parallax image data. Therefore, unlike the case where the value of the single stereoscopic adjustment parameter C is applied to the entire captured image data, the amount of parallax adjustment is intentionally different between the main subject and the subject that is not the main subject. be able to. Therefore, since the parallax amount can be suppressed in the subject that is not the main subject while the stereoscopic subject is left as the main subject, it is possible to reduce a sense of discomfort and fatigue during viewing.

In addition, for a subject whose parallax amount exceeds a threshold value among a plurality of subjects included in the image, the value of the stereoscopic adjustment parameter C is determined so that the parallax amount is adjusted to be less than the threshold value. A sense of discomfort and fatigue can be reliably reduced.

Further, when at least some of the plurality of subjects included in the image are continuous in the depth direction, the value of the three-dimensional adjustment parameter C is determined so that the adjusted parallax amount is continuous in the depth direction with respect to these subjects. Is done. Therefore, it is possible to prevent the parallax amount from changing discretely between subjects that are continuous in the depth direction. Therefore, it is possible to reliably reduce a sense of incongruity and fatigue during viewing.

Further, when at least some of the subjects included in the image are continuous in the depth direction, the stereoscopic adjustment parameter C is set so that the differential value of the adjusted parallax amount is continuous in the depth direction with respect to these subjects. The value of is determined. Therefore, the change in the amount of parallax between subjects that are continuous in the depth direction can be smoothed. Therefore, it is possible to reliably reduce a sense of incongruity and fatigue during viewing.

In addition, since the subject for which the amount of parallax is to be suppressed is specified by the user from among a plurality of subjects included in the image by applying the value of the stereoscopic adjustment parameter C, the amount of parallax of the specified subject is suppressed. Accordingly, it is possible to reliably reduce a sense of incongruity and fatigue during viewing.

Also, the brightness value of each of the plurality of subjects included in the image is calculated, and the value of the stereoscopic adjustment parameter C is determined so that the parallax amount decreases as the brightness value decreases. Therefore, it is possible to prevent a large parallax from being given to a region having a small luminance value in the image, that is, a region having low brightness. Therefore, it is possible to reliably reduce a sense of incongruity and fatigue during viewing.

In the above embodiment, the TV monitor 80 has been described as an example of the image processing apparatus. However, the image processing apparatus can take various forms. For example, a device such as a PC, a mobile phone, or a game device that includes or is connected to the display unit can be an image processing apparatus.

Further, although the above embodiment has been described on the premise of moving image shooting, the configuration of outputting image data in which the amount of parallax is adjusted based on the detected depth information can of course be applied to still image shooting. The still image shot in this way does not cause extreme parallax between the left and right images, and does not give the viewer a sense of incongruity.

In the above embodiment, the target subject is accepted by the user's instruction, but the control unit 201 may automatically select the subject. For example, the control unit 201 can set only a human image included in the scene as a target subject through the human recognition process.

Further, in the above embodiment it has been described as generating the parallax image data (Lt _c disparity plane data and Rt _c disparity plane data) from the photographed image data. However, parallax image data may be generated from image data (such as a CG image) that is not a captured image.

In the above embodiment, the determination unit 232 is described as determining the value of the three-dimensional adjustment parameter C using one lookup table 2310. However, a plurality of lookup tables 2310 may be stored in the determination unit 232, and the value of the stereoscopic adjustment parameter C may be determined using the lookup table 2310 selected by the photographer. In this case, the amount of parallax adjustment for each subject can be selected. Note that selecting the lookup table 2310 is equivalent to selecting a function used to determine the target value of the parallax amount, that is, a function such as the adjusted parallax amount curve 1623.

In this case, variations in the amount of parallax adjustment can be increased. Note that the plurality of lookup tables 2310 can be generated from adjusted parallax amount curves having different shapes.

In the above embodiment, the value of the stereo adjustment parameter C is determined using the lookup table 2310. However, the function of the adjustment parallax amount curve 1623 is stored in the digital camera 10 for each imaging condition in accordance with the above-described principle, and the stereoscopic adjustment parameter is obtained using these functions and information on the subject distribution and the focus position in the depth direction. The value of C may be determined. Furthermore, in this case, a plurality of functions of the adjustment parallax amount curve 1623 are stored in the digital camera 10 for the same shooting condition, and the user operates the operation unit 208 to change any one of the functions. You may select as an object of use.

Further, in the above embodiment, when some subjects are continuous in the depth direction of the scene, the three-dimensional adjustment parameter C is set so that the differential value of the adjusted parallax amount is continuous in the depth direction for these continuous subjects. Described as determining the value. However, in such a case, it is not always necessary to determine the value of the three-dimensional adjustment parameter C so that the differential values are continuous.

For example, as shown in FIG. 24, among the plurality of subjects included in the image, subjects whose unadjusted parallax amount is not included in the range of −m to + m, that is, in the region surrounded by the frame W1 in the figure. For the subject located at, the value of the stereoscopic adjustment parameter C may be determined so that the parallax amount is adjusted to ± m. In this case, the differential value is discontinuous at the boundary between the frame W1 and the frame W2. In this case, it is only necessary to adjust the amount of parallax that is not included in the range of −m to + m, so that the processing becomes simple. For the subject included in the range of −m to + m, the parallax is not suppressed, so Will not be lost.

Further, in the above embodiment, when the subjects included in the image are continuous in the depth direction of the scene, the parallax amount is smoothly shifted between these subjects in the depth direction. However, the subject that is continuous in the depth direction of the scene may be a part of the subject included in the image.

In the above embodiment, the allowable parallax threshold has been described as ± m. However, values having different absolute values may be used for the upper limit value and the lower limit value of the allowable amount of parallax.

Each processing flow described in this embodiment is executed by a control program that controls the control unit. The control program is recorded in a built-in nonvolatile memory, and is appropriately expanded in the work memory to execute each process. Alternatively, the control program recorded in the server is transmitted to each device via the network, and is expanded in the work memory to execute each process. Alternatively, a control program recorded on the server is executed on the server, and each device executes processing in accordance with a control signal transmitted via the network.

As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

The execution order of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior”. It should be noted that they can be implemented in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for the sake of convenience, it means that it is essential to carry out in this order. is not.

10 digital camera, 20 taking lens, 21 optical axis, 22 aperture, 50 eyeball, 51 right eye, 52 left eye, 40 display unit, 61, 62, 71, 72 subject, 80 TV monitor, 81 memory card IF, 82 remote control, 100 Image sensor, 104, 105, 106 opening, 110 basic grid, 201 control unit, 202 A / D conversion circuit, 203 memory, 204 drive unit, 205 image processing unit, 207 memory card IF, 208 operation unit, 209 display unit , 210 LCD drive circuit, 220 memory card, 230 depth information detection unit, 231 calculation unit, 232 determination unit, 233 parallax image data generation unit, 234 video generation unit, 301 adult, 302 boy, 303 girl, 310 frame, 320 title 322 center line, 16 0, 1611 Contrast curve, 1620, 1621, 1622, 1624, 1626, 1628 Parallax amount curve, 1623, 1625 Adjusted parallax amount curve, 1627, 1629 curve, 1804, 1805, 1807, 1808 Distribution curve, 1806, 1809 Composite distribution curve , 1901 Lt distribution curve, 1902 Rt distribution curve, 1903 2D distribution curve, 1905 adjusted Lt distribution curve, 1906 adjusted Rt distribution curve, 2083 touch panel, 2310 lookup table

Claims

An acquisition unit for acquiring image data;
A calculation unit for calculating a parallax amount of a subject included in the image of the image data;
A determination unit configured to determine an adjustment value for adjusting the parallax amount, which is applied when generating parallax image data from the image data, according to the parallax amount;
An image processing apparatus comprising: a generation unit configured to apply the adjustment value to the image data and generate parallax image data that becomes an image having the adjusted parallax amount.
The determination unit determines the adjustment value so that the parallax amount is adjusted to be less than the threshold when the absolute value of the parallax amount calculated by the calculation unit exceeds a predetermined threshold. The image processing apparatus according to 1.
The said determination part determines the said adjustment value so that the said parallax amount adjusted regarding the said subject may be continued in the said depth direction when the said subject continues in the depth direction of a scene. Image processing apparatus.
The image processing according to claim 3, wherein the determination unit determines the adjustment value so that a differential value of the adjusted parallax amount is continuous in the depth direction when the subject is continuous in a depth direction of the scene. apparatus.
The calculating unit calculates a parallax amount of each of the plurality of subjects;
5. The image processing apparatus according to claim 1, wherein the determination unit determines the adjustment value for each of the plurality of subjects according to the calculated amount of parallax.
The image processing apparatus according to claim 1, further comprising a selection unit that causes a user to select a function used by the determination unit in order to determine the adjusted amount of parallax.
The image processing apparatus according to any one of claims 1 to 6, further comprising a designation unit that allows a user to designate a subject for which the amount of parallax is to be suppressed by applying the adjustment value.
A calculation unit for calculating a luminance value of the subject;
The image processing apparatus according to claim 1, wherein the determination unit determines the adjustment value so that the parallax amount decreases as the luminance value decreases.
The image data includes first parallax image data and second parallax image data having parallax with each other,
The generating unit applies the adjustment value to the first parallax image data and the second parallax image data, and a parallax amount between the images of the first parallax image data and the second parallax image data. The image processing apparatus according to claim 1, wherein the third parallax image data and the fourth parallax image data are generated so that images having different parallax amounts are obtained.
A first parallax pixel group that receives a first partial light beam that is displaced in a first direction with respect to an optical axis of the optical system among incident light beams that pass through the optical system, and a second direction that is different from the first direction. An imaging device including a second parallax pixel group that receives the displaced second partial light beam;
The image data including the first parallax image data generated based on the output of the first parallax pixel group and the second parallax image data generated based on the output of the second parallax pixel group is obtained by the acquisition unit. An imaging apparatus comprising: the image processing apparatus according to claim 1 to be acquired.
An acquisition unit for acquiring image data;
A calculation unit for calculating a parallax amount of a subject included in the image of the image data;
A detection unit that detects a distance in the depth direction of the subject based on the amount of parallax;
A determination unit that adjusts the amount of parallax for a portion of the subject in which the distance in the depth direction exceeds a predetermined range;
An image processing apparatus comprising: a generation unit that generates parallax image data having the adjusted parallax amount.
The said determination part adjusts the said amount of parallax so that the amount of parallax may become less than the predetermined range with respect to the part of the said subject where the distance of the said depth direction exceeds the predetermined range. The image processing apparatus described.
The image processing apparatus according to claim 11 or 12, wherein the determination unit adjusts the parallax amount while maintaining a positional relationship of the subject in the depth direction.
An acquisition step of acquiring image data;
A calculation step of calculating a parallax amount of a subject included in the image of the image data;
A determination step for determining an adjustment value for adjusting the parallax amount, which is applied when generating parallax image data from the image data, for the parallax amount;
An image processing program for causing a computer to execute a generation step of generating parallax image data that is an image having the parallax amount adjusted to each other by applying the adjustment value to the image data.