JP2012165207A - Three-dimensional imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional imaging apparatus allowing for comfortable three-dimensional imaging by performing optical axis displacement corrections of a left-eye imaging system and a right-eye imaging system, camera shake corrections during imaging, and parallax adjustment between a left-eye image and a right-eye image.SOLUTION: An optical axis displacement determination section 121 determines an optical axis displacement from first and second taken images stored in a storage section 133. An optical axis displacement correction region determination section 122 determines an image region used for an optical axis displacement correction. A parallax adjustment section 123 determines whether to perform a parallax adjustment of the first and second taken images. A parallax adjustment region determination section 124 determines an image region used for the parallax adjustment. A camera shake correction determination section 125 determines necessity for a camera shake correction. A camera shake correction region determination section 126 determines an image region used for the camera shake correction. An output section 142 outputs the taken images obtained after removing the foregoing three image regions from the first and second taken images.

Description

  The present invention relates to a 3D image capturing apparatus, and more particularly, corrects a shift in the optical axis of an imaging system for a left eye and an imaging system for a right eye, and performs parallax adjustment and camera shake appropriate for an image for a left eye and an image for a right eye. The present invention relates to a 3D image capturing apparatus that performs correction.

  In recent years, two images with parallax in the horizontal direction are displayed on the display as an image for the left eye and an image for the right eye, and the observer independently uses the left eye image for the left eye and the right eye image for the right eye. Thus, the development of technologies related to 3D images that can be perceived as if the subject displayed on the display exists three-dimensionally has been actively conducted.

As a 3D image display and observation method, a left-eye image and a right-eye image having linearly polarized light that are orthogonal to each other are displayed on the display so that the observer can use the left eye with glasses with a polarization filter. A method of independently observing a left-eye image and a right-eye image with the right eye is well known.
In addition, the left eye image and the right eye image are alternately displayed on the display, and the left eye image is displayed with the left eye using an eyeglass with a liquid crystal shutter in which the observer alternately shields the left and right fields of view. Also, a method of independently observing the right eye image with the right eye is well known.

In these 3D image technologies, it is perceived as if the subject image in the 3D image is projected forward from the display surface as viewed from the observer, or how far the object image is retracted from the display surface. Is determined by the horizontal parallax between the left-eye image and the right-eye image. On the other hand, a 3D image in which the parallax in the horizontal direction of the left-eye image and the right-eye image is not properly adjusted is, for example, when the subject image protrudes forward or retracts too much, It has been pointed out that this may cause eye strain and image sickness.
Therefore, techniques relating to an imaging apparatus that captures a 3D image while appropriately adjusting the parallax in the horizontal direction of the image for the left eye and the image for the right eye have been actively developed.

Regarding the parallax adjustment in the horizontal direction of the image for the left eye and the image for the right eye, a method of adjusting the convergence angle between the imaging system for the left eye and the imaging system for the right eye mounted on the imaging device, and solid There is a method of adjusting the range of the image output from each of the left-eye photographic image data and the right-eye photographic image data temporarily stored in the memory via the image sensor.
As an example of the latter, Patent Document 1 discloses that the aspect ratio of the image output from each of the left-eye captured image data and the right-eye captured image data is changed according to the distance of the subject, A technique related to a photographing apparatus that adjusts the parallax of a stereoscopic image is disclosed.

JP 2010-147784 A

By the way, in a so-called multi-lens imaging device having two imaging systems, a left-eye imaging system and a right-eye imaging system, it is possible to perform 3D image capturing after correcting the deviation of the optical axes of both imaging systems. It is important for shooting. If the optical axes of the two imaging systems are deviated in the vertical direction, the scenery and the position of the subject captured by the left-eye image and the right-eye image are deviated. This is because experiments have reported that when such a vertical shift occurs, the landscape or subject image is difficult to perceive as a 3D image, and when the deviation increases, it cannot be perceived as an appropriate 3D image (3D). Research report on content Digital Content Association).

  Further, in shooting a 3D image, it is important to correct camera shake at the time of shooting as in the case of shooting a 2D image. Camera shake in 3D images is not easy to see due to normal camera shake, and changes in the vertical and horizontal directions of the left and right eye images directly lead to changes in parallax. It is also a cause of image loss and may induce image sickness.

  From the above, in 3D image shooting, correction of the optical axis shift of the left-eye imaging system and right-eye imaging system, correction of camera shake during shooting, and parallax between the left-eye image and the right-eye image The problem is how to apply each of these adjustments appropriately.

  The present invention has been made in view of such a problem, and by adjusting the cutout range of an image from an image sensor, correction of the optical axis shift of the left-eye imaging system and the right-eye imaging system, and imaging 3D image shooting capable of taking a comfortable 3D image that is less likely to induce eye strain and image sickness by correcting camera shake at the time and adjusting the parallax between the image for the left eye and the image for the right eye An object is to provide an apparatus.

  To achieve the above object, the present invention provides a first imaging unit (L100) that captures a subject and outputs a first captured image, and a second imaging unit (R100) that outputs a second captured image. A storage unit (133) for storing the first captured image and the second captured image, and an optical axis between the first image capturing unit (L100) and the second image capturing unit (R100). When the optical axis deviation determination unit (121) that determines the presence or absence of deviation and the optical axis deviation determination unit (121) determine that the optical axis is deviated, the first photographed image and the second photographed image An optical axis misalignment image area determination unit (122) that determines an optical axis misalignment correction image area for correcting the optical axis misalignment from the image, and an area of the optical axis misalignment correction image from the first captured image A third captured image excluding the image and the optical axis deviation corrected image from the second captured image A parallax adjustment unit (123) that determines whether or not to adjust the parallax in the horizontal direction with respect to the fourth captured image excluding the region, and a determination that the parallax adjustment unit (123) adjusts the parallax in the horizontal direction In this case, a parallax adjustment image region determination unit (124) that determines a region of a parallax adjustment image for adjusting the parallax in the horizontal direction from the third shooting image and the fourth shooting image, and the third shooting image Determining whether to correct camera shake of a fifth captured image obtained by removing the parallax adjusted image region from the captured image and a sixth captured image obtained by removing the parallax adjusted image region from the fourth captured image When the camera shake correction determination unit (125) and the camera shake correction determination unit determine to correct camera shake, an area of a camera shake correction image for camera shake correction is determined from the fifth captured image and the sixth captured image. Image stabilization area to be determined A portion (126), a seventh photographed image obtained by removing the image stabilization image area from the sixth photographed image, and an eighth photographed image obtained by removing the image stabilization image area from the sixth photographed image. An output unit (142) for outputting a 3D image.

  The 3D image capturing apparatus described above further includes an output image region determining unit (127), and the output image region determining unit (127) includes the first captured image, the second captured image, and the third captured image. And the fourth photographed image, the fifth photographed image and the sixth photographed image, or the combination of the seventh photographed image and the eighth photographed image. On the other hand, the areas of these two captured images may be determined in consideration of the aspect ratio.

  The optical axis deviation determination unit (122) of the 3D image capturing apparatus described above is configured when the magnitude of the vertical difference vector between the first captured image and the second captured image exceeds a predetermined threshold. It may be determined that the optical axis between the first imaging unit (L100) and the second imaging unit (R100) is shifted.

  The parallax adjusted image region determining unit (124) of the 3D image capturing device described above is based on the display surface width of the image display device in which the output unit outputs the seventh captured image and the eighth captured image. You may determine the magnitude | size of a parallax adjustment image area | region.

  According to the 3D image capturing device of the present invention, by adjusting the cutout range of the image from the image sensor, correction of the optical axis misalignment of the left-eye imaging system and the right-eye imaging system, and camera shake during shooting A correction and adjustment of parallax between the image for the left eye and the image for the right eye can be performed, and a comfortable 3D image that hardly induces eye strain or image sickness can be taken.

1 is a block diagram illustrating an internal configuration of an imaging apparatus (digital video camera) 1 according to an embodiment of the present invention. It is a figure which shows the structure of the liquid crystal monitor 143 with which the imaging device 1 is provided. 6 is a flowchart for describing 3D image shooting processing by the imaging apparatus 1. Left-eye photographic image data and right-eye photographic image data in the case where a vertical shift occurs between the optical axis AL100 of the left-eye imaging system L100 and the optical axis AR100 of the right-eye imaging system R100 included in the imaging device 1 It is a conceptual diagram for demonstrating. It is a conceptual diagram for demonstrating the process of the optical axis offset determination by the optical axis offset determination part 121 with which the imaging device 1 is provided. It is a conceptual diagram for determining the area | region used for the correction | amendment of the optical axis deviation by the optical axis deviation correction area | region determination part 122 with which the imaging device 1 is provided. It is a figure for demonstrating the relationship of the parallax of the to-be-photographed object image in the image for left eyes displayed on a monitor, and the image for right eyes, and the protrusion to the front with respect to the display surface of a monitor, or back retracting. It is a conceptual diagram for determining the area | region used for the parallax adjustment by the parallax adjustment area | region determination part 124 with which the imaging device 1 is provided. 6 is a conceptual diagram for determining an area used for camera shake correction by a camera shake correction area determination unit 126 included in the imaging apparatus 1; FIG.

  A preferred embodiment of a 3D image capturing device according to the present invention will be described below with reference to the drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

FIG. 1 is a block diagram illustrating an example of an internal configuration of an imaging apparatus (digital video camera) 1 according to an embodiment of the present invention. The imaging device 1 can capture a moving image and a still image.
Although the imaging apparatus according to the present embodiment has two imaging units and can adjust the convergence angle formed by each optical axis, the convergence angle cannot be adjusted by fixing both optical axes to be parallel. The present invention can also be adopted in an imaging apparatus. Further, the present invention is not limited to digital still cameras, mobile phones, PHS (Personal Handyphone System), PDA (Personal Digital Assistant), and other electronic devices capable of taking pictures as long as they have a plurality of imaging systems. Can also be employed. It can also be used when a 3D image is captured using a plurality of cameras.

  The CPU 120 controls the overall operation of the imaging apparatus 1 such as shooting, display, and recording. Further, the CPU 120 controls each unit according to a predetermined control program based on an input from the operation unit 145.

  The imaging apparatus 1 is provided with a pair of left and right image capturing units L100 and R100 for right eye that are separated by a predetermined interval (for example, 5 cm) slightly shorter than the interval between human eyes. The left-eye image capturing unit L100 and the right-eye image capturing unit R100 include zoom lenses L101 and R101, focus lenses L102 and R102, stops L103 and R103, and solid-state image sensors L104 and R104, respectively.

  The zoom lenses L101 and R101 are moved along the optical axes AL100 and AR100 by a zoom actuator (not shown). The focus lenses L102 and R102 are moved along the optical axes AL100 and AR100 by a focus actuator (not shown). The diaphragms L103 and R103 operate by being driven by a diaphragm actuator (not shown).

  The left-eye image capturing unit L100 and the right-eye image capturing unit R100 are connected to convergence angle actuators L109 and R109, respectively. The convergence angle actuators L109 and R109 receive commands from the CPU 120 to drive both image capturing units, The convergence angle formed by the axes AL100 and AR100 is adjusted.

The ROM 131 is connected to the CPU 120 via the bus 127, and stores a control program executed by the CPU 120, various data necessary for control, and the like. The flash ROM 132 stores various setting information related to the operation of the imaging apparatus 1, such as user setting information.
The SDRAM 133 is used as a calculation work area for the CPU 120 and also as a temporary storage area for image data. The VRAM 134 is used as a temporary storage area for display image data.

The photographing of a stereoscopic image using the photographing device 1 is performed according to the following procedure. The left-eye solid-state image pickup device L104 and the right-eye solid-state image pickup device R104 photoelectrically convert the light that has passed through the left-eye image pickup portion L100 and the right-eye image pickup portion R100, respectively. An analog imaging signal is generated.
After the analog signal processing units L105 and R105 amplify both analog imaging signals, the A / D converters L106 and R106 convert the amplified signals into digital data. The image input controllers L107 and R107 take in the digital data output from the A / D converters L106 and R106 and store them in the SDRAM 133.

Based on a command from the CPU 120, the digital signal processing units L108 and R108 take in digital data stored in the SDRAM 133, perform predetermined signal processing, and generate a signal composed of a luminance signal and a color difference signal.
The digital signal processing units L108 and R108 also perform various digital corrections such as offset processing, white balance adjustment processing, gamma correction processing, RGB interpolation processing, noise reduction processing, contour correction processing, color tone correction processing, and light source type determination processing.

The optical axis deviation determination unit 121 determines the optical axes of the left-eye imaging system L100 and the right-eye imaging system R100 from the vertical deviation of the left-eye photographic image data and the right-eye photographic image data temporarily stored in the SDRAM 133. The presence or absence of vertical displacement is determined.
The optical axis deviation correction area determination unit 123 corrects the optical axis deviation of the left eye imaging system L100 and the right eye imaging system R100 from the left eye imaging image data and the right eye imaging image data temporarily stored in the SDRAM 133. Determine the data for the area to be used.

The parallax adjustment unit 123 adjusts the parallax in the horizontal direction of the left-eye captured image data and the right-eye captured image data temporarily stored in the SDRAM 133.
The parallax adjustment area determination unit 124 determines area data to be used for parallax adjustment from the left-eye captured image data and the right-eye captured image data stored in the SDRAM 133.

The camera shake correction determination unit 125 determines whether or not to perform camera shake correction on the left eye captured image data and the right eye captured image data temporarily stored in the SDRAM 133.
The camera shake correction area determination unit 126 determines data of an area used for camera shake correction from the left-eye captured image data and the right-eye captured image data stored in the SDRAM 133.

  The output image area determination unit 126 outputs data of an area to be output to the HDD I / F 138, the card I / F 140, or the input / output I / F from the left-eye captured image data and the right-eye captured image data stored in the SDRAM 133. decide.

  The compression / decompression processing unit 135, media control unit 136, audio / image processing unit 137, HDD I / F 138, card I / F 140, and input / output I / F 142 are connected to the data bus 127.

The compression / decompression processing unit 135 performs compression processing in a predetermined format on the data stored in the SDRAM 133 in accordance with a command from the CPU 120 to generate compressed image data. Further, in accordance with a command from the CPU 120, the compressed image data stored in the card type recording medium 141 or the like is subjected to a predetermined format expansion process to generate uncompressed image data.
Note that in the imaging apparatus 1 according to the present embodiment, a compression method conforming to the JPEG standard is used for still images, and an MPEG2 standard or AVC / H. A compression method conforming to the H.264 standard is adopted.

The media control unit 136 controls writing of data to the HDD 139 through the HDD I / F 138 or reading of data from the card-type recording medium 141 through the card I / F 140 in accordance with a command from the CPU 120.
The liquid crystal monitor 143, the speaker 144, the operation unit 145, and the input / output terminal 146 are connected to the input / output I / F 142.

FIG. 2 is a diagram for explaining an example of the structure of the liquid crystal monitor 143 capable of displaying a 3D image. In FIG. 2, the following will be described with respect to the liquid crystal monitor 140 in the normal use posture of the imaging apparatus 1 with the horizontal direction as the X axis, the vertical direction as the Y axis, and the orthogonal direction of the liquid crystal monitor 140 as the Z axis.
A lenticular lens LL20 is disposed between the liquid crystal monitor 143 and the left eye LE20 and right eye RE20 of the observer. The lenticular lens LL20 is configured by connecting a plurality of cylindrical convex lenses in the X-axis direction in FIG.

The display area of the 3D image displayed on the liquid crystal monitor 143 includes a strip image display area L for the left eye and a strip image display area R for the right eye. The left-eye strip image display area L and the right-eye strip image display area R each have an elongated strip shape in the Z-axis direction of FIG. 2 and are alternately arranged in the X-axis direction.
Each convex lens constituting the lenticular lens LL20 includes a set of strip image display areas L for the left eye and a strip image display area R for the right eye, respectively, with reference to a predetermined observation point of the observer. It is formed at a position corresponding to.

  In each convex lens constituting the lenticular lens LL20, the left eye strip image displayed in the left eye strip image display area L of the liquid crystal monitor 140 is incident on the viewer's left eye LE20, and the viewer's right eye The curvature and the like are set so that the right eye strip image displayed in the right eye strip image display area R of the liquid crystal monitor 140 is incident on the eye RE20.

  Therefore, the left eye of the observer observes only the strip image for the left eye and the right eye observes only the strip image for the right eye, and the observer views the image taken by the imaging device 1 through the liquid crystal monitor 143. It can be perceived as a 3D image.

Although the example in which the lenticular method shown in FIG. 2 is used as the structure of the liquid crystal monitor 143 for displaying a 3D image has been described, the present invention is not limited to the lenticular method, and the parallax barrier method and the light direction control. Another method for displaying 3D images, such as a method, can also be adopted.
The liquid crystal monitor 140 not only displays the captured image in 3D, but can also display either the left-eye image or the right-eye image in 2D. Also, both the left and right eye images can be displayed in 2D side by side by the so-called side-by-side method.

The operation unit 145 includes an operation key including a relays switch and a power switch (not shown), a cross key, a joystick, a touch panel superimposed on the liquid crystal monitor 143, and the like. Accept.
The input / output terminal 146 is connected to a television monitor, a PC (Personal Computer) or the like (not shown).

  For example, the sound / image processing unit 137 performs predetermined image processing on the digital imaging signal read from the SDRAM 133. In response to the instruction from the CPU 120, the sound / image processing unit 137 generates image data for various processes, superimposes the image data on the original image data read from the SDRAM 133, and outputs the image data to the liquid crystal monitor 143. . With this output, the image displayed on the liquid crystal monitor 143 is a combination of various image data.

<Shooting a 3D image by the imaging apparatus 1>
FIG. 3 is a flowchart for explaining a process when the imaging apparatus 1 captures a 3D image. This process is executed according to a program by the overall control of the CPU 120 in FIG. 1 when the user presses a recording button (not shown) on the operation unit 145. Further, when the user turns on the power of the imaging apparatus 1 and selects a shooting mode using a button (not shown) of the operation unit 142 or when the recording button is pressed halfway, This process may be started.

  As described above, a landscape and a subject are photographed through the left-eye imaging system L100 and the right-eye imaging system R100 (step S101). The photographed scenery and subject are temporarily recorded in the SDRAM 133 as left-eye photographic image data and right-eye photographic image data via the left-eye solid-state image sensor L104 and the right-eye solid-state image sensor R104. (Step S102).

  Next, the optical axis deviation determination unit 121 determines whether the left-eye imaging system L100 and the right-eye are detected based on whether or not the left-eye photographic image data and the right-eye photographic image data are temporarily stored in the SDRAM in step S102. The presence / absence of vertical deviation of the optical axis of the imaging system R100 is detected, and the necessity for correction is determined (step S103). In a multi-lens imaging device such as the imaging device 1, the optical axis AL100 and the right eye of the imaging system L100 for the left eye depend on the characteristics and settings of the device or the user's shooting state such as the orientation and temperature of the device at the time of shooting. The optical axis AR100 of the image pickup system R100 may be displaced in the vertical direction. As described above, when the deviation of the optical axis causes a vertical deviation between the captured image for the left eye and the captured image for the right eye, the subject is hardly perceived as a 3D image, and in some cases cannot be perceived as a 3D image. It is.

  FIG. 4 shows left-eye photographic image data and right-eye photographic image data in the case where a vertical shift occurs between the optical axis AL100 of the left-eye imaging system L100 and the optical axis AR100 of the right-eye imaging system R100. The conceptual diagram for demonstrating is shown. Due to the vertical deviation between the optical axis AL100 and the optical axis AR100, the subject image data LP102 and RP102 in the left-eye photographed image data LP101 and right-eye photographed image data RP101 temporarily stored in the SDRAM 133 are displaced in the vertical direction. ing. As a result, the subject image is hardly perceived as a 3D image, or in some cases, it is not perceived as a 3D image.

  Therefore, the optical axis deviation determination unit 121 uses, for example, a technique that applies an algorithm for calculating a so-called motion vector that specifies how much the same subject in MPEG moves between frames, and uses the left-eye photographed image data LP101 and the right eye. The optical axis AL100 and the optical axis AR100 detect the presence / absence of the vertical deviation from the presence / absence of the vertical deviation of the eye image data RP101.

In MPEG, which is a moving image compression technique, an algorithm for detecting a motion vector based on block matching is used. Here, the motion vector is a vector representing the displacement of the same subject between two frame data.
The current frame data and past frame data are compared, the most similar blocks of the same size are extracted from each, and a motion vector is calculated from the positional relationship between them.

  The optical axis deviation determination unit 121 specifies the same subject image data in the left-eye photographic image data and the right-eye photographic image data by using the algorithm for calculating the motion vector. A point corresponding to the pixel-to-pixel point is extracted from the left-eye captured image data and the right-eye captured image data, and a difference vector between them is calculated as the parallax of the subject image in the left-eye image and the right-eye image.

More specifically, as shown in FIG. 5, the optical axis deviation determination unit 121 performs subject image data in the right eye image data RP101 as the same subject image data as the subject image data LP102 in the left eye photographed image data LP101. RP102 is specified. Next, the optical axis deviation determination unit 121 extracts the reference point LP103 from the subject image data LP102 in the left-eye photographed image data LP101. Then, the optical axis deviation determination unit 121 extracts the corresponding point RP103 corresponding to the reference point LP103 from the right-eye captured image captured image data RP101, and detects the difference vector VV101 between the reference point LP103 and the corresponding point RP103. To do. When the difference vector is detected, the optical axis determination unit 121 determines that the optical axis AL100 and the optical axis AR100 are shifted in the vertical direction and needs to be corrected. When the difference vector is not detected (when the difference vector is zero) ) Does not deviate from the optical axis AL100 and the optical axis AR100, it is determined that correction is unnecessary.
The optical axis deviation determination unit 121 can also extract a reference point from the right-eye photographic image data and a corresponding point from the left-eye photographic image data, and calculate a difference vector.

  The optical axis deviation determination unit 121 determines that correction is necessary when the magnitude of the detected difference vector is greater than or equal to a predetermined threshold value, and determines that correction is not necessary when the magnitude is less than the predetermined threshold value. Good. For example, it has been experimentally reported that when the size of the vertical shift between the left-eye image and the right-eye image is 0.7% or more of the height of the display screen, it cannot be perceived as an appropriate 3D image ( 3D content research report (Digital Content Association of Japan), whether the size of the difference vector is 0.7% or more of the height of the external monitor connected via the liquid crystal monitor 143 or the input / output terminal 146 Whether or not may be used with a predetermined threshold.

  Returning to FIG. 3, when the optical axis deviation determination unit 121 determines that the optical axis deviation needs to be corrected (YES in step S <b> 103), the optical axis deviation correction region determination unit 122 determines the left-eye captured image data or A region to be used for correcting the optical axis shift is determined from at least one of the right-eye photographed image data (S104).

  FIG. 6 is a conceptual diagram for determining an area used for correcting an optical axis deviation by the optical axis deviation correction area determining unit 122. The optical axis deviation correction area determining unit 122 converts an area having the same height as the difference vector VV101 from the left-eye captured image data LP101 (an area indicated by hatching in FIG. 6) to the optical axis deviation correction area LP104. Decide to cut out as. Similarly, the optical axis deviation correction area determination unit 122 converts an area having the same height as the difference vector VV101 from the right-eye captured image data RP101 (an area indicated by hatching in FIG. 6) to the optical axis deviation. It is determined to cut out as the correction region RP104. By performing the correction as described above, it is possible to correct the vertical shift between the left-eye shot image data and the right-eye shot image data due to the shift of the optical axis between the optical axis AL100 and the optical axis AR100.

  In FIG. 6, it has been decided to cut out the optical axis deviation correction region only from the vertical direction of the left-eye photographic image data LP101 and the right-eye photographic image data RP101, but the left-eye image image to be output to the liquid crystal monitor 143 and the right In order to adjust the aspect ratio of the ophthalmic image, the output image area determination unit 127 may cut out a certain area from the horizontal direction.

  When the correction of the optical axis deviation is completed in step S104 in FIG. 3 or when the optical axis deviation determination unit 121 determines that the correction is unnecessary in step S103 (NO in step S103), the parallax adjustment unit 123 captures the left-eye photographed image. It is determined whether or not the data and the right-eye photographic image data need to be adjusted (step S105). When the horizontal parallax between the respective subjects in the left-eye image and the right-eye image is large, the subject image is perceived as protruding too far forward of the display surface as viewed by the observer, or It will be perceived as being retracted too far behind the display surface. This is because if these degrees are too large, a great burden is placed on the observer.

  Similar to the detection of the optical axis shift in step S103 described above, the parallax adjustment unit 123 uses a technique that applies an algorithm for calculating a so-called motion vector to obtain a difference between the captured image data for the left eye and the captured image data for the right eye. The parallax in the horizontal direction is extracted as a difference vector, and the presence or absence of subject image data that places a burden on the observer is detected.

  Referring to FIG. 7, regarding the subject images in the left-eye image and the right-eye image displayed on the liquid crystal monitor 143 or the external monitor connected via the input / output terminal, the left-eye image and the right-eye image The relationship between the parallax between the images and the forward projection or the rearward withdrawal with respect to the display surface of the monitor will be described.

  In FIG. 7, the display surface of the external monitor connected via the liquid crystal monitor 143 or the input / output terminal 146 is DP70, the distance between the observer and the display surface DP70 is D, and the distance between the left eye LE70 and the right eye RE70 of the observer. Is E.

In FIG. 7A, when the display position of the subject image in the image for the left eye on the display surface DP70 is L _f and the display position of the subject image in the image for the right eye is R _f , the observer is left eye LE70. in L _f, when observing the R _f with the right eye RE70, both the subject image is imaged at the position of P _f. As a result, in the 3D image, the observer perceives the subject image as if it protrudes forward with respect to the display surface DP70.

In this case, the parallax image L _f and R _f and V _f, the distance Z _f from the display surface DP70 to the imaging position P _f is as the following equation (1).

(E + V _f ) Z _f = D · V _f (1)

Here, the observer perceives the subject image at a place different from the display surface D70. On the other hand, since the focus is on the display surface DP70, if the difference between the two is large, visual fatigue and discomfort may occur for the observer. On the other hand, if the parallax V _f is too large, the subject is not perceived as popping out and may be perceived as a simple double image.

  The distance E between the left eye LE70 and the right eye RE70 of the observer is 6.5 cm which is a general human pupil distance, and the viewing distance D from the observer to the display surface DP70 is a standard viewing distance. When the distance is three times the long side, the amount of parallax between the left-eye image and the right-eye image that can be comfortably recognized as a 3D image without imposing a burden on the observer's eyes, 2.9% (parallax angle about 1 degree) or less is reported to be desirable ("3DC safety guidelines for the spread of 3D that are friendly to humans" issued by the 3D Consortium Safety Guidelines Committee April 20, 2010 Revised version).

Therefore, the parallax adjustment unit 123 needs to adjust the parallax when the subject image in which the display surface width DW and the parallax V _f illustrated in FIG. Judge that there is.

V _f > 0.029 DW (2)

  Here, as the display surface width DW, the width of the liquid crystal monitor 143 included in the imaging device 1 is adopted as a preset value. When an image is displayed on an external monitor such as a TV display owned by the user, the width may be input and the value may be adopted.

In FIG. 7B, if the display position of the subject image in the left-eye image on the display surface DP70 is L _b , and the position of the subject image in the right-eye image is R _b , the observer uses the left eye LE70. When R _b is observed with L _b and the right eye RE 70, the subject image is formed at the position P _b . Thus, in the 3D image, the observer perceives the subject image as if it was retracted backward with respect to the display surface DP70.

At this time, assuming that the parallax between the images L _b and R _b is V _f , the distance Z _f from the display surface DP 70 to the image forming position is expressed by the following equation (3).

(E−V _b ) Z _b = D · V _b (3)

  Even when the subject image is perceived as if it is retracted with respect to the display surface, depending on the parallax between the left and right eye images, there is a risk of visual fatigue or discomfort to the observer, and the subject is drawn in. It is not perceived as if the subject is moving, but may be perceived as a mere double image, as in the case where the subject image is perceived as if it is projected forward with respect to the display surface. .

Therefore, the parallax adjustment unit 123 needs to adjust the parallax when the subject image in which the display surface width DW and the parallax V _f illustrated in FIG. Judge that there is.

V _b > 0.029 DW (4)

  Here, as the display surface width DW, the width of the liquid crystal monitor 143 included in the imaging device 1 is adopted as a preset value. When an image is displayed on an external monitor such as a TV display owned by the user, the width may be input and the value may be adopted.

  Furthermore, in 3D images that are perceived as if they are drawn from the display surface, it is desirable that the horizontal parallax on the display should not exceed 5 cm. 3DC Safety Guidelines for the 3D Consortium Safety Guidelines Subcommittee revised version on April 20, 2010). Since both eyes of a person do not open outward, observing a subject that is more than the distance between the pupils of both eyes independently with the left and right eyes induces eye strain and 3D image failure. This is because there is a fear.

  Therefore, the parallax adjustment unit 123 may determine that parallax adjustment is necessary when a subject image with Vb exceeding 5 cm is included. The determination whether Vb is 5 cm or more and the adjustment to 5 cm are performed by calculating the number of pixels corresponding to 5 cm from the DW value set or inputted in advance.

  In the above example, the parallax adjustment unit 123 determines whether or not parallax adjustment is necessary from the viewpoint of reducing the burden on the viewer. However, the parallax adjustment unit 123 determines whether the left eye shot image and the right eye shot image are included in the imaging device 1. When the setting for adjusting the parallax in the horizontal direction is made in advance, it is determined that the parallax adjustment is necessary, and when the setting for not adjusting the parallax is made, the parallax adjustment is unnecessary. May be determined.

  Returning to FIG. 3, when the parallax adjustment unit 123 determines that it is necessary to adjust the parallax in the horizontal direction between the left-eye photographic image data and the right-eye photographic image data (YES in step S <b> 105), the parallax adjustment region determination unit 124. Determines a region to be used for parallax adjustment from at least one of the left-eye captured image data and the right-eye captured image data (S106).

  FIG. 8 is a conceptual diagram for determining an area used for parallax adjustment by the parallax adjustment area determining unit 124. The parallax adjustment unit 124 determines to cut out the parallax adjustment region LP105 from the left-eye shot image data LP101 and to cut out the parallax adjustment region RP105 from the right-eye shot image data RP101. Here, the sizes of the parallax adjustment regions LP105 and RP105 are such that the parallax amounts of the subject images LP102 and RP102 are the parallax amounts between the left-eye image and the right-eye image that can be comfortably recognized as a 3D image as described above. Alternatively, it may be determined based on a value recorded in advance in the imaging apparatus 1.

  In FIG. 8, it has been decided to cut out the parallax adjustment region only from the horizontal direction of the left-eye photographic image data LP101 and the right-eye photographic image data RP101, but the left-eye image image and right-eye image output to the liquid crystal monitor 143 and the like. In order to adjust the aspect ratio of the image, the output image area determination unit 127 may cut out a certain area from the vertical direction.

  When the parallax adjustment is completed in step S106 of FIG. 3 or when the parallax adjustment unit 123 determines that the parallax adjustment is unnecessary in step S105 (NO in step S105), the parallax adjustment unit 123 sets the left-eye captured image data and It is determined whether camera shake correction of the right-eye photographic image data is necessary (step S107). Camera shake in a 3D image is unstable in the perceived stereoscopic degree because changes in the vertical and horizontal directions of the left-eye image and right-eye image directly lead to a change in parallax in addition to the difficulty in viewing the image due to normal camera shake. It is also a cause of image loss and may induce image sickness.

  The camera shake correction determination unit 125 determines whether camera shake correction is necessary from at least one of the output result of the gyro sensor 147 or the vector calculation of the left eye captured image data and the right eye captured image data by a known method. .

  When the camera shake correction determination unit 125 determines that camera shake correction is necessary (YES in step S108), the camera shake correction region determination unit 126 performs camera shake correction from at least one of the left-eye captured image data and the right-eye captured image data. An area to be used for the process is determined (S108).

FIG. 9 is a conceptual diagram for determining an area used for camera shake correction by the camera shake correction area determination unit 126. The camera shake correction area determination unit 126 determines to cut out the camera shake correction area LP106 from the left-eye captured image data LP101 and to extract the camera shake correction area RP106 from the right-eye captured image data RP101.
The ranges of the camera shake correction regions LP106 and RP106 are determined based on the zoom magnification at the time of shooting, and are determined so as to satisfy, for example, the following expressions (5) and (6).

(1-Vb / Va) × 100 = (zoom magnification + 2)% (5)

(1−Hb / Ha) × 100 = (zoom magnification + 2)% (6)

  When the camera shake correction is completed in step S108 or the camera shake correction determination unit 125 determines that the camera shake correction is unnecessary (NO in step S107), the output image area determination unit adjusts the aspect ratio and finally. The output image data for the left eye and the output image data for the right eye to be output are determined, and these image data are output to the HDD I / F 138, the card I / F 140, or the input / output I / F 142 (step S109). Note that the camera shake correction area determination unit 126 can secure a predetermined camera shake correction area regardless of the zoom magnification.

  When the user releases the recording button (not shown) of the operation unit 145 and finishes shooting, the imaging apparatus 1 ends the 3D image shooting process. Alternatively, when the user turns off the power of the imaging apparatus 1 or releases the recording button that has been half-pressed, the process for the through image is terminated.

  As described above, according to the imaging apparatus 1 according to the present invention, the optical axes of the left-eye imaging system and the right-eye imaging system are adjusted by adjusting the cutout range of the images from the solid-state imaging devices L104 and R104. Correction of image shift, correction of camera shake at the time of shooting, and adjustment of parallax between the image for the left eye and the image for the right eye, and taking a comfortable 3D image that is less likely to induce eye strain and image sickness be able to.

1 Imaging device 120 CPU
121 optical axis deviation determination unit 122 optical axis deviation correction region determination unit 123 parallax adjustment unit 124 parallax adjustment region determination unit 125 camera shake correction determination unit 126 camera shake correction region determination unit 127 output image region determination unit 133 SDRAM
138 HDD I / F
139 Card I / F
142 I / O I / F
143 LCD monitor 145 Operation unit

Claims (4)

A first imaging unit that images a subject and outputs a first captured image; and a second imaging unit that outputs a second captured image;
A storage unit for storing the first captured image and the second captured image;
An optical axis deviation determination unit that determines the presence or absence of an optical axis deviation between the first imaging unit and the second imaging unit;
When the optical axis deviation determination unit determines that the optical axis is deviated, an optical axis deviation correction image area for correcting the optical axis deviation from the first photographed image and the second photographed image. An optical axis misalignment image region determination unit for determining
A third photographed image obtained by removing the optical axis deviation corrected image area from the first photographed image, and a fourth photographed image obtained by removing the optical axis deviation corrected image area from the second photographed image. A parallax adjustment unit that determines whether or not to adjust the horizontal parallax between,
When the parallax adjustment unit determines to adjust the parallax in the horizontal direction, the parallax adjustment that determines the region of the parallax adjustment image for adjusting the parallax in the horizontal direction from the third captured image and the fourth captured image An image region determination unit;
Whether to correct camera shake of the fifth photographed image obtained by removing the region of the parallax adjustment image from the third photographed image and the sixth photographed image obtained by removing the region of the parallax adjustment image from the fourth photographed image A camera shake correction determination unit for determining whether or not,
When the camera shake correction determination unit determines to correct camera shake, a camera shake correction region determination unit that determines a region of the camera shake correction image for camera shake correction from the fifth captured image and the sixth captured image;
An output unit for outputting a seventh photographed image obtained by removing the image stabilization image area from the sixth photographed image and an eighth photographed image obtained by removing the image stabilization image area from the sixth photographed image; A 3D image photographing apparatus comprising: The 3D image capturing apparatus further includes an output image region determination unit,
The output image area determination unit includes the first captured image and the second captured image, the third captured image and the fourth captured image, the fifth captured image and the sixth captured image, or A region of the two photographed images is determined in consideration of an aspect ratio with respect to a photographed image of at least one of the combinations of the seventh photographed image and the eighth photographed image. The 3D image photographing device according to claim 1.   The optical axis deviation determination unit is configured to detect the first imaging unit and the first imaging unit when the magnitude of a vertical difference vector between the first captured image and the second captured image exceeds a predetermined threshold. The 3D image capturing apparatus according to claim 1, wherein it is determined that the optical axis between the second imaging unit is shifted.   The parallax adjustment image region determination unit determines the size of the parallax adjustment image region based on a display surface width of an image display device from which the output unit outputs the seventh captured image and the eighth captured image. The 3D image photographing device according to any one of claims 1 to 3, wherein the 3D image photographing device is provided.

Images