JP5718149B2 - Image correction method and stereoscopic image photographing apparatus - Google Patents

Description

  The present invention provides an image correction method for correcting each image acquired from two cameras built in a three-dimensional image capturing device that captures a three-dimensional image, and each image acquired from the two cameras is the above image. The present invention relates to a stereoscopic image photographing apparatus that performs correction according to a correction method.

  Since a stereoscopic image (stereoscopic image) can give a user a novelty, a sense of presence, and the like, development of a technique for creating and displaying a stereoscopic image has been actively performed.

  In Patent Document 1, a displacement amount and direction of a positional shift between left and right images are detected from two images (left and right images) taken by two left and right cameras, and based on a displacement vector indicating the displacement amount and direction. A technique for correcting a positional deviation between left and right images so as to obtain a desired parallax is disclosed.

  Patent Document 2 discloses a technique for correcting the horizontal direction of the left and right images by trimming in order to reduce the horizontal disparity shift between the left and right images.

  Japanese Patent Application Laid-Open No. 2004-228561 obtains a phase difference between left and right images, adjusts motion information of the left and right images (time-series images) using the obtained phase difference, and trims the left and right images based on the adjusted motion information. Is disclosed.

JP 2011-71604 A (published April 7, 2011) JP 2011-29700 A (published February 10, 2011) JP 2003-92768 A (published March 28, 2003)

  In the techniques disclosed in Patent Documents 1 to 3, there is a risk of vertical displacement between the acquired two left and right images depending on the distance from the camera to the subject (hereinafter referred to as “subject distance”). The problem occurs.

  Hereinafter, the above problem will be described.

  Factors that cause the vertical displacement between the left and right images include the parallel movement of each camera built in the stereoscopic image capturing device (the displacement of the mounting position of each camera in the vertical direction), the tilt of each camera (each camera's tilt). The tilt of the optical axis in the vertical direction) and the rotation of each camera (rotation shift around the optical axis of the camera) can be mentioned. In the case where the mounting displacement of these cameras with respect to the stereoscopic image capturing apparatus is an error caused in accordance with the assembly accuracy of the camera and / or the mounting position accuracy of the camera, it is difficult to eliminate structurally.

  The above-described vertical shift caused by the parallel movement of the camera and the above-described vertical shift caused by the rotation of the camera differ in the amount of deviation depending on the subject distance. On the other hand, the vertical shift due to camera tilt is a constant shift amount regardless of the subject distance.

  Here, in the technique disclosed in Patent Document 1, the vertical shift due to the parallel movement of the camera and the vertical shift due to the rotation of the camera are distinguished from the vertical shift due to the camera tilt. I can't. Further, in the technique disclosed in Patent Document 1, since the positional deviation between the left and right images is corrected based on the deviation vector, it is not possible to correct the above-described vertical deviation in which the amount of deviation differs depending on the subject distance.

  Therefore, with the technique disclosed in Patent Document 1, even when the vertical shift is corrected based on a specific subject distance, the vertical shift may occur at another subject distance.

  Further, in the technique disclosed in Patent Document 2, there is no mention of vertical trimming of left and right images.

  Further, the technique disclosed in Patent Document 3 has the same problem as the technique disclosed in Patent Document 1.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image correction method capable of reducing vertical displacement between two acquired left and right images according to the subject distance. And providing a stereoscopic image capturing apparatus.

In order to solve the above problems, an image correction method of the present invention includes a first image acquired from a first camera that constitutes a camera group including a plurality of cameras, and a first camera that constitutes the camera group. An image correction method for trimming at least one of a first image and a second image to obtain a stereoscopic image from a second image acquired from another second camera,
(1) A first reference image obtained by photographing a first subject and a second subject having different distances to the camera group with the first camera and the second camera, and the first camera photographing the first subject. A second reference image obtained by photographing the first subject with the second camera, a third reference image obtained by photographing the second subject with the first camera, and a second reference image obtained by the second camera. A first step of acquiring a fourth reference image obtained by
(2) At least two feature points of the first subject are detected from the first reference image and the second reference image, and feature points of the second subject are detected from the third reference image and the fourth reference image. A second step of detecting at least one;
(3) From each feature point of the first subject detected in the second step, the first centered on the optical axis of the second camera relative to the rotation angle of the first camera about the optical axis of the first camera. A third step of calculating a mutual rotation angle difference, which is a difference between rotation angles of the two cameras, and correcting at least one of the first reference image and the second reference image so as to cancel the mutual rotation angle difference;
(4) The first reference image and the second reference image are rotated by the same predetermined angle with each other, and the feature points of the first subject in the vertical direction between the first reference image and the second reference image A fourth step of calculating a first vertical shift that is a shift;
(5) At least one of the third reference image and the fourth reference image is corrected so as to cancel the mutual rotation angle difference and the first vertical shift, and rotated by the same predetermined angle as each other, A fifth step of calculating a second vertical shift, which is a vertical shift between the third reference image and the fourth reference image, of the feature point of the second subject detected in the two steps;
(6) including a sixth step of performing the fourth step and the fifth step for a plurality of the predetermined angles to obtain the predetermined angle and the first vertical shift, wherein the second vertical shift is minimized,
Trimming at least one of the first image and the second image so as to cancel out the predetermined angle and the first vertical shift that minimizes the second vertical shift obtained in the sixth step. It is said.

  There are four factors (factor A) to (factor D) that cause the subject to shift between the first image and the second image.

  (Factor A) Deviation due to a difference in rotation angle between the first camera and the second camera.

  (Factor B) Deviation caused by rotation of the first camera and the second camera in the same direction by the same rotation angle, or deviation in the vertical mounting position between the first camera and the second camera.

  (Factor C) Deviation due to vertical tilt between the optical axes of the first camera and the second camera.

  (Factor D) Deviation due to tilt in the left-right direction between the optical axes of the first camera and the second camera.

  The vertical displacement of the camera according to (Factor B) is the same as that of the camera according to (Factor B), assuming that the straight line passing through the center positions of the lenses of the first camera and the second camera is horizontal. It can be treated as equivalent to a shift caused by rotation at the same rotation angle.

  Among (Factor A) to (Factor D), it is (Factor B) and (Factor C) that cause the vertical shift between the first image and the second image. The vertical shift according to (Factor B) varies in the vertical direction depending on the subject distance, whereas the vertical shift according to (Factor C) does not depend on the subject distance. Therefore, it is necessary to perform correction separately for (Factor B) and (Factor C), but these cannot be distinguished only by photographing a subject at a specific distance.

  According to said structure, it becomes possible to correct | amend (factor A) by a 3rd process. Moreover, according to said structure, it becomes possible to correct | amend (factor B) by a 4th process. Moreover, according to said structure, it becomes possible to correct | amend (factor C) by a 5th process.

  Then, in the sixth step, the correction of (factor B) and (factor C) is optimized by repeating the fourth step and the fifth step while rotating the first reference image and the second reference image at the same angle. Is possible.

  As described above, according to the image correction method of the present invention, the vertical displacement of the image can be corrected by separately correcting (Factor B) and (Factor C). It is possible to reduce the vertical shift between the two right and left images.

  By the way, according to the above configuration, it is necessary to detect two or more feature points of the first subject.

  On the other hand, in order to calculate the mutual rotation angle difference from the first reference image and the second reference image, the first reference can be used to specify the rotation angle of the second reference image with respect to the first reference image. It is sufficient to detect from the image and the second reference image. “Those that can specify the rotation angle of the second reference image with respect to the first reference image” are lines that appear in both the first reference image and the second reference image. Note that this line need not be a subject.

  On the other hand, in order to calculate the first vertical shift from the first reference image and the second reference image, it is necessary to detect one or more feature points of the first subject.

That is, in order to solve the above-described problem, the image correction method of the present invention includes a first image acquired from a first camera constituting a camera group including a plurality of cameras, and a first image constituting the camera group. An image correction method for trimming at least one of a first image and a second image in order to obtain a stereoscopic image from a second image acquired from a second camera different from the camera,
(1) The first and second subjects having different distances to the camera group are photographed by the first camera and the second camera, and the first camera includes the first subject and the periphery of the first subject. A first reference image obtained by photographing the region, a second reference image obtained by photographing the subject region by the second camera, and a third reference image obtained by photographing the second subject by the first camera. A first step in which the second camera acquires a fourth reference image obtained by photographing the second subject;
(2) At least one feature point of the first subject and a line appearing in the subject area are detected from the first reference image and the second reference image, and from the third reference image and the fourth reference image. A second step of detecting at least one feature point of the second subject;
(3) The optical axis of the second camera with respect to the rotation angle of the first camera around the optical axis of the first camera from the feature point of the first subject detected in the second step and the line appearing in the subject area. Calculating a mutual rotation angle difference, which is a difference between rotation angles of the second camera around the center, and correcting at least one of the first reference image and the second reference image so as to cancel the mutual rotation angle difference. 3 steps,
(4) The first reference image and the second reference image are rotated by the same predetermined angle, and the line between the first reference image and the second reference image of the feature point of the first subject and the line appearing in the subject area A fourth step of calculating a first vertical shift that is a shift in the vertical direction at
(5) At least one of the third reference image and the fourth reference image is corrected so as to cancel the mutual rotation angle difference and the first vertical shift, and rotated by the same predetermined angle as each other, A fifth step of calculating a second vertical shift, which is a vertical shift between the third reference image and the fourth reference image, of the feature point of the second subject detected in the two steps;
(6) including a sixth step of performing the fourth step and the fifth step for a plurality of the predetermined angles to obtain the predetermined angle and the first vertical shift, wherein the second vertical shift is minimized,
Trimming at least one of the first image and the second image so as to cancel out the predetermined angle and the first vertical shift that minimizes the second vertical shift obtained in the sixth step. It can also be.

  According to the above configuration, instead of detecting two feature points from the first subject, one feature point is detected from the first subject, and a line that appears in the subject region, in other words, the second reference point for the first reference image. The same method can be implemented even if a reference image whose rotation angle can be specified is detected.

  Further, the image correction method of the present invention includes a shift of the first subject in the left-right direction between the first reference image and the second reference image, and a third reference image and a fourth reference of the second subject. A seventh step of calculating at least one of a shift in the left-right direction with respect to the image, wherein at least one of the first image and the second image is offset so as to cancel the shift obtained in the seventh step. Trimming is preferred.

  According to the above configuration, it is possible to reduce the shift in the left-right direction between the acquired two left and right images according to the subject distance.

  In the image correction method of the present invention, in the first step, the first camera and the second camera are photographed so that both the first subject and the second subject appear in one image. At least one may be performed.

  That is, only the first camera may perform shooting so that the first subject and the second subject appear in one image, or only the second camera shows the first subject and the second subject in one image. Alternatively, both the first camera and the second camera may shoot so that the first subject and the second subject appear in one image.

  The stereoscopic image capturing apparatus of the present invention includes a trimming unit that trims at least one of the first image and the second image by the image correction method of the present invention, and the camera group. .

  According to said structure, it becomes possible to implement | achieve the stereo image imaging device which show | plays the effect similar to the image correction method of this invention.

  The stereoscopic image capturing apparatus of the present invention is obtained by trimming the first image by the trimming unit when the mounting position of the first camera is shifted in the vertical direction with respect to the second camera. Both the image and the image obtained by trimming the second image by the trimming unit are rotated at the same rotation angle.

  According to the image correction method of the present invention, when the mounting position of the first camera in the stereoscopic image capturing apparatus is deviated vertically with respect to the second camera, the center of the first camera and the second camera are accordingly moved. As a result of performing trimming on the basis of a straight line passing through the center, both the image obtained by trimming the first image and the image obtained by trimming the second image rotate at the same rotation angle. A phenomenon occurs. The stereoscopic image capturing apparatus in which the phenomenon occurs may be considered as the stereoscopic image capturing apparatus of the present invention.

  As described above, in the image correction method of the present invention, the first image acquired from the first camera constituting the camera group including a plurality of cameras, and the second camera different from the first camera constituting the camera group. An image correction method for trimming at least one of a first image and a second image in order to obtain a stereoscopic image from a second image acquired from the first subject and the second subject having different distances to the camera group Is captured by the first camera and the second camera, and the first reference image obtained by photographing the first subject by the first camera and the second reference image obtained by photographing the first subject by the second camera. A first step of obtaining a third reference image obtained by photographing the second subject by the first camera, and a fourth reference image obtained by photographing the second subject by the second camera; Reference image and second reference A second step of detecting at least two feature points of the first subject from the image and detecting at least one feature point of the second subject from the third reference image and the fourth reference image; The rotation angle of the second camera around the optical axis of the second camera relative to the rotation angle of the first camera around the optical axis of the first camera from each feature point of the first subject detected in two steps A third step of calculating at least one of the first reference image and the second reference image so as to cancel the mutual rotation angle difference, and calculating the mutual rotation angle difference that is a difference between the first reference image and the first reference image; The second reference image is rotated by the same predetermined angle as each other, and a first vertical shift, which is a vertical shift between the first reference image and the second reference image, of each feature point of the first subject is calculated. And a fourth step to In the second step, at least one of the third reference image and the fourth reference image is corrected so as to cancel the mutual rotation angle difference and the first vertical shift and rotate by the same predetermined angle. A fifth step of calculating a second vertical shift that is a vertical shift between the third reference image and the fourth reference image of the detected feature point of the second subject, and a plurality of the predetermined angles, Including the sixth step of performing the fourth step and the fifth step, and obtaining the predetermined angle and the first vertical deviation, wherein the second vertical deviation is minimized, and obtaining the second vertical deviation obtained in the sixth step. At least one of the first image and the second image is trimmed so as to cancel out the predetermined angle and the first vertical shift that minimize the shift.

  In addition, the image correction method of the present invention acquires a first image acquired from a first camera constituting a camera group including a plurality of cameras, and a second camera separate from the first camera constituting the camera group. An image correction method for trimming at least one of the first image and the second image to obtain a stereoscopic image from the second image, wherein the first subject and the second subject having different distances to the camera group are A first reference image obtained by photographing with the first camera and the second camera, the first camera photographing a subject region including the first subject and the periphery of the first subject, and a second camera capturing the subject region. A second reference image obtained by photographing, a third reference image obtained by photographing the second subject with the first camera, and a fourth reference image obtained by photographing the second subject with the second camera are acquired. The first construction And detecting at least one feature point of the first subject and a line appearing in the subject region from the first reference image and the second reference image, and from the third reference image and the fourth reference image, From the second step of detecting at least one feature point of the second subject and the feature point of the first subject detected in the second step and a line appearing in the subject region, the optical axis of the first camera is the center. Calculating the mutual rotation angle difference, which is the difference of the rotation angle of the second camera around the optical axis of the second camera with respect to the rotation angle of the first camera, and canceling out the mutual rotation angle difference. A third step of correcting at least one of the first reference image and the second reference image, and the first reference image and the second reference image are rotated by the same predetermined angle, and the feature point and the subject area of the first subject A fourth step of calculating a first vertical shift, which is a vertical shift between the first reference image and the second reference image, and at least one of the third reference image and the fourth reference image; The third reference image of the feature point of the second subject detected in the second step, corrected so as to cancel out the mutual rotation angle difference and the first vertical shift, and rotated by the same predetermined angle as each other A fifth step of calculating a second vertical shift that is a vertical shift between the first reference image and the fourth reference image, and the fourth step and the fifth step for the plurality of the predetermined angles, And a sixth step for obtaining the predetermined angle and the first vertical deviation, which minimizes the deviation, and the predetermined angle and the first vertical deviation, which are obtained in the sixth step, so that the second vertical deviation is minimized. To offset Trimming at least one of the first image and the second image.

  Therefore, there is an effect that it is possible to reduce the vertical shift between the acquired two left and right images according to the subject distance.

It is a flowchart which shows the flow of the image correction method which concerns on this invention. FIGS. 2A and 2B are views schematically showing the appearance of a mobile phone as a stereoscopic image capturing apparatus according to the present invention. FIG. 3 is a functional block diagram showing a main configuration of the mobile phone shown in FIG. 2 related to stereoscopic image shooting. It is a figure which shows the vertical error and rotation error which arise between the image acquired from the 1st imaging | photography part, and the image acquired from the 2nd imaging | photography part. It is a figure which shows the prior art example of the technique which correct | amends both the images acquired from the 1st imaging | photography part and the 2nd imaging | photography part so that it may become optimal arrangement | positioning. It is a figure which shows the principle which a vertical shift generate | occur | produces between the image for left eyes, and the image for right eyes. It is another figure which shows the principle which a vertical shift generate | occur | produces between the image for left eyes, and the image for right eyes. It is another figure which shows the principle which a vertical shift generate | occur | produces between the image for left eyes, and the image for right eyes. It is another figure which shows the principle which a vertical shift generate | occur | produces between the image for left eyes, and the image for right eyes. It is a figure which shows that the rotation angle which the image for left eyes and the image for right eyes becomes infinite exists infinitely. (A) to (d) of FIG. 11 are diagrams showing the principle that a phenomenon corresponding to the parallel movement of the camera occurs with the rotation of the camera. It is a figure which shows an example of correction | amendment of a general image conventionally. FIG. 13 is a conceptual diagram illustrating the movement of the left camera and the right camera when the image correction according to FIG. 12 is replaced with the positions of the left camera and the right camera. It is a figure which shows the state which the up-and-down shift has generate | occur | produced between the correction image for left eyes, and the correction image for right eyes. It is a figure which shows a mode that rotation of the camera of a mutually different rotation angle generate | occur | produces and an angle shift | offset | difference generate | occur | produces between the image for left eyes, and the image for right eyes. It is a figure which shows the point which trims an image when rotation of the camera of a mutually different rotation angle generate | occur | produces. It is a figure which shows a mode that the rotation of the camera of the same direction and the same rotation angle mutually, or the parallel movement of a camera generate | occur | produces, and a vertical shift arises between the image for left eyes, and the image for right eyes. It is a figure which shows the point which trims an image when the rotation of the camera of the same direction and the same rotation angle or the parallel movement of a camera generate | occur | produces. It is a figure which shows a mode that the optical axis of one camera has shifted | deviated to the up-down direction with respect to the optical axis of the other camera, and the vertical shift generate | occur | produced between the image for left eyes, and the image for right eyes. It is a figure which shows the point which trims an image when the shift | offset | difference to the up-down direction of an optical axis generate | occur | produced. It is a figure which shows a mode that a left-right shift | offset | difference arises between the image for left eyes, and the image for right eyes. It is a figure which shows the point which trims an image when the shift | offset | difference to the left-right direction of an optical axis generate | occur | produced. It is a table | surface which shows the list of the coordinate of the feature point detected in an image correction method. It is a table | surface which shows the list of the cutout coordinates of an image. It is a table | surface which shows the list (an example) of the constant used in an image correction method. It is a table | surface which shows the list of the variables used in an image correction method. FIG. 27A is a diagram illustrating a state in which feature points are detected from the left side of the image A, and FIG. 27B is a diagram illustrating a state in which feature points are detected from the right side of the image A. FIG. ) Is a diagram illustrating a state in which feature points are detected from the left side of the image B, and FIG. 27D is a diagram illustrating a state in which feature points are detected from the right side of the image B.

[Configuration example of stereoscopic image capturing apparatus]
FIGS. 2A and 2B are views schematically showing the appearance of a mobile phone as a stereoscopic image capturing apparatus according to the present invention.

  As shown in FIG. 2B, the mobile phone (stereoscopic image capturing device) 1 includes a display unit (for example, a liquid crystal display unit) 2 on the front surface. In addition, as shown in FIG. 2A, the mobile phone 1 includes a first camera lens 3 and a second camera lens 4 on the back surface.

  The display unit 2 may have a touch panel function, and the user can input various instructions to the mobile phone 1 by touching the display unit 2.

  The first camera lens 3 and the second camera lens 4 are separated from each other by a predetermined interval (about several centimeters). As a result, when a subject scene (subject) is photographed through the first camera lens 3 and the second camera lens 4 when a stereoscopic image is photographed, two images with parallax can be acquired.

  The mobile phone 1 is not limited to the above configuration, and a configuration in which a foldable display unit is combined with an operation unit in which a plurality of buttons for inputting various instructions by a user is arranged, or the above operation unit is used. A configuration that can be slid in parallel with the display unit may be employed.

  In addition, the stereoscopic image capturing device according to the present invention can be mounted not only on a mobile phone but also in all portable electronic devices such as a digital camera and a PDA (Personal Digital Assistant). Furthermore, the stereoscopic image capturing device according to the present invention can be mounted on a television (3D television) capable of stereoscopic display.

  FIG. 3 is a functional block diagram showing the main configuration of the mobile phone 1 related to stereoscopic image shooting.

  As shown in FIG. 3, the mobile phone 1 includes a first photographing unit (first camera) 11, a second photographing unit (second camera) 12, a first control unit 13, a second control unit 14, an image storage unit 15, A trimming unit 16 and a display unit 17 are provided. The 1st imaging | photography part 11 and the 2nd imaging | photography part 12 comprise the camera group provided with the several camera.

  The first imaging unit 11 includes a first camera lens 3, an image sensor, an AD (Analog / Digital) converter, and the like. The second imaging unit 12 includes a second camera lens 4, an image sensor, an AD converter, and the like. Examples of the imaging element include a solid-state imaging element constituted by a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

  Each of the first control unit 13 and the second control unit 14 includes a DSP (Digital Signal Processor) and the like.

  The first control unit 13 controls the time (shutter time) when the imaging device of the first imaging unit 11 is exposed by the scene to be imaged, the output timing of the imaging signal by the imaging device, and the like, and the output imaging signal To generate image data with adjusted gain, image quality, and the like. Further, the first control unit 13 detects accompanying information such as lens position information (focus information) of the first camera lens 3. The second control unit 14 also has the same configuration as the first control unit 13 except that the control target is changed from the first imaging unit 11 to the second imaging unit 12.

  The image storage unit 15 is configured by, for example, a RAM (Random Access Memory). The image storage unit 15 includes image data generated by the first control unit 13 (hereinafter referred to as “first image data”) and image data generated by the second control unit 14 (hereinafter referred to as “second image data”). Stored). In addition, the image storage unit 15 includes image data obtained by performing trimming on the first image data (first image) by the trimming unit 16 (hereinafter referred to as “first corrected image data”), and second image data ( Image data (hereinafter referred to as “second corrected image data”) obtained by trimming the second image) by the trimming unit 16 is stored.

  The trimming unit 16 trims the first image data and the second image data based on image correction parameters acquired in advance. That is, the trimming unit 16 generates the first corrected image data by performing image processing for cutting out an image outside the desired region on the first image data. Similarly, the trimming unit 16 generates second corrected image data by performing image processing on the second image data to cut out an image outside the desired region. A detailed description of the image correction parameters will be described later.

  The display unit 17 displays various image data stored in the image storage unit 15. Specifically, the display unit 17 uses the first image data and the second image data, or the first corrected image data and the second corrected image data, to display a stereoscopic image that can be visually recognized by the user with the naked eye.

〔background〕
The image correction method described in the embodiment relates to the mobile phone 1 in order to realize a desired parallax between the two left and right images respectively acquired from the first photographing unit 11 and the second photographing unit 12. This is a method of obtaining image correction parameters for trimming an image.

  In order to obtain a stereoscopic image by photographing with the first photographing unit 11 and the second photographing unit 12, an arbitrary subject distance that appears in each image obtained from the first photographing unit 11 and the second photographing unit 12 It is required that the subject located at (the distance to the camera group) has no angle shift or vertical shift between both images. Further, regarding the first photographing unit 11 and the second photographing unit 12 as two cameras, the optical axes need to be parallel or intersect at a desired convergence point.

  However, it is difficult to eliminate errors due to camera assembly accuracy and mounting position accuracy by structural improvements of the camera.

  That is, most stereoscopic image capturing apparatuses have a dimensional error of the mechanism, that is, a manufacturing tolerance.

  In order to reduce the manufacturing tolerance, a high-precision processing technique is required, and if a high-precision processing technique is employed, the cost of the stereoscopic image capturing apparatus is increased. In addition, since a mechanism for holding each component constituting the stereoscopic image capturing apparatus becomes large, reducing the manufacturing tolerance causes an increase in size and cost of the stereoscopic image capturing apparatus.

  For this reason, it is necessary to adjust these errors in the image acquired from the camera by cutting out (trimming) the image acquired from the camera. That is, the amount of deviation of each camera in the three-dimensional image capturing device immediately after assembly varies from device to device. For this reason, it is necessary to correct the above-described vertical shift by performing photographing for each device, confirming the variation for each device, and cutting out a region corresponding to the variation for each device.

  As for the above-mentioned vertical shift, even if the vertical shift is corrected by rotating and extracting a camera image with respect to a subject at a specific subject distance, the vertical shift may occur when a subject at a different subject distance is photographed. .

[Principle of image correction in stereoscopic imaging device]
As factors that cause parallel movement and tilt of the camera, the assembly accuracy of each of the first photographing unit 11 and the second photographing unit 12 and / or attachment of the first photographing unit 11 and the second photographing unit 12 to the mobile phone 1 It is mentioned that the accuracy cannot be increased due to the balance between downsizing and cost reduction of the stereoscopic image capturing apparatus. Since these precisions cannot be increased, there is a possibility that a vertical error and a rotation error (vertical shift) may occur between the image acquired from the first imaging unit 11 and the image acquired from the second imaging unit 12. is there. The vertical error is a vertical shift generated so that one image is shifted in the vertical direction with respect to the other image, and is indicated by reference numeral 41 in FIG. On the other hand, the rotation error is a vertical shift generated so that one image rotates with respect to the other image on a plane substantially parallel to the one image (around the optical axis). Is shown.

  Therefore, conventionally, in order to correct the vertical error and the rotation error described above, in the production process of the mobile phone 1, both images acquired from the first photographing unit 11 and the second photographing unit 12 mainly by software are optimally arranged. There is known a technique for correcting so as to be (see FIG. 5). In FIG. 5, so-called ISP (Image Signal Processor) processing such as trimming, rotation, and enlargement / reduction of each image is performed to correct each image.

  The “left-eye photographed image” indicates a left-eye image (here, an image acquired from the first photographing unit 11) described later. Further, the “right-eye photographed image” indicates a right-eye image (here, an image acquired from the second photographing unit 12) described later.

[Details of the principle of vertical displacement]
The details of the principle of vertical displacement will be described below. For convenience of description, in the following description with reference to FIGS. 6 to 9, it is assumed that the difference in arrangement of the left and right cameras in the horizontal direction is zero in the back surface portion of the mobile phone 1.

  FIG. 6 illustrates an image acquired from the left camera 60 that captures an image for the left eye as the first imaging unit 11 and an image acquired from the right camera 61 that captures an image for the right eye as the second imaging unit 12. It is a figure which shows the principle which a vertical shift generate | occur | produces between.

  Hereinafter, an image acquired from the left camera 60 is referred to as a “left-eye image (first image)”, and an image acquired from the right camera 61 is referred to as a “right-eye image (second image)”.

  In FIG. 6, when the left camera 60 is used as a reference, the right camera 61 undergoes an upward parallel movement (movement amount y <b> 1) with respect to the left camera 60. Further, the right camera 61 is tilted upward (tilt angle β1) with respect to the left camera 60.

  Here, consider a case where the left camera 60 and the right camera 61 are used to photograph the center 63 of the test chart 62 at the same height as the optical axis of the left camera 60 that is horizontal. In the case of the example shown in FIG. 6, due to the parallel movement of the right camera 61, a straight line passing through the center of the lens of the right camera 61 and the center 63 is inclined so that the center 63 side is down.

  As a result, as shown in the finder screen F in FIG. 6, the left-eye image 63 </ b> L at the center 63 is positioned at the center of the finder screen F, while the right-eye image 63 </ b> R at the center 63 is lower than the center of the finder screen F. Located in. The distance between the right-eye image 63R and the lower side of the finder screen F is t1.

  In FIG. 6, the apparent tilt angle of the right camera 61 (that is, the substantial tilt angle when shooting the center 63 below the right camera 61) in consideration of the inclination of the straight line is defined as the inclination angle β2. Yes.

  FIG. 7 is another diagram illustrating the principle of vertical displacement between the left-eye image and the right-eye image.

  In FIG. 7, when the left camera 60 is used as a reference, the right camera 61 does not move in parallel with the left camera 60 (movement amount y2 = 0). In FIG. 7, it is assumed that the right camera 61 has an upward tilt (the same tilt angle β2 as in FIG. 6) with respect to the left camera 60.

  Here, the movement amount y1 and the movement amount y2 are different, and the inclination angle β1 and the inclination angle β2 are different. However, the position of the center 63 in the left-eye image (corresponding to the first image data) and the position of the center 63 in the right-eye image (corresponding to the second image data) are the same as the example shown in FIG. 6 and the example shown in FIG. Match.

  That is, as shown in the finder screen F in FIG. 7, the left-eye image 63 </ b> L at the center 63 is positioned at the center of the finder screen F, while the right-eye image 63 </ b> R at the center 63 extends downward from the center of the finder screen F. To position. Here, if the distance between the right-eye image 63R and the lower side of the finder screen F is t2, the tilt angle β2 of the right camera 61 is the same in FIGS. 6 and 7, and therefore the distance t2 is equal to the distance t1.

  That is, when a subject at a certain subject distance is photographed, even if translation and tilting occur in the left camera 60 and / or the right camera 61, between the images acquired from both cameras, At first glance, the vertical misalignment seems to match.

  That is, if only a subject at a certain subject distance is photographed, the vertical shift between the two acquired images is caused by the parallel movement in the left camera 60 and / or the right camera 61, or due to tilt. I can't tell you are doing it.

  On the other hand, when a plurality of subjects at different subject distances are photographed, whether the vertical shift between the acquired images is caused by the parallel movement in the left camera 60 and / or the right camera 61, or is tilted. It becomes possible to distinguish whether it is caused or not. This principle will be described below with reference to FIGS.

  FIG. 8 is still another diagram illustrating the principle of vertical displacement between the left-eye image and the right-eye image.

  FIG. 9 is another diagram illustrating the principle of vertical displacement between the left-eye image and the right-eye image.

  The parallel movement and tilt of the right camera 61 are the same in the example shown in FIG. 6 and the example shown in FIG. 8, and the same in the example shown in FIG. 7 and the example shown in FIG. 9.

  On the other hand, each example shown in FIGS. 8 and 9 is different from each example shown in FIGS. 6 and 7 in that a test chart 64 is arranged at a position different from the test chart 62. 8 and 9, the test chart 64 is disposed closer to the left camera 60 and the right camera 61 than the test chart 62, and the center is the center 65.

  Here, a case where the centers 63 and 65 are photographed by the left camera 60 and the right camera 61 is considered.

  In the case of the example shown in FIG. 8, due to the parallel movement of the right camera 61, a straight line passing through the center of the lens of the right camera 61 and the center 63 is inclined so that the center 63 side is down. In FIG. 8, the inclination angle of the right camera 61 with respect to the test chart 62 in consideration of the inclination of the straight line (that is, a substantial tilt angle when photographing the center 63 below the right camera 61) is expressed as the inclination angle. β2_1.

  Further, in the case of the example shown in FIG. 8, due to the parallel movement of the right camera 61, a straight line passing through the center of the lens of the right camera 61 and the center 65 is inclined so that the center 65 side is down. In FIG. 8, the inclination angle of the right camera 61 with respect to the test chart 64 in consideration of the inclination of the straight line (that is, a substantial tilt angle when the center 65 under the right camera 61 is photographed) is expressed as the inclination angle. β2_2.

  Here, the straight line passing through the center of the lens of the right camera 61 and the center 65 has a larger inclination than the straight line passing through the center of the lens of the right camera 61 and the center 63. Accordingly, the inclination angle β2_2 is different from the inclination angle β2_1 and is larger than the inclination angle β2_1.

  As a result, as shown in the finder screen F in FIG. 8, the left-eye image 63 </ b> L at the center 63 is positioned at the center of the finder screen F, while the right-eye image 63 </ b> R at the center 63 extends downward from the center of the finder screen F. To position. The positions of the left-eye image 63L and the right-eye image 63R on the finder screen F are the same as those in FIG. 6 if the inclination angle β2_1 and the inclination angle β2 are equal. On the other hand, the left-eye image 65L at the center 65 is positioned at the center of the finder screen F, while the right-eye image 65R at the center 65 is positioned further below the right-eye image 63R from the center of the finder screen F.

  On the other hand, in FIG. 9, when the left camera 60 is used as a reference, the right camera 61 does not move parallel to the left camera 60 (movement amount y2 = 0). In FIG. 9, the right camera 61 has an upward tilt (inclination angle β <b> 2) with respect to the left camera 60. The tilt angle of the right camera 61 due to the tilt in the upward direction is the tilt angle β2 (equal) with respect to both the test chart 62 and the test chart 64.

  As a result, as shown in the finder screen F in FIG. 9, the left-eye image 63L at the center 63 is positioned at the center of the finder screen F, while the right-eye image 63R at the center 63 is located below the center of the finder screen F. Located in. On the other hand, the left-eye image 65L at the center 65 is positioned at the center of the finder screen F, while the right-eye image 65R at the center 65 has the same inclination angle β2 with respect to the optical axis of the center 63 and the right camera 61 at the center 65. Therefore, the position is the same as that of the right-eye image 63R.

  In this way, the position of the left-eye image 65L at the center 65 matches the example shown in FIG. 8 and the example shown in FIG. 9, but the position of the right-eye image 65R at the center 65 corresponds to the example shown in FIG. Shifts up and down from the example shown in FIG.

  That is, when only the camera tilt occurs, the tilt is the same angle regardless of the subject distance, as shown in FIG. On the other hand, when the vertical translation of the camera occurs, as shown in FIG. 8, an apparent tilt angle of the camera that varies depending on the subject distance is generated. Therefore, the vertical deviation of the captured image due to the camera tilt is constant regardless of the subject distance, but the vertical deviation of the captured image due to the parallel movement of the camera depends on the subject distance. I can see that they are different.

  That is, by photographing a plurality of subjects having different subject distances, it is possible to distinguish between parallel movement and tilt in the left camera 60 and / or the right camera 61 from the vertical shift between the acquired images. .

  In an actual camera, the left camera 60 and the right camera 61 are arranged apart from each other in the left-right direction, so that the position in the left-right direction is shifted between the left-eye image 63L and the right-eye image 63R on the finder screen F. It will be. In particular, as shown in FIG. 8, when the test charts 62 and 64 are arranged at different distances, the position in the left-right direction differs depending on the distance of the test chart. For example, the left-eye image 63L and the right-eye image 63R in the finder screen F in FIG. 8 are actually shifted in the left-right direction.

  Although details will be described later, in the actual image correction parameter calculation, each of the left camera 60 and the right camera 61 uses a distant subject (test chart 62) and a near subject (test chart 64). A total of four images obtained by photographing are used. Then, by subtracting the coordinates of the same feature point appearing in each image, the movement amount of the left camera 60 and / or the right camera 61 due to the parallel movement of the camera and the tilt angle due to the tilt of the camera are calculated.

  In addition, the amount of vertical displacement between both images to be calculated from the left-eye image and the right-eye image includes the movement amount due to the parallel movement of the camera, the tilt angle due to the camera tilt, and the rotation angle due to the camera rotation. . Regarding the rotation of the camera, since there are an infinite number of rotation angles at which the left-eye image and the right-eye image are parallel, it is not possible to determine an appropriate image correction parameter simply by photographing a subject at a single subject distance. .

  FIG. 10 is a diagram illustrating that the rotation angle at which the left-eye image and the right-eye image are parallel exists infinitely.

  In FIG. 10, “left-eye photographed image” indicates a left-eye image. In FIG. 10, “right-eye photographed image” indicates a right-eye image. In FIG. 10, “left-adjusted image” indicates an image obtained by correcting the left-eye photographed image. In FIG. 10, “right-adjusted image” indicates an image obtained by correcting the right-eye photographed image.

  FIG. 10 shows two plans for correcting the left-eye shot image and the right-eye shot image, which are referred to as correction plan 1 and correction plan 2, respectively.

  In both of the correction plan 1 and the correction plan 2, the left-eye image and the right-eye image are corrected so that the subjects 100 appearing in both the left-eye image and the right-eye image are parallel to each other, and the left-adjusted image And a right-adjusted image. That is, in both the correction plan 1 and the correction plan 2, the subject 100 is parallel between the left adjusted image and the right adjusted image.

  However, the top and bottom of the subject 100 are different between the correction plan 1 and the correction plan 2. Accordingly, at least one of the correction plan 1 and the correction plan 2 is not corrected for the image even though the subject 100 is corrected in parallel between the left adjusted image and the right adjusted image. Approved as appropriate. Here, since the broken line indicating the trimming area of the correction plan 1 is inclined, the correction plan 1 has a rotational shift.

  In addition, the vertical shift of the captured image caused by the rotation of the camera differs depending on the subject distance, similarly to the vertical shift of the captured image caused by the parallel movement of the camera. Hereinafter, this reason will be described.

  (A) to (d) of FIG. 11 are diagrams showing the principle that a phenomenon corresponding to the parallel movement of the camera occurs with the rotation of the camera.

  FIGS. 11A and 11C show a state in which the left camera 60 and the right camera 61 do not generate any parallel movement or tilt, but rotate. 11A and 11C, the straight line passing through the center 111 that is the center of the lens 110 of the left camera 60 and the center 113 that is the center of the lens 112 of the right camera 61, that is, the straight line 114 is horizontal. It is illustrated as follows.

  On the other hand, FIGS. 11B and 11D show the states shown in FIGS. 11A and 11C, respectively, so that the left camera 60 is upright. 11 (a) and FIG. 11 (b) are equivalent, and FIG. 11 (c) and FIG. 11 (d) are equivalent.

  According to (b) and (d) of FIG. 11, when the left camera 60 is in an upright state, the straight line 114 does not coincide with the horizontal direction (the direction perpendicular to the vertical direction) in the left-eye image. . The fact that the straight line 114 is inclined with respect to the horizontal direction in the image for the left eye means that the center 113 is shifted in the vertical direction with respect to the center 111. In other words, a phenomenon corresponding to the parallel movement of the right camera 61 relative to the left camera 60 occurs. Although not shown, the same applies to the case where the right camera 61 is upright.

  When the image correction parameter candidate calculated from one image is applied to an image of a subject photographed at a different subject distance, if the parameter for the rotation angle is incorrect, a vertical shift occurs in the corrected image. In the case of the correct parameter, no vertical deviation occurs. Therefore, as will be described in detail later, even in the production process of the mobile phone 1, shooting is performed at a plurality of subject distances, and the vertical shift between both images is minimized in both the left-eye image and the right-eye image. A method of selecting a parameter as the correct parameter is selected.

[Summary about vertical deviation]
A so-called two-lens 3D camera that captures a stereoscopic image with two cameras on the left and right has a subject misalignment between the acquired images for the left and right eyes, depending on the accuracy of the camera itself and the mounting position of the two cameras. Can occur. The main factors of the positional deviation include camera tilt and camera rotation. In addition, the positional deviation of the camera in the left-right direction may be the cause of the positional deviation.

  The positional deviation can be corrected by rotating, translating, and trimming the left-eye image and the right-eye image.

  Conventionally, with respect to one of the left-eye image and the right-eye image as a reference, if the other image has an angle shift, the rotation angle is corrected, and if there is a vertical / left-right position shift, a correction by parallel movement has been performed.

  FIG. 12 is a diagram showing an example of conventional image correction. FIG. 12 shows an example in which the right-eye image 121 is corrected using the left-eye image 120 as a reference.

  Since the left-eye image 120 is a reference, trimming is performed while maintaining the rotation angle and the vertical and horizontal positions, and the left-eye correction image 122 is obtained.

  On the other hand, the rotation angle of the right-eye image 121 is corrected so that the rotation angle is equal to that of the left-eye image 120, and correction by parallel movement is performed so that the vertical and horizontal positions are equal to those of the left-eye image 120. . In other words, the right-eye image 121 is trimmed so as to reflect the correction of the rotation angle and the correction by the parallel movement, and becomes the right-eye correction image 123.

  Note that the result of the correction of the rotation angle varies depending on where the rotation center is set, and accordingly, the correction amount by the correction by the parallel movement also changes. For example, in the example shown in FIG. 12, in the correction of the rotation angle, when the right image is rotated to the left with the center of the image as the rotation center, the correction amount with respect to the vertical direction of the subject becomes very small, almost zero. Become. As described above, when there is a rotation angle correction, the rotation correction and the parallel movement are not independent items. As a result, the upper and lower axes of the image itself are rotated by the rotation correction.

  FIG. 13 is a conceptual diagram illustrating the movement of the left camera 60 and the right camera 61 when the image correction according to FIG. 12 is replaced with the positions of the left camera 60 and the right camera 61.

  FIG. 13 shows that the same image as that obtained when the camera tilt or camera rotation does not occur in the left camera 60 and the right camera 61 by the image correction according to FIG.

  However, the left camera 60 and the right camera 61 may have a rotation error that causes a vertical shift as described in the item [Principle of image correction in the stereoscopic image capturing apparatus]. In this case, even if the subject that appears in one of the left-eye image and the right-eye image matches the subject that appears in the other image, a straight line that passes through the center of the lens of the left camera 60 and the center of the lens of the right camera 61. (That is, the straight line 114 shown in FIGS. 11A and 11B) does not always coincide with the horizontal direction in the left-eye correction image 122 and the right-eye correction image 123 (FIGS. 11A to 11D). reference).

  The left-eye correction image 122 and the right-eye correction image 123 that have been subjected to the correction shown in FIGS. 12 and 13 in a state where the straight line does not coincide with the horizontal direction have different vertical shift amounts between subjects depending on the subject distance. If the vertical displacement occurs, the vertical displacement may not be eliminated by the correction by the parallel movement shown in FIG.

  FIG. 14 is a diagram illustrating a state in which a vertical shift occurs between the left-eye correction image 122 and the right-eye correction image 123.

  The subjects 141 and 142 have a vertical shift between the left-eye correction image 122 and the right-eye correction image 123. The subject 141 and the subject 142 have different subject distances, but the vertical shift amount of the subject 141 is different from the vertical shift amount of the subject 142.

[Overview of how to correct vertical misalignment]
Hereinafter, an outline of a method for correcting vertical displacement between the left-eye image and the right-eye image will be described.

  By adjusting the trimming of the left-eye image and the right-eye image, it is possible to obtain a stereoscopic image from these images.

  In the left camera 60 and the right camera 61, when a rotation angle difference (relative rotation difference) occurs between the cameras, an angle shift occurs between the left-eye image and the right-eye image (see FIG. 15). .

  FIG. 16 is a diagram showing a procedure for trimming an image when the cameras having different rotation angles are generated.

  According to FIG. 16, the subjects 164 and 165 shown in the left-eye image 160 are oriented in a different direction from the subjects 164 and 165 shown in the right-eye image 161. As described above, when the subjects 164 and 165 are in different directions between the left-eye image 160 and the right-eye image 161, the subjects 164 and 165 do not appear as stereoscopic images.

  Accordingly, when the orientations of the subjects 164 and 165 shown in the left-eye image 160 are aligned with the orientations of the subjects 164 and 165 shown in the right-eye image 161 by correcting the rotation angle of the image, the subjects 164 and 165 appear to be a stereoscopic image. . FIG. 16 shows a left-eye correction image 162 obtained by trimming the left-eye image 160 according to the correction of the rotation angle of the image, and a right-eye correction image 163 obtained by trimming the right-eye image 161 according to the correction of the rotation angle of the image. ing. In order to perform correction so that the orientations of the subjects 164 and 165 are aligned between the left-eye image 160 and the right-eye image 161, the subjects 164 and 165 need to have two or more feature points.

  In addition, when the left camera 60 and the right camera 61 are rotated in the same direction and at the same rotation angle, or the camera is moved in parallel, vertical displacement occurs between the left-eye image and the right-eye image ( FIG. 17).

  FIG. 18 is a diagram illustrating a procedure for trimming an image when the rotation of the camera in the same direction and the same rotation angle or the parallel movement of the camera occurs.

  According to FIG. 18, the subjects 164 and 165 shown in the left-eye image 160 have a vertical shift with respect to the subjects 164 and 165 shown in the right-eye image 161. As described above, when vertical shift occurs in the subjects 164 and 165 between the left-eye image 160 and the right-eye image 161, the subjects 164 and 165 cannot be seen as a stereoscopic image. In addition, the vertical shift according to FIG. 18 changes in the vertical shift amount according to the subject distance, and thus the shift amount differs between the subject 164 and the subject 165.

  When the vertical shift amount changes according to the subject distance, the vertical shift according to FIG. 18 cannot be corrected appropriately only by performing correction by parallel movement of the image. Therefore, by correcting the rotation angle of the image, it is possible to appropriately correct the vertical shift. FIG. 18 shows a left-eye correction image 162 obtained by trimming the left-eye image 160 according to the correction of the rotation angle of the image, and a right-eye correction image 163 obtained by trimming the right-eye image 161 according to the correction of the rotation angle of the image. ing. In order to correct the rotation angle of the image, a plurality of subjects having different subject distances are required as shown as subjects 164 and 165.

  Also, in the left camera 60 and the right camera 61 due to camera tilt or the like, if the optical axis of one camera is shifted in the vertical direction with respect to the optical axis of the other camera, the image for the left eye and the image for the right eye Vertical displacement occurs between the images (see FIG. 19).

  FIG. 20 is a diagram illustrating a procedure for trimming an image when the optical axis is displaced in the vertical direction.

  According to FIG. 20, the subjects 164 and 165 shown in the left-eye image 160 have a vertical shift with respect to the subjects 164 and 165 shown in the right-eye image 161. As described above, when vertical shift occurs in the subjects 164 and 165 between the left-eye image 160 and the right-eye image 161, the subjects 164 and 165 cannot be seen as a stereoscopic image. However, since the vertical shift according to FIG. 20 has the same vertical shift amount regardless of the subject distance, the subject 164 and the subject 165 have the same shift amount.

  Accordingly, by correcting by parallel movement of the image, the vertical positions of the subjects 164 and 165 shown in the left-eye image 160 and the vertical positions of the subjects 164 and 165 shown in the right-eye image 161 are made uniform, so that FIG. It is possible to eliminate the vertical shift related to. FIG. 20 shows a left-eye correction image 162 obtained by trimming the left-eye image 160 according to the correction by the parallel movement of the image, and a right-eye correction image 163 obtained by trimming the right-eye image 161 according to the correction by the parallel movement of the image. ing.

  Further, when the optical axis of the left camera 60 and the optical axis of the right camera 61 are not parallel or do not intersect at a desired convergence point, a left-right shift occurs between the left-eye image and the right-eye image ( (See FIG. 21).

  FIG. 22 is a diagram illustrating a procedure for trimming an image when the optical axis is shifted in the left-right direction.

  According to FIG. 22, when there is a large lateral shift between the left-eye image 160 and the right-eye image 161 in the subjects 164 and 165, the subjects 164 and 165 are not visible as a stereoscopic image or desired It may appear in front or behind the position.

  Therefore, by correcting by the parallel movement of the image, the horizontal positions of the subjects 164 and 165 shown in the left-eye image 160 and the horizontal positions of the subjects 164 and 165 shown in the right-eye image 161 are aligned, so that the horizontal direction It is possible to eliminate the deviation. FIG. 22 shows a left-eye correction image 162 obtained by trimming the left-eye image 160 and a right-eye correction image 163 obtained by trimming the right-eye image 161. In this case, the subjects 164 and 165 need to have a known subject distance.

  In the image correction method described in the embodiment, two or more subjects at different subject distances are used, and one of the subjects has one or more feature points, and the other of the subjects has 2 Must have at least one feature point.

[Points of image correction method]
In the image correction method described in the embodiment, the “tilted rectangle” is cut out from the image for the left eye and the image for the right eye, so that the two left and right cameras (that is, the left camera 60 and the right camera 61) are separated. Correct the angle deviation and vertical deviation.

  In the conventional method, the relative deviation between the left-eye image and the right-eye image is corrected. When the left and right two cameras are rotated in the same direction and at the same angle, vertical displacement occurs between the left-eye image and the right-eye image. However, in the conventional method, the left-eye image and / or the right-eye image The vertical shift is corrected by shifting the trimming position in the vertical direction.

  However, when the two left and right cameras are rotated in the same direction and at the same angle, the amount of vertical displacement varies depending on the subject distance. Therefore, in the conventional method, the vertical displacement is adjusted according to the subject at any subject distance. It could not be corrected.

  Therefore, in the image correction method described in the embodiment, a straight line passing through the center of the lens of the left camera and the center of the lens of the right camera is used as an absolute reference line, and the image is corrected with respect to the reference line. Correct the deviation. As a result, it is possible to correct the vertical shift regardless of the distance of the subject.

  An image correction procedure according to the image correction method described in the embodiment is as follows with reference to FIG.

  (Procedure 1: S1) For two different distances (subject distances) A and B, a subject a (first subject) having at least two detectable feature points at a distance A and a distance B. A subject b (second subject) having at least one detectable feature point is photographed by two cameras on the left and right. An image obtained by photographing (acquiring) the subject a by the left camera (first camera) is defined as a left-eye image a (first reference image), and an image obtained by photographing the subject a by the right camera (second camera) is represented by a right-eye image a ( A second reference image), an image obtained by photographing the subject b by the left camera as an image b for the left eye (third reference image), and an image obtained by photographing the subject b by the right camera as an image for the right eye b (fourth reference image). To do. Note that the left-eye image a and the left-eye image b may be different images or the same image. Similarly, the right-eye image a and the right-eye image b may be different images or the same image. That is, at least one of shooting with the left camera and shooting with the right camera may be performed so that both the subject a and the subject b are captured. In other words, only the left camera may shoot so that the subject a and the subject b appear in one image, or only the right camera shoots so that the subject a and the subject b appear in one image. Alternatively, the left and right cameras may shoot so that the subject a and the subject b appear in one image.

  (Procedure 2: S2) Two or more feature points of the subject a are detected from the left eye image a and the right eye image a, and one or more feature points of the subject b are detected from the left eye image b and the right eye image b.

  (Procedure 3: S3) The optical axis of the right camera with respect to the relative angular difference between the two left and right cameras from the two feature points of the subject a, that is, the rotation angle of the left camera around the optical axis of the left camera The difference of the rotation angle of the right camera centering on (Δθ: mutual rotation angle difference) is calculated. Then, with respect to the left-eye image a and the right-eye image a, the rotation angle is corrected by Δθ so as to cancel Δθ. That is, at this stage, the rotation angle correction coefficient for the left-eye image a is set to Δθ / 2, and the rotation angle correction coefficient for the right-eye image a is set to −Δθ / 2.

  (Procedure 4: S4) After the correction in (Procedure 3), both the left-eye image a and the right-eye image a are rotated by an arbitrary angle (Δφ: predetermined angle). Then, the difference (ΔY: first vertical shift) in the vertical direction between the left-eye image a and the right-eye image a of the two feature points of the subject a is calculated.

(Procedure 5: S5) Both the left-eye image b and the right-eye image b are corrected based on Δθ, Δφ, and ΔY so as to cancel Δθ and ΔY and rotate Δφ. That is, the correction coefficient for the left-eye image b is
Rotation angle: Δθ / 2 + Δφ
Vertical movement: ΔY / 2
And the correction coefficient for the right-eye image b is
Rotation angle: -Δθ / 2 + Δφ
Vertical movement: -ΔY / 2
And Thereafter, a shift (ΔY ′: second vertical shift) in the vertical direction between the left-eye image b and the right-eye image b of the feature point of the subject b is calculated.

  (Procedure 6: S61 and S62) The operations shown in (Procedure 4) and (Procedure 5) are repeated for a plurality of Δφs to obtain Δφ and ΔY that minimize ΔY ′.

  (Procedure 7: S7) The subjects a appearing in the left-eye image a and the right-eye image a and the subjects b appearing in the left-eye image b and the right-eye image b overlap at an infinite distance or a desired convergence point. Then, the distance (ΔX) to be corrected is obtained.

As a result of performing (Procedure 1) to (Procedure 7), the correction coefficient for the image for the left eye (corresponding to the first image data) acquired from the left camera (first imaging unit 11) is:
Rotation angle: Δθ / 2 + Δφ
Vertical movement: ΔY / 2
Left / right movement: ΔX / 2
The correction coefficient for the right-eye image (corresponding to the second image data) acquired from the right camera (second imaging unit 12) is:
Rotation angle: -Δθ / 2 + Δφ
Vertical movement: -ΔY / 2
Left / right movement: -ΔX / 2
And

  Thereafter, the correction coefficient for the left-eye image and the correction coefficient for the right-eye image are supplied to the trimming unit 16 (see FIG. 3) as image correction parameters. With reference to the left-eye image as the first image data and the right-eye image as the second image data respectively stored in the image storage unit 15, the trimming unit 16 (based on the image correction parameter) The left eye image and the right eye image are trimmed so as to reflect the correction defined by the parameters, and the first corrected image data and the second corrected image data are generated. Thereby, the mobile phone 1 that corrects each of the left-eye image and the right-eye image can be realized by the image correction method according to the present invention.

  In step 3, the rotation angle is corrected by Δθ for both the left-eye image a and the right-eye image a. However, a method for correcting either the left-eye image a or the right-eye image a is used. It may be adopted.

  Further, in the procedure 5, both the left-eye image b and the right-eye image b are corrected based on Δθ, Δφ, and ΔY. Of these, regarding Δθ and ΔY, the left-eye image b and the right-eye image A method of correcting either one of the images b may be employed. On the other hand, regarding Δφ, since rotation correction of the left-eye image b and the right-eye image b is performed in the same direction and at the same angle, either one cannot be corrected, and both need to be corrected.

  Furthermore, in each of the above procedures, the description has been given on the assumption that both the left-eye image and the right-eye image are trimmed. However, if a stereoscopic image can be acquired by trimming, either the left-eye image or the right-eye image is selected. May be trimmed.

  Here, points when performing the image correction method will be described.

  As described above, there are four factors (Factor A) to (Factor D) that cause the subject to shift between the left-eye image and the right-eye image.

  (Factor A) Deviation due to a difference in rotation angle (relative rotation difference) between the left camera and the right camera.

  (Factor B) Deviation caused by rotation of the left camera and the right camera in the same direction by the same rotation angle, or deviation in the vertical mounting position between the left camera and the right camera.

  (Factor C) Deviation due to vertical tilt between the optical axes of the left camera and the right camera.

  (Factor D) Deviation due to tilt in the left-right direction between the optical axes of the left camera and the right camera.

  The vertical mounting position shift of the camera according to (Factor B) is the same direction and the same rotation angle of the camera according to (Factor B), assuming that a straight line passing through the center position of each lens of the two cameras is horizontal. It can be treated as equivalent to a shift caused by rotation of.

  Among (Factor A) to (Factor D), it is (Factor B) and (Factor C) that cause a vertical shift between the left-eye image and the right-eye image. The vertical shift according to (Factor B) varies in the vertical direction depending on the subject distance, whereas the vertical shift according to (Factor C) does not depend on the subject distance. Therefore, it is necessary to perform correction separately for (Factor B) and (Factor C), but these cannot be distinguished only by photographing a subject at a specific distance. If the image correction method is used, it is possible to adjust the image in such a manner that (Factor B) and (Factor C) are corrected separately.

  In the above (procedure 2), two feature points are detected from the subject a. On the other hand, in the above (Procedure 2), instead of detecting two feature points from the subject a, one feature point is detected from the subject a and the rotation angle of the right eye image a relative to the left eye image a can be specified. A similar method can be carried out even if an object is detected from the left-eye image a and the right-eye image a. Note that this line need not be a subject. In other words, in the left-eye image a and the right-eye image a, the subject a and its surroundings (background, etc.) are photographed as subject areas, and lines appearing in the subject area are detected from the left-eye image a and the right-eye image a. do it.

  Specifically, in order to calculate Δθ from the left-eye image a and the right-eye image a, the rotation angle of the right-eye image a relative to the left-eye image a is specified from the left-eye image a and the right-eye image a. It is sufficient to detect what is possible, that is, the lines that appear in both the left eye image a and the right eye image a. On the other hand, in order to calculate ΔY from the left-eye image a and the right-eye image a, it is necessary to detect one or more feature points of the subject a.

  The timing for supplying the correction coefficient for the left-eye image and the correction coefficient for the right-eye image to the trimming unit 16 as image correction parameters may be before shipment as a product of the mobile phone 1. The timing may be after the mobile phone 1 is shipped, for example, when the mobile phone 1 is repaired.

  Procedure 1 is the first step, Procedure 2 is the second step, Procedure 3 is the third step, Procedure 4 is the fourth step, Procedure 5 is the fifth step, Procedure 6 is the sixth step, Procedure 7 corresponds to the seventh step.

[Specific examples of image correction methods]
FIG. 23 is a table showing a list of feature point coordinates detected in the image correction method.

  FIG. 24 is a table showing a list of image cutout coordinates.

  FIG. 25 is a table showing a list (one example) of constants used in the image correction method.

  FIG. 26 is a table showing a list of variables used in the image correction method. For convenience, variables not described in FIG. 26 are defined in the following steps.

(Step 1: Shooting subjects with subject distance A and subject distance B with two cameras on the left and right, and detecting the coordinates of feature points in each image)
First, an image obtained by photographing a subject at a subject distance A (first subject) with a left camera (first camera) was photographed with A left (first reference image), and a subject at a subject distance A was photographed with a right camera (second camera). The image is A right (second reference image).

  Subsequently, an image obtained by photographing the subject at the subject distance B (second subject) with the left camera is B left (third reference image), and an image obtained by photographing the subject at the subject distance B with the right camera is B right (fourth reference image). ).

  Subsequently, feature points A1 and B1 are detected from the left side of the image A (see FIG. 27A), and feature points A2 and B2 corresponding to the feature points A1 and B1 are detected from the right side of the image A (FIG. 27). (See (b)). Subsequently, the feature point A3 is detected from the left side of the image B (see FIG. 27C), and the feature point A4 corresponding to the feature point A3 is detected from the right side of the image B (see FIG. 27D). It should be noted that detection of each of these feature points can be realized by a well-known conventional technique, and thus detailed description thereof is omitted.

(Step 2: Calculate the angle difference between the left image A and the right image A in the rotation direction, and calculate the rotation angle at which the left image A and the right image A are horizontal)
(A) The inclination of a straight line connecting two feature points calculated from the left of the image A is calculated. That is,
The slope of the straight line connecting point A1 and point B1 with respect to the horizontal:
θ1 = ATAN2 ((X12-X11), (Y12-Y11))
Is calculated.

(B) The slope of a straight line connecting two feature points calculated from the right side of the image A is calculated. That is,
The slope of the straight line connecting point A2 and point B2 with respect to the horizontal:
θ2 = ATAN2 ((X22-X21), (Y22-Y21))
Is calculated. ATAN2 is a general function for obtaining the arc tangent of a trigonometric function, and obtains the inclination (angle, unit is radians) of a line connecting the coordinates (x, y) of a certain point and the origin.

(C) The adjustment rotation angles (θL1, θR1) of the image acquired from the left camera and the image acquired from the right camera are calculated from the calculation results of (b) and (b) above. That is,
Left image (first image acquired from the left camera) Adjustment rotation angle: θL1 = (θ2-θ1) / 2
Right image (second image acquired from the right camera) Adjustment rotation angle: θR1 = − (θL1)
Is calculated.

(Step 3: Calculate the optimal rotation angle common to the left and right cameras)
As a calculation procedure, the rotation correction amount Δφ common to the image acquired from the left camera and the image acquired from the right camera is changed, and the following (A) to (E) are repeated.

  As an example, the range and steps for changing the rotation correction amount Δφ are as follows.

Starting angle: -10 °
End angle: 10 °
Change step: 0.01 °
However, the range in which the rotation correction amount Δφ is changed and the step of changing the rotation correction amount Δφ are arbitrary. In principle, there is no problem even if the rotation correction amount Δφ is changed in any fineness (step) within a range of 90 ° to the left and right.

(A) An angle obtained by adding the rotation correction amount Δφ to the left image adjustment rotation angle and the right image adjustment rotation angle calculated in step 2 is calculated. That is,
Left image adjustment rotation angle with rotation deviation Δφ common to left and right cameras: θL2 = θL1 + Δφ
Right image adjustment rotation angle with rotation deviation Δφ common to left and right cameras: θR2 = θR1 + Δφ
Is calculated.

  (B) The coordinates when the point A1 and the point B1 are rotated at the rotation angle θL2, and the coordinates when the point A2 and the point B2 are rotated at the rotation angle θR2 are calculated.

First, rotation is performed assuming that the center axis of rotation is the center of the image. The center coordinates (Xcenter, Ycenter) of the left image A and the right image A are calculated. That is,
Xcenter = Size_O_x / 2
Ycenter = Size_O_y / 2
Is calculated.

Subsequently, the coordinates of the movement destinations of the points A1 and B1 are calculated with the rotation center coordinates (Xcenter, Ycenter) as the center. That is,
The coordinates of the point A1 ′ (X11 ′, Y11 ′) to which the point A1 is rotated:
X11 '= (X11-Xcenter) * COS (θL2)-(Y11-Ycenter) * SIN (θL2) + Xcenter
Y11 '= (X11-Xcenter) * SIN (θL2) + (Y11-Ycenter) * COS (θL2) + Ycenter
Coordinates of point B1 ′ (X12 ′, Y12 ′) to which point B1 is rotated:
X12 '= (X12-Xcenter) * COS (θL2)-(Y12-Ycenter) * SIN (θL2) + Xcenter
Y12 '= (X12-Xcenter) * SIN (θL2) + (Y12-Ycenter) * COS (θL2) + Ycenter
Is calculated.

Subsequently, the coordinates of the movement destinations of the points A2 and B2 are calculated with the rotation center coordinates (Xcenter, Ycenter) as the center. That is,
The coordinates of the point A2 ′ (X21 ′, Y21 ′) to which the point A2 is rotated:
X21 '= (X21-Xcenter) * COS (θR2)-(Y21-Ycenter) * SIN (θR2) + Xcenter
Y21 '= (X21-Xcenter) * SIN (θR2) + (Y21-Ycenter) * COS (θR2) + Ycenter
The coordinates of the point B2 ′ (X22 ′, Y22 ′) to which the point B2 is rotated:
X22 '= (X22-Xcenter) * COS (θR2)-(Y22-Ycenter) * SIN (θR2) + Xcenter
Y22 '= (X22-Xcenter) * SIN (θR2) + (Y22-Ycenter) * COS (θR2) + Ycenter
Is calculated.

  (C) From the coordinate values calculated in (b) above, the deviation due to the tilt in the X direction (horizontal tilt) and the shift due to the tilt in the Y direction (vertical tilt) between the left image A and the right image A Is calculated.

First, from the coordinates of the points A1 ′ and A2 ′, which are the coordinates after rotation correction, the amount of deviation in the X direction and the Y direction between the image A left and the image A right is calculated. That is,
Deviation in the X direction between the points A1 ′ and A2 ′: ΔX = X21′−X11 ′
Deviation in the Y direction between the points A1 ′ and A2 ′: ΔY = Y21′−Y11 ′
Is calculated.

Subsequently, correction amounts in the X direction and the Y direction in each of the image acquired from the left camera and the image acquired from the right camera are calculated. That is,
Left image X direction correction amount: ΔXL = ΔX / 2
Left image Y direction correction amount: ΔYL = ΔY / 2
Right image X direction correction amount: ΔXR = − (ΔXL)
Right image Y direction correction amount: ΔYR = − (ΔYL)
Is calculated.

  (D) The coordinates when the point A3 is rotated at the rotation angle θL2 and the coordinates when the point A4 is rotated at the rotation angle θR2 are calculated.

First, rotation is performed assuming that the center axis of rotation is the center of the image. Centering on the rotation center coordinates (Xcenter, Ycenter), the coordinates of the movement destinations of the points A3 and A4 are calculated. That is,
The coordinates of the point A3 ′ (X31 ′, Y31 ′) to which the point A3 is rotated:
X31 '= (X31-Xcenter) * COS (θL2)-(Y31-Ycenter) * SIN (θL2) + Xcenter
Y31 '= (X31-Xcenter) * SIN (θL2) + (Y31-Ycenter) * COS (θL2) + Ycenter
The coordinates of the point A4 ′ (X41 ′, Y41 ′) to which the point A4 is rotated:
X41 '= (X41-Xcenter) * COS (θR2)-(Y41-Ycenter) * SIN (θR2) + Xcenter
Y41 '= (X41-Xcenter) * SIN (θR2) + (Y41-Ycenter) * COS (θR2) + Ycenter
Is calculated.

Subsequently, the coordinates when the rotated points A3 ′ and A4 ′ are moved in the Y direction are calculated. That is,
Y coordinate (Y31 ″) of point A3 ″ to which point A3 is moved: Y31 ″ = Y31 ′ + ΔYL
Y coordinate (Y41 ″) of point A4 ″ to which point A4 is moved: Y41 ″ = Y41 ′ + ΔYR
Is calculated.

  (E) The error between the Y coordinates in the coordinates after movement calculated in (d) above is calculated.

An error in the Y direction between the points Y31 '' and Y41 '' is calculated. That is,
Y direction deviation between Y31 ″ and Y41 ″: ΔY ′ = Y41 ″ −Y31 ″
Is calculated.

  (F) The above values (A) to (E) are repeated while changing the value of the rotation correction amount Δφ.

  (G) The above steps (a) to (f) are repeatedly executed, and the value of the rotation correction amount Δφ when ΔY ′ calculated in (e) is minimized is determined to be the optimum adjustment rotation angle. The correction amount is set to an optimum correction amount.

  First, the rotation correction amount Δφ that minimizes the Y-coordinate error (ΔY ′) is found, and the angle is set as the optimum common adjustment angle Δθ.

Subsequently, the optimum adjustment rotation angle of the image acquired from the left camera and the image acquired from the right camera is calculated. That is,
Optimal left image adjustment rotation angle: θL3 = θL1 + Δθ
Optimal right image adjustment rotation angle: θR3 = θR1 + Δθ
Is calculated.

  Subsequently, when the common rotation correction amount Δφ is made equal to Δθ, the X direction correction amounts (ΔXL and ΔXR) and the Y direction correction amounts (ΔYL and ΔYR) are optimally corrected for the left side of the image A and the right side of the image A. Decide with quantity.

(Step 4: Calculate the reverse correction amount)
The reverse correction value of the optimum adjustment amount obtained in step 3 (g) is calculated. That is,
Optimal left camera side adjustment rotation angle (reverse correction value): θL4 = −θL3
Optimal right camera side adjustment rotation angle (reverse correction value): θR4 = −θR3
Left camera side X direction correction amount (reverse correction value): ΔXL ′ = − ΔXL
Right camera side X direction correction amount (reverse correction value): ΔXR ′ = − ΔXR
Left camera side Y direction correction amount (reverse correction value): ΔYL ′ = − ΔYL
Right camera side Y direction correction amount (reverse correction value): ΔYR ′ = − ΔYR
Is calculated.

(Step 5: Calculate the cut-out coordinates of the left and right images)
(A) A reference point (cutout coordinates when the correction amounts are all 0) for calculating the cutout coordinates is calculated from the size and cutout size of the original image. However, since the reference point is essentially a constant, no calculation is necessary.

First, the cutout reference points of the left image are A5, B5, C5, and D5. Further, the cut-out reference points of the right image are A6, B6, C6, and D6. And for A5, B5, C5, D5,
Point A5 coordinates (X51, Y51):
X51 = (Size_O_x−Size_S_x) / 2
Y51 = (Size_O_y−Size_S_y) / 2
Point B5 coordinates (X52, Y52):
X52 = X51 + (Size_S_x−1)
Y52 = Y51
Point C5 coordinates (X53, Y53):
X53 = X51
Y53 = Y51 + (Size_S_y-1)
Point D5 coordinates (X54, Y54):
X54 = X52
Y54 = Y53
Is calculated. Similarly, with respect to A6, B6, C6, and D6, the coordinates of the point A6 (X61, Y61), the coordinates of the point B6 (X62, Y62), the coordinates of the point C6 (X63, Y63), and the coordinates of the point D6 (X64, Y64) is calculated respectively.

(B) The coordinates of the left and right reference points (total of 8 points) calculated in (a) above are used as the X direction correction amount (reverse correction value) and Y direction correction amount (reverse correction value) calculated in step 4 above. Calculate the coordinates when moved. That is,
The coordinates (X51 ′, Y51 ′) of the point A5 ′ to which the point A5 is moved:
X51 ′ = X51 + ΔXL ′
Y51 '= Y51 + ΔYL'
(Similarly, the Y coordinate of point B5 ′ / point C5 ′ / point D5 ′ is calculated)
The coordinates (X61 ′, Y61 ′) of the point A6 ′ to which the point A6 is moved:
X61 '= X61 + ΔXR'
Y61 '= Y61 + ΔYR'
(Similarly, the Y coordinate of point B6 ′ / point C6 ′ / point D6 ′ is calculated)
Is calculated.

(C) The coordinates obtained by rotating the eight points obtained in (a) above with the optimum adjustment rotation angle (reverse correction value) calculated in step 4 above, centering on the center coordinates (Xcenter, Ycenter) of the image. calculate. That is,
The coordinates of the point A5 ″ (S_X51, S_Y51) to which the point A5 is rotated:
S_X51 = (X51'-Xcenter) * COS (θL4)-(Y51'-Ycenter) * SIN (θL4) + Xcenter
S_Y51 = (X51'-Xcenter) * SIN (θL4) + (Y51'-Ycenter) * COS (θL4) + Ycenter
(Similarly, the coordinates of point B5 ″ / point C5 ″ / point D5 ″ are calculated)
The coordinates of the point A6 ″ (S_X61, S_Y61) to which the point A6 is rotated:
S_X61 = (X61'-Xcenter) * COS (θR4)-(Y61'-Ycenter) * SIN (θR4) + Xcenter
S_Y61 = (X61'-Xcenter) * SIN (θR4) + (Y61'-Ycenter) * COS (θR4) + Ycenter
(Similarly, the coordinates of point B6 "/ point C6" / point D6 "are calculated)
(Step 6: End)
The cut-out coordinates of the left and right images determined in step 5 are supplied to the trimming unit 16 (see FIG. 3) as image correction parameters. With reference to the left image as the first image data and the right image as the second image data respectively stored in the image storage unit 15, the trimming unit 16 determines the left image and the right image based on the image correction parameter. The first correction image data and the second correction image data are generated. Thus, the mobile phone 1 that corrects the left and right images can be realized by the image correction method according to the present invention.

  The stereoscopic image capturing apparatus of the present invention is obtained by trimming the first image by the trimming unit when the mounting position of the first camera is shifted in the vertical direction with respect to the second camera. Both the image and the image obtained by trimming the second image by the trimming unit are rotated at the same rotation angle.

  According to the image correction method of the present invention, when the mounting position of the first camera in the stereoscopic image capturing apparatus is deviated vertically with respect to the second camera, the center of the first camera and the second camera are accordingly moved. As a result of performing trimming on the basis of a straight line passing through the center, both the image obtained by trimming the first image and the image obtained by trimming the second image rotate at the same rotation angle. A phenomenon occurs. The stereoscopic image capturing apparatus in which the phenomenon occurs may be considered as the stereoscopic image capturing apparatus of the present invention.

  In principle, both a distant subject and a near subject are captured at the same time with one shot each on the left and right (two images on the left and right), and from each of the images, the feature points of the distant subject, It is also possible to detect feature points of nearby subjects. For example, in each of these images, when shooting a distant subject in the upper half and a close subject in the lower half, and detecting feature points, the top and bottom of the image is divided into halves. Detection may be performed.

  If the focus can be adjusted well, or if the feature point can be detected even if the focus is weak, it is possible to simultaneously photograph a short-distance subject and a long-distance subject.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  The present invention relates to an image correction method for correcting each image acquired from two cameras built in a three-dimensional image capturing device that captures a three-dimensional image, and each image acquired from two cameras by this image correction method. It can be used for a three-dimensional image photographing apparatus that corrects the above.

1 Mobile phone (stereoscopic image capturing device)
11 First photographing unit (first camera)
12 Second imaging unit (second camera)
16 Trimming section

Claims (5)

To obtain a stereoscopic image from a first image acquired from a first camera constituting a camera group including a plurality of cameras and a second image obtained from a second camera different from the first camera constituting the camera group. An image correction method for trimming at least one of the first image and the second image,
A first reference image obtained by photographing a first subject and a second subject having different distances to the camera group with the first camera and the second camera, and the first camera photographing the first subject; A second reference image obtained by photographing the first subject with two cameras, a third reference image obtained by photographing the second subject with the first camera, and a second reference image obtained by photographing the second subject with the second camera. A first step of acquiring a fourth reference image;
At least two feature points of the first subject are detected from the first reference image and the second reference image, and at least one feature point of the second subject is detected from the third reference image and the fourth reference image. A second step to detect;
The second camera centered on the optical axis of the second camera with respect to the rotation angle of the first camera centered on the optical axis of the first camera from each feature point of the first subject detected in the second step. A third step of calculating a mutual rotation angle difference, which is a difference in rotation angle, and correcting at least one of the first reference image and the second reference image so as to cancel the mutual rotation angle difference;
The first reference image and the second reference image are rotated by the same predetermined angle as each other, and each feature point of the first subject is shifted in the vertical direction between the first reference image and the second reference image. A fourth step of calculating the first vertical shift;
At least one of the third reference image and the fourth reference image is corrected so as to cancel out the mutual rotation angle difference and the first vertical shift, and is rotated by the same predetermined angle as each other, and in the second step A fifth step of calculating a second vertical shift, which is a vertical shift between the third reference image and the fourth reference image, of the detected feature point of the second subject;
A sixth step of performing the fourth step and the fifth step for a plurality of the predetermined angles to obtain the predetermined angle and the first vertical shift, wherein the second vertical shift is minimized,
The reference point is calculated from the size of the original image that is at least one of the first image and the second image and the cutout size in the trimming, and the second vertical shift obtained in the sixth step is minimized, The coordinates of the reference point are moved with the reverse correction value related to the vertical movement so as to cancel the predetermined angle and the first vertical shift, and then rotated with the reverse correction value related to the rotation, and the cutout coordinates of the original image And trimming at least one of the first image and the second image based on the coordinates ,
The reverse correction value related to the vertical movement is a value obtained by changing the sign of the correction amount used for the correction for canceling the first vertical shift,
The reverse correction value related to the rotation is a value obtained by changing the sign of the correction amount used for correction for canceling the mutual rotation angle difference and rotating the same predetermined angle.
An image correction method characterized by the above. To obtain a stereoscopic image from a first image acquired from a first camera constituting a camera group including a plurality of cameras and a second image obtained from a second camera different from the first camera constituting the camera group. An image correction method for trimming at least one of the first image and the second image,
The first and second subjects having different distances to the camera group are photographed by the first camera and the second camera, and the first camera photographs the subject region including the first subject and the periphery of the first subject. A first reference image obtained by photographing the subject area with the second camera, a third reference image obtained by photographing the second subject with the first camera, and a second reference image obtained by photographing the second subject with the second camera. A first step in which a camera acquires a fourth reference image obtained by photographing a second subject;
At least one feature point of the first subject and a line appearing in the subject area are detected from the first reference image and the second reference image, and from the third reference image and the fourth reference image, the first reference image and the second reference image are detected. A second step of detecting at least one feature point of two subjects;
The optical axis of the second camera is centered on the optical axis of the first camera with respect to the rotation angle of the first camera about the optical axis of the first camera from the feature point of the first subject detected in the second step and the line appearing in the subject area. A third step of calculating a mutual rotation angle difference, which is a difference in rotation angle of the second camera, and correcting at least one of the first reference image and the second reference image so as to cancel the mutual rotation angle difference; ,
The first reference image and the second reference image are rotated by the same predetermined angle as each other, and the feature point of the first subject and the line appearing in the subject region are vertically moved between the first reference image and the second reference image. A fourth step of calculating a first vertical shift that is a shift in the direction;
At least one of the third reference image and the fourth reference image is corrected so as to cancel out the mutual rotation angle difference and the first vertical shift, and is rotated by the same predetermined angle as each other, and in the second step A fifth step of calculating a second vertical shift, which is a vertical shift between the third reference image and the fourth reference image, of the detected feature point of the second subject;
A sixth step of performing the fourth step and the fifth step for a plurality of the predetermined angles to obtain the predetermined angle and the first vertical shift, wherein the second vertical shift is minimized,
The reference point is calculated from the size of the original image that is at least one of the first image and the second image and the cutout size in the trimming, and the second vertical shift obtained in the sixth step is minimized, The coordinates of the reference point are moved with the reverse correction value related to the vertical movement so as to cancel the predetermined angle and the first vertical shift, and then rotated with the reverse correction value related to the rotation, and the cutout coordinates of the original image And trimming at least one of the first image and the second image based on the coordinates ,
The reverse correction value related to the vertical movement is a value obtained by changing the sign of the correction amount used for the correction for canceling the first vertical shift,
The reverse correction value related to the rotation is a value obtained by changing the sign of the correction amount used for correction for canceling the mutual rotation angle difference and rotating the same predetermined angle.
An image correction method characterized by the above. Deviation in the left-right direction between the first reference image and the second reference image of the first subject, and deviation in the left-right direction between the third reference image and the fourth reference image of the second subject. A seventh step of calculating at least one of
In order to cancel the deviation obtained in the seventh step, the coordinates of the reference point are moved with the reverse correction value related to the left-right movement, and then rotated with the reverse correction value related to the rotation, and the original image , And trimming at least one of the first image and the second image based on the coordinates ,
The reverse correction value related to the left-right movement is a value obtained by changing the sign of the correction amount used for correction to cancel the deviation obtained in the seventh step.
The image correction method according to claim 1, wherein the image correction method is an image correction method.   In the first step, at least one of photographing with the first camera and photographing with the second camera is performed so that both the first subject and the second subject are captured in one image. Item 4. The image correction method according to any one of Items 1 to 3. A trimming unit that trims at least one of the first image and the second image by the image correction method according to claim 1,
A stereoscopic image capturing apparatus comprising the camera group.

Images