JP2013074473A - Panorama imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To take 3D panorama photograph that has a wide field view in horizontal and vertical directions, from a plurality of 3D images photographed with a twin-lens camera.SOLUTION: A plurality of 3D photographs are taken by changing a field view with the 3D camera (twin-lens camera), and a distance information map is calculated from the left and right images of the same field view. Then, a combined image in which the characteristic points corresponding to the left (or right) images of different field views are coupled so as to match is formed. Also, a distance information connection map coupled so as to have the same positional relation as the coupling method is formed. Thereby, 3D panorama photographing is realized.

Description

  The present invention relates to a panoramic imaging apparatus that obtains a panoramic image by combining a plurality of captured images having different fields of view.

  It is possible to shoot a 3D image by shooting a left image of the left line of sight and a right image of the right line of sight with two cameras (two-lens cameras) having optical axis intervals that are approximately the distance between human eyes.

  FIG. 2A shows an example of a portable terminal with a conventional 3D camera. A first camera 2 and a second camera 3 are arranged apart from each other in a housing 1 of the imaging apparatus. A display element 5 for displaying a 3D captured image is disposed on the rear surface of the housing 1. For example, the display element can perform 3D display using a parallax barrier type naked-eye 3D liquid crystal or the like.

  As a method for obtaining a 3D panoramic image having a wider angle of view than the angle of view of the camera, for example, there is a method described in Patent Document 1.

  An omnidirectional panoramic image having a wide angle of view in the horizontal direction is obtained by photographing a plurality of cameras arranged side by side on the circumference. Further, a distance estimation camera is arranged at the upper center of the circle to capture a horizontal image. The distance measuring camera can be photographed in all directions by rotating with respect to the central axis of the circle. The distance information of each subject captured by the camera on the circumference is obtained from the parallax of the subject of each camera image arranged on the circumference in the horizontal direction and the camera image for distance measurement. Thereby, an omnidirectional panoramic image and its distance information can be acquired.

On the other hand, a panoramic image synthesis method with a wide field of view is known by combining two images so that the common feature points coincide with each other in an overlapping portion of two images taken in two (or a plurality of) different directions. It has been. (For example, Patent Document 2)
The first panorama image can be generated from the first camera image photographed in a plurality of different directions using the panorama image synthesis method. Similarly, a second panorama image can be generated from a plurality of second camera images using the panorama image synthesis method. Thereby, a 3D panoramic image composed of the first panoramic image and the second panoramic image can be obtained.

  However, in order to obtain a 3D panoramic image by this method, it is necessary to shoot by moving the reference line 16 connecting the optical axis centers of the 3D camera so as not to tilt.

  This is because, as shown in FIG. 2B, when the reference line 16 connecting the optical axis centers of the 3D camera is tilted, the vertical displacement 15 of the subject of the first camera image (B) and the second camera image (B) occurs. Because it does.

JP 2010-239416 A JP 2006-20111 A

  In the method of 3D shooting with a conventional twin-lens camera facing a plurality of different directions and synthesizing a 3D image using the panoramic image synthesis method, the reference line connecting the optical axis centers of the twin-lens camera is adjusted so that it does not tilt, Since it is necessary to shoot a plurality of 3D images, there is a problem that operability is poor.

  In addition, when panorama synthesis of the left and right cameras is performed independently and this combination is repeated using two or more images, the difference in the left and right synthesis accuracy is accumulated, so that the subject of the left and right cameras As the 3D image is accumulated and a part that cannot be stereoscopically viewed is generated, a panoramic 3D image having a wide angle of view cannot be generated.

  Further, the method of performing 3D panoramic photographing using cameras arranged on a plurality of cylinders in Patent Document 1 requires a plurality of cameras, and thus is large and expensive.

  The present invention has been made to solve the above-described problems. A 3D panoramic image can be taken with a twin-lens camera without using a plurality of cameras. An object of the present invention is to realize a panoramic imaging apparatus capable of capturing a 3D panoramic image having a wide field angle.

  Next, means for solving the above problems will be described.

  The panorama imaging apparatus of the present invention includes a first camera, a second camera, distance information detection means for generating a distance information map from the parallax of the first camera and the second camera, and a plurality of images corresponding to each image. Image combining means for combining images after performing alignment so that the points coincide with each other, taking a plurality of 3D images composed of a plurality of first camera images and a plurality of second camera images having different fields of view, and said distance Generating a plurality of the distance information maps of the plurality of first camera images, generating a first camera combined image obtained by combining the plurality of first camera images by the image combining unit; A distance information combined map is generated by combining the distance information map with the same positional relationship as the first camera combined image combining method.

  With this configuration, a panoramic image including distance information can be obtained from a plurality of 3D images of a twin-lens camera.

  In the panoramic imaging device of the present invention, the distance information value of the distance information combination map in each pixel of the common area of the second distance information map photographed adjacent to the first distance information map of the first camera image is The intermediate value of the two values of the distance information of the first distance information map and the second distance information map.

  With this configuration, it is possible to increase the accuracy of the value of the distance information map of the pixels in the common area.

  The panorama imaging apparatus of the present invention generates a 3D panoramic image composed of a left combined image and a right combined image from the distance information combined map and the first camera combined image.

  With this configuration, a 3D panoramic image can be obtained.

  In addition, the panoramic imaging apparatus of the present invention can extract a cut-out 3D image including a cut-out image for left and a cut-out image for right from the cut-out map and cut-out image obtained by cutting out a part of the distance information combination map and the first camera combined image. Generate.

  With this configuration, it is possible to generate a 3D image when viewed from an arbitrary inclination at an arbitrary angle of view.

  In addition, the panorama imaging apparatus of the present invention includes a 3D display unit and a posture sensor, detects a change in the normal direction of the 3D display unit by the posture sensor, and cuts out according to the change in the normal direction. The cut-out 3D image generated by shifting the cut-out position of the image is displayed on the 3D display means.

  With this configuration, it is possible to display a panoramic 3D image with a wide angle of view using a 3D display with a narrow angle of view.

  According to the present invention, it is possible to generate a panoramic 3D image with a wide angle of view that can be stereoscopically viewed even if the horizontal direction of the binocular camera is tilted during shooting.

1 is a schematic structural diagram and a block configuration diagram of a 3D imaging device according to a first embodiment of the present invention. Explanatory drawing of the subject of the conventional panorama 3D imaging method The figure which shows the panorama 3D image generation method of Embodiment 1 of this invention. The figure which shows the continuation of the panorama 3D image generation method of Embodiment 1 of this invention The figure which shows the production | generation method of the cut-out 3D image of Embodiment 1 of this invention The figure which shows the determination method of the cutout position of Embodiment 1 of this invention The figure which shows the parallax direction of the cut-out 3D image of Embodiment 1 of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
FIG. 1 is a schematic structural diagram and a block configuration diagram of a 3D imaging apparatus according to the present embodiment.

  Hereinafter, the first embodiment will be described assuming a portable terminal device including a single case. However, in the case where the portable terminal device is mounted on an electronic apparatus having a small camera such as a folding portable terminal or a digital still camera. Is the same.

  As shown in FIG. 1A, a two-lens 3D camera 4 including a first camera 2 and a second camera 3 is disposed on the back surface of the display element 5 of the rectangular housing 1 of the mobile terminal device. Here, the left camera is assumed to be the first camera, and the right camera is assumed to be the second camera.

  FIG. 1 (3) is a block diagram of the mobile terminal device. The first camera 2 and the second camera 3 are controlled by the camera control unit 6, store the captured image in the storage unit 7, and display the captured image using the display element 5.

  The display element 5 is, for example, a naked eye 3D liquid crystal having a barrier liquid crystal, but may be a frame sequential 3D display element using electronic shutter glasses or the like.

  The distance information detection means 8 has a function of detecting distance information of the subject from the images of the first camera 2 and the second camera 3 and generating a distance information map. Here, the distance information map refers to a map in which a one-dimensional amount including the distance between the camera and the subject or the distance information between the subjects is defined for each pixel of the captured image.

  The image combining means 9 has a so-called panoramic synthesis function for detecting feature points corresponding to a plurality of images and connecting the plurality of images so that the feature points match to generate a composite image.

  Furthermore, this portable terminal has a built-in posture sensor 10 including an acceleration sensor, a direction sensor, and the like, and can detect the posture of the terminal.

  FIG. 2 shows an example of an image taken when the reference line 16 connecting the optical axis centers of the conventional 3D camera 4 is tilted.

  As shown in FIG. 2 (2), when the reference line 16 connecting the optical axis center of the 3D camera is inclined, the vertical displacement 15 of the subject in the first camera image (B) and the second camera image (B) is conventionally generated. Occur.

  FIG. 3 shows a panorama 3D image generation method according to the first embodiment.

  FIG. 3 (1) shows the image when the distance information map (A) 19 is generated from the first camera image (A) 11 and the second camera image (A) 12 of the 3D image (A) 17 of the first scene. An example is shown.

  FIG. 3 (2) shows the distance information map (B) 20 generated from the first camera image (B) 13 and the second camera image (B) 14 of the 3D image (B) 18 of the second scene. An example of an image is shown.

  The distance information map is a monochrome image that is obtained by replacing the subject distance with the luminance value of the image, and the closer to the pixel, the more white the pixel, and the farther the pixel, the black.

  FIG. 4 shows the continuation of the panorama 3D image generation method of the first embodiment. FIG. 4 (2) shows a first camera combined image 21 created by performing panoramic synthesis using a distance information detection means from a second camera image (B) taken in a different direction from the first camera image (A). And

  For example, when the upper left point of the first camera image (A) 11 is O1, and the upper left point of the first camera image (B) 13 is O2, the relative position of the point O1 and the point O2 is x in the horizontal direction and the vertical direction. And the angle of the horizontal line of the first camera image (B) with respect to the first camera image (A) is θ.

  Next, panorama synthesis of the distance information map (A) 19 of the first camera image (A) 11 and the distance information map (B) 20 of the first camera image (B) 13 is performed by the following method.

  For example, in the case of the first camera combined image 21, the upper left point of the distance information map (A) is O3, and the upper left point of the distance information map (B) is O4, as shown in FIG. Then, the relative positions of the points O3 and O4 are shifted by x in the horizontal direction and y in the vertical direction, and the angle of the horizontal line of the distance information map (B) with respect to the distance information map (A) is tilted by θ to combine the images. The first camera distance information combination map 22 is obtained.

  That is, the distance information map (A) 19 and the distance information map (B) 20 are combined so that the positional relationship is the same as the positional relationship between the first camera image (A) and the first camera image (B).

  Distance information of the pixel point A3 of the first camera distance information combination map 22 corresponding to the pixels in the overlapping region of the distance information map (A) and the distance information map (B), for example, the points A1 and A2 in FIG. Is calculated as follows.

  When the distance information of the point A1 is L1, the distance information of the point A2 is L2, and the distance information of the point A3 is L3, for example, L3 is an average value of L1 and L2 L3 = (L1 + L2) / 2. Thereby, accurate distance information L3 can be obtained.

  If L3 is an intermediate value between L1 and L2, it may not be an average value.

  Furthermore, a panoramic 3D image 23 can be obtained by obtaining a panoramic image of the right image and the left image from the first camera combined image 21 and the first camera distance information combined map 22 that are left images.

  Since the right image is generated as the parallax information of the first camera distance information combination map 22, a 3D image with no vertical shift is obtained. That is, the conventional problem of FIG.

  That is, the first 3D image (A) 17 photographed so that the reference line connecting the optical axis centers of the first camera and the second camera is horizontal, and the second photographed with the reference line tilted from the horizontal. In the panorama 3D image 23 obtained by combining the 3D images (B) 18, the common part of the first 3D image (A) and the second 3D image (B) is removed from the 3D image (B). The parallax direction of the 3D image (B) is completely equal to the horizontal direction of the first 3D image, and vertical displacement can be eliminated.

  Next, a 3D display method for a panoramic 3D image will be described.

  A part of the image is extracted from the first camera combined image 21 and the first camera distance information combined map 22 obtained by the above-described method, and a 3D image is generated and then displayed on the display element, thereby displaying a panoramic 3D image. Realize.

  As illustrated in FIG. 5, a cutout image 24 is cut out from the first camera combined image 21, and a cutout map 25 having an angle of view corresponding to the cutout image 24 is cut out from the first camera distance information combined map 22.

  A cut-out 3D image 26 is obtained from the cut-out image 24 and the cut-out map 25 corresponding thereto.

  FIG. 6 shows a method for determining the position of the cutout image 24 from the first camera combined image 21.

  As shown in FIG. 6A, the upper left position of the cut-out image is changed according to the direction of the normal vector in the normal direction of the display of the display element 5. The normal vector is a unit vector.

  The direction of the normal vector can be measured by an attitude sensor 10 built in the housing 1 of the mobile terminal. A normal vector calculated from the attitude of the mobile terminal when the cutout image 24 is displayed is a normal vector 29.

  If the normal vector when the orientation of the mobile terminal is changed is the second normal vector 30, and the cutout image at this time is the second cutout image 27, the movement amount 28 of the cutout position is the normal vector 29 and the second cutout image 27. The amount of change of the normal vector 30 is changed according to the amount of change 31 of the normal vector.

  For example, if the movement amount 28 vector at the upper left cutout position is D1 and the normal vector change amount 31 vector is D2, D1 = a · D2 (a is a constant).

  Next, FIG. 6B shows a method for determining the angle of the cut-out horizontal line when the horizontal line of the display of the portable terminal is tilted.

  From the attitude sensor 10 such as an acceleration sensor, the vertical direction with respect to the horizontal line of the display of the portable terminal can be known. This vertical vector is defined as a vertical vector 32.

  The vertical vector of the mobile terminal at the time of the second cutout image 27 is set as a second vertical vector 33. When the angle between the vertical vector 32 and the second vertical vector 33 is θ, the second cutout image 27 is cut out so that the horizontal line of the second cutout image 27 is θ with respect to the horizontal line of the cutout image 24.

  A 3D image is generated from the cutout image and the cutout map and displayed on the display element.

  As described above, a cut-out 3D image is generated, the display image can be changed according to the orientation of the display of the mobile terminal, and a panoramic 3D image with a wide angle of view can be displayed.

  As shown in FIG. 7, the direction of parallax when generating a 3D image from the cutout image 24 and the cutout map 25 can be freely set.

  When the horizontal line 37 of the left and right eyes of the user is parallel to the horizontal line of the cutout image, as shown in FIG. 7A, the second cutout 3D in which the parallax of the left and right images is in the horizontal line direction of the cutout image. Let it be an image 34.

  In addition, when the horizontal line 37 of the left and right eyes of the user is perpendicular to the gravity direction 36, as shown in FIG. The third cut-out 3D image 35 is used. The gravity direction can be measured from the posture sensor 10.

  As described above, even if the binocular camera is tilted in the horizontal direction, it is possible to generate a panoramic 3D image with a wide angle of view that can be stereoscopically viewed, and display a panoramic 3D image with a wide angle of view.

  The present invention is useful as a 3D camera, and can be used for various electronic devices having a camera such as a mobile phone, a mobile terminal, a digital still camera, and the like.

DESCRIPTION OF SYMBOLS 1 Housing | casing 2 1st camera 3 2nd camera 4 3D camera 5 Display element 6 Camera control means 7 Memory | storage means 8 Distance information detection means 9 Image coupling | bonding means 10 Posture sensor 11 1st camera image (A)
12 Second camera image (A)
13 First camera image (B)
14 Second camera image (B)
15 Vertical displacement 16 Reference line 17 3D image (A)
18 3D image (B)
19 Distance information map (A)
20 Distance information map (B)
21 first camera combined image 22 first camera distance information combined map 23 panorama 3D image 24 cutout image 25 cutout map 26 cutout 3D image 27 second cutout image 28 moving amount of cutout position 29 normal vector 30 second normal vector 31 Change amount of normal vector 32 Vertical vector 33 Second vertical vector 34 Second cut-out 3D image 35 Third cut-out 3D image 36 Gravitational direction 37 Horizontal line of eye

Claims (6)

The first camera and the second camera, the distance information detecting means for generating a distance information map from the parallax of the first camera and the second camera, and aligning the plurality of images so that the corresponding feature points of the images match. Image combining means for combining images after performing
A plurality of 3D images composed of a plurality of first camera images and a plurality of second camera images with different fields of view are photographed, and a plurality of the distance information maps of the plurality of first camera images are generated by the distance information detecting means. A first camera combined image obtained by combining the plurality of first camera images by the image combining means, and a plurality of distance information maps combined with the same positional relationship as the first camera combined image combining method. A panoramic imaging device that generates a combined map. The distance information value of the distance information combination map in each pixel of the common area of the second distance information map photographed adjacent to the first distance information map of the first camera image is the first distance information map and the first distance information map. The panorama imaging apparatus according to claim 1, wherein the panorama imaging apparatus is an intermediate value between two values of the distance information in the two-distance information map. 3. The panorama imaging apparatus according to claim 1, wherein a 3D panoramic image including a left combined image and a right combined image is generated from the distance information combined map and the first camera combined image. The cut-out 3D image including the cut-out image for the left and the cut-out image for the right is generated from the cut-out map and the cut-out image cut out from the distance information combination map and the first camera combined image. The panoramic imaging device according to any one of the above. It has 3D display means and a posture sensor, and it is generated by detecting a change in the normal direction of the 3D display means with the posture sensor and shifting the cutout position of the cutout image according to the change in the normal direction. The panoramic imaging apparatus according to claim 1, wherein the cut-out 3D image is displayed on the 3D display unit. A distance information detecting means for generating a distance information map from parallax between the first camera and the second camera, and between the optical axis centers of the first camera and the second camera; In the third panorama 3D image obtained by combining the first 3D image photographed so that the reference line to be connected is horizontal and the second 3D image photographed in a state where the reference line is tilted from the horizontal, the first 3D image and A panoramic imaging device in which a parallax direction of a non-common part 3D image part obtained by removing a common part of the second 3D image from the second 3D image is equal to a horizontal direction of the first 3D image.

Images