JP2014102265A - 3d imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a 3D camera having a camera with optical zoom function at one side capable of achieving a high resolution 3D photography at a high magnification.SOLUTION: In a 3D photography using an asymmetrical camera including a first camera (optical zoom) and a second camera (electronic zoom), when the zoom magnification N is smaller than N0, a right and left 3D image is generated using an optical zoom image taken by a first camera and an electronic zoom image taken by a second camera image; and when the zoom magnification N is larger than N0, after estimating the distance using the right and left cameras, a right and left 3D image is generated using an image taken by the first camera. Thus, 3D photographing of moving picture is possible at a low magnification, and a high resolution 3D photographing of still picture with little resolution difference is possible even at a high magnification.

Description

  The present invention relates to a portable electronic device with a camera such as a mobile phone or a digital still camera having a 3D camera capable of 3D shooting.

  As a 3D (stereoscopic) shooting method, the left image of the left line of sight and the right image of the right line of sight are respectively displayed with two cameras (binocular cameras) having an optical axis separated by about 6 cm to 8 cm, which is the distance between human eyes. There is a way to shoot.

  FIG. 9 shows an example of a conventional structure of a 3D camera composed of two cameras. For example, Patent Document 1 describes details.

  As shown in FIGS. 9 (1) and 9 (2), two cameras with an optical zoom are arranged apart from each other on the back surface of the display element 5 of the housing 1 of the imaging apparatus.

  FIG. 9 (3) shows a block diagram. The first camera and the second camera with the optical zoom are controlled by the camera control means 8. During zooming, the lens positions of the first camera and the second camera are adjusted based on the adjustment values in the look-up table so that the optical zoom magnifications of the first camera and the second camera match.

  Next, a second conventional example of the 3D camera will be described. For example, Patent Document 2 describes details.

  Two cameras are arranged apart from each other on the back surface of the display element 5 of the housing 1. The first camera is an optical zoom and the second camera is a single-focus camera without optical zoom. Further, the number of pixels of the first camera is larger than the number of pixels of the second camera.

  FIG. 9 (4) shows a block diagram. The first camera with optical zoom and the single-focus second camera are controlled by the camera control means 8. During zooming, the optical zoom magnification of the first camera and the electronic zoom magnification of the second camera are adjusted so that the zoom magnifications of the first camera and the second camera match.

  Since the resolution of the second camera is lower than the resolution of the first camera, the resolution of the second camera image is increased as follows.

  An area of the first camera image (called a corresponding block) corresponding to a part of the second camera image (called a block) is searched by image processing, and the block of the second camera image is made a corresponding block of the first camera image. replace.

  As described above, the resolution of the second camera image can be increased, and the difference in resolution between the first camera image and the second camera image can be reduced.

  As another method for generating a 3D image from a 3D camera, for example, as described in Patent Document 3, there is a method using parallax information (or distance information). After measuring the parallax from the main image and the sub image of the binocular camera composed of the main image pickup device and the sub image pickup device that are spatially separated, a 3D image can be generated from the parallax information and the main image.

Japanese Patent No. 3303254 Japanese Patent Laying-Open No. 2005-210217 Japanese Patent Laid-Open No. 2005-20606

  The conventional 3D optical zoom camera described in Patent Document 1 requires two expensive and large optical zoom cameras, which causes a problem that the mobile terminal is expensive and large.

  On the other hand, the conventional 3D optical zoom camera described in Patent Document 2 is a single-focus camera on one side, and thus can be reduced in cost and can be reduced in size. However, it takes time to perform high-resolution software processing of a single-focus camera image. It was difficult to shoot video.

  In some cases, the corresponding block (or corresponding pixel) between the optical zoom camera image and the single focus camera image cannot be found. In this case, the single focus camera image is a partially degraded image with a low resolution.

  In addition, since the 3D image generation method using the 3D camera described in Patent Document 3 takes a long time to generate a 3D image, it is difficult to shoot a moving image.

  An object of the present invention is to enable 3D shooting with a high zoom magnification while maintaining a high resolution with an inexpensive camera configuration.

  In the above description, it is assumed that the first camera is an optical zoom. However, the same problem occurs also in the case of a 3D camera including a high-resolution third camera and a low-resolution fourth camera. The problem will be described below.

  An image cut from the third camera having the maximum pixel number AL is defined as a third camera generation image, and the pixel number is defined as BL (BL ≦ AL). On the other hand, an image cut from the fourth camera (AR <AL) having the maximum pixel number AR is defined as a fourth camera generation image, and the pixel number is defined as BR (BR ≦ AR). Here, for simplicity, it is assumed that the aspect ratio of the image is constant.

  When the ratio of the number of pixels BR to the number of pixels BL (BR / BL) decreases, the 3D image having the third camera generation image and the fourth camera generation image as the left and right images becomes a 3D image that is difficult to see due to a large resolution difference. .

  Therefore, the present invention solves the problem of image quality degradation of 3D images due to the difference in resolution between left and right images.

  Next, means for solving the above problems will be described.

  The 3D imaging apparatus of the present invention is a 3D imaging apparatus having a first camera with an optical zoom, a single-focus type second camera, and parallax extracting means for the first camera image and the second camera image. When the magnification is smaller than a predetermined magnification, a 3D image is generated from the optical zoom image of the first camera and the electronic zoom image of the second camera. Further, when the zoom magnification is larger than the predetermined magnification, a 3D image is generated from the parallax information which is an output result of the parallax extracting means and the first camera image.

  With this configuration, at low magnification, a 3D image can be generated in a short time by generating a 3D image from the images of the first camera and the second camera. Further, by generating a 3D image from an optical zoom image at a high magnification, 3D shooting with a high zoom magnification can be realized while maintaining a high resolution with an inexpensive camera configuration.

  The 3D imaging apparatus of the present invention includes a third camera, a fourth camera having a smaller number of pixels than the third camera, and a parallax extraction unit for the third camera image and the fourth camera image. When the number of pixels of the generated 3D image is smaller than the predetermined number of pixels, the 3D image is generated from the third camera image and the fourth camera image. Further, when the number of pixels of the 3D image to be generated is larger than the predetermined number of pixels, a 3D image is generated from the parallax information that is the output result of the parallax extracting means and the third camera image.

  With this configuration, at a low magnification, a 3D image can be generated in a short time by generating a 3D image from the images of the third camera and the fourth camera. Further, by generating a 3D image from the third camera image at a high magnification, 3D shooting with a high zoom magnification can be realized while maintaining a high resolution with an inexpensive camera configuration.

The 3D imaging apparatus of the present invention includes a third camera, a fourth camera having a smaller number of pixels than the third camera, and a parallax extraction unit for the third camera image and the fourth camera image. Because
At the time of 3D moving image shooting, the 3D image is generated from the third camera image and the fourth camera image. At the time of 3D still image shooting, a 3D image is generated from the parallax information which is the output result of the parallax extracting means and the third camera image.

  With this configuration, during moving image shooting, a 3D moving image can be generated in a short time by generating a 3D image from the images of the third camera and the fourth camera. Further, at the time of still image shooting, by generating a 3D image from the third camera image, it is possible to realize 3D still image shooting with a high zoom magnification while maintaining high resolution with an inexpensive camera configuration.

  According to the present invention, it is possible to realize 3D shooting with a high zoom magnification while maintaining a high resolution with an inexpensive camera configuration. In addition, image quality deterioration of a 3D image having a large resolution difference between the left and right images is suppressed.

1 is a schematic structural diagram of a 3D imaging apparatus according to Embodiment 1 of the present invention. The figure which shows the production | generation method of the 3D image at the time of zoom of Embodiment 1 of this invention The figure which shows the subjective evaluation result of asymmetric 3D image Flowchart of 3D imaging according to Embodiment 1 of the present invention Flowchart of second 3D imaging according to the first embodiment of the present invention The figure which shows the production | generation method of 3D image of Embodiment 2 of this invention. The figure which shows the relationship between the number of pixels and the resolution ratio in the 1st 3D image generation method Flowchart of 3D shooting according to the second embodiment of the present invention Schematic structure diagram of a conventional 3D imaging device

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
FIG. 1 is a schematic structural diagram of the 3D imaging apparatus according to the present embodiment.

  Hereinafter, the first embodiment will be described on the assumption that a portable terminal device including a single casing is used. However, other electronic devices having a small camera such as a foldable portable terminal or a digital still camera (DSC) may be used. The same applies to the case of wearing.

  As shown in FIGS. 1A and 1B, a 3D camera 4 including a first camera 2 and a second camera 3 is disposed on the back surface of the display element 5 of the rectangular casing 1 of the mobile terminal device. The first camera 2 is a camera with an optical zoom function, and the second camera 3 is a single focus camera without an optical zoom function. Since the second camera is cheaper and smaller than the first camera, the portable terminal can be made smaller or thinner than when the two cameras with optical zoom shown in FIG. 9A are used.

  In FIG. 1A, the first camera with optical zoom is a left image camera, but it may be a right image camera.

  FIG. 1 (3) is a block diagram of the mobile terminal device. The first camera 2 and the second camera 3 are controlled by the camera control unit 8, and the captured image is stored in the storage unit 10 or displayed using the display element 5.

  The first camera includes a sensor 7 and a lens 6 and has an optical zoom function, and a desired zoom magnification is obtained by controlling the position of the lens 6 with a zoom control signal 9.

  FIG. 2 shows a 3D image generation method during zooming.

  First, a 3D image generation method when the zoom magnification N is smaller than the switching magnification N0 will be described. This is called a first 3D image generation method.

  The first camera (left camera) acquires an optical zoom image according to the zoom magnification N. Further, an image having the same angle of view as the optical zoom image is cut out from the image of the second camera (right camera), and then an electronic zoom image in which the pixel adjustment (= resizing) is performed to the same number of pixels as the optical zoom image is created. A 3D image is generated by setting the optical zoom image [1] as a left image and the electronic zoom image [2] as a right image.

  Note that the resolution of the electronic zoom image relatively decreases as the magnification N increases as compared with the optical zoom image. Therefore, the first 3D image generation method is performed only when the zoom magnification N is smaller than N0.

  If the number of vertical pixels of the original image indicating the resolution of the right image and the left image is YL, YR, and the number of horizontal pixels is XL, XR, then YR (or XR) is larger than YL (or XL) when the zoom magnification is increased. Get smaller.

  FIG. 3 shows the result of the subjective evaluation by the present inventors on whether or not the 3D left and right images have different resolutions as to whether or not the 3D images look uncomfortable.

  For four types of 3D images, whether the left image is full HD (1920 × 1080 dots) or not when the number of pixels of the right image is smaller than full HD, there is no sense of discomfort as a 3D image. The evaluation results are shown.

  The vertical pixel number ratio represents the ratio of the vertical pixel number YR of the right image to the vertical pixel YL (= 1080 dots of full HD) of the left image, and the smaller the value, the larger the resolution difference between the left and right images. .

  From the results of subjective evaluation by the inventors, it was found that if YR / YL is about 0.4 or more, a decrease in the resolution of the right image is not an issue, and a 3D image with less discomfort can be obtained.

  Therefore, if a zoom magnification at which YR / YL is, for example, 0.4 or more is N0, it is understood that a 3D image can be generated by the first 3D image generation method when N <N0. It was.

  In the case of N> N0, the resolution of the right image is lowered, so that a good 3D image cannot be obtained.

  Next, a 3D image generation method when the zoom magnification is greater than N0 will be described. This will be referred to as a second 3D image generation method.

  The second 3D image generation method is a method of calculating or estimating distance information (depth information) for each pixel or minute region, and generating a 3D image from the distance information.

  As a method for calculating the depth information, there is a method of estimating from the brightness of the pixel, but a more accurate measurement method is to calculate parallax information (= distance information) for each pixel or minute area from the left and right images of the binocular camera. This method is described in Patent Document 3, for example.

  Here, a method of obtaining parallax information (= distance information) for each pixel or minute area from the left and right images of the binocular camera is employed.

  From the left and right images [3] and [4] in FIG. 2, each pixel (or minute area) is correlated using image processing correlation calculation or the like, so that disparity information (distance information) of each pixel is obtained. [5] can be calculated.

  Left and right images [6] and [7] are obtained from the first camera left image [3] and the parallax information [5].

  Since the 3D image generation uses the first camera image [3] having a high resolution, both the left and right images can be converted to a high resolution image.

  Therefore, when the magnification is high (N> N0), a high-resolution 3D image can be obtained by using the second 3D image generation method.

  FIG. 4 shows a 3D shooting flowchart during zooming according to Embodiment 1 of the present invention.

  As described above, by combining the first 3D image generation method and the second 3D image generation method, high-resolution 3D imaging is possible in a wide zoom magnification range.

  Even when the magnification is low (N <N0), the second 3D image generation method can be used. However, since the second 3D image generation method requires processing time, the resolution of the second camera image is allowed to be reduced. If so, the first 3D image generation method is more desirable than the second 3D image generation method.

  Also in the case of 3D moving image shooting, the second 3D image generation method takes more processing time and the frame rate of the moving image becomes smaller. Therefore, the first 3D image generation method is preferable to the second 3D image generation method.

  Therefore, instead of switching between the first 3D image generation method and the second 3D image generation method according to the zoom magnification, a method of switching between a moving image and a still image is also effective.

  FIG. 5 shows a 3D shooting flowchart at this time. In the case of moving image shooting, the first 3D image generation method having a high processing speed is adopted, and in the case of still image shooting, a high-resolution second 3D image generation method is adopted.

  Thereby, a 3D image can be generated without reducing the frame rate of the 3D moving image.

(Embodiment 2)
In the second embodiment, a method of changing the 3D image generation method according to the number of pixels of the 3D image to be generated will be described.

  In the second embodiment, the description will be made assuming a portable terminal device including a single casing, but the portable terminal device is mounted on an electronic device having a small camera such as a foldable portable terminal or a digital still camera (DSC). The same applies to the case.

  FIG. 6A is a block configuration diagram of the 3D imaging apparatus according to the second embodiment.

  Here, the third camera 11 and the fourth camera 12 do not have an optical zoom function, but may have an optical zoom function.

  The 3D image generation method shown in FIG. 6 (2) is the same as the first 3D image generation method of the first embodiment.

  Hereinafter, the number of pixels refers to the number of vertical pixels × the number of horizontal pixels, and the resolution means the number of pixels in the original image. For simplicity, it is assumed that the aspect ratio of the image is constant.

  An image generated from the third camera having the maximum pixel number AL ([6]) is defined as a third camera generated image ([8]), and the pixel number is defined as BL (BL ≦ AL). On the other hand, an image generated from the fourth camera having the maximum number of pixels AR (AR <AL) ([7]) is a fourth camera generation image ([9]), and the number of pixels is BR (BR ≦ AR). To do.

  [8] An image that has been resized to an image having the number of pixels C is a [10] image, and [9] an image that has been resized to an image having the number of pixels C is an [11] image, and from [10] and [11] A 3D image is obtained.

  FIG. 7 shows an example of a graph of the resolution ratio P ((right resolution) / (left resolution)) with respect to the number C of pixels of the 3D image.

  When the number of pixels C is smaller than BR, the resolution of [10] decreases from BL to C, and the resolution of [11] also decreases from BR to C. Therefore, the ratio of the left and right resolution is (left resolution) / (right resolution) = C / C = 1.

  On the other hand, when the number of pixels C is larger than BR and smaller than BL, the resolution (number of pixels) of [10] decreases from BL to C, and the resolution of [11] remains BR. Therefore, the ratio between the left and right resolutions is (right resolution) / (left resolution) = BR / C> BR / BL.

  On the other hand, when the number of pixels C is larger than BL, the resolution of [10] remains BL and the resolution of [11] also remains BR. Accordingly, the ratio P between the left and right resolutions is (left resolution) / (right resolution) = BL / BR.

  Here, assuming that a resolution at which the resolution ratio P is, for example, 0.3 × 0.3 = 0.09 or more is set as P0, and the resolution C of the 3D image at this time is C0, FIG. 3 of the first embodiment. As can be seen from the result, if C <C0, the resolution ratio P is larger than P0, so there is no problem with the 3D image using the first 3D image generation method.

  On the other hand, when C> C0, the first 3D image generation method is not suitable because the resolution difference is large at a resolution ratio P0 or less. Therefore, when C> C0, a 3D image is generated using the second 3D image generation method.

  FIG. 6 (3) shows a second 3D image generation method.

  When C> C0, distance information [12] is calculated from the third camera image [10] and the fourth camera image [11], and the third camera image [10] and the distance information [10] with high resolution and the number of pixels C are calculated. 12] to obtain a 3D image. The generation method is the same as the second 3D image generation method of the first embodiment. Thereby, a 3D image with the same resolution of the left and right images is obtained.

  As described above, by changing the 3D image generation method according to the number of pixels C of the 3D image to be generated, a 3D image with a small resolution difference between the left and right images can be obtained.

  The present invention is useful as a 3D camera, and can be used for various electronic devices having a camera such as a mobile phone, a mobile terminal, a digital still camera, and the like.

DESCRIPTION OF SYMBOLS 1 Housing | casing 2 1st camera 3 2nd camera 4 3D camera 5 Display element 6 Lens 7 Sensor 8 Camera control means 9 Zoom control signal 10 Memory | storage means 11 3rd camera 12 4th camera

Claims (3)

A 3D imaging device comprising: a first camera with an optical zoom; a single-focus second camera; and a parallax extracting means for the first camera image and the second camera image,
When the zoom magnification is smaller than a predetermined magnification, a 3D image is generated from the optical zoom image of the first camera and the electronic zoom image of the second camera,
A 3D imaging device that generates a 3D image from parallax information that is an output result of the parallax extracting unit and the first camera image when the zoom magnification is larger than the predetermined magnification. A 3D imaging device having a third camera, a fourth camera having a smaller number of pixels than the third camera, and a parallax extracting means for the third camera image and the fourth camera image,
When the number of pixels of the 3D image to be generated is smaller than the predetermined number of pixels, the 3D image is generated from the third camera image and the fourth camera image,
When the number of pixels of the generated 3D image is larger than the predetermined number of pixels, an imaging apparatus that generates a 3D image from the parallax information that is an output result of the parallax extracting unit and the third camera image. A 3D imaging device having a third camera, a fourth camera having a smaller number of pixels than the third camera, and a parallax extracting means for the third camera image and the fourth camera image,
At the time of 3D video shooting, the 3D image is generated from the third camera image and the fourth camera image,
A 3D imaging device that generates a 3D image from parallax information that is an output result of the parallax extraction unit and the third camera image at the time of 3D still image shooting.

Images