JP2013115467A - Stereoscopic photographing device and portable terminal device using the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stereoscopic photographing device capable of stereoscopic image photographing of a still image or moving image by using a main camera for front photographing and a sub-camera for photographing a photographer, and to provide a portable terminal device using the same.SOLUTION: The image data processing part 135 determines a structure from an image of the main camera 110, and cuts it out from a background image. Structure position information is extracted from an image of the sub-camera 111 and, based on this, the structure is moved from a camera position obtained by a main camera. The background and a hidden object are corrected. In addition, image information about a portion hidden in an image photographed by the main camera 110 is created from information about the image of the sub-camera 111. Accordingly error-free correction is possible.

Description

  The present invention relates to a stereoscopic imaging apparatus that can mount a camera and can capture a stereoscopic image, and a portable terminal apparatus using the same.

  In recent years, mobile phones having a camera function equipped with a solid-state imaging device such as a CCD have appeared, and still images and moving images can be taken. In addition, those that can shoot stereoscopic images have also appeared.

Next, the principle that an image looks 3D (3 dimensions) is shown in the figure.
Take the scenery as shown in FIG. 11 as an example. FIG. 12 shows a landscape that a human can see from the left eye and a landscape that can be seen from the right eye. Since there is a distance between the left eye and the right eye as shown in FIG. 12, a stereoscopic effect is generated by feeling a difference (parallax) between the scenery seen on the left and right.

  In order to display a 2D (2Dimension) photograph or moving image in 3D, it is necessary to prepare a background (image) visible from the left eye and a background (image) visible from the right eye as shown in FIG. When the background (image) for the left eye is visible only with the left eye and the background (image) for the right eye is visible only with the right eye, the image appears three-dimensional. As a technique for obtaining a left-eye image and a right-eye image for 3D conversion, there are the following methods.

  The following (1) to (3) are examples of the prior art of 3D shooting by hardware. (4) is an example of the prior art of 3D conversion by software.

(1) Method of mounting two identical imaging units in parallel on an imaging device (see Patent Documents 1 and 2)
As shown in FIG. 13, two identical imaging units are separated from each other by a parallax distance and mounted in parallel in the imaging apparatus, so that images of both the left eye and the right eye can be obtained. The feature is that both still images and moving images can be obtained.

(2) Method of photographing two images in one imaging unit having two lens units As shown in FIG. 14, the left eye and the right eye are mounted on the imaging device by separating the two lenses by a distance of parallax and mounting them in parallel. Both images are input to one sensor. The feature is that both still images and moving images can be obtained.

(3) A method of continuously capturing images by translating the imaging device with an imaging device having only one imaging unit As shown in FIG. 15, the imaging device is imaged at the left eye position to obtain an image. Next, it is moved manually and photographed at the right eye position to obtain an image. A 3D image is obtained using the image of the left eye position and the image of the right eye position.

(4) 3D conversion by software In order to convert an image into 3D, a process of converting an original image into an image viewed from the right eye based on an image viewed from the left eye is performed. A schematic flow is shown in FIG. In the figure, there are two structures, and the structure is a process for making the structures appear three-dimensional.
(A) Identify the composition and obtain information on the angle of view and depth.
(B) Identify and cut out structures A and B at each depth.
(C) Based on the depth information of (a), the structure is moved by the amount of parallax between the left-eye viewpoint image and the right-eye viewpoint image.
(D) For the left-eye viewpoint image and the right-eye viewpoint image, the structure is moved to complement the missing background and the hidden part of the structure.

JP 2005-250396 A JP 2006-345246 A

However, the above prior art has the following problems.
(1) Method of mounting two identical image pickup units in parallel on an image pickup device It is necessary to equip two same image pickup units, which is expensive and has the disadvantage that the device is enlarged.
(2) Method of taking two images in one imaging unit having two lens units In order to obtain images of both eyes in one sensor, it is necessary to separate the respective images. In addition, the vertical resolution (number of pixels) of the sensor is less than half. In general, since the lens becomes small, there is a drawback that sensitivity is lowered.

(3) A method of continuously capturing images by translating the image capturing apparatus in an image capturing apparatus having only one image capturing unit. Since there is a difference in capturing time between the image at the left eye position and the image at the right eye position, the moving object If you shoot, you will not get the correct 3D image. Moreover, there is a drawback that a moving image cannot be obtained.

(4) 3D conversion by software If there is an error in the identification of the composition in (a), a 3D image cannot be generated correctly. Similarly, if the structure (b) is mistakenly identified, a 3D image cannot be generated correctly.
In addition, since the processes (a) to (d) are executed by software, it takes a very long time, and a high processing capacity is required for the processing apparatus to create 3D moving image data in real time.

  Furthermore, in order to perform 3D imaging of images including moving images, two identical imaging units are prepared as in (1), or the left eye, right eye, and two images are input to one imaging element. There is a need.

  Many common mobile terminal devices include an imaging unit having a large image sensor for photographing on the back side and two imaging units having a small image sensor used for a TV phone or the like. If two parts are arranged, three imaging parts are required, which is not realistic.

  SUMMARY OF THE INVENTION In view of such circumstances, the present invention provides a stereoscopic camera capable of shooting a stereoscopic image with good image characteristics using a main camera that performs front-side shooting and a sub-camera for self-shooting, and a mobile terminal device using the same. The purpose is to provide.

The present invention provides a stereoscopic imaging device having a main imaging unit for frontal imaging and a sub-imaging unit for self-imaging in a housing.
The sub-imaging is performed using the main imaging unit and the sub-imaging unit arranged in the same direction and using the same-direction arrangement structure in which the distance between the imaging units is parallax and an image captured by the main imaging unit as original image data. An image data processing unit that acquires parallax information from an image captured by a unit and generates an image having parallax of the sub-imaging unit based on the parallax information based on the original image data. To do.

  In the stereoscopic imaging device of the present invention, the image data processing unit performs a structure analysis on the original image data, extracts a structure and a background image, and extracts structure position information from the image of the sub-imaging unit. Extracting and moving and arranging the structure of the original image based on the parallax information and the structure position information.

  Further, in the stereoscopic imaging apparatus of the present invention, the image data processing unit extracts hidden part information of the original image data, and extracts image information from the image of the sub-imaging unit, thereby generating a hidden part image. It is characterized by doing.

  The present invention is a portable terminal device including the stereoscopic photographing device.

According to the present invention, since the same image as the characteristics of the main imaging unit is generated from the parallax information of the image captured by the sub-imaging unit based on the image captured by the main imaging unit, a stereoscopic image with good image characteristics is acquired. can do.
In addition, the structure and the background image are extracted from the original image data, the parallax information and the structure position information from the image of the sub-imaging unit are extracted, and the structure of the original image is moved and arranged, so that the accurate parallax An image can be generated.
In addition, since the image information is extracted from the image of the sub-imaging unit for the information of the hidden portion of the original image data, an accurate parallax image can be generated.

It is a figure which shows the 1st Example in a foldable portable terminal. It is a figure which mounts an acceleration sensor in the 1st example and detects a horizontal state. It is a figure which shows the 2nd Example in a foldable portable terminal. It is a figure which shows the 3rd Example in a foldable portable terminal. It is a block diagram which shows the function structure of the portable terminal which has a main camera and a sub camera, and enables three-dimensional photography. It is a figure which shows the example of 3D conversion by the image analysis of the software on the basis of the main camera image. It is a flowchart which shows the operation | movement of transfer to 3D imaging | photography mode using a sensor. It is a flowchart which shows the process which produces | generates the image of a sub camera, referring the image information of a sub camera based on the image data of a main camera. It is a flowchart following FIG. It is a flowchart following FIG. It is a figure which shows the scenery which is imaging | photography object. It is a figure which shows the scenery seen from a left eye and the scenery seen from a right eye. It is a figure which shows the conventional imaging device which mounts two identical imaging parts in parallel, and obtains a stereo image. It is a figure which shows the conventional imaging device which mounts two lenses by the distance of parallax, and mounts them in parallel, and acquires a stereo image. It is a figure which shows the conventional imaging device which mounts only one imaging part and obtains a stereo image. It is a figure which shows the conventional 3D system which converts into the image seen from a right eye based on the image seen from a left eye from an original image.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

[Summary of Invention]
It is an object of the present invention to obtain a highly accurate 3D image regardless of whether it is a still image or a moving image using two non-identical image sensors. In other words, using the one with the better captured image characteristics, the other image is regenerated to obtain an image with better characteristics. Therefore, “(3) a method of continuously capturing images by translating the imaging device with an imaging device having only one imaging unit” is not relevant to the present invention. In addition, since “(1) a method of mounting two identical imaging units in parallel in an imaging apparatus” in the prior art has the same imaging unit, a 3D image is obtained by the same processing for each imaging unit. However, providing two identical image pickup units restricts the size and shape design of the terminal if the main image pickup device and the sub image pickup device have the same image pickup unit. Further, if the main imaging element has the same imaging unit and is separated from the sub imaging unit, three imaging units are provided, which is not practical in terms of cost.

  The present invention realizes that a 3D image is obtained by an imaging apparatus having two imaging units, a main camera for frontal imaging and a sub camera for self-imaging. Regarding the conventional technique “(2) Method of photographing two images on one imaging unit using a conversion lens for giving two parallaxes to the lens unit”, it is necessary to provide a dedicated structure. It is not related to the invention.

  As for 3D conversion by software, a structure analysis is performed on an image obtained from one imaging unit, and 3D conversion is performed. However, in the present invention, one image is obtained from two non-identical imaging units. Is the object of the image to be converted to 3D, and the other is information for assisting the structural analysis for converting to 3D and for improving accuracy.

[Imaging unit location and terminal structure]
A portable terminal will be described as an example of an embodiment using the stereoscopic imaging device of the present invention. When 3D shooting is performed using two image sensors, it is necessary to provide parallax by horizontally aligning the imaging units of the main camera and the sub camera. As an implementation example, there are structures as shown in FIGS. This will be described in order below.

<First embodiment>
FIG. 1 is a diagram showing a first embodiment of a foldable portable terminal.
The mobile terminal 10 includes an upper housing 11 and a lower housing 12. In the upper housing 11, a main screen 13 and a sub screen 14 are arranged on the front and back. The upper housing 11 can be rotated 180 degrees with respect to the lower housing 12 when the mobile terminal 10 is opened and closed by the rotary hinge 15 (see FIGS. 1B and 1F). Furthermore, the upper casing 11 can be turned upside down by rotating 180 degrees with respect to the surface of the lower casing 12 with the portable terminal 11 opened at the time of 3D shooting by the rotary hinge 16 (FIG. 1 (c). ) To (e)).

The main camera (main imaging unit) 17 is provided on the back surface of the lower housing 12 and shoots toward a general subject. In shooting with the main camera 17, the user takes a picture with the main screen 13 as a viewfinder in the state of (b) or (f) of FIG.
The sub camera 18 is provided on the upper portion of the upper housing 11 and is a camera used mainly for photographing a user himself such as a TV phone. The sub camera 18 is inferior to the main camera 17 in characteristics such as resolution.

  FIG. 1E shows a state in which the mobile terminal 10 is opened and rotated 180 degrees with respect to the lower housing 12, and the main camera 17 and the sub camera 18 are arranged in the same direction. At this time, the distance between the main camera 17 and the sub camera 18 is a degree corresponding to the parallax distance. At the time of 3D photographing, the user photographs the sub screen 14 as a viewfinder in a state as shown in FIG. In FIG. 1, the cameras 16 and 17 are arranged in the vertical direction, but the mobile terminal 10 is placed horizontally and the cameras 16 and 17 are placed horizontally. The main screen 13 can be protected in the state shown in FIG. 1A by the rotation of the rotary hinge 15, and the portable terminal 10 can be used in the states shown in FIGS. 1B and 1F. The state of FIG. 1E is a state used only for 3D shooting.

  The same direction detector 19 for detecting whether or not the main camera 17 and the sub camera 18 are in the 3D shooting mode in the same direction is provided in the rotary hinge 16 portion. For example, a magnet 19a is provided on the lower housing 12 side of the rotary hinge 16, and a magnetic sensor 19b is provided on the upper housing 11 side. And then. In the state of FIG. 1 (e), the magnet 19a and the magnetic sensor 19b are close to each other and face each other, and the 3D shooting mode can be detected. Based on the detection result, a function of performing “display for notifying that the camera is in the 3D camera shooting mode” or “display for asking whether to enter the 3D camera shooting mode” on the sub-screen 14 serving as a viewfinder Can be implemented. In the state other than FIG. 1E, the 3D shooting mode is not set.

  Further, as shown in FIG. 2A, by mounting an acceleration sensor (horizontal detection unit) 20, it is detected whether the main camera 17 and the sub camera 18 are in the horizontal position. A function of displaying the detection angle on the sub screen 14 serving as a finder can be implemented. As shown in FIG. 2A, if the level is suitable for 3D shooting, it may be displayed on the sub-screen 14 that 3D shooting is possible.

  In addition, as shown in FIG. 2B, a “warning message prompting to be level” can be displayed on the sub-screen 14 when an angle at which 3D shooting becomes difficult during the 3D camera shooting mode is reached. Also, if the result of detection by the same direction detection unit 19 results in a state other than that shown in FIG. 1E during the 3D camera shooting mode, 3D shooting is not possible, so the 3D camera shooting mode is automatically canceled. Function can be implemented.

<Second embodiment>
FIG. 3 is a diagram illustrating a second embodiment of the folding portable terminal.
The mobile terminal 30 includes an upper housing 31 and a lower housing 32. The upper casing 31 further includes a first upper casing 31a and a second upper casing 31b. The first upper casing 31a has a sub screen 34 disposed on the surface side, and the second upper casing 31a. A main screen 33 is disposed on the surface side of the housing 31b (the surface opposite to the surface of the first upper housing 31a). The upper casing 31 has a rotating hinge 35 and can rotate 180 degrees with respect to the lower casing 32 when the portable terminal 30 is opened and closed (see FIG. 3B). Furthermore, it has a cycloid rotation structure between the first upper casing 31a and the second upper casing 31b, and can rotate 90 degrees (see FIGS. 3C to 3E). The main camera 36 is provided on the back surface of the lower housing 32, and when shooting with the main camera 36, the main screen 33 is shot using the main screen 33 as a viewfinder in the state of (c) or (e) of FIG.

  The sub camera 37 is a camera that is disposed on the upper portion of the second upper casing 31b and is used mainly for photographing the user himself / herself such as a TV phone. At the time of 3D shooting, shooting is performed using the sub screen 34 as a viewfinder in the state shown in FIG. In the state of FIG. 3G, the cameras 36 and 37 are arranged in the horizontal direction, and shooting is possible in the 3D shooting mode.

A magnet 39a and a magnetic sensor 39b are provided as the same direction detection unit, the magnet 39a is provided in the lower casing 32, and the magnetic sensor 39b is provided in the second upper casing 31b. Since its function is the same as that of the first embodiment, detailed description thereof is omitted.
Further, since an acceleration sensor is provided as a horizontal direction detection unit as in the first embodiment, detailed description thereof is omitted.

<Third embodiment>
FIG. 4 is a diagram showing a third embodiment of the folding portable terminal.
The mobile terminal 50 includes an upper housing 51 and a lower housing 52. In the upper casing 51, a main screen 53 and a sub screen 54 are arranged on the front and back sides. The upper casing 51 can rotate 180 degrees with respect to the lower casing 52 when the portable terminal 50 is opened and closed by the rotary hinge 55.
The main camera (main imaging unit) 57 is provided on the back surface of the lower housing 52 and shoots toward a general subject. In photographing with the main camera 17, the user photographs using the main screen 53 as a viewfinder.

  The sub camera 18 is provided on the upper portion of the upper housing 11 and is a camera used mainly for photographing a user himself such as a TV phone.

  The sub camera 58 is provided on the upper part of the upper casing 51, and is a camera used mainly for photographing a person such as a TV phone. In the state of (a) for the person photographing, the sub screen 54 is photographed as the viewfinder in the state of (c) during the 3D photographing.

  In this embodiment, a rotation mechanism is mounted on the sub camera 58 for 3D shooting, and rotation by 180 degrees is possible. Therefore, the direction can be changed from the front surface to the back surface of the upper casing 51 ((a) to (c)). It is also possible to automatically enter the 3D shooting mode when the orientation of the sub camera 58 is detected by a sensor and the camera faces in the direction (c).

  In the embodiment, the portable terminal is a foldable type, but it is not particularly necessary to be a foldable type. In addition, since the rotation mechanism is implemented in the sub camera 58, the main screen can be used as a viewfinder unlike in FIGS. 1 and 3, so that more comfortable shooting is possible.

  When the main screen is not used as a finder and the sub screen is used as a finder, the main camera may have a rotation mechanism. Also in this structure, as shown in FIG. 2 (b), a warning message for prompting to be horizontal can be issued when an angle at which 3D shooting becomes difficult is reached during the 3D camera shooting mode. Further, when the state other than the state shown in FIG. 4C occurs during the 3D camera shooting mode, 3D shooting is impossible, so that the mode before the 3D camera shooting can be automatically returned.

[Functional configuration of mobile terminal]
A functional configuration of the mobile terminal described above will be described.
FIG. 5 is a block diagram illustrating a functional configuration of a mobile terminal that has a main camera and a sub camera and enables stereoscopic shooting. The portable terminal 100 includes a main camera (main imaging unit) 110, a sub camera (sub imaging unit) 111, a same direction detection unit 120, a horizontal detection unit 121, a processing unit 130, display units 150 and 151, a RAM 160, a ROM 161, and an auxiliary device. It is composed of functional blocks of the storage device 162. The main camera 110 includes an image sensor 112, and the sub camera 111 includes an image sensor 113. The processing unit 130 includes video I / Fs 131 and 132, a sensor I / F 133, an image data processing unit 135, a CPU 136, and display I / Fs 137 and 138.
Each part will be described below.

・ Camera 110, 111
As described above, the main camera 110 is an imaging unit for front-side shooting. On the other hand, the sub camera 111 is an image capturing unit that captures an image of the user.
The main camera 110 and the sub camera 111 are provided with image sensors 112 and 113, respectively. The image sensors 112 and 113 are semiconductor elements that take in light incident from a lens and convert the light into an electric signal.
As shown in FIG. 5, the elements are arranged in a matrix (ROW-COLMN), the subject is imaged on the light receiving surface of the sensor, the light and darkness of the imaged image is converted into an electrical signal, and the converted signals are sequentially By reading, it can be obtained as image data.
In general, the larger the light receiving area of the element, the higher the light receiving sensitivity, and the larger the number of elements, the higher the resolution of the image. A terminal with an imaging unit on both the back and front, such as a portable terminal, uses a large-area, high-resolution imaging unit on the back side for each purpose, and has a small area and low resolution on the front side compared to the back side. An imaging unit is often used.

The same direction detection unit 120 and the horizontal detection unit 121
The same direction detection unit 120 is a sensor that detects whether the main camera 110 and the sub camera 111 are arranged in the same direction, and corresponds to the magnet and the magnetic sensor shown in FIG. The same direction detection unit 120 detects whether the mobile terminal 110 can be set to the 3D shooting mode.
The horizontal detection unit 121 corresponds to the acceleration sensor shown in FIG. 1 and detects whether the main camera 110 and the sub camera 111 are in a horizontal position suitable for 3D shooting.

Processing Unit The video I / Fs 131 and 132 receive image data detected from the main camera 110 and the sub camera 111 and send them to the image data processing unit 135.
The sensor I / F 133 receives detection results from the same direction detection unit 120 and the horizontal detection unit 121 and sends the detection results to the CPU 136.
The image data processing unit 135 acquires image data from the image sensors 112 and 113 in accordance with parameters and instructions given from the CPU 136 and outputs them to the RAM 160 and the CPU 136. In order to perform the display, the data acquired from the image sensors 112 and 113 and the image data from the CPU 136 / RAM 160 are output to the display unit (main screen 150, sub screen 151). Also, color information and resolution are converted when image data is input and output.
The CPU 136 processes and executes a program stored in the ROM 161, and activates the 3D imaging system of the mobile terminal 100. Then, the program stored in the auxiliary storage device 162 is transferred to the RAM 160, and the program temporarily stored in the RAM 160 is processed and executed.
The display I / Fs 137 and 138 send the processing data of the image data processing unit 135 to the display units 150 and 151.

-ROM (Read Only Memory) 161
A basic program (IPL: Initial Program Loader / OS: Operating System) for operating the computer and fixed data are stored.

-RAM (Random Access Memory) 160
This is a storage device for temporarily storing a work area for executing a program and image data obtained from the image sensors 112 and 113. The program used in the present invention is temporarily read from the auxiliary storage device 162 and stored in order to execute at high speed. The processing unit 130 reads the program from the RAM 160 and executes it.

Auxiliary storage device 162
Generally, it refers to a magnetic recording device HDD (Hard Disk Drive), a semiconductor recording device SSD (Solid State Drive), an optical recording device (CD, DVD, BLU-RAY Disk) and the like. Stores programs, image data, and moving image data. Therefore, a 3D processing program used in the present invention is also stored. Information for converting the color and resolution of the image sensors 112 and 113 is also stored.

Display unit (main screen 150, sub screen 151)
Data for display is received from the processing unit 130 and displayed. For example, data stored in the RAM 160 obtained from the image sensors 112 and 113 is output to the display unit 150 and displayed.

[3D image data processing]
In a method for obtaining a 3D image by arranging two imaging units having different characteristics in parallel in an imaging apparatus, a 3D image can be obtained without requiring image analysis by software. However, the resolution / color characteristics of the sub camera are basically 3D images based on the poor characteristics. In order to make 3D images with high resolution and high (color characteristic) quality, it is better to use the image of the main camera as a reference.

FIG. 6 is a diagram showing an example of 3D conversion by software image analysis based on the main camera image. The processing procedure for 3D conversion is the same as that for 3D conversion by software shown in FIG.
(A) is a left-eye image taken by the main camera, which is a source for creating a 3D right-eye image. (E) is a sub-camera photographed image at the right eye position photographed simultaneously with the main camera photographing. Conventionally, the composition was extracted by analyzing the composition from the image data of (a), but further, the structure can be reliably obtained by extracting the image difference corresponding to the parallax from the image captured by the sub camera of (f). .

  In (b), the structure is confirmed and cut out from the background image. In (g), structure position information is extracted from the image of the sub camera, and in (c), the structure is moved from the image position obtained by the main camera based on the extracted position information. Conventionally, the moving position is structurally analyzed and moved by estimating the moving position from the depth information. Therefore, the accuracy is not sufficient. However, by performing such processing, the accuracy can be improved.

  In (d), the background and the hidden object are corrected. Conventionally, since there is no image information for the hidden part, the correction was made by guessing from the surrounding information, and there was a possibility that the correction was incorrect, but as shown in (h), the image of the sub-camera image Since the image information of the hidden part is generated from the information captured by the main camera from the information, correction without error is possible.

[Operation in 3D shooting]
FIG. 7 is a flowchart showing an operation of shifting to the 3D shooting mode using the sensor. Description will be made assuming that the functional block is FIG. 5 and the terminal structure is FIG. 1 and FIG.

  The CPU 136 detects ON / OFF of the magnetic sensor 19b which is the same direction detection unit 120 (step S1). In step S2, if the magnetic sensor 19b is ON (if the magnetic sensor 19b is close to the magnet 19a), the CPU 136 uses the mobile terminal 100 (in FIG. 1, the mobile terminal 10 in the acceleration sensor 20). ) Is detected (step S3). In step S4, if the portable terminal 100 is in the horizontal position (if it is within the range of angles capable of 3D shooting), the process proceeds to step S5. If NO in steps S2 and S4, the process returns to step S1.

  In step S5, the CPU 136 automatically shifts to the 3D shooting mode. Alternatively, the CPU 136 causes the image data processing unit 135 to generate a message indicating whether or not to shift to the 3D shooting mode, and displays the message on one of the display units 150 and 151.

  Again, the CPU 136 detects ON / OFF of the magnetic sensor 19b which is the same direction detection unit 120 (step S6). In step S7, if the magnetic sensor 19b is ON (if the magnetic sensor 19b is close to the magnet 19a), the CPU 136 uses the mobile terminal 100 (in FIG. 1, the mobile terminal 10 in the acceleration sensor 20). ) Is detected (step S8). In step S9, if the portable terminal 100 is in the horizontal position (if it is within the range of angles capable of 3D shooting), the process proceeds to step S10, and the CPU 136 maintains the 3D shooting mode. If NO in step S9, the process returns to step S6.

  If NO in step S7, the CPU 136 automatically returns from the 3D shooting mode to the original mode in step S11. Alternatively, the CPU 136 causes the image data processing unit 135 to generate a message indicating whether or not to shift to the original shooting mode, and displays the message on one of the display units 150 and 151.

  FIG. 8, FIG. 9, and FIG. 10 are flowcharts showing processing for generating an image of the sub camera based on the image data of the main camera while referring to the image information of the sub camera. Although there are differences in resolution and color information between the image of the main camera and the image of the sub camera, it can be referred to as image data on the right eye side, so it is possible to create right eye image data for 3D with high accuracy from the main camera image. . This will be described with reference to the functional block of FIG. 5 and FIG.

  In step S <b> 21, the image data processing unit 135 performs a structure analysis of the captured image data (FIG. 6A) captured by the main camera 110. In step S22, an object (structure) that can be converted to 3D is extracted from the captured image data of the main camera 110, and the shape and coordinates of the object are extracted (FIG. 6B). The number of objects that can be converted to 3D = M. In FIG. 6B, there are two structures A and B. Data such as the shape and coordinates of each object is stored in the RAM 160 as an array MO (M).

  In step S23, the structure analysis of the captured image data of the sub camera 111 (FIG. 6E) is performed. In step S24, an object that can be converted into 3D from the sub camera is extracted (FIG. 6F). Extraction of the shape and coordinates of the object is extracted, and the number of objects that can be converted to 3D = S. Data such as the shape and coordinates of each object is stored in the RAM 160 as an array SO (S).

  In step S25, the image data processing unit 135 sets variable K = 0 and array variable MK (0: M−1) = 0. Here, the variable K represents the number of 3D objects. The array variable MK (0: M−1) indicates the object number to be converted into 3D. If MK (0: M−1) = 0, it means “not 3D target”, and MK (0: M− If 1) = 1, it means “3D object”.

  In step S26, the image data processing unit 135 sets I = 0 for the loop counter. In step S27, it is confirmed whether I = M. If NO, the process proceeds to step S28, and if YES, the process proceeds to step S35.

  In step S28, the image data processing unit 135 sets J = 0 for the loop counter. In step S29, it is confirmed whether J = S. If NO, the process proceeds to step S30, and if YES, the process proceeds to step S33.

In step S30, the image data processing unit 135 checks whether the main camera object MO (I) and the sub camera object SO (J) have the same shape. If they have the same shape, in step S31, it is confirmed whether the object MO (I) of the main camera and the object SO (J) of the sub camera are at positions where the coordinates for the parallax are shifted. If there is a position where the parallax coordinates are shifted, the process proceeds to step S32.
If NO in steps S30 and 31, J = J + 1 in step S34.

  In step S32, K = K + 1 and MK (I) = 1. In step S33, I = I + 1 is set, and the process returns to step S27.

In step S35, I = 0, and in step S36, it is confirmed whether K = I. If yes, the process ends.
If K = I is not satisfied, in step S37, the coordinates of the object data of MO (MK (I)) are changed to the coordinates of the object data of SO (MK (I)).

  In step S <b> 38, the image data processing unit 135 corrects the data hidden in the moved MO (MK (I)) object data coordinates with reference to the surrounding data of SO (MK (I)). In step S39, K = K + 1 is set, and the process returns to step S36.

  In this way, since the same image as the characteristics of the main camera is generated from the information of the image captured by the sub camera based on the image captured by the main camera, a stereoscopic image with good image characteristics can be acquired.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and the design and the like within the scope of the present invention are also within the scope of the claims. include.

10, 30, 50, 100 Mobile terminal 11, 31, 51 Upper casing 12, 32, 52 Lower casing 13, 33, 53, 150 Display unit (main screen)
14, 34, 54, 151 Display section (sub-screen)
15, 16, 35 Rotating hinges 17, 36, 57, 110 Main camera (main imaging unit)
18, 37, 58, 111 Sub camera (sub imaging unit)
19a Magnet 19b Magnetic sensor 20 Acceleration sensors 112 and 113 Image sensor 120 Same direction detection unit 121 Horizontal detection unit 130 Processing unit 135 Image data processing unit 136 CPU

Claims (4)

In a stereoscopic imaging device having a main imaging unit for frontal imaging and a sub-imaging unit for self-imaging in a housing,
A same-direction arrangement structure in which the main imaging unit and the sub-imaging unit are arranged in the same direction and the distance between the imaging units is parallax;
The image captured by the main imaging unit is used as original image data, parallax information is acquired from the image captured by the sub imaging unit, and the parallax of the sub imaging unit is obtained based on the parallax information. An image data processing unit for generating an image having;
A stereoscopic photographing apparatus comprising:   The image data processing unit performs a structure analysis on the original image data, extracts a structure and a background image, extracts structure position information from an image of the sub-imaging unit, and extracts the parallax information and the structure. The stereoscopic imaging apparatus according to claim 1, wherein the structure of the original image is moved and arranged based on position information.   The image data processing unit extracts information of a hidden portion of the original image data from an image of the sub-imaging unit, and generates an image of the hidden portion. The stereoscopic photographing apparatus according to 2.   A portable terminal device comprising the stereoscopic photographing device according to any one of claims 1 to 3.

Images