TWI613904B - Stereo image generating method and electronic apparatus utilizing the method - Google Patents

Abstract

A method for generating a stereo image and an electronic device using the method. The electronic device includes a first camera and a second camera suitable for capturing stereo images, and a resolution of the first camera is greater than a resolution of the second camera. In this method, a first camera is used to capture a first image, a second camera is used to capture a second image, and then the second image is enlarged to the resolution of the first camera, and a depth map is generated using the first image and the enlarged second image. . Then, referring to the depth map, the first image is reprojected to reconstruct a reference image of the second image. Then, the masked area in the reference image is detected, and the masked area is filled with the enlarged second image. Finally, a stereo image including a first image and a filled reference image is generated.

Description

Stereoscopic image generation method and electronic device using the method

The present invention relates to an image processing method and device, and more particularly, to a method for generating a three-dimensional image and an electronic device using the method.

With the increasing progress of image capture technology, the digital camera's pixels have increased significantly, but the camera size has been relatively reduced, and it can be deployed on portable electronic devices such as mobile phones and tablets, allowing users to shoot images anytime, anywhere. At present, some mobile phones on the market are further equipped with a dual-lens camera. The dual-lens camera can capture images with different angles of left and right, which can generate stereo images for users to watch. In addition, according to the aberrations between the corresponding objects in the left and right images, the camera can further calculate the depth information of each object in the image and apply it to advanced functions such as object detection, depth of field adjustment, and focus switching to increase the use of the camera .

However, arranging a dual lens with the same specifications on the camera doubles the cost, which will greatly increase the cost of the device with a dual lens camera. Therefore, how to realize the functions similar to those provided by the dual-lens camera at the least cost is really one of the problems that the industry players are eager to solve.

The present invention provides an electronic device and a method for generating a stereo image. By processing images captured by a low-resolution camera and a high-resolution camera, a high-resolution stereo image can be generated.

The method for generating a stereo image of the present invention is applicable to an electronic device including a first camera and a second camera, wherein the first camera and the second camera are suitable for capturing a stereo image, and the resolution of the first camera is greater than the resolution of the second camera. In this method, a first camera is used to capture a first image, a second camera is used to capture a second image, and the second image is upscaled to the resolution of the first camera, and the first image and the enlarged second image are used. Generate a depth map. Then, referring to the depth map, the first image is reprojected to reconstruct a reference image of the second image. Then, the masked area in the reference image is detected, and the masked area is filled with the enlarged second image. Finally, a stereo image including a first image and a filled reference image is generated.

In an embodiment of the present invention, the step of re-projecting the first image to reconstruct the reference image of the second image in the reference depth map further includes depth information in the reference depth map and between the first camera and the second camera. Baseline distance, calculate the aberration information between the first image and the reference image of the second image, and extract the image data of the first image re-projected in the three-dimensional space based on this aberration information as the first Reference image for two images.

In an embodiment of the present invention, in the above reference depth map, the step of reprojecting the first image to reconstruct the reference image of the second image further includes detecting a binocular distance of the user of the electronic device, and according to the binocular distance and Baseline distance between the first camera and the second camera, calculate the baseline distance adjustment value, and then refer to the depth map and this baseline distance adjustment value to reproject the first image to reconstruct the reference image of the second image so that the reference image and the The aberration information between an image matches the distance between the two eyes.

In an embodiment of the present invention, the step of reprojecting the first image to reconstruct the reference image of the second image with the reference depth map and the baseline distance adjustment value includes adjusting the position of the second camera and calculating the adjustment according to the baseline distance adjustment value. The position coordinates of the camera, retrieve the depth information in the depth map, and convert the coordinates of the depth information to the world coordinate system, and according to the converted depth information and the adjusted position coordinates of the second camera, An image is reprojected to reconstruct a reference image of a second image.

In an embodiment of the present invention, the step of detecting a masked area in the reference image includes matching the reference image with the enlarged second image, and taking an area in the reference image that does not match the second image as the masked area. .

In an embodiment of the present invention, the step of detecting a masked area in the reference image includes detecting a blank area in the reference image as a masked area.

In an embodiment of the present invention, after detecting the occluded area in the reference image and using the enlarged second image to fill the occluded area, the method further detects the first image and the second image. The main object, and adjusting the aberration between the reference image of the second image and the first image, so that the main object is gathered on the display plane of the electronic device.

The electronic device of the present invention includes a first camera, a second camera, and an image processing circuit. The image processing circuit is coupled to the first camera and the second camera, and is configured to process the first image and the second image captured by the first camera and the second camera, respectively. The image processing circuit includes a resolution amplification module, a depth generation module, an image reconstruction module, a masked area detection module, a masked area filling module, and a three-dimensional image generation module. The first camera and the second camera are respectively used to capture one of a left-eye image and a right-eye image required for generating a stereo image, and a resolution of the first camera is greater than a resolution of the second camera. The resolution enlargement module is used to enlarge the second image to the resolution of the first camera. The depth generation module is used to generate a depth map using the first image and the enlarged second image. The image reconstruction module is used to reference the depth map, and reprojects the first image to reconstruct the reference image of the second image. The masked area detection module is used to detect the masked area in the reference image, and the masked area filling module is used to fill the masked area with the enlarged second image. The stereo image generation module is used to generate a stereo image including a first image and a filled reference image.

In an embodiment of the present invention, the image reconstruction module mentioned above refers to the depth information in the depth map and the baseline distance between the first camera and the second camera to calculate the reference image between the first image and the second image. The aberration information of the image and capture the image data of the first image re-projected in the three-dimensional space accordingly to reconstruct the reference image of the second image.

In an embodiment of the present invention, the image processing circuit further includes a binocular distance detection module, which is used to detect binocular distance of a user of the electronic device. The above image reconstruction module calculates the baseline distance adjustment value based on the distance between the two eyes and the baseline distance between the first camera and the second camera, and refers to the depth map and the baseline distance adjustment value to reproject the first image to The reference image of the second image is reconstructed so that the aberration information between the reference image and the first image conforms to the binocular distance.

In an embodiment of the present invention, the image reconstruction module further adjusts the position of the second camera and calculates the adjusted position coordinates according to the baseline distance adjustment value, retrieves the depth information in the depth map, and uses the depth information The coordinates of the image are converted to the world coordinate system, and the first image is re-projected to reconstruct the reference image of the second image according to the converted depth information and the adjusted position coordinates of the second camera.

In an embodiment of the present invention, the above-mentioned masked area detection module includes matching the reference image with the enlarged second image, and selecting a region in the reference image that does not match the second image as a masked area.

In an embodiment of the present invention, the above-mentioned masked area detection module includes detecting a blank area in the reference image as a masked area.

In an embodiment of the present invention, the image processing circuit further includes an object detection module and an aberration adjustment module, wherein the object detection module is used to detect a main object in the first image and the second image. The aberration adjustment module is used to adjust the aberration between the reference image of the second image and the first image, so that the main objects are gathered on the display plane of the electronic device.

Based on the above, the electronic device and the method for generating a stereo image of the present invention calculate the depth map of the image by enlarging the image captured by the low-resolution camera and combining the image captured by the high-resolution camera, and reconstructing the The high-resolution reference image corresponding to the image captured by the low-resolution camera can be combined with the high-resolution image captured by the high-resolution camera to generate a high-resolution stereo image.

In order to make the above features and advantages of the present invention more comprehensible, embodiments are hereinafter described in detail with reference to the accompanying drawings.

The invention adopts the structure of a low-resolution camera and a high-resolution camera, thereby reducing the device cost. Wherein, the present invention uses a high-resolution image taken by a high-resolution camera to reconstruct an image of a low-resolution camera, and uses a low-resolution image taken by a low-resolution camera to fill a masked area in the reconstructed image, and finally reconstructs this The image is combined with the image taken by a high-resolution camera to produce a high-resolution stereo image. The invention further detects the distance between the eyes of the user, and adjusts the baseline distance of the camera, and reconstructs the image of the low-resolution camera accordingly, so that the reconstructed image and the image captured by the high-resolution camera The aberration information conforms to the binocular distance of the user, and makes the generated stereoscopic image suitable for the user to view.

FIG. 1 is a block diagram of an electronic device according to an embodiment of the invention. Referring to FIG. 1, the electronic device 10 of this embodiment is, for example, a digital camera, a digital video camcorder (DVC), or a portable electronic device such as a mobile phone or a tablet computer, which can provide a camera function. The electronic device 10 includes a first camera 12, a second camera 14, and an image processing circuit 16. The functions are described as follows:

The first camera 12 and the second camera 14 include components such as an optical lens, an actuator, an aperture, a shutter, and an image sensor. The optical lens is a combination of a plurality of meniscus lenses, and is composed of a stepping motor or a sound sensor. An actuator such as a Voice Coil Motor (VCM) is driven to change the relative position between the lenses, thereby changing the focal length of the camera. The shutter is used to control the length of time that light enters the camera. The combination of the shutter and the aperture will affect the exposure of the image captured by the image sensor. The image sensor is equipped with a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS) element, or other types of light sensing elements, and can sense the intensity of light entering the optical lens To produce an image. In this embodiment, the first camera 12 and the second camera 14 are respectively one of a left camera and a right camera for capturing a stereo image, and are used for capturing a left-eye image and a right-eye image required for generating a stereo image. , Where the resolution of the first camera 12 is preset to be greater than the resolution of the second camera 14.

The image processing circuit 16 includes a resolution amplification module 161, a depth generation module 162, an image reconstruction module 163, a masked area detection module 164, a masked area filling module 165, and a three-dimensional image generation module 166. In an embodiment, each module in the image processing circuit 16 is implemented by an integrated circuit (Integrate Circuit, IC), for example, so as to implement the method for generating a three-dimensional image according to the embodiment of the present invention. In another embodiment, the image processing circuit 16 described above includes, for example, a storage device and a processor. The storage device is, for example, a hard disk or a memory having a data storage function, and the processor is, for example, a Central Processing Unit. , CPU), Microprocessor, Digital Signal Processor, Programmable Controller, Application Specific Integrated Circuits (ASIC), Programmable Logic Device (PLD), or other Device with data calculation function. Each module in the image processing circuit 16 is, for example, a computer program stored in a storage device, and these programs can be loaded by a processor to execute the three-dimensional image generating method according to the embodiment of the present invention.

Specifically, FIG. 2 is a flowchart of a method for generating a three-dimensional image of an electronic device according to an embodiment of the present invention. Please refer to FIG. 1 and FIG. 2 at the same time. The method in this embodiment is applicable to the electronic device 10 in FIG. 1 described above. The following describes the detailed steps of the method for generating a stereo image in this embodiment with various components of the electronic device 10 in FIG. 1. :

First, a first image is captured by the first camera 12 and a second image is captured by the second camera 14 (step S202). The electronic device 10 triggers the first camera 12 and the second camera 14 to capture images at the same time after the user presses the shutter button, for example.

Next, the resolution enlargement module 161 enlarges the second image captured by the second camera 14 to the resolution of the first camera 12 (step S204). The resolution enlargement module 161 uses, for example, interpolation to enlarge the second image with a low resolution to the same resolution as the first image with a high resolution. That is, the size of the enlarged second image will be the same as the size of the first image.

The above-mentioned second image enlarged by the resolution enlargement module 161 is transmitted to the depth generating module 162 together with the first image captured by the first camera 12, and the depth generating module 162 uses this first image and enlargement A depth map is generated after the second image (step S206). In detail, for example, the depth generation module 162 directly calculates aberrations of corresponding pixels in the first image and the enlarged second image, and captures the first image and the second image according to the first camera 12 and the second camera 14 The focal length of the image, the baseline distance between the cameras, and the aberration of each pixel estimate the depth of each pixel. The depth generation module 162 calculates the displacement of each pixel between the first image and the enlarged second image based on the position of each pixel in the first image and the enlarged second image, as aberration.

The depth map generated by the depth generation module 162 is transmitted to the image reconstruction module 163, and the image reconstruction module 163 refers to the depth map and reprojects the first image to reconstruct the reference image of the second image (step S208). In detail, the image reconstruction module 163, for example, refers to the depth information in the depth map and the baseline distance between the first camera 12 and the second camera 14 to calculate the aberration between the first image and the reference image of the second image. Information, and acquire the image data of the first image re-projected in the three-dimensional space accordingly to reconstruct the reference image of the second image. Thereby, a high-resolution image captured by the position of the second camera 14 can be simulated.

It should be noted that, in this embodiment, the image reconstruction module 163 is a reference image that reconstructs the second image with reference to the depth map generated by the depth generation module 162. However, in other embodiments, the image reconstruction module 163 may also refer to a disparity map generated by an aberration generating module (not shown) to reconstruct a reference image of the second image. The above-mentioned aberration generating module calculates, for example, aberrations of respective pixels in the first image and the enlarged second image (that is, the displacement of each pixel between the first image and the enlarged second image) to generate Aberration diagram.

Then, the masked area detection module 164 detects a masked area in the reference image, and the masked area filling module 165 fills the masked area with the enlarged second image (step S210). In one embodiment, since the areas covered by the reference image and the enlarged second image are the same, the masked area can be found by comparing the difference between the reference image and the enlarged second image. According to this, for example, the masked area detection module 164 matches the reference image with the aforementioned enlarged second image, and selects a region in the reference image that does not match the second image as a masked area. In another embodiment, since the reference image is re-projected from the first image, the area that was originally masked in the first image will be blank in the reconstructed image. Accordingly, the masked area detection module 164 can directly detect a blank area in the reference image as a masked area. The masked region filling module 165 directly uses the image of the region corresponding to the masked region in the enlarged second image to fill the masked region, and finally generates a reference image of the high-resolution second image.

For example, FIG. 3A to FIG. 3D are examples of masked regions according to an embodiment of the present invention. 3A and FIG. 3B respectively show an image taken by a left camera (which has a higher resolution by default) 32 and an enlarged image 34 of an image taken by a right camera (which has a lower resolution). , Where the resolution of the enlarged image 34 is the same as the resolution of the image 32. Comparing the relative positions of object A and object B in Figs. 3A and 3B, it can be seen that because the left camera is positioned to the left, an image with a relatively left angle of view can be captured, and because the right camera is positioned to the right, an angle of view can be captured Relatively right image. If the image 32 is analogized to the above-mentioned first image with higher resolution, when the electronic device wants to reconstruct the reference image 36 of the image 34 from this image 32, it will be because of the lower right part of the object A in the image 32 32a (as shown in FIG. 3C) is blocked by the object B, and the image of the region 36a overlapping with the region 32a cannot be reconstructed in the reference image 36. At this time, the electronic device can take the image data of the area corresponding to the area 36a in the image 34 to fill the area 36a, and finally generate a high-resolution reference image 36.

Returning to the flow of FIG. 2, after the filling of the masked area is completed, the final stereo image generating module 166 can generate a first image captured by the first camera 12 and a reference image filled by the masked area filling module 165. Stereo image (step S212).

With the above method, the electronic device of this embodiment can still generate a stereo image of left and right eye images with high resolution even when a low-resolution camera is used.

It should be noted that, due to factors such as small size and complicated component configuration, the electronic device may limit the configuration space of the camera. Therefore, a mobile phone with a dual-lens camera generally has a camera distance of only 1 to 2 cm, which is a distance from the average eye distance of 7.7 cm. And this gap will make the aberration of the stereo image captured by the electronic device insufficient, so that users viewing the stereo image will think that the stereo effect is not good. In view of this problem, the present invention provides a method for adaptively adjusting the aberration between the reconstructed image of the second image and the first image based on the configuration of the high- and low-resolution cameras, so as to compensate the above-mentioned gap.

In detail, FIG. 4 is a block diagram of an electronic device according to an embodiment of the invention. Referring to FIG. 4, the electronic device 40 in this embodiment is, for example, a digital camera, a digital video camera, or a portable electronic device such as a mobile phone or a tablet computer, which can provide a camera function. The electronic device 40 includes a first camera 42, a second camera 44, an image processing circuit 46, a front camera 48, and a display device 50. The first camera 42, the second camera 44, and the first camera 12, The two cameras 14 are the same or similar, so the detailed content will not be repeated here.

Different from the embodiment in FIG. 1, the electronic device 40 in this embodiment additionally includes a front camera 48 and a display device 50. The front camera 48 is, for example, disposed on the front surface of the electronic device 40 (relative to the rear surfaces of the first camera 42 and the second camera 44), and is used to capture an image of a user of the electronic device 40. The display device 50 is, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, or another display. The display device 50 can display a three-dimensional image captured by the electronic device 40, and can also be touched with The control panel is combined with a touch screen, and while the captured image is displayed, the user can operate the electronic device 40 by touching it.

On the other hand, the image processing circuit 46 of this embodiment includes a resolution amplification module 461, a depth generation module 462, an image reconstruction module 463, a masked area detection module 464, a masked area filling module 465, and a stereo image. The generating module 466, the first image processing module 467, the second image processing module 468, and the binocular distance detection module 469. Among them, the functions of the resolution amplification module 461, the depth generation module 462, the masked area detection module 464, the masked area filling module 465, and the three-dimensional image generation module 466 are the same as those of the resolution magnification module 161 of the foregoing embodiment. The depth generation module 162, the masked area detection module 164, the masked area filling module 165, and the three-dimensional image generation module 166 are the same or similar, so the detailed contents are not repeated here.

Different from the embodiment in FIG. 1, the image processing circuit 46 in this embodiment additionally includes a first image processing module 467, a second image processing module 468, and a binocular distance detection module 469. The first image processing module 467 and the second image processing module 468 process, for example, image signals captured by the first camera 42 and the second camera 44 to obtain a first image and a second image. The first image processing module 467 and the second image processing module 468 can also perform brightness, contrast, color temperature, white balance, sharpness or vividness adjustments, or noise reduction processing on this image to make the processed image Suitable for subsequent stereo image generation. In addition, the binocular distance detection module 469 receives, for example, a user image captured by the front lens 48, and can calculate the binocular distance of the user by performing image processing such as face recognition and eyeball recognition. The binocular distance calculated by the binocular distance detection module 469 will be transmitted to the image reconstruction module 463 and used as a reference for reconstructing the reference image of the second image.

Specifically, FIG. 5 is a flowchart of a method for generating a three-dimensional image of an electronic device according to an embodiment of the present invention. Please refer to FIG. 4 and FIG. 5 at the same time. The method in this embodiment is applicable to the electronic device 40 in FIG. :

First, a first image is captured by the first camera 42 and a second image is captured by the second camera 44 (step S502). After that, the resolution enlargement module 461 enlarges the second image captured by the second camera 44 to the resolution of the first camera 42 (step S504), and the depth generation module 462 uses the first image and the enlarged image The second image generates a depth map (step S506). The implementation manners of steps S502 to S506 are the same as or similar to the implementation manners of steps S202 to S206 in the foregoing embodiment, so the details thereof will not be repeated here.

The difference from the embodiment of FIG. 1 is that, in this embodiment, while the first camera 42 and the second camera 44 are capturing images, the front camera 48 is also used to capture the user's image, and the binocular distance detection module 469 uses This image is analyzed to detect the distance between the eyes of the user of the electronic device 40 (step S508). This binocular distance is transmitted to the image reconstruction module 463, and the image reconstruction module 463 calculates a baseline distance adjustment value based on the binocular distance and the baseline distance between the first camera 42 and the second camera 44. For example, if the baseline distance between the first camera 42 and the second camera 44 is 1.5 cm, and the detected distance between the two eyes is 6.5 cm, the baseline distance adjustment value may be 5 cm.

Then, the image reconstruction module 463 refers to the depth map generated by the depth generation module 162 and the above-mentioned baseline distance adjustment value, and reprojects the first image to reconstruct the reference image of the second image, so that the reference image and the first image are different. The aberration information is consistent with the binocular distance (step S512). In detail, the image reconstruction module 463, for example, adjusts the position of the second camera 42 and calculates the adjusted position coordinates according to the baseline distance adjustment value. It also acquires depth information in the depth map and converts the coordinates of the depth information. To the world coordinate system, finally, the first image is re-projected to reconstruct the reference image of the second image according to the converted depth information and the adjusted position coordinates of the second camera 42.

For example, FIG. 6A and FIG. 6B are examples of reconstructing a reference image according to a baseline distance adjustment value according to an embodiment of the present invention. Please refer to FIG. 6A first. The baseline distance between the left camera 62 and the right camera 64 of the electronic device in this embodiment is d1, where the left-eye image 62a is an image captured by the left camera 62 and the right-eye image 64a is The reference image of the image captured by the right camera 64 reconstructed by the electronic device according to the method of the above embodiment. 4B, the distance between the left eye 66 and the right eye 68 of the user is d2. The electronic device can calculate the baseline distance adjustment value by using the above-mentioned baseline distance d1 and the user's binocular distance d2, and adjust the reconstructed reference image. In detail, comparing the left-eye image 62a and the left-eye image 62b shows that the position of the object has not changed, while comparing the right-eye image 64a and the right-eye image 64b, the position of the object has been shifted to the left, which will make these objects The aberration of the lens is increased to meet the user's binocular distance d2. It should be noted that, in this embodiment, the electronic device only adjusts the reconstructed right-eye image 64a according to the binocular distance d2, so that the aberration of the left-eye image 62b and the right-eye image 64b matches the binocular distance d2 of the user. However, in other embodiments, the electronic device may also adjust the left-eye image 62a and the right-eye image 64a at the same time, or adjust only the left-eye image 64a, so that the aberrations of the adjusted left-eye and left-eye images match the user's two eyes. The distance d2 is not limited here.

Returning to the flow of FIG. 5, after the image adjustment is completed, the masked area detection module 464 detects the masked area in the reference image, and the masked area filling module 465 uses the enlarged second image to fill the mask. Area (step S514). Finally, the stereo image generation module 466 generates a stereo image including the first image captured by the first camera 42 and the reference image filled by the masked area filling module 465 (step S516). The display is transmitted to the display device 50 and provided to the user for viewing. The implementation manners of steps S514 to S516 are the same as or similar to the implementation manners of steps S210 to S212 in the foregoing embodiment, so the detailed content is not repeated here.

With the above method, the user can view a stereo image with a better stereo effect on the display device 50 of the electronic device while shooting the stereo image using the electronic device.

It should be noted that, in order to create a stereoscopic effect for the eyes of the user, the electronic device displays two images (ie, a left-eye image and a right-eye image) having a difference in perspective (parallax) on the same display plane, and The objects appearing in the image will converge at different positions relative to the display plane according to their disparity. For example, objects with negative parallax will be presented in front of the display plane; objects with positive parallax will be presented behind the display plane; and objects with zero parallax will be presented on the display plane. With the above-mentioned difference in parallax, the user will perceive the distance of these objects, and then produce a three-dimensional feeling. Among them, the user's eyes will try to focus on the display plane, but some objects in the image will converge to other positions than the display plane, which will cause the user to easily feel dizzy.

In this regard, based on the fact that when viewing a stereo image, the user usually focuses on the main objects in the image, such as the face of an actor in a movie. Therefore, if the face area of the actor appearing in the left and right eye images is adjusted to zero parallax, Will provide users with a relatively comfortable viewing experience. Accordingly, the present invention provides a method for adjusting a generated stereo image according to the position of a main object, so as to provide a stereo image suitable for viewing by a user.

Specifically, FIG. 7 is a block diagram of an electronic device according to an embodiment of the invention. Referring to FIG. 7, the electronic device 70 in this embodiment is, for example, a digital camera, a digital video camera, or a portable electronic device such as a mobile phone or a tablet computer, which can provide a camera function. The electronic device 70 includes a first camera 72, a second camera 74, an image processing circuit 76, and a display device 78. The first camera 72, the second camera 74, and the first camera 12 and the second camera 14 in the embodiment of FIG. The same or similar, so its detailed content will not be repeated here.

Different from the embodiment in FIG. 1, the electronic device 40 in this embodiment additionally includes a display device 78, which is, for example, a liquid crystal display, a light-emitting diode display, or other display, and can display a stereo image captured by the electronic device 70. The display device 78 can also be combined with a touch panel to form a touch screen, and at the same time that the captured image is displayed, the user can operate the electronic device 70 by touching it.

On the other hand, the image processing circuit 76 of this embodiment includes a resolution amplification module 761, a depth generation module 762, an image reconstruction module 763, a masked area detection module 764, a masked area filling module 765, and a stereo image. The generating module 766, the first image processing module 767, the second image processing module 768, the object detection module 769, and the aberration adjustment module 770. Among them, the functions of the resolution amplification module 761, the depth generation module 762, the image reconstruction module 763, the masked area detection module 764, the masked area filling module 465, and the stereo image generation module 466 are the same as those of the embodiment in FIG. The resolution amplification module 161, depth generation module 162, image reconstruction module 163, masked area detection module 164, masked area filling module 165, and stereo image generation module 166 are the same or similar, so the detailed content is as follows This will not be repeated here.

Different from the embodiment in FIG. 1, the image processing circuit 76 in this embodiment additionally includes a first image processing module 767, a second image processing module 768, an object detection module 769, and an aberration adjustment module 770. The first image processing module 767 and the second image processing module 768 process, for example, the image signals captured by the first camera 72 and the second camera 74 to obtain the first image and the second image. The first image processing module 767 and the second image processing module 768 can also perform brightness, contrast, color temperature, white balance, sharpness or vividness adjustments, or noise reduction processing on this image to make the processed image Suitable for subsequent stereo image generation. In addition, the object detection module 769 receives the first image and the second image processed by the first image processing module 767 and the second image processing module 768, and detects the first image and the second image. The main object is provided to the aberration adjustment module 770. The aberration adjustment module 770 adjusts the aberration between the reference image of the second image and the first image for the main object detected by the object detection module 769, so that the main object is gathered on the display of the electronic device 70 flat.

Specifically, FIG. 8 is a flowchart of a method for generating a three-dimensional image of an electronic device according to an embodiment of the present invention. Please refer to FIG. 7 and FIG. 8 at the same time. The method in this embodiment is applicable to the electronic device 70 in FIG. 7 described above. The following describes the detailed steps of the method for generating a stereo image in this embodiment with each component of the electronic device 10 in FIG. 7. :

The difference between this embodiment and the embodiment of FIG. 2 is that this embodiment additionally includes step S812, in which the main object in the first image and the second image is detected by the object detection module 769, and adjusted by the aberration The module 770 adjusts the aberration between the reference image of the second image and the first image, so that the main objects can be gathered on the display plane of the electronic device. This can provide users with a comfortable viewing experience.

For example, FIGS. 9A and 9B are examples of adjusting image aberrations based on main objects according to an embodiment of the present invention. FIG. 9A illustrates the relationship between the objects viewed by the left eye 92 and the right eye 94 of the user and the display plane. Among them, the object C with negative parallax will be presented in front of the display plane; the object D with positive parallax will be presented behind the display plane. Assume that the object 96 is the main object in the stereo image, and because the object 96 has negative parallax, the object 96 viewed by the user's left eye 92 and right eye 94 in the stereo image is located in front of the display plane, which will cause the user Easily feel dizzy. In this regard, the electronic device in this embodiment adjusts the aberrations of the left and right eye images in the stereo image so that the parallax of the object 96 is zero. Please refer to FIG. 9B. After adjusting the aberration of the stereo image, the objects 96 viewed by the user's left eye 92 and right eye 94 in the adjusted stereo image will be located on the display plane, thereby providing the user with a more comfortable viewing Experience.

To sum up, the method for generating a stereo image of the present invention and the electronic device using the method adopt the architecture of a high-resolution camera and a low-resolution camera, and use the high-resolution image captured by the high-resolution camera to reconstruct the low-resolution camera. Image, and the low-resolution image captured by the low-resolution camera is used to fill the occluded area in the reconstructed image, so as to generate a high-resolution stereo image. The invention can further adjust the aberration of the image according to the binocular distance of the user and the depth information of the main objects in the image, so as to provide a stereo image with better stereo effect or more comfortable viewing.

Although the present invention has been disclosed as above with the examples, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some modifications and retouching without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be determined by the scope of the attached patent application.

10, 40, 70: electronic devices 12, 42, 72: first camera 14, 44, 74: second camera 16, 46, 76: image processing circuits 161, 461, 761: resolution amplification modules 162, 462, 762: Depth generation module 163, 463, 763: Image reconstruction module 164, 464, 764: Masked area detection module 165, 465, 765: Masked area filling module 166, 466, 766: Stereo image generation module 32: image 32a, 36a: area 34: enlarged image 36: reference image 467, 767: first image processing module 468, 768: second image processing module 469: binocular distance detection module 48: front camera 50 , 78: Display device 62: Left camera 64: Right camera 62a, 62b: Left eye image 64a, 64b: Right eye image 66, 92: Left eye 68, 94: Right eye 769: Object detection module 770: Aberration Adjustment modules A, B, C, D, 96: Object d1: Baseline distance d2: Two-eye distance S202-S212, S502-S516, S802-S814: Steps of a stereo image generating method according to an embodiment of the present invention.

FIG. 1 is a block diagram of an electronic device according to an embodiment of the invention. FIG. 2 is a flowchart of a method for generating a stereo image of an electronic device according to an embodiment of the present invention. 3A to 3D are examples of masked regions according to an embodiment of the present invention. FIG. 4 is a block diagram of an electronic device according to an embodiment of the invention. FIG. 5 is a flowchart of a method for generating a stereo image of an electronic device according to an embodiment of the present invention. 6A and 6B are examples of reconstructing a reference image according to a baseline distance adjustment value according to an embodiment of the present invention. FIG. 7 is a block diagram of an electronic device according to an embodiment of the invention. FIG. 8 is a flowchart of a method for generating a stereo image of an electronic device according to an embodiment of the present invention. FIG. 9A and FIG. 9B are examples of adjusting image aberrations based on main objects according to an embodiment of the present invention.

S202-S212: Steps of a method for generating a stereo image according to an embodiment of the present invention.

Claims (14)

A stereo image generating method is applicable to an electronic device including a first camera and a second camera, wherein the first camera and the second camera are suitable for capturing stereo images, and the resolution of the first camera is greater than that of the second camera. Resolution, the method includes the following steps: using the first camera to capture a first image, and using the second camera to capture a second image; upscale the resolution of the second image to the first camera; using the first An image and the enlarged second image generate a depth map; referring to the depth map, reprojecting the first image to reconstruct a reference image of the second image; detecting a masked area in the reference image, and using the enlarged image The second image fills the masked area; and generates the stereo image including the first image and the filled reference image. The method according to item 1 of the scope of patent application, wherein the step of referring to the depth map and reprojecting the first image to reconstruct the reference image of the second image further includes: referring to the depth information in the depth map and the first A baseline distance between a camera and the second camera, calculating aberration information between the first image and the reference image of the second image; and capturing and reprojecting the three-dimensional space based on the aberration information Image data of the first image to reconstruct the reference image of the second image. The method according to item 1 of the scope of patent application, wherein the step of re-projecting the first image to reconstruct the reference image of the second image with reference to the depth map further comprises: detecting both eyes of a user of the electronic device Distance; calculating a baseline distance adjustment value according to the distance between the eyes and a baseline distance between the first camera and the second camera; and re-projecting the first image to reconstruct the reference image with reference to the depth map and the baseline distance adjustment value The reference image of the second image makes the aberration information between the reference image and the first image consistent with the binocular distance. The method of claim 3, wherein the step of reprojecting the first image to reconstruct the reference image of the second image with reference to the depth map and the baseline distance adjustment value includes: adjusting the value according to the baseline distance , Adjusting the position of the second camera and calculating the adjusted position coordinates; acquiring depth information in the depth map, and converting the coordinates of the depth information into a world coordinate system; and according to the converted depth Information and the adjusted position coordinates of the second camera, reproject the first image to reconstruct the reference image of the second image. The method according to item 1 of the scope of patent application, wherein the step of detecting the occluded area in the reference image includes: matching the reference image with the enlarged second image; and taking a reference image that is not The area matched by the second image is used as the masked area. The method according to item 1 of the scope of patent application, wherein the step of detecting the masked area in the reference image includes: detecting a blank area in the reference image as the masked area. The method according to item 1 of the patent application scope, wherein after the step of detecting a masked area in the reference image and filling the masked area with the enlarged second image, the method further includes: detecting the first image And the main object in the second image; and adjusting the aberration between the reference image and the first image of the second image so that the main object is gathered on the display plane of the electronic device. An electronic device includes: a first camera and a second camera, respectively capturing one of a left-eye image and a right-eye image required for generating a stereo image, wherein a resolution of the first camera is greater than a resolution of the second camera An image processing circuit, which is coupled to the first camera and the second camera, and processes the first image and the second image respectively captured by the first camera and the second camera, and includes: a resolution magnifying module, which magnifies the first camera and the second camera; The resolution from the two images to the first camera; a depth generation module that uses the first image and the enlarged second image to generate a depth map; an image reconstruction module that references the depth map to reproject the first image to Reconstructing a reference image of the second image; a masked area detection module that detects a masked area in the reference image; a masked area filling module that uses the enlarged second image to fill the masked area; and a stereo image generation model Group to generate the stereo image including the first image and the filled reference image. The electronic device according to item 8 of the scope of patent application, wherein the image reconstruction module includes calculating the first image and the reference image by referring to the depth information in the depth map and the baseline distance between the first camera and the second camera. Aberration information between the reference images of the second image, and according to this, image data of the first image re-projected in the three-dimensional space is retrieved to reconstruct the reference image of the second image. The electronic device according to item 8 of the patent application scope, wherein the image processing circuit further includes: a binocular distance detection module that detects a binocular distance of a user of the electronic device, wherein the image reconstruction module includes The binocular distance and the baseline distance between the first camera and the second camera, calculate a baseline distance adjustment value, and refer to the depth map and the baseline distance adjustment value to reproject the first image to reconstruct the second image The reference image such that the aberration information between the reference image and the first image conforms to the binocular distance. The electronic device according to item 10 of the scope of patent application, wherein the image reconstruction module further adjusts the position of the second camera and calculates the adjusted position coordinates according to the baseline distance adjustment value to capture the depth in the depth map. Information, and transform the coordinates of the depth information into the world coordinate system, and reproject the first image to reconstruct the reference of the second image according to the converted depth information and the adjusted position coordinates of the second camera. image. The electronic device according to item 8 of the scope of patent application, wherein the masked area detection module includes matching the reference image with the enlarged second image, and taking the reference image that does not match the second image Area as the masked area. The electronic device according to item 8 of the scope of patent application, wherein the masked area detection module includes detecting a blank area in the reference image as the masked area. The electronic device according to item 8 of the scope of patent application, wherein the image processing circuit further includes: an object detection module that detects main objects in the first image and the second image; and an aberration adjustment module, Aberrations between the reference image and the first image of the second image are adjusted so that the main object is gathered on a display plane of the electronic device.