TWM626646U - Electronic apparatus - Google Patents

Abstract

An electronic apparatus is provided. The electronic apparatus includes an interpupillary distance detection device, a storage device and a processor. A two-dimensional (2D) original image corresponding to a first view angle is obtained by the processor, and a depth map of the 2D original image is estimated by the processor. Interpupillary distance information of a user is detected by the interpupillary distance detection device. A pixel shift processing is performed for the 2D original image according to the interpupillary distance information and the depth map by the processor to generate a reference image corresponding to a second viewing angle. An image inpainting processing is performed on the reference image by the processor to obtain a restored image. The restored image and the 2D original image are merged by the processor to generate a three-dimensional image conforming to a three-dimensional image format and including image content corresponding to different view angles.

Description

electronic device

The present invention relates to an electronic device, and particularly to an electronic device that provides stereoscopic images.

With the advancement of display technology, displays supporting three-dimensional (3D) image playback have gradually become popular. The difference between 3D display and two-dimensional (two dimension, 2D) display is that 3D display technology allows viewers to feel the three-dimensional sense in the image, such as the three-dimensional facial features and depth of field of characters, etc. 2D images cannot show this effect. The principle of the 3D display technology is to allow the viewer's left eye to view the left-eye image and the viewer's right eye to view the right-eye image, so that the viewer can experience the 3D visual effect. With the vigorous development of 3D stereoscopic display technology, it can provide people with a visually immersive experience. It can be known that a 3D display needs to use a corresponding 3D display technology to play an image of a specific 3D image format, otherwise the display will not be able to display the image correctly. In addition, the 3D image content that the user can obtain by himself is not very sufficient at present, so even if the user has a naked-view 3D display, the user cannot fully and arbitrarily enjoy the display effect brought by the naked-view 3D display.

In view of this, the present invention proposes an electronic device for providing a stereoscopic image, which can convert a two-dimensional image into a stereo image conforming to a stereoscopic image format according to a user's real interpupillary distance.

The novel creative embodiment provides an electronic device, which includes a pupil distance tracking device, a storage device, and a processor. The processor connects the interpupillary distance tracking device and the storage device and is configured to perform the following steps. A 2D original image corresponding to the first viewing angle is acquired, and a depth map of the 2D original image is estimated. The pupil distance information of the user is detected by the pupil distance tracking device. A reference image corresponding to the second viewing angle is generated by performing pixel offset processing on the two-dimensional original image according to the pupil distance information and the depth map. An image restoration process is performed on the reference image to obtain a restored image. The restored image and the 2D original image are combined to generate a stereoscopic image conforming to a stereoscopic image format, wherein the stereoscopic image includes image content corresponding to different viewing angles.

Based on the above, in the embodiment of the present invention, a corresponding depth map can be generated according to a two-dimensional original image, and then a stereoscopic image conforming to the stereoscopic image format can be generated according to the viewer's interpupillary distance and the depth map. Accordingly, the novel creation embodiment can greatly expand the 3D content that can be displayed on the 3D display, and provide a more comfortable stereoscopic viewing experience.

In order to make the above-mentioned features and advantages of the novel creation more obvious and easy to understand, the following embodiments are given and described in detail with the accompanying drawings as follows.

Part of the embodiments of the present invention will be described in detail with reference to the accompanying drawings in the following. The component symbols quoted in the following description, when the same component symbols appear in different drawings, will be regarded as the same or similar components. These embodiments are only a part of the present invention, and do not disclose all possible implementations of the present invention. Rather, these embodiments are only examples of devices within the scope of the patent application of the present invention.

FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. Referring to FIG. 1 , the electronic device 10 may include a pupil distance detection device 110 , a storage device 120 and a processor 130 . The processor 130 is coupled to the storage device 120 . In one embodiment, the electronic device 10 and the stereoscopic (3D) display 20 can form a 3D display system. The 3D display 20 may be a naked-view 3D display or a glasses-type 3D display. On the other hand, the 3D display can be a head-mounted display device or a computer screen, a desktop screen or a TV that provides a 3D image display function. The 3D display system can be a single integrated system or a separate system. Specifically, the 3D display 20 , the interpupillary distance detection device 110 , the storage device 120 and the processor 130 in the 3D display system can be implemented as an all-in-one (AIO) electronic device, such as a head-mounted display device, Laptops, smartphones, tablets or game consoles, etc. Alternatively, the 3D display 20 can be connected to the processor 130 of the computer system through a wired transmission interface or a wireless transmission interface, such as a head mounted display device, a desktop screen or an electronic signboard connected to the computer system.

The interpupillary distance detection device 110 can be used to detect the interpupillary distance (IPD) information of the user, such as an eye tracking device, an eye tracker, or an image capture device, etc. related devices that can obtain eyeball information. In some embodiments, a device with computing capabilities (such as the processor 130 ) can receive the user image captured by the image capture device, and can calculate the user's Interpupillary distance information. Alternatively, in some embodiments, the eye tracking device can calculate the pupillary distance information by emitting an infrared beam to obtain the pupil position.

The storage device 120 is used to store data such as images, data, and code (eg, operating system, application program, driver program) for the processor 130 to access, and it can be, for example, any type of fixed or removable random access memory random access memory (RAM), read-only memory (ROM), flash memory (flash memory), hard disk or a combination thereof.

The processor 130 is coupled to the interpupillary distance detection device 110 and the storage device 120 , such as a central processing unit (CPU), an application processor (AP), or other programmable general-purpose or special Microprocessor (microprocessor), digital signal processor (digital signal processor, DSP), image signal processor (image signal processor, ISP), graphics processing unit (graphics processing unit, GPU) or other similar devices, integrated circuits and their combinations. The processor 130 can access and execute the code and software modules recorded in the storage device 120, so as to realize the stereoscopic image generation method in the novel creative embodiment.

In order to allow the user to experience the 3D visual effect through the 3D display 20 , the 3D display 20 can allow the user's left eye and right eye to view image content corresponding to different viewing angles (ie, the left eye image and the right eye image) based on various designs. . In the embodiment of the present invention, the electronic device 10 can generate a stereoscopic image conforming to a stereoscopic image format according to a 2D planar image corresponding to a single viewing angle, and the stereoscopic image can include image contents corresponding to different viewing angles. Accordingly, the 3D display 20 can display a stereoscopic image conforming to the stereoscopic image format based on its 3D display technology, so that the user can view the content of the stereoscopic image.

FIG. 2 is a flowchart of a method for generating a stereoscopic image according to an embodiment of the present invention. Referring to FIG. 2 , the method of this embodiment is applicable to the electronic device 10 in the above-mentioned embodiment. The following describes the detailed steps of this embodiment in conjunction with various elements in the electronic device 10 .

In step S210, the processor 130 acquires the 2D original image corresponding to the first viewing angle, and estimates the depth map of the 2D original image. The two-dimensional original image may be a photo taken by a common camera device according to a single viewing angle. The 2D original image can also be the image content generated by the graphics software. On the other hand, the 2D raw image can also be the image content provided by an application operating in full screen mode. Alternatively, the 2D original image can also be the image content displayed on the display by the application. Alternatively, the 2D original image can also be a single frame image in the video stream. The processor 130 may use a screen capture technology such as the "Desktop Duplication API" of the Windows operating system to obtain the 2D raw image. Alternatively, the processor 130 can acquire the 2D original image through any image transmission technology. It should be particularly noted that the two-dimensional original image is image data suitable for display using a two-dimensional display technology.

In some embodiments, the processor 130 may utilize various monocular depth estimation techniques to estimate the depth map of the 2D raw image. The monocular depth estimation technology can use the DenseNet model, SeNet model or MiDaS model in the convolutional neural network architecture, or the MegaDepth model or the CycleGAN model in the generative adversarial network architecture to estimate the depth information of the two-dimensional original image. In other words, the processor 130 can input the 2D raw image to the trained deep learning model, so that the deep learning model can generate the depth map of the 2D raw image accordingly. The model parameters of the trained deep learning model (such as the number of neural network layers and the weight of each neural network layer, etc.) have been determined by pre-training and stored in the storage device 120 . Alternatively, in some embodiments, the processor 130 may estimate the depth map of the two-dimensional original image by using techniques such as Structure From Motion (SFM).

In step S220, the processor 130 detects the interpupillary distance information of the user through the interpupillary distance detection device 110. Interpupillary distance information will vary from person to person, but it is mainly related to race, gender, and age. In other words, different users will have different interpupillary distances. Generally speaking, the 3D display system performs stereoscopic display according to the information of the user's interpupillary distance. For example, the processor 130 controls the hardware configuration of the 3D display 20 or performs corresponding image processing according to the user's interpupillary distance information, so as to perform stereoscopic display.

In step S230 , the processor 130 performs pixel offset processing on the two-dimensional original image according to the interpupillary distance information and the depth map to generate a reference image corresponding to the second viewing angle. The depth map generated based on the 2D original image includes a plurality of depth values, and the depth values correspond to a plurality of pixels in the 2D original image on a one-to-one basis. In order to generate the image data corresponding to the second viewing angle, the processor 130 may refer to the depth map and the user's interpupillary distance information to create a reference image corresponding to the second viewing angle, and the first viewing angle is different from the second viewing angle. More specifically, the processor 130 can determine a pixel offset (ie, aberration information) corresponding to each pixel in the two-dimensional original image according to the interpupillary distance information and each depth value in the depth map, and extract two A reference image corresponding to the second viewing angle is established by dimensioning the image data of the original image. It can be known that the deeper the depth is, the smaller the pixel offset is, and the shallower the depth is, the larger the pixel offset is.

In step S240, the processor 130 performs image restoration processing on the reference image to obtain the restored image. In detail, since the reference image includes the image content corresponding to the new second viewing angle, some scene information originally occluded in the 2D original image may be revealed in the reference image, which cannot be obtained from the corresponding scene information. obtained from the 2D original image from the first viewing angle. In addition, the image edge of the reference image corresponding to the second viewing angle also includes scene information that does not originally exist in the two-dimensional original image corresponding to the first viewing angle. Therefore, the reference image created based on the depth map and the 2D original image will include image missing blocks. In some embodiments, the processor 130 may perform image restoration processing to fill in missing image blocks (or holes) in the reference image. In some embodiments, the processor 130 may use pixel information around the missing image blocks to fill in the missing image blocks in the reference image. Alternatively, in some embodiments, the processor 130 may use a convolutional neural network model for image inpainting. For example, the processor 130 may use constant color filling, horizontal extrapolation using depth information, variational inpainting using depth information, or other related methods. Algorithms to perform image restoration processing on reference images.

In step S250, the processor 130 combines the repaired image and the 2D original image to generate a stereoscopic image conforming to the stereoscopic image format. It can be known that the stereoscopic image includes a two-dimensional original image corresponding to the first viewing angle and a restored image corresponding to the second viewing angle. In other words, the 2D original image and the repaired image can be a left-eye image and a right-eye image, respectively. Stereoscopic image formats include Side by Side (SBS) format or Top and Bottom (TB) format. Based on the foregoing, it can be seen that the restored image corresponding to the second viewing angle is generated according to the interpupillary distance information detected by the interpupillary distance detection device 110 , so the electronic device 10 of the new creative embodiment can be based on the interpupillary distance information of different users produce different stereoscopic images. In other words, the electronic device 10 according to the new creative embodiment can generate a stereoscopic image conforming to the user's real interpupillary distance according to the two-dimensional original image, thereby improving the viewing experience of the stereoscopic image.

It should also be noted that, in some embodiments, the processor 130 needs to perform other image processing on the stereoscopic image conforming to the stereoscopic image format according to the interpupillary distance information of the user, so as to generate image data suitable for playing on the 3D display 20 . For example, when the 3D display 20 is a naked-eye 3D display, it provides two images with parallax to the left eye and the right eye through the lens refraction principle or the grating technology, so that the viewer can experience the stereoscopic display Effect. Therefore, the processor 130 performs image weaving processing on the stereoscopic image, so as to stagger the pixel data of the left-eye image and the pixel data of the right-eye image to generate a single frame image suitable for playing by the naked-eye 3D display. In order to accurately provide the left eye image and the right eye image to the left eye and the right eye, respectively, the processor 130 needs to perform image weaving processing according to the user's interpupillary distance information, so as to determine how to stagger the pixel data of the left eye image and the right eye image. Pixel data of the eye image. Accordingly, since the stereoscopic image generated by the novel creative embodiment is generated according to the user's real interpupillary distance information, and the processor 130 performs image weaving processing according to the consistent interpupillary distance information, the content of the stereoscopic image can be improved. Viewing comfort and allow users to experience better stereoscopic 3D visual effects.

FIG. 3 is a flowchart of a method for generating a stereoscopic image according to an embodiment of the present invention. FIG. 4 is a schematic diagram of a method for generating a stereoscopic image according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 4 , the method of this embodiment is applicable to the electronic device 10 in the above-mentioned embodiment, and the detailed steps of this embodiment are described below in conjunction with various elements in the electronic device 10 .

In step S310, the processor 130 acquires the initial image Img_int. For example, the processor 130 can capture the desktop image of the electronic device 10 to obtain the initial image Img_int. In step S320 , the processor 130 detects the interpupillary distance information IPD_1 of the user through the interpupillary distance detecting device 110 . In step S330, the processor 130 determines whether the initial image Img_int includes the stereoscopic image Img_3D1 conforming to the stereoscopic image format. In some embodiments, the stereoscopic image Img_3D1 conforming to the stereoscopic image format is the initial image Img_int. In some embodiments, the stereoscopic image Img_3D1 conforming to the stereoscopic image format is a partial image block in the original image Img_int.

If the determination in step S330 is yes, then step S380 is continued, and the processor 130 performs image weaving processing on the stereoscopic image Img_3D1 according to the interpupillary distance information IPD_1 to obtain image data suitable for being played by the 3D display device 20 . As described above, the processor 130 will rearrange the pixel data of the left-eye image and the pixel data of the right-eye image in the stereoscopic image Img_3D1 to generate image data playable by the 3D display device 20 .

If the determination in step S330 is NO, in step S340, the processor 130 acquires the 2D original image Img_2D corresponding to the first viewing angle, and estimates the depth map d_m of the 2D original image Img_2D. The processor 130 can directly use the original image Img_int as the two-dimensional original image Img_2D, or extract some image blocks from the original image Img_int to obtain the two-dimensional original image Img_2D. The depth map d_m includes depth values corresponding to each pixel in the two-dimensional original image Img_2D, and these depth values may be within a predetermined initial range, such as 0 to 255.

In step S350, the processor 130 performs pixel offset processing on the two-dimensional original image Img_2D according to the interpupillary distance information IPD_1 and the depth map d_m to generate a reference image Img_ref corresponding to the second viewing angle. Step S350 may include sub-steps S351-S352.

In step S351, the processor 130 obtains the pixel offset according to the interpupillary distance information IPD_1 and the depth value corresponding to the first pixel in the two-dimensional original image Img_2D in the depth map d_m. Specifically, the processor 130 may determine the pixel offset of each first pixel in the two-dimensional original image Img_2D according to the interpupillary distance information IPD_1 and each depth value in the depth map d_m, respectively.

In some embodiments, the processor 130 may normalize each depth value in the depth map d_m to a predetermined value range, such as 0 to 1, to generate depth information suitable for calculating the pixel offset. That is, the processor 130 can normalize the depth value within a predetermined initial range to a predetermined value range.

In some embodiments, the pixel offset may be a multiplication result between the interpupillary distance information IPD_1 and the depth value in the depth map d_m. That is, the processor 130 can obtain the pixel offset of each first pixel in the two-dimensional original image Img_2D by multiplying each depth value in the depth map d_m by the user's interpupillary distance information IPD_1. In some embodiments, when each depth value in the depth map d_m is normalized to be between 0 and 1, the processor 130 may determine the pixel offset according to integer processing such as rounding, unconditional rounding, or unconditional rounding. quantity.

In some embodiments, the pixel offset may be the multiplication result of the interpupillary distance information IPD_1 and the output value of a function related to the depth value in the depth map d_m. In other words, the processor 130 can first input a certain depth value into a function to generate a function output value related to the depth value, and then multiply the function output value by the interpupillary distance information IPD_1 to determine the corresponding pixel offset . This function can be, for example, a first-order linear function. That is, the pixel offset can be generated by formula (1). Pixel offset = pupil distance × f(d) formula (1) Among them, f(˙) is a function that takes the depth value d as the input value. Similarly, when the pixel offset generated by the formula (1) is not an integer, the processor 130 can determine the pixel offset according to integer processing such as rounding, unconditional rounding, or unconditional carry.

In step S352, the processor 130 translates the first pixel in the two-dimensional original image Img_2D along the predetermined axis according to the pixel offset to obtain the second pixel in the reference image Img_ref. The above-mentioned preset axial direction may include a positive X-axis direction or a negative X-axis direction. That is, the processor 130 can shift the first pixel in the 2D original image Img_2D to the right according to the pixel offset to obtain the second pixel in the reference image Img_ref. In this case, the 2D original image Img_2D is used as the right eye image And create a left eye image. Alternatively, the processor 130 may shift the first pixel in the 2D original image Img_2D to the left according to the pixel offset to obtain the second pixel in the reference image Img_ref. In this case, the 2D original image Img_2D is used as the left eye image to obtain the second pixel. Create a right eye image.

In some embodiments, after the first pixel in the two-dimensional original image Img_2D is translated to obtain the second pixel, the processor 130 may determine whether the pixel coordinates of the second pixel fall within the reference image Img_ref. In response to the pixel coordinates of the second pixel not falling within the reference image Img_ref, the processor 130 may discard the second pixel. For example, assuming that the pixel coordinate of the first pixel is (0,0) and the pixel offset is Δs, if the processor 130 translates the first pixel along the negative X-axis, the pixel coordinate obtained is (-Δs,0 ) of the second pixel. Based on this, the processor 130 can determine that the second pixel does not fall within the reference image Img_ref and discard the second pixel whose pixel coordinate is (-Δs, 0).

In some embodiments, the processor 130 may translate the first pixel in the two-dimensional original image Img_2D along a predetermined axis according to the pixel offset. The processor 130 may further translate another first pixel in the two-dimensional original image Img_2D along a predetermined axis according to another pixel offset. In response to both the first pixel and the other pixel corresponding to the pixel coordinates of the second pixel, the processor 130 may select to set the first pixel as the second pixel in the reference image Img_ref. In other words, if the plurality of first pixels are shifted according to the corresponding pixel displacements and all correspond to the same pixel coordinates, the processor 130 may select one of the first pixels as the second pixel in the reference image Img_ref. In some embodiments, the processor 130 may determine the second pixel in the reference image Img_ref according to the depth value corresponding to each first pixel. Alternatively, in some embodiments, the processor 130 may determine the second pixel in the reference image Img_ref according to the calculation sequence corresponding to each first pixel.

In step S360, the processor 130 performs image restoration processing on the reference image Img_ref to obtain the restored image Img_rec. In step S370, the processor 130 combines the repaired image Img_rec with the 2D original image Img_2D to generate a stereoscopic image Img_3D2 conforming to the stereoscopic image format. FIG. 5 is a schematic diagram of an example of generating a stereoscopic image Img_3D2 conforming to a stereoscopic image format according to an embodiment of the present invention. Referring to FIG. 5 , the processor 130 may side-by-side the restored image Img_rec and the 2D original image Img_2D to obtain a stereoscopic image Img_3D2 conforming to the side-by-side format. In some embodiments, the processor 130 may also perform image scaling processing on the repaired image Img_rec and the 2D original image Img_2D before combining the two scaled images. Then, in step S380 , the processor 130 performs an image weaving process on the stereoscopic image Img_3D2 according to the interpupillary distance information IPD_1 to obtain image data suitable for being played by the stereoscopic display device 20 .

It should be noted that, in the embodiment of the present invention, the pixel offsets are generated according to the user's real interpupillary distance information, so the image between the reference image and the two-dimensional original image is established based on these pixel offsets. The difference information can match the real distance between the eyes of the user. Therefore, when the 3D display system performs other hardware configurations required for 3D display or other subsequent image processing according to the real pupil distance information, since the stereoscopic images conforming to the stereoscopic image format are also generated based on the real pupil distance information, the viewing can be greatly improved. Comfort and feel better three-dimensional effect.

To sum up, in the novel creative embodiment, the user can convert the 2D flat image into the 3D image conforming to the 3D image format, so as to enrich the 3D content that can be displayed on the 3D display. In addition, since the stereoscopic image is generated according to the user's real interpupillary distance information, it can be ensured that the interpupillary distance information used in the image weaving processing in the naked-view 3D display technology is consistent with the interpupillary distance information for generating the stereoscopic image. Accordingly, viewing comfort can be greatly improved and users can feel a better three-dimensional effect.

Although the present invention has been disclosed above with examples, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the new creation should be determined by the scope of the appended patent application.

10: Electronics 110: Interpupillary distance detection device 120: Storage Device 130: Processor 20: 3D display Img_int: initial image IPD_1: Interpupillary Distance Information Img_3D1, Img_3D2: Stereoscopic image Img_2D: 2D original image d_m: depth map Img_ref: reference image Img_rec: repaired image S210～S250, S310～S380: Steps

FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. FIG. 2 is a flowchart of a method for generating a stereoscopic image according to an embodiment of the present invention. FIG. 3 is a flowchart of a method for generating a stereoscopic image according to an embodiment of the present invention. FIG. 4 is a schematic diagram of a method for generating a stereoscopic image according to an embodiment of the present invention. FIG. 5 is an exemplary schematic diagram of generating a stereoscopic image conforming to a stereoscopic image format according to an embodiment of the present invention.

10: Electronics

110: Interpupillary distance detection device

120: Storage Device

130: Processor

20: 3D display

Claims (9)

An electronic device, comprising: an interpupillary distance detection device; a storage device that records a plurality of modules; and a processor, connected to the interpupillary distance detection device and the storage device, configured to: acquiring a two-dimensional original image corresponding to the first viewing angle, and estimating a depth map of the two-dimensional original image; detecting the interpupillary distance information of a user through the interpupillary distance detection device; performing a pixel offset process on the two-dimensional original image according to the interpupillary distance information and the depth map to generate a reference image corresponding to the second viewing angle; performing an image restoration process on the reference image to obtain a restored image; and A stereoscopic image conforming to a stereoscopic image format is generated by combining the repaired image and the two-dimensional original image, wherein the stereoscopic image includes image contents corresponding to different viewing angles. The electronic device of claim 1, wherein the processor is further configured to: perform an image weaving process on the stereoscopic image according to the interpupillary distance information to obtain image data suitable for being played by a stereoscopic display device. The electronic device of claim 2, wherein the stereoscopic image format includes a side-by-side format or a side-by-side format. The electronic device of claim 2, wherein the processor is further configured to: obtaining a pixel offset according to the interpupillary distance information and a depth value of the depth map corresponding to a first pixel in the two-dimensional original image; and A second pixel in the reference image is obtained by translationally shifting the first pixel in the two-dimensional original image along a predetermined axis according to the pixel offset. The electronic device of claim 4, wherein the pixel offset is a multiplication result between the interpupillary distance information and the depth value in the depth map. The electronic device of claim 4, wherein the pixel offset is a multiplication result of the interpupillary distance information and a function output value related to the depth value in the depth map. The electronic device of claim 5, wherein the processor is further configured to: normalize each depth value in the depth map to a predetermined value range. The electronic device of claim 4, wherein the processor is further configured to: determining whether the pixel coordinates of the second pixel fall within the reference image; and In response to the pixel coordinates of the second pixel not falling within the reference image, the second pixel is discarded. The electronic device of claim 4, wherein the processor is further configured to: translate the first pixel in the two-dimensional original image along the predetermined axis according to the pixel offset; translate another first pixel in the 2D original image along the predetermined axis according to another pixel offset; and In response to the pixel coordinates of the first pixel and the other pixel both corresponding to the second pixel, the first pixel is selected and set as the second pixel in the reference image.

Images