JP2024062935A - Method of creating solid vision display content and device of them - Google Patents

Abstract

To provide a method of creating a high quality solid vision image and a video, and provide a device of them.SOLUTION: A method of generating a solid vision display content, contains steps of: acquiring a first RGB image and a depth image from a RGB plus distance (RGB-D) image by using a processor; determining a first parallax map in accordance with the RGB-D image on the basis of a depth value in the depth image; determining a second parallax map and a third parallax map by converting the first parallax map by using a parallax distribution ratio; and generating a pair of solid vision images containing a second RGB image and a third RGB image by the processor. The second RGB image is generated by shifting a first pixel set in the first RGB image on the basis of the second parallax map, and the third RGB image is generated by shifting the set of a second pixel in the first RGB image on the basis of the third parallax map.SELECTED DRAWING: Figure 3

Description

This disclosure relates to stereoscopic vision, and in particular to generating stereoscopic display content.

Virtual reality (VR), augmented reality (AR), and mixed reality (MR), as the next generation of human-computer interaction methods, are highly immersive and intuitive. Producing high-quality stereoscopic images and videos is necessary to provide the best immersive VR, AR, and MR viewing experience.

Currently, the perception of three-dimensional depth can be achieved by using two or more cameras to generate two slightly different images for each eye. However, this can be a complex and computationally intensive process. Furthermore, without accurate depth information, the generated VR, AR, and MR environments cannot provide people with a good viewing experience.

This specification discloses implementations of methods, devices, and systems for generating stereoscopic display content.

In one aspect, a method of generating stereoscopic display content is disclosed. The method includes using a processor to obtain a first red-green-blue (RGB) image and a depth image from a red-green-blue-plus-distance (RGB-D) image; determining a first disparity map according to the RGB-D image based on depth values in the depth image, the first disparity map including a plurality of disparity values for the first RGB image that are converted into a pair of stereoscopic images; determining a second disparity map and a third disparity map by transforming the first disparity map using a disparity distribution ratio; and generating, by the processor, a pair of stereoscopic images including the second RGB image and the third RGB image, the second RGB image being generated by shifting a first set of pixels in the first RGB image based on the second disparity map, and the third RGB image being generated by shifting a second set of pixels in the first RGB image based on the third disparity map.

In another aspect, an apparatus for generating stereoscopic display content is disclosed. The apparatus includes a non-transitory memory and a processor, the non-transitory memory including instructions executable by the processor to obtain a first red-green-blue (RGB) image and a depth image from a red-green-blue-plus-distance (RGB-D) image; determine a first disparity map according to the RGB-D image based on depth values in the depth image, the first disparity map including a plurality of disparity values for the first RGB image that is transformed into a pair of stereoscopic images; determine a second disparity map and a third disparity map by transforming the first disparity map using a disparity distribution ratio; and generate a pair of stereoscopic images including the second RGB image and the third RGB image, the second RGB image being generated by shifting a first set of pixels in the first RGB image based on the second disparity map, and the third RGB image being generated by shifting a second set of pixels in the first RGB image based on the third disparity map.

In another aspect, a non-transitory computer-readable storage medium configured to store a computer program for generating stereoscopic display content is disclosed. The computer program includes instructions executable by a processor to obtain a first red-green-blue (RGB) image and a depth image from a red-green-blue-plus-distance (RGB-D) image; determine a first disparity map according to the RGB-D image based on depth values in the depth image, the first disparity map including a plurality of disparity values for the first RGB image that are converted into a pair of stereoscopic images; determine a second disparity map and a third disparity map by transforming the first disparity map using a disparity distribution ratio; and generate, by the processor, a pair of stereoscopic images including the second RGB image and the third RGB image, the second RGB image being generated by shifting a first set of pixels in the first RGB image based on the second disparity map, and the third RGB image being generated by shifting a second set of pixels in the first RGB image based on the third disparity map.

The present disclosure is best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to common practice, the various features of the drawings are not drawn to scale. Conversely, dimensions of various features have been arbitrarily expanded or reduced for clarity.

FIG. 1 is an exemplary block diagram of an apparatus for computing and communications.

FIG. 1 is an exemplary diagram for explaining the principle of binocular stereoscopic vision.

1 is a flowchart of an example process for generating stereoscopic display content according to some implementations of the present disclosure.

1 is an example for determining disparity values for a left and right human eye, according to some implementations of the present disclosure.

FIG. 5 is an example flow diagram for generating a pair of stereoscopic images according to some implementations of the present disclosure.

DETAILED DESCRIPTION Virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies have been developed in several application areas, such as virtual tourism and travel, digital virtual entertainment (e.g., VR games and VR movies), virtual training and education, VR exposure therapy, etc. Meanwhile, VR/AR/MR devices, such as VR headsets, VR helmets, AR/MR apps and glasses, are used to simulate 3D immersive environments in which people can participate. When a user wearing a VR/AR/MR headset moves his/her head, the simulated 3D environment follows the user's movements and is displayed in front of the user.

A simulated 3D immersive environment can be achieved by binocular vision. The left and right eyes of a human see objects from slightly different perspectives. The various observed two-dimensional (2D) images are processed by the brain to generate the perception of 3D depth. Based on binocular vision, VR/AR/MR stereoscopic vision is generated by using two 2D images (e.g., one image for the left eye and one image for the right eye) as inputs for the left and right eyes, respectively. The two 2D images are acquired from different perspectives by two cameras for the same scene. Traditionally, stereoscopic image pairs (e.g., one image for the left eye and one image for the right eye) used in virtual reality (VR)/augmented reality (AR)/mixed reality (MR) helmets/glasses are generated using an inverse rectification process. Because the 2D images do not contain distance/depth information, the 3D VR/AR/MR display content generated by such processing may cause discomfort and 3D dizziness due to inaccurate distance estimation.

According to an implementation of the present disclosure, a method is used to generate 3D display content for VR/AR/MR using three-dimensional red-green-blue plus distance (RGB-D) images with accurate distance/depth information recorded from an RGB-D sensor. The RGB-D sensor can include, for example, a structured light-based RGB-D sensor, an active/passive stereo vision-based RGB-D sensor, a time-of-flight RGB-D sensor, or any combination thereof. A conventional red-green-blue (RGB) image is a function of x and y coordinates and represents only the distribution of RGB color values in a 2D image. For example, a pixel at (x,y) coordinates with display colors red=1, green=1, and blue=1 can be represented as Pixel(x,y)=(1,1,1), which displays a black pixel at x and y coordinates on the image. The RGB-D image recorded from the RGB-D sensor provides additional depth information for each pixel of the RGB image. For example, a pixel at (x,y,z) coordinates with display colors Red=1, Green=1, Blue=1 can be represented as Pixel(x,y)=(1,1,1,z), which displays a black pixel at the x and y coordinates on the image, one z unit distance away (e.g., millimeters).

According to an implementation of the present disclosure, an RGB-D image can be generated using an RGB-D sensor to generate stereoscopic display content. Based on the RGB-D image, a corresponding RGB image and a depth image can be obtained. The depth image indicates distance information of an object corresponding to a pixel in the RGB image. Based on the triangulation relationship, a global disparity map can be generated for the RGB image using the distance, focal length, and interpupillary distance of each pixel in the RGB image. The global disparity map is a 2D matrix, and each element indicates the disparity value of a pixel in the RGB image. The left disparity map can be determined by the disparity distribution ratio k and the global disparity map. The right disparity map can be determined by the disparity distribution ratio k and the global disparity map. Thus, a pair of stereoscopic images can be generated from the RGB image based on the left disparity map and the right disparity map. The pair of stereoscopic images includes a left eye image and a right eye image. The left and right eye images can be zoomed, cropped, or resized to generate left and right view images according to the display requirements of an augmented reality (AR), virtual reality (VR), or mixed reality (MR) device.

It should be noted that the application and implementation of the present disclosure is not limited to the examples, and alternatives, variations, or modifications of the implementation of the present disclosure can be achieved for any computing environment. Details of the disclosed method, apparatus, and system are described below after an overview of the system and coding structure. Details of the disclosed method and server are described below.

1 is an exemplary block diagram illustrating internal components of an apparatus 100 for computing and communication according to an implementation of the present disclosure. As shown in FIG. 1, the apparatus 100 for computing and communication can include a memory 104, a processor 106, a communication unit 108, an input/output (I/O) component 110, a sensor 112, a power supply 114, and a bus 102. The bus 102 can be used to distribute internal signals. The bus 102 represents what may be one or more buses (such as an address bus, a data bus, or a combination thereof). The apparatus may be implemented by any configuration of one or more computing devices, such as a Red Green Blue Plus Distance (RGB-D) camera, a bridge camera, a film camera, a smartphone camera, a fisheye camera, a microcomputer, a mainframe computer, a general purpose computer, a database computer, a special purpose/dedicated computer, a remote server computer, a personal computer, a tablet computer, a laptop computer, a mobile phone, an embedded computing/edge computing device, a single board computer, an ASIC (Application Specific Integrated Circuit) chip, an FPGA (Field Programmable Gate Array) chip, a SoC (System on Chip) chip, a cloud computing device/service, or a wearable computing device. In some implementations, the different apparatuses may be implemented in the form of multiple groups of RGB-D cameras that are in different geographic locations and can communicate with each other via a network or the like. In some implementations, the different apparatuses are configured for different operations. In some implementations, the apparatus for computing and communication may perform one or more aspects of the methods and systems described herein. For example, a special purpose processor in an RGB-D camera that includes a specialized chip may be used to implement one or more aspects or elements of the methods and systems described herein.

1 shows that the device 100 for computing and communication includes a memory 104, a processor 106, a communication unit 108, an input/output (I/O) component 110, a sensor 112, a power supply 114, and a bus 102. In some implementations, the device 100 for computing and communication can include any number of memory units, processor units, communication units, input/output (I/O) components, sensor units, power supply units, and bus units.

Memory 104 includes, but is not limited to, non-transitory computer-readable media for long-term storage of program code and/or data, such as secondary or permanent long-term storage. Memory 104 may retrieve data, store data, or both. Memory 104 here may be a read-only memory (ROM) device, a hard drive, a random access memory (RAM), a flash drive, a solid-state drive (SSD), an embedded multimedia card (EMMC), an optical/magnetic disk, a security digital (SD) card, or any combination of suitable types of storage devices.

The processor 106 can be used to manipulate or process information that can be received from the memory 104, the communication unit 108, the I/O components 110, the sensors 112, or a combination thereof. In some implementations, the processor 106 can include a digital signal processor (DSP), a central processor (e.g., a central processing unit or CPU), an application specific instruction set processor (ASIP), an embedded computing/edge computing device, a single board computer, an ASIC (application specific integrated circuit) chip, an FPGA chip (field programmable gate array), a SoC (system on chip) chip, a cloud computing service, a graphics processor (a graphics processing unit of a GPU). The processor 106 can access computer instructions stored in the memory 104 via the bus 102. In some implementations, one or more processors can be used to speed up data processing, including executing or processing computer instructions to perform one or more aspects of the methods and systems described herein. Output data from the processor 106 can be distributed to the memory 104, the communication unit 108, the I/O components 110, and the sensors 112 via the bus 102. The processor 106 may be any type of device or devices operable to control the apparatus 100 for computing and communication to perform one or more configured or embedded operations.

In addition to the processor 106 and memory 104, the device 100 may include a sensor 112. For example, one or more conditions of the operating environment of the device 100 may be detected, captured, or determined by the sensor 112. In some implementations, the sensor 112 may include one or more charge-coupled devices (CCDs), active pixel sensors (CMOS sensors), or other visible or non-visible light detection and capture units. Data captured about sensed aspects of the operating environment of the device 100 for computing and communication may be transmitted from the sensor 112 to the memory 104, the processor 106, the communication unit 108, the input/output (I/O) components 110, the power supply 114, and the bus 102. In some implementations, multiple sensors may be included in the device 100, such as, for example, a lidar unit, a microphone, an RGB-D sensing device, an ultrasonic unit, or a pressure sensor. The above-mentioned sensors may capture, detect, or determine one or more conditions of the operating environment of the device 100 for computing and communication.

In addition to the processor 106 and memory 104, the device 100 may include an I/O component 110. The I/O component 110 may receive user input. The I/O component 110 may transmit the user input to the bus 102, the power supply 114, the memory 104, the communication unit 108, the sensor 112, the processor 106, or a combination thereof. The I/O component 110 may provide a visual or display output to an individual. In some implementations, the I/O component 110 may be formed from a communication device for transmitting signals and/or data.

In addition to the processor 106 and the memory 104, the device 100 may include a communication unit 108. Using the communication unit 108, the device 100 may communicate with another device using wired or wireless communication protocols over one or more communication networks, such as a cellular data network, a wide area network (WAN), a virtual private network (VPN), or the Internet.

In addition to the processor 106 and memory 104, the device 100 may include a power supply 114. The power supply 114 may provide power to other components in the device 100, such as the bus 102, the memory 104, and the memory 104. In some implementations, the power supply 114 may be a battery, such as a rechargeable battery. In some implementations, the power supply 114 may include a power input connection capable of receiving energy from an external power source.

In addition to the processor 106 and memory 104, the device 100 may include a bus 102. Power signals and internal data signals from a power supply 114 may be distributed via the bus 102 to the memory 104, the communication unit 108, the sensors 112, the processor 106, the I/O components 110, and the power supply 114.

It should be noted that the portions or components of the apparatus and systems for generating stereoscopic display content may include elements not limited to those shown in FIG. 1. Without departing from the scope of this disclosure, the apparatus and systems for generating stereoscopic display content may include more or fewer parts, components, and hardware or software modules for performing various functions in addition to or related to generating stereoscopic display content.

2 shows an exemplary diagram 200 for explaining the binocular stereoscopic principle. The diagram 200 includes a left image 230, a right image 240, a left optical center O'(0,0), a right optical center O''(0,0), a left focal point L=( _XL , _YL , _ZL ), a right focal point R=( _XR , _YR , _ZR ), and a target point P=( _XC , _YC , _ZC ). The left optical center O' is a pixel point at the center of the left image 230. The right optical center O'' is another pixel point at the center of the right image 240. The pixel coordinate for the left optical center O' is (0,0) in the left image 230. The pixel coordinate for the right optical center O'' is (0,0) in the right image 240. A target point P as a world coordinate point (e.g., a 3D point) can be transformed and projected as a 2D coordinate point P′=(X _left ,Y) in the left image 230 through a left focal point L. A target point P can be transformed and projected as another 2D coordinate point P″=(X _right ,Y) in the right image 240 through a right focal point R. The distance between the left focal point L and the right focal point R is the baseline b.

The 2D coordinate point P′ and the 2D coordinate point P″ are two projected points in the left image 230 and the right image 240, respectively, for the same target point P. The difference in horizontal coordinates of P′ and P″ in the left image 230 and the right image 240 (e.g., disparity: d= _Xleft − _Xright ) can be used to estimate the distance between the target point P and two focal points (e.g., the left focal point L and the right focal point R). In some implementations, the target point P is a 3D world coordinate point in a 3D object. Each 3D world coordinate point in the 3D object can be projected in both the left image 230 and the right image 240. Corresponding pixels of the 3D object can be found and matched between the left image 230 and the right image 240. The disparity of each pixel (e.g., disparity for the target point P: d= _Xleft − _Xright ) can be calculated, and a disparity map for the 3D object can be generated based on the calculated disparity. The disparity map can be used to reconstruct the 3D object in the world coordinate system.

In some implementations, the human's left eye can be the left focal point L. The human's right eye can be the right focal point R. The human's left and right eyes have slightly different views of the surrounding world. In that case, the baseline b is the interpupillary distance between the left and right eyes (e.g., 50-75 mm). The target point P can be any world coordinate point observed by the human. The target point P can be projected onto both the human's left and right eye images. The disparity of corresponding pixels between the left and right eye images can be used to calculate the distance between the target point P and the human. The left and right eye images can then be used by the human's brain as a pair of stereoscopic images to generate a stereoscopic view of the surrounding world.

In some implementations, two cameras (e.g., left and right cameras) at different positions can generate left image 230 and right image 240 that contain different 2D pixels for the same 3D object. The focus of the left camera can be a left focus L. The focus of the right camera can be a right focus R. The distance between the two focuses of the left and right cameras can be a baseline b. In some cases, if the left and right cameras are not positioned horizontally, the left and right images 230 and 240 can be calibrated to correctly show the disparity maps for all pixels in both the left and right images 230 and 240. The disparity maps for the left and right images 230 and 240 can be used to generate depth information for each pixel to reconstruct the 3D environment captured by the left and right cameras.

In some implementations, a stereo camera with two or more image sensors can be used to generate left image 230 and right image 240 that contain different 2D pixels for the same 3D object. For example, if the stereo camera includes two image sensors (e.g., a left image sensor and a right image sensor), the stereo camera can be used to reconstruct a 3D object with depth information. The left image sensor can be used to generate the left image 230. The right image sensor can be used to generate the right image 240. The horizontal distance between the left image sensor and the right image sensor can be the baseline b. A disparity map can be calculated based on the left image 230 and the right image 240 that represent slightly different views of the surrounding world.

In general, binocular stereoscopic vision is realized based on the principle of parallax (e.g., disparity). For example, in FIG. 2, two images (e.g., left image 230 and right image 240) are aligned in a row, which means that the left image 230 and the right image 240 are in the same plane. A target point P can be projected in the left image 230 and the right image 240 with different pixel coordinates, respectively. The difference in pixel coordinates (e.g., disparity: d=X _left -X _right ) can be used to calculate the distance between the target point P and the two images (e.g., left image 230 and right image 240). The calculated distance information can be used to reconstruct 3D objects in the world.

3 is a flow chart of an exemplary process 300 for generating stereoscopic display content according to some implementations of the present disclosure. The process 300 can be implemented as a software and/or hardware module in the device 100 of FIG. 1. For example, the process 300 can be implemented as a software module stored in the memory 104 as instructions and/or data executable by the processor 106 of a camera such as the device 100 of FIG. 1. In another example, the process 300 can be implemented in hardware as a specialized chip that stores instructions executable by the specialized chip. Some or all of the operations of the process 300 can be implemented using a disparity map as described below in connection with FIG. 4. As mentioned above, all or some of the aspects of the disclosure described herein can be implemented using a general-purpose computer/processor with a computer program that, when executed, executes any of the respective techniques, algorithms, and/or instructions described herein. Additionally or alternatively, a special-purpose computer/processor can be utilized that may include, for example, specialized hardware for executing any of the techniques, algorithms, or instructions described herein.

In operation 302, a first red-green-blue (RGB) image and a depth image can be obtained from the red-green-blue plus distance (RGB-D) image using a processor. For example, the processor can be the processor 106 of FIG. 1. In some cases, the sensor 112 of the device 100 of FIG. 1 can be used to obtain an RGB-D image in the operating environment of the device 100. The RGB-D image can be transmitted to the processor 106 via the bus 102 to obtain an RGB image and a depth image. The depth image indicates distance information of a corresponding object (or multiple corresponding objects) in the RGB image.

Using FIG. 5 as an example, the RGB-D image can be acquired by an RGB-D sensor 502. The RGB-D image can be processed by any technique to obtain an RGB image 512 and a depth image 514. In some implementations, the RGB-D image can be captured by an RGB-D sensor. For example, the RGB-D sensor can be the sensor 112 of FIG. 1. The RGB image 512 can include various objects, such as, for example, humans, animals, sofas, desks, and other objects. In the depth image 514, different shading is used in FIG. 5 to indicate different distances, with darker shading indicating closer distances. The depth image 514 indicates the distance of the corresponding object in the RGB image 512.

In some implementations, a pixel in the depth image indicates the distance between the RGB-D sensor and a corresponding object captured in the RGB-D image. For example, a pixel in the RGB-D image can correspond to a pixel in the depth image. The pixel in the RGB-D image indicates a point that belongs to an object. A corresponding pixel at the same location in the depth image can indicate the distance between the corresponding object and the RGB-D sensor.

5, pixels in depth image 514 indicate the distance between RGB-D sensor 502 and a corresponding object captured in RGB image 512. The corresponding object may include, for example, object 516 (e.g., a toy bear). Each pixel in RGB image 512 may be associated with an object (e.g., object 516). The corresponding pixel in depth image 514 for each pixel in RGB image 512 indicates the distance between RGB-D sensor 502 and the corresponding object.

Returning to FIG. 3, in operation 304, a first disparity map based on the RGB-D image can be determined based on depth values in the depth image, the first disparity map including a plurality of disparity values for the first RGB image that are converted into a pair of stereoscopic images. In some cases, the first disparity map includes a plurality of disparity values for the first RGB image, and the disparity values in the first RGB image can be used to generate a pair of stereoscopic images.

The disparity value of each pixel can be determined based on the depth value in the depth image using FIG. 4 as an example. FIG. 4 is a diagram illustrating an example of determining disparity values for the left and right eyes of a human according to some implementations of the present disclosure. For example, in FIG. 4, the distance of the target point O is distance Z, and the disparity value for the target point O is f*b/Z, where f is the focal length, b is the interpupillary distance between the left eye _E1 and the right eye _E2 , and Z is the distance between the target point O and the RGB-D sensor. From the triangulation relationship in FIG. 4, for each pixel in the first RGB image, a corresponding disparity value can be determined (e.g., f*b/Z). In general, based on the triangulation relationship, the depth value, focal length, and interpupillary distance of each pixel in the depth image can be used to determine the disparity value of each pixel in the first RGB image (e.g., RGB image). According to FIG. 4, the disparity value in the first disparity map can be determined, for example, using Equation (5) described below.

In the example of FIG. 5, an RGB-D image can be acquired by an RGB-D sensor 502. A pixel in a depth image 514 indicates a distance (i.e., depth) between a corresponding object in the RGB image 512 and the RGB-D sensor. For example, the distance for an object 516 in the RGB image 512 is displayed in the depth image 514. Based on the depth of each pixel in the depth image 514, a global disparity map 522 for the RGB image 512 can be determined. In some implementations, the global disparity map 522 can be determined using the interpupillary distance between the left and right eyes, the depth value of each pixel, and the focal length of the RGB-D sensor. For example, the global disparity map 522 can be determined using equation (5), as described below. For example, the object 516 is shown in the global disparity map 522 of FIG. 5 as a disparity value represented in grayscale. The global disparity map 522 can then be used to convert the RGB image into a pair of stereoscopic images (e.g., left-eye image 542 and right-eye image 544), as described below.

In some implementations, the first disparity map is a two-dimensional (2D) matrix, with each element indicating a disparity value. Using FIG. 5 as an example, the first disparity map (e.g., global disparity map 522) can be determined based on the depth image 514 and the RGB image 512. The global disparity map 522 can be a 2D matrix, with each element indicating a disparity value for a pixel in the RGB image 512.

In some implementations, the first disparity map can be determined using at least one of the focal length f or the interpupillary distance b. Using FIG. 4 as an example, a disparity value (e.g., f*(b/(z(x,y)))) in the first disparity map for the target point O can be determined based on the focal length f, the interpupillary distance b between the left eye _E1 and the right eye _E2, and the distance Z. For example, the disparity value can be determined using equation (5) described below.

In the example of FIG. 5, pixels in the RGB image 512 are associated with distances in the depth image 514. The focal length f or interpupillary distance b can be predefined from public data or set by manual input. The focal length f and interpupillary distance b with distance can be used to determine a global disparity map 522 for the RGB image 512.

ただし、ｄ_Ｌ（ｘ，ｙ）は第２のパリティマップ内の視差値であり、ｄ_Ｒ（ｘ，ｙ）は第３パリティマップ内の視差値である。ｄ（ｘ，ｙ）は第１のパリティマップの視差値であり、ｚ（ｘ，ｙ）はＲＧＢ－ＤセンサとＲＧＢ画像内のピクセル（ｘ，ｙ）に関連付けられた対応する物体との間の距離を示し、ｋは視差分配比である。ただし、視差分配比は、左目と右目との間の観察点の位置を示す定数値であり得る。いくつかの実装形態では、視差分配比ｋは、事前に設定された一定値であり得る。 Returning to FIG. 3 , in operation 306, a second disparity map and a third disparity map can be determined by transforming the first disparity map using a disparity distribution ratio. In other words, the second disparity map and the third disparity map can be determined based on the same original disparity map using a disparity distribution ratio. In some implementations, the first disparity map can be transformed into the second disparity map, for example using equation (1), and then transformed into the third disparity map based on the disparity distribution ratio k, for example using equation (2) below.

where d _L (x,y) is the disparity value in the second parity map, d _R (x,y) is the disparity value in the third parity map, d(x,y) is the disparity value of the first parity map, z(x,y) indicates the distance between the RGB-D sensor and the corresponding object associated with pixel (x,y) in the RGB image, and k is the disparity distribution ratio, where the disparity distribution ratio may be a constant value indicating the position of the observation point between the left eye and the right eye. In some implementations, the disparity distribution ratio k may be a preset constant value.

In some implementations, the second and third disparity maps can be determined from the first disparity map in other ways without using equations (1) and (2). For example, the second and third disparity maps can be determined using an offset in addition to the disparity distribution ratio k.

The disparity value d(x,y) for the first parity map can be determined, for example, using equation (5) below, where f is the focal length and b is the interpupillary distance between the left and right eyes.

4 as an example, the disparity value d(x,y) in the first parity map for the target point O can be determined based on the focal length f (e.g., f= _f1 = _f2 ), the interpupillary distance b, and the distance Z. Based on the disparity distribution ratio k, the disparity value _dL (x,y) in the second parity map and the disparity value _dR (x,y) in the third parity map for the target point O can be determined as described above using equations (1) and (2).

In the example of FIG. 5, a global disparity map 522 can be determined based on the RGB image 512 and the depth image 514. Based on the disparity distribution ratio k, a left disparity map 534 and a right disparity map 536 can be determined using equations (3) and (4), respectively, described below. The left disparity map 534 and the right disparity map 536 can be used to convert the RGB image into a pair of stereoscopic images.

Returning to FIG. 3, at operation 308, a pair of stereoscopic images including a second RGB image and a third RGB image may be generated by a processor. The second RGB image is generated by shifting a first set of pixels in the first RGB image based on the second disparity map. The third RGB image is generated by shifting a second set of pixels in the first RGB image based on the third disparity map.

The disparity values in the second and third disparity maps can be used to shift pixels in the first RGB image horizontally to the left or right to generate the second and third RGB images. In some implementations, a processor (e.g., processor 106) can generate the second RGB image (e.g., right disparity map 536 of FIG. 5 ) by shifting a first set of pixels in the first RGB image (e.g., RGB image 532 of FIG. 5 ) based on the second disparity map (e.g., left disparity map 534 of FIG. 5 ) using equation (3). The processor can generate the third RGB image (e.g., right-eye image 544 of FIG. 5 ) by shifting a second set of pixels in the first RGB image (e.g., RGB image 532 of FIG. 5 ) based on the third disparity map (e.g., right disparity map 536 of FIG. 5 ) using equation (4).

In equations (3) and (4), Pixel _L (x,y) is pixel (x,y) in the second RGB image, Pixel _R (x,y) is pixel (x,y) in the third RGB image, Pixel(x,y) is pixel (x,y) in the first RGB image, (R(x,y),G(x,y),B(x,y)) is the RGB color for pixel (x,y), _dL referring to _dL (x,y) in equation (1) indicates the disparity value in the second disparity map, and _dR referring to _dR (x,y) indicates the disparity value in the third disparity map.

In some implementations, the disparity values in the second and third disparity maps can be determined in other ways without using equations (3) and (4). In some implementations, in addition to the horizontal shift described above, for example, an additional pixel or additional pixels can be added to the top or bottom to determine the disparity values. In some implementations, in addition to the horizontal shift, additional pixel(s) can be added to the left or right.

5 as an example, RGB image 532 may be a first RGB image. Left disparity map 534 may be a second disparity map. Right disparity map 536 may be a third disparity map. Left disparity map 534 and right disparity map 536 may be determined by transforming global disparity map 522 based on disparity distribution ratio k, as described above. Left eye image 542 may be generated by transforming a first set of pixels in RGB image 532 based on left disparity map 534. For example, equation (3) may be used with left disparity map 534 to generate left eye image 542. Equation (4) may be used with right disparity map 536 to generate right eye image 544. Left eye image 542 and right eye image 544 may be a pair of stereoscopic images.

In some implementations, a pair of adjusted display images resized for the display requirements of an augmented reality (AR), virtual reality (VR), or mixed reality (MR) device can be generated by a processor (e.g., processor 106) based on the pair of stereoscopic images. Using FIG. 5 as an example, the pair of stereoscopic images includes a left eye image 542 and a right eye image 544. The pair of adjusted display images resized for the display requirements of an augmented reality (AR), virtual reality (VR), or mixed reality (MR) device can include, for example, a left display image 552 and a right display image 554, which can be generated based on the left eye image 542 and the right eye image 544.

4 is a diagram of an example disparity calculation 400 for the left and right eyes of a human being according to some implementations of the present disclosure. FIG. 4 may include a left eye E ₁ , a right eye E ₂ , a target point O, an interpupillary distance b between the left eye E ₁ and the right eye E ₂ , a distance Z between the target point O and an RGB sensor, a focal length f ₁ for the left eye E ₁ , a focal length f ₂ for the right eye E ₂ , a projection point O ₁ ' of the target point O in the image plane of the left eye E ₁ , a projection point O ₂ ' of the target point O in the image plane of the right eye E ₂ , an origin C ₁ ' in the image plane of the left eye E ₁ , and an origin C ₂ ' in the image plane of the right eye E ₂ . Without loss of generality, the focal length f ₁ of the left eye is equal to the focal length f ₂ of the right eye, and both f ₁ and f ₂ are equal to f.

The left eye _E1 and the right eye _E2 of a human being are separated by an interpupillary distance b in the horizontal direction. This allows the target point O to be projected to different positions (e.g., projection points _O1 ' and _O2 ') in the image of the left eye _E1 and the image of the right eye _E2 , respectively. The projection point _O1 ' is projected to the left of the origin _C1 ' in the image plane of the left eye _E1 . The pixel distance between the projection point _O1 ' and the origin _C1 ' in the image plane of the left eye _E1 is U1. The projection point _O2 ' is projected to the right of the origin _C2 ' in the image plane of the right eye _E2 . The pixel distance between the projection point _O2 ' and the origin _C2 ' in the image plane of the right eye _E2 is U2. The difference in pixel positions is the disparity value of the target point O. Every pixel in the image plane of the left eye _E1 can be matched with a pixel at the same position in the image plane of the right eye _E2 . A disparity map can be generated based on the difference in pixel positions between the left eye _E1 image plane and the right eye _E2 image plane.

In some implementations, each pixel in the depth image indicates the distance between the RGD sensor and the corresponding object. For example, in FIG. 4, the distance for the target point O is distance Z. The pixel distance difference between the projection points O ₁ ' and O ₂ ' is |U1|+|U2|. From the triangulation relationship in FIG. 4, |U1|+|U2| is equal to (b*f)/Z, where b is the interpupillary distance between the left eye E ₁ and the right eye E ₂ , f is the focal length for the left eye E ₁ and the right eye E ₂ , and Z is the distance between the target point O and the RGB sensor. Thus, b/Z*f is the disparity value for the target point O. The disparity value for each pixel in the RGB image can be determined using the triangulation relationship utilizing the depth value, focal length, and interpupillary distance for each pixel in the depth image. A disparity map can be obtained for all pixels in the image planes of the left eye E ₁ and the right eye E ₂ , for example, using the following equation:

In equation (5), z(x,y) denotes the distance between the RGB-D sensor and the corresponding object associated with pixel (x,y) in the RGB image. z(x,y) can be obtained from a depth image generated by the RGB-D sensor. f (e.g., f= _f1 = _f2 ) in equation (5) is the focal length for the left eye _E1 and the right eye _E2 . d(x,y) denotes each element of the disparity map. In some implementations, the calculation of the disparity map according to FIG. 3 can be performed in operation 304.

5 is an example workflow for generating a pair of stereoscopic images according to some implementations of the present disclosure. An RGB-D image can be acquired using one or more RGB-D sensors (e.g., RGB-D sensor 502). An RGB image 512 and a depth image 514 can be acquired from the acquired RGB-D image. The depth image 514 shows the distance of a corresponding object in the RGB image 512. For example, an object 516 is displayed in the RGB image 512, and the distance to the object 516 is shown in the depth image 514. In some implementations, for example, according to FIG. 3, acquiring the RGB-D image can be performed in operation 302.

The global disparity map 522 can be determined for the RGB image 512 based on, for example, the distance in the depth image 514. The disparity values in the global disparity map 522 for the RGB image 512 can be calculated based on the distance in the depth image 514, the focal length, and the interpupillary distance (e.g., focal length f= _f1 = _f2 , interpupillary distance b in FIG. 4). The disparity values of the global disparity map 522 for the RGB image 512 can be calculated using, for example, Equation (5) having a triangulation relationship based on the distance in the depth image 514, the focal length, and the interpupillary distance. For example, some pixels for the object 516 in the global disparity map 522 indicate a disparity value for the object 516. In some implementations, for example according to FIG. 3, the determination of the global disparity map 522 can be performed in operation 304.

The left disparity map 534 can be determined based on the disparity distribution k by transforming the global disparity map 522. The right disparity map 536 can be determined based on the disparity distribution k by transforming the global disparity map 522. Based on the disparity distribution k, the disparity values in the global disparity map 522 can be assigned to the left disparity map 534 and the right disparity map 536 of a portion. For example, the left disparity map 534 and the right disparity map 536 can be determined using the disparity distribution k. As mentioned above, equations (1) and (2) can be used to determine the disparity maps. In some implementations, for example according to FIG. 3, the determination of the left disparity map 534 and the right disparity map 536 can be performed in operation 306.

A pair of stereoscopic images can be generated based on the left disparity map 534 and the right disparity map 536. The left eye image 542 can be generated based on the left disparity map 534 by transforming a set of pixels in the RGB image 532 (e.g., the RGB image 512). The right eye image 544 can be generated based on the right disparity map 536 by transforming another set of pixels in the RGB image 532 (e.g., the RGB image 512). The left eye image 542 and the right eye image 544 are a pair of stereoscopic images. The left eye image 542 can be generated using equation (3) to horizontally shift a set of pixels in the RGB image 532. The right eye image 544 can be generated using equation (4) to horizontally shift a set of pixels in the RGB image 532. In some implementations, for example according to FIG. 3, the generation of the pair of stereoscopic images can be performed in operation 308.

The left-eye image 542 and the right-eye image can be zoomed and cropped to generate left display image 552 and right display image 554 that meet the display requirements of an augmented reality (AR), virtual reality (VR), or mixed reality (MR) device.

Aspects of the disclosure described herein can be described in terms of functional block components and various processing operations. The disclosed processes and sequences can be performed alone or in any combination. The functional blocks can be realized by any number of hardware and/or software components performing the specified functions. For example, the described aspects can use various integrated circuit components, such as, for example, memory elements, processing elements, logic elements, look-up tables, etc., capable of performing various functions under the control of one or more microprocessors or other control devices. Similarly, where elements of the described aspects are implemented using software programming or software elements, the disclosure can be implemented using various algorithms implemented using any combination of data structures, objects, processes, routines, or other programming elements using any programming or scripting language, such as C, C++, Java, Assembler, etc. Functional aspects can be implemented in algorithms executed on one or more processors. Additionally, aspects of the disclosure can use any number of conventional techniques for electronic configuration, signal processing and/or control, data processing, etc. The terms "mechanism" and "element" are used broadly and are not limited to mechanical or physical implementations or aspects, but can also include software routines that interface with processors or the like.

Implementations or parts of implementations of the above disclosure may take the form of a computer program product accessible, for example, from a computer usable or computer readable medium. The computer usable or computer readable medium may be, for example, any device that can tangibly contain, store, communicate, or transport a program or data structure for use by or in association with any processor. The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media may also be used. Such computer usable or computer readable media may be referred to as non-transitory memory or media, and may include RAM or other volatile memory or storage that may change over time. The memory of the devices described herein need not be physically contained in the device, unless otherwise specified, but may be memory that can be remotely accessed by the device and need not be contiguous with other memories that may be physically contained within the device.

Any of the individual or combination of functions described herein as being performed as examples of the present disclosure may be implemented using machine-readable instructions in the form of code to operate any or any combination of the aforementioned hardware. The computational code may execute the individual or combination of functions as a computational tool. It may be implemented in the form of one or more modules, with input and output data of each module being passed to and from one or more further modules during operation of the methods and systems described herein.

Information, data, and signals may be represented using a variety of different technologies and techniques. For example, any data, instructions, commands, information, signals, bits, symbols, and chips referred to herein may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, other items, or combinations of the foregoing.

The term "example" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as an "example" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" is intended to illustrate a concept. Moreover, use of the terms "an embodiment" or "one embodiment" throughout this disclosure is not intended to refer to the same embodiment or implementation unless so described.

The term "or" as used in this disclosure is intended to mean an inclusive "or" rather than an exclusive "or" for the two or more elements it binds. That is, unless otherwise specified or unless the context clearly dictates otherwise, "X includes A or B" is intended to mean any of its natural inclusive permutations. In other words, if X includes A, if X includes B, or if X includes both A and B, then "X includes A or B" is satisfied in any of the foregoing cases. Similarly, "X includes any one of A and B" is intended to be used in the same sense as "X includes A or B". The term "and/or" as used in this disclosure is intended to mean "and" or an inclusive "or". That is, unless otherwise specified or unless the context clearly dictates otherwise, "X includes A, B, and/or C" is intended to mean that X may include any combination of A, B, and C. In other words, if X includes A, if X includes B, if X includes C, if X includes both A and B, if X includes both B and C, if X includes both A and C, or if X includes all of A, B, and C, then "X includes A, B, and/or C" is satisfied in any of the foregoing cases. Similarly, "X includes at least one of A, B, and C" is intended to be used equivalently to "X includes A, B, and/or C."

The use of the terms "including" or "having" and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof, as well as additional items. Depending on the context, the word "if" as used herein can be interpreted as "at the time," "during," or "depending on."

The use of the terms "a" and "an" and "the" and similar referents in the context of describing this disclosure (especially in the context of the claims) should be interpreted to include both the singular and the plural. Furthermore, unless otherwise stated herein, the recitation of a range of values herein is intended only to serve as a shorthand method of individually referring to each individual value within the range, and each individual value is incorporated into the specification as if it were individually set forth herein. Finally, the operations of any method described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. Any examples provided herein, or the use of language indicating that an example is described (e.g., "etc.") are intended solely for the purpose of providing a better understanding of the disclosure, and do not impose limitations on the scope of the disclosure, unless otherwise specified.

This specification has been described under various headings and subheadings. These have been included to enhance readability and to facilitate the process of locating and referencing material within the specification. These headings and subheadings are not intended, and should not be used, to affect the interpretation of the claims or to limit their scope in any way. The specific implementations shown and described herein are illustrative examples of the disclosure and are not intended to limit the scope of the disclosure in any way.

All references cited in this specification, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

Although the disclosure has been described in connection with particular embodiments and implementations, it should be understood that the disclosure is not limited to the disclosed implementations, but on the contrary, is intended to cover various modifications and equivalent arrangements included. Within the scope of the appended claims, their scope should be accorded the broadest interpretation permitted under law so as to encompass all such modifications and equivalent arrangements.

Claims (20)

1. A method for generating stereoscopic display content, comprising:
obtaining, using a processor, a first red-green-blue (RGB) image and a depth image from the red-green-blue-plus-distance (RGB-D) image;
determining a first disparity map according to the RGB-D image based on depth values in the depth image, the first disparity map including a plurality of disparity values for the first RGB image that are converted into a pair of stereoscopic images;
determining a second disparity map and a third disparity map by transforming the first disparity map using a disparity distribution ratio;
generating, by the processor, the pair of stereoscopic images comprising a second RGB image and a third RGB image, wherein the second RGB image is generated by shifting a first set of pixels in the first RGB image based on the second disparity map, and the third RGB image is generated by shifting a second set of pixels in the first RGB image based on the third disparity map. The method of claim 1, further comprising: generating, by the processor, a pair of adjusted display images based on the pair of stereoscopic images, the pair of adjusted display images being resized to fit display requirements of an augmented reality (AR), virtual reality (VR), or mixed reality (MR) device. The method of claim 1, wherein the first disparity map is a two-dimensional (2D) matrix, each element of which represents a disparity value. The method of claim 1, wherein the RGB-D image is captured by an RGB-D sensor. The method of claim 4, wherein a pixel in the depth image indicates a distance between the RGB-D sensor and a corresponding object captured in the RGB image. Determining the first disparity map includes:
The method of claim 5 , comprising using at least one of a focal length f or an interpupillary distance b to determine the first disparity map. Determining the second disparity map by transforming the first disparity map using the disparity distribution ratio is based on:

Determining the third disparity map by transforming the first disparity map using the disparity distribution ratio is based on:

7. The method of claim 6, wherein d _L (x,y) is the disparity value of the second disparity map, d _R (x,y) is the disparity value of the third parity map, d(x,y) is the disparity value of the first parity map, z(x,y) is the distance between the RGB-D sensor and an object corresponding to the pixel (x,y) in the RGB-D image, and k is the disparity distribution ratio, which is a constant value indicating the position of an observation point between the left eye and the right eye. Shifting the first set of pixels in the first RGB image based on the second disparity map based on:

Shifting the second set of pixels in the first RGB image based on the third disparity map based on:

8. The method of claim 7, wherein Pixel _L (x,y) is pixel (x,y) in the second RGB image, Pixel _R (x,y) is pixel (x,y) in the third RGB image, Pixel(x,y) is pixel (x,y) in the first RGB image, and (R(x,y),G(x,y),B(x,y)) is the RGB color for pixel (x,y). An apparatus for generating stereoscopic display content, comprising:
A non-transient memory;
a processor, the non-transitory memory comprising:
obtaining a first red-green-blue (RGB) image and a depth image from a red-green-blue plus distance (RGB-D) image;
determining a first disparity map according to the RGB-D image based on depth values in the depth image, the first disparity map including a plurality of disparity values for the first RGB image that are converted into a pair of stereoscopic images;
determining a second disparity map and a third disparity map by transforming the first disparity map using a disparity distribution ratio;
generating the pair of stereoscopic images including a second RGB image and a third RGB image, the second RGB image being generated by shifting a first set of pixels in the first RGB image based on the second disparity map, and the third RGB image being generated by shifting a second set of pixels in the first RGB image based on the third disparity map;
The apparatus further comprises instructions executable by the processor. The instructions executable by the processor include:
10. The device of claim 9, further comprising instructions for generating a pair of adjusted display images based on the pair of stereoscopic images, the pair of adjusted display images being resized to display requirements of an augmented reality (AR), virtual reality (VR), or mixed reality (MR) device. The apparatus of claim 9, wherein the first disparity map is a two-dimensional (2D) matrix, each element of which represents a disparity value. The device of claim 9, wherein the RGB-D image is captured by an RGB-D sensor. The device of claim 12, wherein a pixel in the depth image indicates a distance between the RGB-D sensor and a corresponding object captured in the RGB image. Determining the first disparity map includes:
The apparatus of claim 13 , comprising using at least one of a focal length f or an interpupillary distance b to determine the first disparity map. Determining the second disparity map by transforming the first disparity map using the disparity distribution ratio is based on:

Determining the third disparity map by transforming the first disparity map using the disparity distribution ratio is based on:

14. The apparatus of claim 13, wherein d _L (x,y) is the disparity value of the second disparity map, d _R (x,y) is the disparity value of the third parity map, d(x,y) is the disparity value of the first parity map, z(x,y) is the distance between the RGB-D sensor and an object corresponding to the pixel (x,y) in the RGB-D image, and k is the disparity distribution ratio, which is a constant value indicating the position of an observation point between the left eye and the right eye. Shifting the first set of pixels in the first RGB image based on the second disparity map based on:

Shifting the second set of pixels in the first RGB image based on the third disparity map based on:

16. The apparatus of claim 15, wherein Pixel _L (x,y) is pixel (x,y) in the second RGB image, Pixel _R (x,y) is pixel (x,y) in the third RGB image, Pixel(x,y) is pixel (x,y) in the first RGB image, and (R(x,y),G(x,y),B(x,y)) is the RGB color for pixel (x,y). 1. A non-transitory computer-readable storage medium configured to store a computer program for generating stereoscopic display content, the computer program comprising:
obtaining a first red-green-blue (RGB) image and a depth image from a red-green-blue plus distance (RGB-D) image;
determining a first disparity map according to the RGB-D image based on depth values in the depth image, the first disparity map including a plurality of disparity values for the first RGB image that are converted into a pair of stereoscopic images;
determining a second disparity map and a third disparity map by transforming the first disparity map using a disparity distribution ratio;
generating, by a processor, the pair of stereoscopic images including a second RGB image and a third RGB image, the second RGB image being generated by shifting a first set of pixels in the first RGB image based on the second disparity map, and the third RGB image being generated by shifting a second set of pixels in the first RGB image based on the third disparity map;
A non-transitory computer-readable storage medium comprising instructions executable by the processor. The instructions executable by the processor include:
20. The non-transitory computer-readable storage medium of claim 17, further comprising instructions for generating, by the processor, a pair of adjusted display images based on the pair of stereoscopic images, the pair of adjusted display images being resized to display requirements of an augmented reality (AR), virtual reality (VR), or mixed reality (MR) device. The non-transitory computer-readable storage medium of claim 17, wherein the first disparity map is a two-dimensional (2D) matrix, with each element indicating a disparity value. The non-transitory computer-readable storage medium of claim 17 , wherein the RGB-D image is captured by an RGB-D sensor.

Images