TWM630947U - Stereoscopic image playback apparatus - Google Patents

Abstract

A stereoscopic image playback apparatus is provided, which includes a processor and a graphics processing unit (GPU). The GPU is configured to create a 3D mesh and its texture to obtain a stereoscopic scene, and capture a planar image of the stereoscopic scene. In response to the planar image not being a side-by-side image, the processor performs image pre-processing on the planar image to obtain a first image. A graphics processing pipeline of the GPU performs depth estimation on the first image to obtain a depth image, and updates the 3D mesh according to a depth setting of the depth image, and project the updated 3D mesh to a corresponding coordinate system according to an eye tracking result of a user. The graphics processing pipeline draws the first image onto the projected 3D mesh to obtain an output 3D mesh, and capture an output side-by-side image from the output 3D mesh. The graphics processing pipeline weaves a left-eye image and a right-eye image of the side-by-side image into an output image, and displays the output image on a stereoscopic image display apparatus.

Description

Stereoscopic video playback device

This creation is about stereoscopic image display technology, especially about a stereoscopic image playback device.

With the advancement of technology, virtual reality devices have become more and more popular. Because the virtual reality scene requires a huge amount of computation, when the user's position in the virtual reality scene changes or the line of sight changes, the traditional stereoscopic image generation device often only uses the processor (software) to recalculate the updated The virtual reality scene is obtained, and the left eye image and the right eye image viewed by the user are obtained from the updated virtual reality scene. The above-mentioned calculation method often causes a great burden on the processor in the stereoscopic video playback device.

Therefore, there is a need for a stereoscopic video playback device to solve the above problems.

The present creation provides a stereoscopic image playback device, including: a processor; and a graphics processing unit for creating a stereoscopic mesh and its material to obtain a stereoscopic scene, and capturing a plane image of the stereoscopic scene; wherein, Since the planar image is not a side-by-side image, the processor performs image preprocessing on the planar image to obtain a first image; wherein a graphics processing pipeline of the graphics processing unit performs depth estimation on the first image to obtain a depth image, and update the three-dimensional grid according to the depth setting of the depth image, and map the three-dimensional grid to the corresponding coordinate system according to the eye tracking result of the user of the three-dimensional image playback device; wherein, the graphics processing pipeline Projecting the first image onto the mapped three-dimensional grid to obtain an output three-dimensional grid, and extracting an output side-by-side image from the output three-dimensional grid, wherein the output side-by-side image includes a left-eye image and a right-eye image wherein, the graphics processing pipeline knits the left eye image and the right eye image into an output image, and plays the output image on a stereoscopic image display device.

In some embodiments, since the two-dimensional image is the side-by-side image, the graphics processing pipeline directly knits the left-eye image and the right-eye image in the side-by-side image into an output image, which is played on a stereoscopic image display device the output image.

In some embodiments, the vertex shader of the graphics processing pipeline projects the first image onto the mapped volumetric mesh to obtain the output volumetric mesh.

In some embodiments, the image preprocessing adjusts the size and format of the planar image to meet the requirements of an artificial intelligence model in the graphics processing pipeline, and the artificial intelligence model performs depth estimation on the first image to obtain the depth image.

In some embodiments, the processor subtracts the minimum depth from the maximum depth in the depth image to obtain the depth setting.

In some embodiments, the graphics processing pipeline uses the depth image as a material for the volumetric mesh to update the volumetric mesh.

The following description is a preferred implementation manner to complete the creation, and its purpose is to describe the basic spirit of the creation, but it is not intended to limit the creation. The actual creative content must refer to the scope of the claims that follow.

It must be understood that words such as "comprising" and "including" used in this specification are used to indicate the existence of specific technical features, values, method steps, operation processes, elements and/or components, but do not exclude the possibility of Plus more technical features, values, method steps, job processes, elements, components, or any combination of the above.

The use of words such as "first", "second" and "third" in the scope of the patent application is used to modify the elements in the scope of the patent application, and is not used to indicate that there is a priority order, an antecedent relationship, or a The precedence of one element over another, or the chronological order in which method steps are performed, is only used to distinguish elements with the same name.

FIG. 1 is a block diagram of a stereoscopic video playback device according to an embodiment of the present invention.

The stereoscopic video playback device 10 can be, for example, a personal computer, a server, a portable electronic device or other electronic devices with similar computing capabilities. As shown in FIG. 1 , the stereoscopic video playback device 10 includes a host 100 and a stereoscopic video display device 180 . The host 100 is connected to the stereoscopic image display device 180 , and the host 100 can, for example, generate an image signal, and transmit the image signal to the stereoscopic image display device 180 for playback. The stereoscopic image display device 180 can play the image signal from the host 100 in a stereoscopic image playing mode or a flat image playing mode according to the format of the image signal from the host 100 .

The host 100 includes a processor 110 , a graphics processing unit 120 , a memory unit 130 , a storage device 140 , a camera 160 and a transmission interface 170 , wherein the above components in the host 100 are coupled to each other through the system bus 111 .

The processor 110 is, for example, a central processing unit (CPU), a general-purpose processor, etc., but the present invention is not limited thereto. The graphics processing unit 120 may be, for example, a graphics processing unit on a display card or a graphics processing unit integrated into the processor 110 , but the present invention is not limited thereto.

The memory unit 130 is a random access memory, such as dynamic Random Access Memory (DRAM) or Static Random Access Memory (SRAM), but the present invention is not limited to this. The memory unit 130 may also be referred to as a system memory, and may be used as an image buffer in addition to temporarily storing data for the processor 110 .

The storage device 140 is a non-volatile memory, such as a hard disk drive, a solid-state disk, and a flash memory. , or a read-only memory, but this creation is not limited to this. For example, the storage device 140 can store an operating system 141 (eg, Windows, Linux, MacOS, etc.), a graphics driver 142 and a stereoscopic image player 143 . For example, the processing unit 110 can read and execute the operating system 141 , the graphics driver 142 and the stereoscopic image playing program 143 into the memory unit 130 . For example, the graphics processing unit 120 can perform the drawing processing of the stereoscopic image player program 143 executed by the processing unit 110 to generate an image signal including one or more images, and transmit the image signal to the stereoscopic image display device 180 through the transmission interface 170 .

The transmission interface 170 may be a wired transmission interface and/or a wireless transmission interface, wherein the wired transmission interface may include: a High Definition Multimedia Interface (HDMI), a DisplayPort (DP) interface, an embedded display port ( embedded DisplayPort (eDP), low voltage differential signaling (LVDS) interface, Universal Serial Bus (USB) interface, USB Type-C interface, Thunderbolt Interface, Digital Video Interface (DVI), Video Graphics Array (VGA) interface, General Purpose Input Output (GPIO) interface, Universal Asynchronous Transceiver Transmitter (UART) interface, Serial Peripheral Interface (SPI) interface, Integrated Circuit Bus (I2C) interface, or a combination thereof, and the wireless transmission interface may include: Bluetooth (Bluetooth), Wi-Fi, Near Field Communication (NFC) interface, etc., but the present invention is not limited thereto.

The camera 160 can be disposed on the host 100 (or the stereoscopic image display device 180 ), for example, to capture the face image of the user in front of the host 100 . In addition, the processor 110 can execute an eye tracking program (not shown), for example, to detect the orientation of the eyes of the user from the facial image to obtain the eye tracking result, which can be used for stereoscopic images The side-by-side image generation stage in the playback program 143 will be described in detail later.

The stereoscopic image display device 180 can be, for example, a head mounted display (HMD) or an autostereoscopic display device for playing a virtual reality image or a stereoscopic image. The stereoscopic image display device 180 can be implemented with different stereoscopic display technologies in the art. For example, the naked-view stereoscopic display technology may include parallax barriers, lenticular lenses, directional backlight, etc., which can play alternately or simultaneously play one of the three-dimensional images. Left eye image and right eye image. The head-mounted display may include, for example, a left-eye display panel and a right-eye display panel, which are used to respectively play the left-eye image and the right-eye image in the stereoscopic image, and pass through the corresponding left-eye lens and right-eye lens to display the left-eye and right-eye lenses on the user's left side. The eye and the right eye are imaged to produce stereoscopic vision. in this creative field Those with ordinary knowledge can understand the relevant playback mechanisms of the head-mounted display and the stereoscopic display device, so the details are not repeated here.

The stereoscopic image playing program 143 includes a side-by-side image detection module 144 , an image preprocessing module 145 , a side-by-side image generation module 146 and an image weaving module 147 . The side-by-side image detection module 144 is used for detecting whether the received planar image is a side-by-side image, wherein the side-by-side image is a side-by-side image of the left-eye image and the right-eye image. The image preprocessing module 145 is used for adjusting the size and/or format conversion of the received planar image, and the size and/or format of the processed planar image conforms to the stereoscopic image player program 143 and the graphics processing unit 120 . The need for depth estimation by an AI model.

The side-by-side image generation module 146 uses the three-dimensional grid updated by the graphics processing unit 120 according to the depth setting and performs coordinate mapping according to the eye tracking result to generate the side-by-side image, the details of which will be described in detail later.

FIG. 2 is a schematic diagram of a plane image, a three-dimensional grid, and a plane grid according to an embodiment of the present invention.

Figure 2 includes regions 210, 220 and 230, wherein region 210 is a schematic diagram of a two-dimensional mesh, region 220 is a schematic diagram of a three-dimensional mesh, and region 230 is a schematic diagram of a three-dimensional scene Schematic. For example, the two-dimensional mesh in the region 210 includes a plurality of triangles arranged in sequence on the X-Y plane, and each triangle has a set of vertices. The graphics processing pipeline 121 in the graphics processing unit 120 will first generate a two-dimensional mesh corresponding to the three-dimensional scene, and give the vertices of each triangle in the two-dimensional mesh a specific height (height) (ie, the Z-axis direction), the three-dimensional grid shown in the area 220 can be obtained, wherein the Z-axis direction represents the scene depth. Finally, the graphics processing pipeline 121 pastes the corresponding materials of each triangle to obtain the three-dimensional scene shown in the area 230 .

FIG. 3 is a schematic diagram of a flow of a method for generating a 3D image using a graphics processing pipeline according to an embodiment of the present invention. Please refer to Figure 1 and Figure 3 at the same time.

Process 300 begins with initialization phase 310 , which includes blocks 312 , 314 and 316 . Block 312: Create a solid grid. For example, when the graphics processing pipeline 121 of the graphics processing unit 120 performs drawing processing, a 3D mesh corresponding to the three-dimensional scene is first created, as shown in the area 220 in FIG. 2 .

Block 314: Create the material. For example, the three-dimensional mesh includes a plurality of triangles, and each triangle has a corresponding texture. The graphics processing pipeline 121 creates materials corresponding to each triangle together, and the vertex shader of the graphics processing pipeline 121 pastes the materials of each triangle when performing drawing processing to obtain a three-dimensional scene.

Block 316: Start image capture. For example, although the graphics processing unit 120 has generated a three-dimensional scene, the host 100 still needs to generate a corresponding stereoscopic image before it can be played on the stereoscopic image display device 180 . At this time, the processor 110 can obtain a plane image of the 3D scene by capturing images of the 3D scene.

Block 318: Determine whether the image has arrived. For example, in the process of FIG. 3, the stereoscopic image can be successfully generated at the block 340, then the block 318 will It is judged that the image has arrived. If the stereoscopic image is not successfully generated at block 340, then block 318 determines that the image has not arrived, so block 320 is executed to stop image capture.

Block 322: Side-by-side image detection. Block 324: Determine whether it is a side-by-side image. For example, the side-by-side image detection module 144 is used to detect whether the planar image generated by the graphics processing unit 120 is a side-by-side image, wherein the side-by-side image is a left-eye image and a right-eye image side by side image. If it is detected at block 324 that the planar image is a side-by-side image, then proceed to block 340 . If it is detected at block 324 that the planar image is not a side-by-side image, then block 326 is entered.

Block 326: Image preprocessing. For example, the image preprocessing module 145 is used to adjust the size and format conversion of the planar image generated by the graphics processing unit 120, and the size and/or format of the processed planar image conform to the stereoscopic image player 143 and graphics The AI model executed by the processing unit 120 needs to perform depth estimation. It should be noted that when the above-mentioned artificial intelligence model performs depth estimation on its input image, the format and/or size (resolution) of the input image must meet the requirements of the above-mentioned artificial intelligence model.

Block 328: Depth estimation. For example, the artificial intelligence model executed by the graphics processing unit 120 has been trained in advance, and it obtains a single input image (ie, a plane image) to determine the corresponding depth of objects in the input image. Therefore, the graphics processing unit 120 can obtain the depth setting of the planar image, wherein the depth setting may be, for example, a depth effect strength parameter, which may be the maximum depth minus the minimum depth of the planar image in the Z-axis direction. In one embodiment, the depth estimation of the present creation may be performed for each frame (Frame). The calculation is to generate a corresponding depth map for each frame of the input image.

Side-by-side image generation stage 330 includes blocks 332 , 334 , 336 and 338 . Block 332: Update the volume grid. For example, the graphics processing pipeline 121 of the graphics processing unit 120 can update the volume mesh according to the depth setting of the planar image, wherein the vertex shader of the graphics processing pipeline 121 can use the depth map obtained at block 328 or The depth image is used as the texture for the corresponding triangles in the 3D mesh. In one embodiment, the user can dynamically adjust the depth setting value. For example, if the depth of the display plane is 0, and the depth value in the direction of the screen is at most -10, the user can set the depth from 0 to -6 according to his own needs. Then the maximum depth value will be -6; in another embodiment, the user can set the depth value range from -3 to -9; accordingly, the graphics processing pipeline 121 can update the 3D mesh according to the depth setting of the plane image grid.

Block 334: Coordinate system mapping. For example, the graphics processing unit 120 performs coordinate mapping on the three-dimensional grid according to the eye tracking result. When the user wears the head-mounted display or watches the stereoscopic display device, the user may adjust the position of the body or head, or adjust the line of sight of the eyeball. A stereoscopic image of a virtual reality scene (ie, a three-dimensional scene). For example, when the user performs the above actions, it can be considered that the positions of the left camera and the right camera (corresponding to the user's left and right eyes) in the virtual reality scene also change, so the graphics processing unit 120 The left and right images of the virtual reality scene captured by the changed positions of the left camera and the right camera need to be recalculated to serve as the left-eye image and the right-eye image viewed by the user, respectively.

Block 336: Project the image to the volume grid. For example, through blocks 332 and 334, the relative position and distance (or depth) of the 3D grid obtained by the user's eyes after updating and mapping can be determined, so the graphics processing pipeline 121 can project the planar image to the on the stereo mesh to get the updated virtual reality scene.

Block 338: Generate side-by-side images. Since the updated virtual reality scene has been obtained at block 336, the processor 110 (or the graphics processing unit 120) can use the left camera and the right camera in the updated virtual reality scene to take pictures at this time to obtain the left eye image and the right eye image, and the left eye image and the right eye image are side-by-side to obtain side-by-side images.

Block 340: Image weaving processing. For example, for some stereoscopic image display devices (eg, a naked-view stereoscopic display device), it is necessary to play the left-eye image and the right-eye image at the same time, so that the user can experience the stereoscopic vision. The input image format of such a stereoscopic image display device needs to be a weaved image, for example, the odd-numbered lines are the left-eye image and the even-numbered lines are the right-eye image, or the odd-numbered lines are the right-eye image and the even-numbered lines are the left-eye image .

In this creation, because the graphics processing pipeline 121 of the graphics processing unit 120 is a dedicated hardware circuit, and when the user's position or line of sight in the virtual reality scene changes, the graphics processing pipeline 121 can quickly update the three-dimensional grid And calculate the coordinate system after mapping, and can paste plane image on the three-dimensional grid after mapping to obtain the updated three-dimensional scene. Therefore, the present creation can use the graphics processing pipeline 121 of the graphics processing unit 120 to quickly calculate the left and right images of the virtual reality scene captured by the left and right cameras at the changed positions, and generate corresponding side-by-side images and woven images for playback by the stereoscopic image display device 180 . In one embodiment, the graphics processing pipeline 121 of the graphics processing unit 120 simultaneously stores each frame of image and generates a corresponding depth map, so that the same graphics processing pipeline 121 simultaneously processes the depth map. ) and each frame captured, the processing speed will be very fast.

FIG. 4 is a flowchart of a method for generating a stereoscopic image according to an embodiment of the present invention. Please also refer to Figures 1, 3 and 4.

In step S410, the graphics processing unit 120 is used to create a three-dimensional mesh and its material. For example, the three-dimensional mesh includes a plurality of triangles, and each triangle has a corresponding texture. The graphics processing pipeline 121 creates materials corresponding to each triangle together, and the vertex shader of the graphics processing pipeline 121 pastes the materials of each triangle when performing drawing processing to obtain a three-dimensional scene.

In step S420, a planar image of the three-dimensional grid is captured. For example, although the graphics processing unit 120 has generated a three-dimensional scene, the host 100 still needs to generate a corresponding stereoscopic image before it can be played on the stereoscopic image display device 180 . At this time, the processor 110 can obtain a plane image of the 3D scene by capturing images of the 3D scene.

In step S430, since the planar image is not a side-by-side image, image preprocessing is performed on the planar image to obtain a first image. For example, the image preprocessing module 145 is used to adjust the size and/or format conversion of the planar image generated by the graphics processing unit 120, and the size and/or format of the processed planar image conform to the stereoscopic image player program 143 and the artificial intelligence model executed by the graphics processing unit 120 (AI model) The need for depth estimation. It should be noted that when the above-mentioned artificial intelligence model performs depth estimation on its input image, the format and/or size (resolution) of the input image must meet the requirements of the above-mentioned artificial intelligence model.

In step S440, depth estimation is performed on the first image using the graphics processing pipeline of the graphics processing unit to obtain a depth image. For example, the artificial intelligence model executed by the graphics processing unit 120 has been trained in advance, and can obtain a single input image (ie, a plane image) to determine the corresponding depth of objects in the input image. Therefore, the graphics processing unit 120 can obtain the depth setting of the planar image, wherein the depth setting may be, for example, a depth effect strength parameter, which may be the maximum depth minus the minimum depth of the planar image in the Z-axis direction. In addition, the graphics processing unit 120 determines the corresponding depth map for each input image.

In step S450, the three-dimensional grid is updated by using the depth image, and the three-dimensional grid is mapped to a corresponding coordinate system according to the user's eye tracking result. For example, the graphics processing unit 120 performs coordinate mapping on the three-dimensional grid according to the eye tracking result. When the user wears the head-mounted display or watches the stereoscopic display device, the user may adjust the position of the body or head, or adjust the line of sight of the eyeball. A stereoscopic image of a virtual reality scene (ie, a three-dimensional scene). For example, when the user performs the above actions, it can be considered that the positions of the left camera and the right camera (corresponding to the user's left and right eyes) in the virtual reality scene also change, so the graphics processing unit 120 It is necessary to recalculate the left side shadow of the virtual reality scene captured by the left camera and the right camera after changing the position. The image and the right image are used as the left-eye image and the right-eye image viewed by the user, respectively.

In step S460, the first image is projected onto the mapped three-dimensional mesh to obtain an output three-dimensional mesh. For example, through steps S440 and S450, the relative positions and distances (or depths) of the 3D mesh obtained by the user's eyes after updating and mapping can be determined, so at this time, the vertex shader in the graphics processing pipeline 121 can The flat image is projected onto the stereo mesh to obtain an updated virtual reality scene (ie, a 3D scene).

In step S470, an output side-by-side image is extracted from the output stereo grid, wherein the output side-by-side image includes a left-eye image and a right-eye image. Since the updated virtual reality scene has been obtained in step S460, the processor 110 (or the graphics processing unit 120) can use the left camera and the right camera in the updated virtual reality scene to shoot to obtain the left eye image and the right eye image, and the left eye image and the right eye image are side-by-side to obtain side-by-side images.

In step S480, the left-eye image and the right-eye image are woven into an output image, and the output image is played on the stereoscopic display device. For example, for some stereoscopic image display devices (eg, a naked-view stereoscopic display device), it is necessary to play the left-eye image and the right-eye image at the same time, so that the user can experience the stereoscopic vision. The input image format of such a stereoscopic image display device needs to be a weaved image, for example, the odd-numbered lines are the left-eye image and the even-numbered lines are the right-eye image, or the odd-numbered lines are the right-eye image and the even-numbered lines are the left-eye image .

To sum up, the present invention provides a stereoscopic image playback device, which can use the artificial intelligence model of the graphics processing pipeline of the graphics processing unit to quickly determine the Cut out the depth of each object in the plane image, and the graphics processing pipeline can quickly update the stereo grid and calculate the mapped coordinate system, and can paste the plane image on the mapped stereo grid to get the updated stereo scene . Therefore, when the user's position in the stereoscopic scene changes or the line of sight changes, the creation can use the graphics processing pipeline of the graphics processing unit to quickly calculate the virtual reality captured by the left camera and the right camera at the changed positions. The left and right images of the environment scene are generated, and corresponding side-by-side images and woven images are generated for the stereoscopic image display device to play, thereby improving the image quality of the output stereoscopic images and increasing the operation speed when playing the stereoscopic images. Compared with the traditional pixel offset method, the original image is pixel-shifted according to the depth map information to obtain the other-eye image, this creation can quickly calculate the other-eye image according to the user's needs and position movement through the three-dimensional grid. without additional calculations.

Although this creation is disclosed above with preferred embodiments, it is not intended to limit the scope of this creation. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of this creation. Therefore, the scope of protection of this creation should be determined by the scope of the appended patent application.

10: Stereoscopic video playback device

100: host

110: Processor

111: System busbar

120: Graphics Processing Unit

121: Graphics processing pipeline

130: Memory unit

140: Storage Device

141: Operating System

142: Graphics driver

143: Stereoscopic video player program

144: Side-by-side image detection module

145: Image preprocessing module

146: Side-by-side image generation module

147: Image weaving module

160: Camera

170: Transmission interface

180: Stereoscopic image display device

210, 220, 230: Area

310: Initialization Phase

312-328: Blocks

330: Side-by-side image generation stage

332-340: Blocks

S410-S480: Steps

FIG. 1 is a block diagram of a stereoscopic video playback device according to an embodiment of the present invention. FIG. 2 is a schematic diagram of a plane image, a three-dimensional grid, and a plane grid according to an embodiment of the present invention. FIG. 3 is a schematic diagram of a flow of a method for generating a 3D image using a graphics processing pipeline according to an embodiment of the present invention. FIG. 4 is a flowchart of a method for generating a stereoscopic image according to an embodiment of the present invention.

10: Stereoscopic video playback device

100: host

110: Processor

111: System busbar

120: Graphics Processing Unit

121: Graphics processing pipeline

130: Memory unit

140: Storage Device

141: Operating System

142: Graphics driver

143: Stereoscopic video player program

144: Side-by-side image detection module

145: Image preprocessing module

146: Side-by-side image generation module

147: Image weaving module

160: Camera

170: Transmission interface

180: Stereoscopic image display device

Claims (7)

A stereoscopic image playback device, comprising: a processor; and a graphics processing unit for creating a three-dimensional mesh and its material to obtain a three-dimensional scene, and capturing a plane image of the three-dimensional scene; Wherein, since the planar image is not a side-by-side image, the processor performs image preprocessing on the planar image to obtain a first image; Wherein, a graphics processing pipeline of the graphics processing unit performs depth estimation on the first image to obtain a depth image, and updates the 3D grid according to the depth setting of the depth image, and according to the user of the 3D image playback device the eye tracking result to map the three-dimensional grid to the corresponding coordinate system; Wherein, the graphics processing pipeline projects the first image onto the mapped three-dimensional grid to obtain an output three-dimensional grid, and extracts an output side-by-side image from the output three-dimensional grid, wherein the output side-by-side image includes a left side-by-side image. an eye image and a right eye image; Wherein, the graphics processing pipeline knits the left-eye image and the right-eye image into an output image, and plays the output image on a stereoscopic image display device. The stereoscopic image playback device of claim 1, wherein since the two-dimensional image is the side-by-side image, the graphics processing pipeline directly knits the left-eye image and the right-eye image in the side-by-side image into an output image, and outputs a The stereoscopic image display device plays the output image. The stereoscopic image playback device of claim 1, wherein the vertex shader of the graphics processing pipeline projects the first image to the mapped stereoscopic mesh to obtain the output stereoscopic mesh. The stereoscopic image playback device of claim 1, wherein the image preprocessing adjusts the size and format of the flat image to meet the requirements of an artificial intelligence model in the graphics processing pipeline, and the artificial intelligence model performs the first image processing. Depth estimation to obtain the depth image. The stereoscopic image playback device of claim 1, wherein the processor subtracts the minimum depth from the maximum depth in the depth image to obtain the depth setting. The stereoscopic image playback device of claim 1, further comprising: a camera for capturing the face image of the user, and the processor detects the line-of-sight direction of the user's eyes from the face image to as the eye tracking result. The 3D image playback device of claim 1, wherein the graphics processing pipeline uses the depth image as a material of the 3D mesh to update the 3D mesh.

Images