WO2020134085A1 - Vr系统中图像显示的控制方法、装置及vr头戴设备 - Google Patents

Vr系统中图像显示的控制方法、装置及vr头戴设备 Download PDF

Info

Publication number
WO2020134085A1
WO2020134085A1 PCT/CN2019/098833 CN2019098833W WO2020134085A1 WO 2020134085 A1 WO2020134085 A1 WO 2020134085A1 CN 2019098833 W CN2019098833 W CN 2019098833W WO 2020134085 A1 WO2020134085 A1 WO 2020134085A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
original
pixel
pixels
pose information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/098833
Other languages
English (en)
French (fr)
Inventor
蔡磊
戴天荣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Goertek Inc
Original Assignee
Goertek Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Goertek Inc filed Critical Goertek Inc
Priority to US16/631,136 priority Critical patent/US20210063735A1/en
Publication of WO2020134085A1 publication Critical patent/WO2020134085A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/18—Image warping, e.g. rearranging pixels individually
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays

Definitions

  • the invention relates to the field of virtual reality technology, and in particular to a method and device for controlling image display in a VR system and a VR headset.
  • the VR Mounted Display is to continuously track the user's head posture in real time through the posture sensor, and then use the posture information to render two 2D images from the virtual 3D world, namely the left eye and the right The picture that the eye should see, and these two images are displayed on the screen.
  • the built-in mechanism of the brain will resolve the three-dimensional sense, so as to achieve VR (Virtual Reality, virtual reality) effect.
  • a core indicator is the system delay.
  • the system delay refers to the length of time from when the user’s head position is obtained until the rendered picture is completely displayed on the screen. Too long system delay will cause the user's physiological senses to be inconsistent with the pictures received by the eyes, resulting in symptoms of motion sickness.
  • the flow of 3D rendering is a pipelined design. See Figure 1. From the first frame, the input data frames U1 to U4 (for example only) flow through the central processing units CPU1 and CPU2. Threads, GPU (Graphics Processing Unit, graphics processor) and screen, and finally produce an output on the screen after screen scanning.
  • the pipelined design improves the utilization of each component and ensures higher throughput. At the same time, it also brings higher system delay. See Figure 1. From the current posture obtained by sampling the IMU to the first frame, The photons displayed on the screen are observed by the user.
  • a typical throughput-oriented rendering process causes a system delay (Motion to Photon latency) of at least four screen refresh cycles. On a screen with a refresh frequency of 90 Hz, It will take 44.4 milliseconds, far exceeding the physiologically tolerable limit of 18 milliseconds.
  • Timewarp algorithm that is used to warp (eg, shift, rotate, adjust, or reproject) image frames to correct for head movement or translation that occurs after rendering the frame, thereby reducing system delay
  • the Timewarp algorithm currently only processes images with 3 degrees of freedom (3DOF) pose, and the processed images by the Timewarp algorithm are not realistic enough to meet the actual needs.
  • the embodiments of the present invention provide an image display control method and device in a VR system and a VR head-mounted device, which expands the scene applicable to the Timewarp algorithm, so that it can be applied to the image display control of a 6-degree-of-freedom pose and enhances The realism of the image improves the image display effect.
  • a method for controlling image display in a VR system including:
  • the sensor data is sampled to obtain the latest pose information of the tracked object, where the pose information includes information indicating the rotation of the tracked object and information indicating the translation of the tracked object;
  • the display of the target frame is triggered when the next synchronization signal arrives.
  • an image display control device in a VR system including:
  • the acquisition module is used to monitor the synchronization signal of the image frame in the VR system and acquire the original 2D image;
  • the sampling module is used to sample the sensor data to obtain the latest pose information of the tracked object at a preset time point before the arrival of the next synchronization signal, where the pose information includes information indicating the rotation of the tracked object and indicating the tracked object Translation information
  • the vector calculation module is used to convert the original 2D image into the corresponding 3D image, and calculate the motion vector corresponding to each pixel of the 3D image according to the latest pose information and the pose information corresponding to the 3D image;
  • the target frame generation module is used to perform position transformation on the pixels of the original 2D image based on the motion vector, and fill in the pixels of the vacant area that appear after the position transformation to obtain the target frame;
  • the trigger module is used to trigger the display of the target frame when the next synchronization signal arrives.
  • a VR headset including: a memory and a processor, the memory and the processor are connected via an internal bus communication, the memory stores program instructions that can be executed by the processor, and the program instructions The method of one aspect of the present application can be implemented when executed by a processor.
  • the synchronization signal of the image frame in the VR system is monitored and the original 2D image is acquired, and the sensor data is sampled at a preset time point before the arrival of the next synchronization signal
  • the tracked object's latest pose information indicating the tracked object's rotation and translation information calculates the motion vector corresponding to each pixel of the 3D image, transforms the position of the pixel of the original 2D image based on the motion vector, and appears after the position transformation Fill in the pixels of the vacant area to get the target frame, and trigger the display of the target frame when the next synchronization signal arrives.
  • the application scope and scene of the Timewarp algorithm are expanded, the display control requirements of 6DOF pose changes are met, and the new position of the pixel point is calculated based on the motion vector corresponding to each pixel point and the pixels of the vacant area are calculated. Filling at the point to get the target frame also improves the picture authenticity and display effect.
  • the VR headset of the embodiment of the present invention applies the time warp Timewarp to the scene where the tracked object 6DOF changes in pose, shortening the system delay and ensuring the authenticity and display effect of the picture, meeting the actual needs and enhancing the product market Competitiveness.
  • Figure 1 is a schematic diagram of the principle of system delay generation
  • FIG. 2 is a schematic diagram of the principle of applying the Timewrap algorithm for image display control according to an embodiment of the present invention
  • Figure 3a is the acquired image before the position of the tracked object moves
  • FIG. 3b is an enlarged schematic diagram of the rectangular frame in FIG. 3a;
  • Figure 4a is the acquired image after the position of the tracked object moves
  • FIG. 4b is an enlarged schematic diagram of the rectangular frame in FIG. 4a;
  • FIG. 5 is a schematic flowchart of a method for controlling image display in a VR system according to an embodiment of the present invention
  • FIG. 6 is a schematic diagram of an embodiment of the present invention after adding a grid to the image shown in FIG. 4a;
  • FIG. 7a is a schematic diagram of the image shown in FIG. 6 after position conversion based on motion vectors
  • FIG. 7b is an enlarged schematic view of the part shown in the box in FIG. 7a
  • FIG. 8a is a schematic view of the image shown in FIG. 6 after position conversion and filling;
  • FIG. 8b is an enlarged schematic view of the portion shown by the rectangular frame in FIG. 8a;
  • 9a is a schematic diagram of the original 2D image filling process after applying the method of the embodiment of the present invention.
  • FIG. 9b is an enlarged schematic view of the portion shown in the rectangular frame in FIG. 9a;
  • FIG. 10 is a block diagram of an image display control device in a VR system according to an embodiment of the present invention.
  • FIG. 11 is a schematic structural diagram of a VR head-mounted device according to an embodiment of the present invention.
  • the design concept of the present invention is to improve the existing Timewrap algorithm and expand the application range of the Timewrap algorithm so that it can be applied to scenes with 6-degree-of-freedom posture changes to meet actual needs.
  • Input collect all user input, such as mouse, keyboard and other input data of various external devices.
  • the IMU Inertial Measurement Unit in Figure 1 is a sensor that collects attitude data.
  • Update update the status of objects in the 3D world based on user input, such as updating the position and orientation of the camera, updating the movement of characters in the controlled game application, and updating other non-player controls in the game application The movement and change of characters and objects.
  • Render The GPU executes the series of rendering instructions generated in the previous step (step 3) one by one, and finally generates a 2D image for the user to observe.
  • FIG 1 I in Figure 1 represents the Input stage, U represents the Update stage, C represents the Commit stage, and R represents the Render stage.
  • Figure 1 illustrates the four frames processed in CPU1, which are in the update phase, respectively U1, U2 to U4, the three frames processed in CPU2 are in the submission phase, respectively C1, C2 to C3, and one frame processed in the GPU is in rendering stage.
  • the screen refresh mechanism when pushing a 2D image to the screen, you need to wait for the next synchronization signal to arrive. When the next synchronization signal arrives, you can push it, so there is no task after the GPU renders the current frame of instructions. Processing, it will enter the Idle (see Idle after R1 in Figure 1) idle state.
  • Step one After the update thread samples the pose, it updates the world state, and then submits the pipeline time. This stage takes 1 screen refresh cycle. For a screen with a refresh frequency of 60 Hz, a refresh cycle of 1000/60 ⁇ 33.3 ms, and a refresh frequency of 90 Hz, a refresh cycle of 1000/90 ⁇ 11.1 ms.
  • Step 2 The render thread submits the rendering instruction to the GPU according to the latest 3D world state and pose. This stage takes 1 screen refresh cycle.
  • Step 3 Rasterize (ie render) the 3D image to generate a 2D image, and wait for the time of the screen synchronization signal.
  • the time-consuming of rasterization depends on the complexity of the scene in the current pose. Usually, the content designer needs to ensure that the time is less than 1 screen refresh cycle to ensure a stable frame rate. This step takes at least 1 screen refresh cycle. If rasterization occupies n.m refresh cycles, the overall time spent in this step is rounded up to obtain (n+1) screen refresh cycles.
  • Step 4 The 2D image data is transmitted to the screen, and the time to scan to the screen plus the time to actually emit photons takes a total of 1 refresh cycle.
  • a typical throughput-oriented rendering process brings a system delay of at least 4 screen refresh cycles, which will take 44.4ms on a 90Hz screen, far exceeding the physiologically tolerable limit of 18ms.
  • the Timewarp algorithm inserts some additional steps after rasterizing the image in step three above and before waiting for the next screen synchronization signal, changing step three into the following sub-steps to reduce system delay, see Figure 2, which mainly includes Proceed as follows:
  • 2D images are generated by rasterization and cached as original images
  • Timewarp is established because the complexity of transforming 2D images depends only on the image resolution and is much smaller than the rasterization time of 3D scenes. Usually this process takes 2 ⁇ 3ms, so as long as the image is refreshed on the screen, as long as 4ms is reserved to change the image with the latest pose, the overall effect of reducing system delay can be achieved.
  • the system delay is reduced from 4 refresh cycles to Timewarp time + 1 refresh cycle.
  • any unconstrained object has 6 independent movements in space, namely 6 degrees of freedom (Degrees of Freedom, DOF for short).
  • the VR device in the rectangular coordinate system oxyz, the VR device can have 3 translational movements and 3 rotational movements.
  • the three translational motions are translational motions along the x, y, and z axes; and the three rotational motions are rotations around the x, y, and z axes, respectively. It is customary to call the above 6 independent movements as 6 degrees of freedom.
  • the Timewarp algorithm based on 3DOF or 3DOF only considers the direction rotation when implementing, and does not consider the position translation. The reason is that when only the direction rotation is processed, the occlusion relationship of the scene in the 2D image does not change.
  • the Timewarp algorithm is easier to implement, but it brings The problem is lack of realism.
  • the Timewarp algorithm is applied to the 6DOF scene, it is much more complicated, because it is necessary to consider both the rotation and the change in the scene occlusion relationship caused by the displacement.
  • FIG. 3a is an acquired image before the position of the tracked object moves, where the tracked object is, for example, the head of a wearer wearing a VR head-mounted device, and FIG. 3a illustrates the VR head-mounted device before the head position does not move
  • Figure 3b illustrates the image after magnifying the black rectangular frame in Figure 3a. You can see that the wall blocks the corner of the bench.
  • FIG. 4a after the wearer's head moves to the left, the camera on the VR headset moves to the left, and the black rectangular frame in the picture taken by the camera exposes the blocked area before the movement.
  • FIG. 4b after magnifying the black rectangular frame in FIG. 4a, you can see that there are two band-shaped black areas within the range enclosed by the rectangular frame. This part is due to the user’s head moving to the left. What the original undisplayed area looks like.
  • the Timewrap algorithm in the prior art does not consider the image processing when the 6DOF pose changes, so no corresponding solution is given.
  • the embodiments of the present invention are directed to the above technical problems, extending the application of the Timewrap algorithm under 6DOF pose changes to fill this gap, and at the same time, improve the authenticity of the reconstructed image and ensure the image display effect.
  • FIG. 5 is a method for controlling image display in a VR system according to an embodiment of the present invention. Referring to FIG. 5, the method includes the following steps:
  • Step S501 monitor the synchronization signal of the image frame in the VR system and obtain the original 2D image
  • Step S502 at a preset time point before the next synchronization signal arrives, the sensor data is sampled to obtain the latest pose information of the tracked object, where the pose information includes information indicating the rotation of the tracked object and indicating the translation of the tracked object information;
  • step S503 the original 2D image is converted into a corresponding 3D image, and the motion vector corresponding to each pixel of the 3D image is calculated according to the latest pose information and the pose information corresponding to the 3D image,
  • Step S504 Perform position transformation on the pixels of the original 2D image based on the motion vector, and fill in the pixels of the vacant area that appear after the position transformation to obtain the target frame;
  • Step S505 when the next synchronization signal arrives, the display of the target frame is triggered.
  • the latest included head rotation is obtained Pose information of information and position translation information, calculate the motion vector of each pixel on the 3D image based on such pose information and the image transformed into the 3D space, adjust the position of the pixel on the 2D image based on the vector, after the position adjustment Fill the vacant area that appears (see the black stripe area in FIG. 4b), complement the information to get the target frame.
  • the target frame is displayed when the next synchronization signal arrives.
  • the time from the sampling of the latest pose to the generation and display of the target frame according to the pose is greatly shortened, that is, the system delay is significantly reduced.
  • the image display control method in the VR system of the embodiment of the present invention is applied to a virtual reality VR head-mounted device.
  • the wearer of the head-mounted device it can move in translation or rotation, which reduces The mobile restrictions of the users have improved user satisfaction.
  • the method according to the embodiment of the present invention fills in the missing image information caused by the translation of the head position to generate a target frame, and the display of the target frame also ensures the realism of the image and the image display effect.
  • the tracked object is not limited to the head, but can also be the hand.
  • control method of image display in the VR system of the embodiment of the present invention is to buffer the depth information (Z-buffer) corresponding to each pixel while rasterizing to generate the original 2D image, and then use the depth information to convert each pixel Transform back to 3D space, calculate the corresponding motion vector (MotionVector) in 3D space, and finally divide the original 2D image into a dense grid, and apply the motion vector to transform each pixel in the original 2D image, while The pixel interpolation between the grids is used to complete the filling of the vacant area to obtain the target frame.
  • Z-buffer depth information
  • MotionVector MotionVector
  • the method for controlling the image display in the VR system is applied to monitor the synchronization signal of the image frame in the VR system, at a preset time point before the arrival of the next synchronization signal, for example, before the arrival of the next synchronization signal
  • a preset time point before the arrival of the next synchronization signal
  • the specific time point depends on the execution time of the Timewarp algorithm, and further depends on the resolution of the original image and the computing power of the GPU hardware, which can be determined according to actual tests.
  • Obtaining the original 2D image here is to rasterize the 3D image captured by the depth camera in world space to generate a 2D original image, and the rasterization also generates Z-buffer depth information.
  • the depth information of the original image is saved and recorded The 6DOF pose (O, P) used at this moment.
  • Z-buffer saves the depth information of each pixel in the original image to restore the 3D coordinates of the pixel.
  • O is the direction information (u, v, w) indicating the rotation of the user's head movement
  • P is the position information (x, y, z) indicating the translation of the user's head position .
  • the latest pose information of the user's head can be obtained by sampling the IMU data of the VR system's inertial measurement unit.
  • the calculated motion vector here is to inversely transform the 2D pixels back to the 3D space to obtain the corresponding motion vector (Motion Vector).
  • an inverse matrix M is generated for the spatial transformation matrix used during rasterization. Obtain the horizontal position information, vertical position information and depth information of each pixel of the original 2D image to obtain the position vector of each pixel of the original 2D image; the inverse matrix of the matrix and the position of each pixel when using the 3D to 2D spatial transformation Vector to calculate the original position of each pixel of the 3D image corresponding to the original 2D image in the 3D space.
  • the position vector is (x, y, z) coordinates.
  • the pixel points in the 2D image and the pixel points in the 3D image essentially describe the same objective objects, but the description methods and the reflected physical characteristics in different spaces are different.
  • the motion vector (n, m, k) is obtained according to the original position (x', y', z') and the new position (x", y", z") of each pixel.
  • the motion vector is applied to the original 2D image, which is based on the motion vector to transform the position of the pixels of the original 2D image, and fills in the pixels of the vacant area that appears after the position transformation to obtain the target frame.
  • a part of pixels are selected from the pixels of the original 2D image to obtain key pixels; for each selected key pixel, position transformation is performed based on the size and direction indicated by the motion vector.
  • selecting some key pixels from the pixels of the original 2D image includes: dividing the original 2D image into a plurality of regular grids, and selecting pixels corresponding to the vertices of the grid as key pixels.
  • a regular grid is created for the original image, for example, a 200x100 grid, and pixel points corresponding to the vertices of the grid are selected as key pixel points. Note: The denser the grid, the greater the amount of calculation and the better the quality of the resulting picture.
  • FIG. 7a is a schematic diagram of the image shown in FIG. 6 after position conversion based on motion vectors
  • FIG. 7b is an enlarged schematic diagram of the portion shown in the box in FIG. 7a. See FIGS. 7a and 7b for the grid shown in FIG. After the pixel positions of the vertices are changed, some pixels in the area enclosed by the vertices of the grid have no color information, so there is a vacant area.
  • the pixels of the vacant area appearing after the position conversion are filled to obtain the target frame.
  • the vacant area in the area enclosed by the mesh vertices after the position transformation is determined; the pixel points of the vacant area are filled with interpolation.
  • the pixel positions between vertices and vertices are calculated by the built-in linear interpolation of the GPU to achieve the reconstruction effect.
  • Figure 8a is a schematic diagram of the position conversion and filling of the image shown in Figure 6;
  • Figure 8b is the rectangular frame in Figure 8a
  • the enlarged schematic diagram of the part shown, as shown in Figures 8a and 8b, while the grid completes the position change, the pixels in the middle vacant area will be automatically filled by the GPU.
  • the Timewarp transformation process is completed, and the final target image is obtained.
  • the content of the target image reflects the image observed by the user in the latest head posture. Subsequent target images will be refreshed to the screen for display.
  • FIG. 9a is a schematic diagram of the original 2D image filling process applied by the method of the embodiment of the present invention
  • FIG. 9b is an enlarged schematic diagram of the rectangular frame in FIG. 9a. Comparing Figure 9b with Figure 4b, it can be seen that the content filled by this method is more realistic and feasible.
  • the blocked area is the texture material of the wooden chair.
  • This method uses the texture of the wooden chair (see the white dotted rectangular frame in Figure 9b) to correct Filling, the edges of the walls and wooden chairs remain vertical, making the picture more authentic. And compared with the existing technical solution that incorrectly uses the texture of the wall to fill the vacant area, causing the wall edge to bend to the left as a whole, and the filling effect is unrealistic, the picture authenticity is improved.
  • the image display control method in the VR system of the embodiment of the present invention expands the application scenarios of Timewarp, the method is simple and easy to operate, the calculation intensity is lower, and no additional hardware resources need to be added, the operation efficiency is high, and the authenticity of the picture is guaranteed The sense enhances the user experience.
  • FIG. 10 is an image display in the VR system according to an embodiment of the present invention.
  • Block diagram of the control device, the image display control device 1000 in the VR system including:
  • the acquisition module 1001 is used to monitor the synchronization signal of the image frame in the VR system and acquire the original 2D image;
  • the sampling module 1002 is configured to sample the sensor data at a preset time point before the next synchronization signal arrives to obtain the latest pose information of the tracked object, where the pose information includes information indicating the rotation of the tracked object and indicating the tracked Object translation information;
  • the vector calculation module 1003 is used to convert the original 2D image into the corresponding 3D image, and calculate the motion vector corresponding to each pixel of the 3D image according to the latest pose information and the pose information corresponding to the 3D image;
  • the target frame generation module 1004 is used to perform position transformation on the pixels of the original 2D image based on the motion vector, and fill in the pixels of the vacant area that appear after the position transformation to obtain the target frame;
  • the trigger module 1005 is used to trigger the display of the target frame when the next synchronization signal arrives.
  • the target frame generation module 1004 is specifically used to divide the original 2D image into a plurality of regular grids (such as dividing the original 2D image into 200x100 regular grids) and select the grid vertices
  • the corresponding pixel is used as a key pixel, and the selected key pixels are transformed based on the size and direction indicated by the motion vector, and filled in the pixels of the vacant area after the position transformation to obtain the target frame.
  • the vector calculation module 1003 is specifically used to calculate the original position of each pixel of the 3D image corresponding to the original 2D image in the 3D space using the inverse matrix of the matrix during the 3D to 2D space transformation; according to the latest The offset between the pose information and the pose information corresponding to the 3D image is used to calculate the new position corresponding to each pixel in the 3D space; using the original position and the new position of each pixel in the 3D space, the 3D is calculated The motion vector corresponding to each pixel of the image.
  • the vector calculation module 1003 is used to obtain the horizontal position information, vertical position information, and depth information of each pixel of the original 2D image to obtain the position vector of each pixel of the original 2D image; using 3D When transforming to 2D space, the inverse matrix of the matrix and the position vector of each pixel point calculate the original position of each pixel point of the 3D image corresponding to the original 2D image in the 3D space.
  • the target frame generation module 1004 is used to select some pixels from the pixels of the original 2D image to obtain key pixels; for each selected key pixel, the size and direction indicated by the motion vector Perform position transformation and fill in the pixels of the vacant area that appears after the position transformation to obtain the target frame.
  • the target frame generation module 1004 is used to determine the vacant area in the area enclosed by the mesh vertices after the position transformation; the pixel points of the vacant area are filled with interpolation.
  • the sampling module 1002 samples the IMU data of the inertial measurement unit of the VR system to obtain the latest pose information of the user's head.
  • the image display control device in the VR system shown in FIG. 10 corresponds to the aforementioned image display control method in the VR system, so the functions implemented by the image display control device in the VR system in this embodiment
  • the functions implemented by the image display control device in the VR system in this embodiment please refer to the related description of the foregoing embodiments of the present invention, which will not be repeated here.
  • FIG. 11 is a schematic structural diagram of a VR head-mounted device according to an embodiment of the present invention.
  • the VR headset includes a memory 1101 and a processor 1102.
  • the memory 1101 and the processor 1102 are connected through an internal bus 1103.
  • the memory 1101 stores program instructions that can be executed by the processor 1102. When executed by the processor 1102, the above-mentioned image display control method in the VR system can be realized.
  • the logic instructions in the above-mentioned memory 1101 can be implemented in the form of software functional units and sold or used as an independent product, and can be stored in a computer-readable storage medium.
  • the technical solution of the present invention essentially or part of the contribution to the existing technology or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium, including Several instructions are used to enable a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code .
  • Another embodiment of the present invention provides a computer-readable storage medium that stores computer instructions, and the computer instructions cause the computer to perform the method described above.
  • the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.
  • computer usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • each flow and/or block in the flowchart and/or block diagram and a combination of the flow and/or block in the flowchart and/or block diagram may be implemented by computer program instructions.
  • These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processing machine, or other programmable data processing device to produce a machine that enables the generation of instructions executed by the processor of the computer or other programmable data processing device

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Computer Graphics (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本发明公开了一种VR系统中图像显示的控制方法、装置及VR头戴设备，方法包括：监测VR系统中图像帧的同步信号并获取原始2D图像；在下一个同步信号到来之前的预设时间点，对传感器数据进行采样得到被跟踪对象最新的位姿信息，将原始2D图像转换为对应的3D图像，根据最新的位姿信息以及3D图像对应的位姿信息计算3D图像的各像素点对应的运动矢量，基于运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧；当下一个同步信号到来时触发目标帧的显示。本发明实施例扩展了Timewarp算法适用的场景，增强了图像的真实感，提高了图像显示效果。

Description

VR系统中图像显示的控制方法、装置及VR头戴设备 技术领域
本发明涉及虚拟现实技术领域，具体涉及一种VR系统中图像显示的控制方法、装置及VR头戴设备。
发明背景
VR头戴设备(Head Mounted Display，简称HMD)是通过姿态传感器实时连续跟踪用户的头部位姿，然后利用位姿信息从虚拟的3D世界中渲染出两幅2D图像，分别为左眼与右眼应当看到的画面，并将这两幅图像显示到屏幕上，用户的视网膜接收到图像内容后，大脑的内建机制会解析出立体感，从而实现VR(Virtual Reality，虚拟现实)效果。在整个过程中，一个核心指标是系统延迟，系统延迟是指从取得用户头部位姿开始，到渲染出的画面完全呈现在屏幕上为止的时间长度。过长的系统延迟会导致用户的生理感官与眼睛接收到的画面不符，产生晕动症症状。
在现代的硬件和软件系统中，3D渲染的流程为流水线式设计，参见图1，从第一帧开始，输入的数据帧U1至U4(仅作示例)流经中央处理器CPU1和CPU2的多个线程以及GPU(Graphics Processing Unit，图形处理器)和屏幕，最终经过屏幕扫描在屏幕上产生一个输出。流水线式的设计提高了各组件的利用率，确保了较高的吞吐量，与此同时，也带来了较高的系统延迟，参见图1，从对IMU采样得到当前姿态到第一帧在屏幕上显示发出的光子被用户观察到，一个典型的面向吞吐量的渲染过程，带来的系统延迟(Motion to Photon latency)至少有四个屏幕刷新周期，而在刷新频率为90Hz的屏幕上，将耗时44.4毫秒，远超过人体生理上能够容忍的极限18毫秒。
针对此问题，有一种时间扭曲Timewarp算法，被用来扭曲(例如，位移、旋转、调整或重投影)图像帧，以校正在渲染帧之后发生的头部运动或平移，由此降低系统延迟，但目前Timewarp算法出于简单起见，仅针对3自由度(3DOF)位姿的图像进行处理并且Timewarp算法处理后图像真实感不足，不能满足实际需求。
发明内容
本发明实施例提供了一种VR系统中图像显示的控制方法、装置及VR头戴设备，扩展了Timewarp算法适用的场景，使其可应用于6自由度位姿的图像显示控制中并增强了图像的真实感，提高了图像显示效果。
根据本申请的一个方面，提供了一种VR系统中图像显示的控制方法，包括：
监测VR系统中图像帧的同步信号并获取原始2D图像；
在下一个同步信号到来之前的预设时间点，对传感器数据进行采样得到被跟踪对象最新的位姿信息，其中，位姿信息包括指示被跟踪对象旋转的信息以及指示被跟踪对象平移的信息；
将原始2D图像转换为对应的3D图像，根据最新的位姿信息以及3D图像对应的位姿信息计算3D图像的各像素点对应的运动矢量，
基于运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧；
当下一个同步信号到来时触发目标帧的显示。
根据本申请的另一个方面，提供了一种VR系统中图像显示的控制装置，包括：
获取模块，用于监测VR系统中图像帧的同步信号并获取原始2D图像；
采样模块，用于在下一个同步信号到来之前的预设时间点，对传感器数据进行采样得到被跟踪对象最新的位姿信息，其中，位姿信息包括指示被跟踪对象旋转的信息以及指示被跟踪对象平移的信息；
矢量计算模块，用于将原始2D图像转换为对应的3D图像，根据最新的位姿信息以及3D图像对应的位姿信息计算3D图像的各像素点对应的运动矢量；
目标帧生成模块，用于基于运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧；
触发模块，用于当下一个同步信号到来时触发目标帧的显示。
根据本申请的又一个方面，提供了一种VR头戴设备，包括：存储器和处理器，存储器和处理器之间通过内部总线通讯连接，存储器存储有能够被处理器执行的程序指令，程序指令被处理器执行时能够实现本申请一个方面的的方法。
应用本发明实施例的VR系统中图像显示的控制方法和装置，监测VR系统中图像帧的同步信号并获取原始2D图像，在下一个同步信号到来之前的预设时间点，对传感器数据进行采样得到被跟踪对象最新的指示被跟踪对象旋转、平 移信息的位姿信息，计算3D图像的各像素点对应的运动矢量，基于运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧，当下一个同步信号到来时触发目标帧的显示。从而与现有技术相比，扩展了Timewarp算法适用的范围和场景，满足了6DOF位姿变化的显示控制需求，并且基于各像素点对应的运动矢量计算像素点的新位置并对空缺区域的像素点处进行填充得到目标帧，也提升了画面真实度和显示效果。本发明实施例的VR头戴设备将时间扭曲Timewarp应用于被跟踪对象6DOF位姿变化的场景，缩短了系统延迟并保证了画面的真实度和显示效果，满足了实际需求，增强了产品的市场竞争力。
附图简要说明
图1是系统延迟产生的原理示意图；
图2是本发明实施例的应用Timewrap算法进行图像显示控制的原理示意图；
图3a是获取的被跟踪对象位置移动之前的图像；
图3b是图3a中矩形框的放大示意图；
图4a是获取的被跟踪对象位置移动之后的图像；
图4b是图4a中矩形框的放大示意图；
图5是本发明一个实施例的VR系统中图像显示的控制方法流程示意图；
图6是本发明一个实施例对图4a所示图像添加网格后的示意图；
图7a是对图6所示图像基于运动矢量进行位置变换后的示意图；
图7b是对图7a中方框所示部分的放大后的示意图
图8a是对图6所示图像进行位置变换以及填充后的示意图；
图8b是对图8a中矩形框所示部分的放大后的示意图；
图9a是应用本发明实施例的方法对原始2D图像填充处理后的示意图；
图9b是图9a中矩形框所示部分的放大后的示意图；
图10是本发明一个实施例的VR系统中图像显示的控制装置的框图；
图11是本发明一个实施例的VR头戴设备的结构示意图。
具体实施方式
为使本发明的上述目的、特征和优点能够更加明显易懂，下面结合附图和具体实施方式对本发明作进一步详细的说明。显然，所描述的实施例是本发明 一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
本发明的设计构思在于，对现有的利用Timewrap算法进行改进，拓展Timewrap算法的应用范围，使其可以应用于6自由度位姿变化的场景中，满足实际需求。
为了便于更好理解本发明实施例的技术方案，这里对Timewrap算法以及Timewrap算法应用于3自由度位姿变化场景的现有技术进行说明。
要实现在屏幕上产生一幅画面，在现代的渲染引擎以及GPU上，基本需要经过以下完整过程：
1.输入(Input):收集用户的所有输入，比如鼠标、键盘等等各种外部设备的输入数据。图1中的IMU(Inertial Measurement Unit，惯性测量单元)，是一种采集姿态数据的传感器。
2.更新(Update):根据用户的输入，对3D世界中的物体进行状态更新，如更新相机Camera的位置和方向、更新所控制的游戏应用中角色的移动、更新游戏应用中其它非玩家控制角色及物体的移动、变化等。
3.提交(Commit)：将整个3D世界物体更新后的状态，转换为一系列的渲染指令，并提交给GPU准备进行渲染。
4.渲染(Render)：GPU对上一步(步骤3)生成的一系列渲染指令一条一条的执行，最终生成一张2D图像供用户观察。
以上是“一帧图像”的完整渲染过程。
参见图1，图1中的I表示Input阶段，U表示Update阶段，C表示Commit阶段，R表示Render阶段。图1中示意了CPU1中处理的四帧，处于更新阶段，分别为U1、U2至U4，CPU2中处理的三帧处于提交阶段，分别为C1、C2至C3，GPU中处理的一帧处于渲染阶段。
由于根据屏幕刷新机制，在将2D图像推送到屏幕时，需要先等待下一次的同步信号到来的当下一次的同步信号到来时才能推送，所以在GPU渲染完当前的一帧的指令后，没有任务处理，它将进入Idle(参见图1中的R1后的Idle)空闲状态。
结合Timewarp来说明，主要包括以下几部分，参见图1：
步骤一，update线程对位姿采样后，更新世界状态，然后提交流水线的时间。 这一阶段耗时1个屏幕刷新周期。如刷新频率为60Hz的屏幕，一个刷新周期为1000/60≈33.3ms，刷新频率为90Hz的屏幕，一个刷新周期为1000/90≈11.1ms。
步骤二，render线程根据最新的3D世界状态以及位姿，提交渲染指令到GPU的时间。这一阶段耗时1个屏幕刷新周期。
步骤三，将3D图像进行光栅化(即渲染)生成2D图像，并等待屏幕同步信号的时间。光栅化的耗时取决于当前位姿下的场景复杂度，通常需要由内容设计人员确保耗时小于1个屏幕刷新周期，才能保证稳定的帧率，这一步至少耗时1个屏幕刷新周期。若光栅化占用了n.m个刷新周期，则这一步整体耗时为，向上取整得到(n+1)个屏幕刷新周期。
步骤四，2D图像数据传输到屏幕，并扫描到屏幕的时间加上实际发射光子的时间，共耗时1个刷新周期。
因此，一个典型的面向吞吐量的渲染过程，带来的系统延迟至少有4个屏幕刷新周期，在90Hz屏幕上将耗时44.4ms，远超过人体生理上能够容忍的极限18ms。
针对此问题，Timewarp算法在上面步骤三里对图像光栅化之后，以及等待下一个屏幕同步信号之前，插入一些额外的步骤，将步骤三变为下列子步骤以降低系统延迟，参见图2，主要包括如下操作：
1)光栅化生成2D图像，并缓存为原始图像；
2)等待一定时间，直到快到达下一次同步信号之前；
3)对位姿进行一次最新采样，得到当前时刻用户头部的位姿；
4)利用最新的位姿，对缓存的原始图像进行变换，生成新的姿态下的所应该看到的图像(即目标图像)；
5)在紧接着的同步信号到来时，将目标图像刷新到屏幕上显示。
Timewarp之所以成立是因为：对2D图像进行变换的复杂度，仅取决于图像分辨率，且远小于3D场景的光栅化时间。通常这一过程耗时为2～3ms，因此在图像刷新到屏幕上之前，只要预留4ms的时间来用最新位姿变换图像，就可以达到整体降低系统延迟的效果。
如图2所示，应用Timewarp算法后，系统延迟从4个刷新周期降低到Timewarp时间+1个刷新周期，对刷新频率为90Hz的屏幕而言，共耗时为11.1+4＝15.1(ms)，15.1ms<18ms，满足了需求。
需要说明的是，图2中示意的U1、U2、C1、C2等与图1中的符号的含义 相同，所以可以参见前述图1部分的相关说明，这里不再赘述，仅对图2与图1的不同之处作简要说明：图2中，Idle(空闲状态)后面的I表示再取一次用户的输入Input(即图2中示意的对IMU采样得到最新姿态)，随后的W表示执行实际的Timewarp过程(即本申请实施例所提供的的时间扭曲填充帧的过程，见图2中示意的箭头连接的Timewrap)，再之后的P表示Present，即在下一次同步信号来临时，将变换后的目标2D图像实际推送(展示)到屏幕上。
再来看自由度，任何一个没有受约束的物体，在空间均具有6个独立的运动，即6自由度(Degrees of Freedom，简称DOF)。以VR设备为例，在直角坐标系oxyz中VR设备可以有3个平移运动和3个旋转运动。3个平移运动分别是沿x，y，z轴的平移运动；3个旋转运动分别是绕x，y，z轴的转动。习惯上把上述6个独立运动称为6个自由度。
基于3自由度即3DOF的Timewarp算法在实现时仅考虑方向旋转，不考虑位置平移，原因是仅处理方向旋转时，2D图像中景物遮挡关系不发生变化这样Timewarp算法较容易实现，但带来的问题是真实感不足。而如果将Timewarp算法应用于6DOF的场景则复杂的多，因为既需要考虑旋转又需要考虑位移带来的景物遮挡关系变化。
结合图3a到图4b对上述6DOF位姿场景中图像的变化进行说明。图3a是获取的被跟踪对象位置移动之前的图像，这里的被跟踪对象例如是佩戴了VR头戴设备的佩戴者的头部，图3a示意了在头部位置没有移动之前，VR头戴设备上的摄像头拍摄的画面，图3b示意了将图3a中的黑色矩形框放大之后的图像，可以看到墙壁遮挡住了长椅的一角。
接着看图4a，佩戴者的头部向左移动之后，VR头戴设备上的摄像头随之向左移动，摄像头拍摄的画面中黑色矩形框中暴露出了移动之前被遮挡区域。参见图4b，在对图4a中的黑色矩形框放大之后，可以看到在该矩形框所框住的范围内有两条带状黑色区域，这一部分就是由于用户头部向左平移运动之后，原来未显示的区域呈现出的样子。由于位移的存在，景物遮挡关系变化，有一些在原始图像(移动之前的图像)中被遮挡的内容在目标图像(移动之后的图像)中将被暴露在视野里，但这部分内容在原始图像中没有保存相应的信息，必须要通过一定的算法和方式来进行重建。
现有技术的Timewrap算法中不考虑6DOF位姿变化时的图像处理，所以未给出对应的解决方案。本发明的实施例正是针对上述技术问题，扩展Timewrap 算法在6DOF位姿变化下的应用，填补这一空白，同时，提高重建的图像的真实度，保证图像显示效果。
图5是本发明一个实施例的一种VR系统中图像显示的控制方法，参见图5，该方法包括下列步骤：
步骤S501，监测VR系统中图像帧的同步信号并获取原始2D图像；
步骤S502，在下一个同步信号到来之前的预设时间点，对传感器数据进行采样得到被跟踪对象最新的位姿信息，其中，位姿信息包括指示被跟踪对象旋转的信息以及指示被跟踪对象平移的信息；
步骤S503，将原始2D图像转换为对应的3D图像，根据最新的位姿信息以及3D图像对应的位姿信息计算3D图像的各像素点对应的运动矢量，
步骤S504，基于运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧；
步骤S505，当下一个同步信号到来时触发目标帧的显示。
由图1所示可知，本实施例的中，在获取原始2D图像后等待时机，即下一个同步信号到来之前的预设时间点，在该预设时间点上获取头部最新的包含方向旋转信息以及位置平移信息的位姿信息，根据这样的位姿信息以及变换到3D空间中的图像计算3D图像上各像素点的运动矢量，基于矢量调整2D图像上像素点的位置，在位置调整后出现的空缺区域(参见图4b中黑色条状区域)中进行填充，补足信息得到目标帧。当下一个同步信号到来时显示目标帧。从而，一方面，从采样得到最新的位姿到根据该位姿生成并显示目标帧时间大大缩短，即显著降低了系统延迟。另一方面，可以在6DOF位姿变化的场景中使用Timewrap算法对目标帧进行恢复和重建，提高了图像的真实感，保证了显示效果，拓宽了VR系统的应用范围，提升了产品的市场竞争力。
以下结合一个具体的应用场景对本发明实施例的实现步骤进行说明。
可以理解，本发明实施例的VR系统中图像显示的控制方法应用于虚拟现实VR头戴设备中，对于该头戴设备的佩戴者而言，其既可以平移运动也可以转动，减少了对佩戴者的移动限制，提高了用户的满意度。同时，根据本发明实施例的方法对由于头部位置平移导致的图像信息缺失进行填充生成目标帧，显示目标帧也保证了图像的真实感和图像显示效果。注：被跟踪对象不限于头部，也可以是手部。
概括而言，本发明实施例的VR系统中图像显示的控制方法是在光栅化生成 原始2D图像的同时，缓存每个像素对应的深度信息(Z-buffer)，随后利用深度信息将每个像素变换回3D空间，并在3D空间中计算出相应的运动矢量(Motion Vector)，最后将原始2D图像划分为密集的网格状，并应用运动矢量对原始2D图像中的各个像素进行变换，同时利用网格间的像素插值完成空缺区域的填充，得到目标帧。
具体而言，应用本发明实施例的VR系统中图像显示的控制方法，监测VR系统中图像帧的同步信号，在下一个同步信号到来之前的预设时间点，比如，下一个同步信号到来前的5ms时的这一时间点上(具体时间点取决于Timewarp算法执行的时间，进一步取决于原始图像的分辨率以及GPU硬件的计算能力，可根据实际测试确定)，执行一次Timewarp，流程如下：
首先，获取原始2D图像；
这里的获取原始2D图像是通过将世界空间中深度相机拍摄得到的3D图像光栅化生成2D原始图像，光栅化同时生成有Z-buffer深度信息，本实施例中保存原始图像的深度信息，并记录此刻所使用的6DOF位姿(O，P)。
注：Z-buffer保存了原始图像中每个像素点的深度信息，以用来恢复像素的3D坐标。6DOF位姿(O，P)中的O为指示使用者头部运动的旋转即方向信息(u，v，w)，P为指示使用者头部位置平移的位置信息(x，y，z)。使用者头部最新的位姿信息可以通过对VR系统的惯性测量单元IMU数据进行采样得到。
其次，计算运动矢量。
这里的计算运动矢量是将2D像素逆变换回3D空间得到对应的运动矢量(Motion Vector)。
例如，利用3D到2D空间变换时矩阵的逆矩阵，计算原始2D图像对应的3D图像各像素点在3D空间中对应的原位置，根据最新的位姿信息以及3D图像对应的位姿信息之间的偏移量，计算出3D空间中各像素点对应的新位置，利用各像素点在3D空间中的原位置与新位置，计算得到3D图像各像素点对应的运动矢量。
具体的，对光栅化时所使用的空间变换矩阵生成逆矩阵M。获取原始2D图像的各像素点的水平位置信息、竖直位置信息以及深度信息，得到原始2D图像的各像素点的位置向量；利用3D到2D空间变换时矩阵的逆矩阵与各像素点的位置向量，计算出原始2D图像对应的3D图像各像素点在3D空间中对应的原 位置。
也就是说，从原始图像读取像素点的水平位置信息、竖直位置信息，即坐标(x,y)，从Z-buffer中读取像素的深度，即z坐标，得到每个像素点在2D空间(即图像空间)下位置向量即(x，y，z)坐标。
利用逆矩阵M对位置向量(x，y，z)执行逆变换，即，进行矩阵与向量乘法(x’，y’，z’)＝M×(x，y，z)，得到原3D空间中像素点的坐标。
需要说明的是，2D图像中的像素点与3D图像中的像素点实质上描述的客观对象是相同的，只是不同空间中描述方式以及所反映的物理特性存在不同。
接下来，利用2D图像对应的位姿信息(O,P)以及采样得到的最新位姿信息(O’,P’)的变化，计算出2D图像上每个像素点(x’，y’，z’)的新位置(x”,y”,z”)。
最后根据每个像素点的原位置(x’，y’，z’)与新位置(x”,y”,z”)得到运动矢量(n，m，k)。
再次，将运动矢量应用到原始2D图像上。
本实施例中，将运动矢量应用到原始2D图像上，是基于运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处填充颜色，得到目标帧。
如果想要达到最佳的图像效果，可以对根据每个像素点的运动矢量对原始2D图像的每个像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处填充颜色，得到目标帧。但是，这样做的缺点也是很明显的，即，计算强度大，效率较低。另外，图像分辨率低时也会浪费系统资源。所以实际应用中可根据图像分辨率、计算强度、图像效果进行权衡，选取部分像素点进行位置变换。
在本发明的一个实施例中，从原始2D图像的像素点中选取部分像素点，得到关键像素点；对选取的各关键像素点，基于运动矢量指示的大小和方向进行位置变换。
这里的从原始2D图像的像素点中选取部分关键像素点包括：将原始2D图像划分为多个规则的网格，并选取网格顶点对应的像素点作为关键像素点。参见图6，为原始图像创建规则的网格，例如200x100网格，选取网格顶点对应的像素点作为关键像素点。注：网格越密集，则计算量越大，相应的生成的画面质量越好。
划分了网格之后，对选取的网格顶点对应的像素点，基于运动矢量指示的大小和方向进行位置变换。图7a是对图6所示图像基于运动矢量进行位置变换后的示意图；图7b是对图7a中方框所示部分的放大后的示意图，参见图7a和7b，对图6所示的网格顶点的像素点位置变换后，由于网格顶点中间围成的区域里部分像素点没有色彩信息，所以出现了空缺区域。
最后，空缺区域的像素点处进行填充。
本实施例中，在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧。具体的，确定位置变换后网格顶点合围而成的区域中的空缺区域；对空缺区域的像素点利用插值进行填充。例如，顶点与顶点间的像素位置通过GPU内建的线性插值完成计算达到重建的效果，图8a是对图6所示图像进行位置变换以及填充后的示意图；图8b是对图8a中矩形框所示部分的放大后的示意图，如图8a和图8b所示，网格完成位置变换的同时，中间空缺区域的像素将被GPU自动填充，由于是网格图，图8b中未体现出较易被人眼识别的色彩效果，但可以明显观察到填充后的网格产生了拉伸的效果，参见图8b中白色圆圈内的部分所示意的拉伸效果。
需要说明的是，实际应用过程中，将运动矢量应用到原始2D图像上与空缺区域的像素点处进行填充的步骤之间没有严格的先后顺序，可以利用GPU的内建的机制同步完成两者。
本次Timewarp变换过程结束，得到最终的目标图像，目标图像的内容体现了用户最新的头部位姿下所观察到的图像。后续目标图像将被刷新到屏幕上进行显示。
图9a是应用本发明实施例的方法对原始2D图像填充处理后的示意图，图9b是图9a中矩形框的放大示意图。将图9b与图4b进行对比可知，本方法填充的内容更真实可行，被遮挡的区域为木椅的纹理材质，本方法采用了木椅纹理(参见图9b中的白色虚线矩形框)进行正确填充，墙壁和木椅的边缘仍然保持竖直的形态，使画面真实度更高。并且与现有的错误采用墙壁的纹理来填充空缺的区域，导致墙壁边缘整体产生向左的弯曲，填充效果不真实的技术方案相比，提升了画面真实度。
由此，本发明实施例的VR系统中图像显示控制方法，扩展了Timewarp的应用场景，方法简单易操作，计算强度更低，且无需附加额外的硬件资源,运行效率高，保证了画面的真实感提升了用户体验。
与前述VR系统中图像显示的控制方法，同属于一个技术构思，本发明实施例还提供了一种VR系统中图像显示的控制装置，图10是本发明一个实施例的VR系统中图像显示的控制装置的框图，VR系统中图像显示的控制装置1000，包括：
获取模块1001，用于监测VR系统中图像帧的同步信号并获取原始2D图像；
采样模块1002，用于在下一个同步信号到来之前的预设时间点，对传感器数据进行采样得到被跟踪对象最新的位姿信息，其中，位姿信息包括指示被跟踪对象旋转的信息以及指示被跟踪对象平移的信息；
矢量计算模块1003，用于将原始2D图像转换为对应的3D图像，根据最新的位姿信息以及3D图像对应的位姿信息计算3D图像的各像素点对应的运动矢量；
目标帧生成模块1004，用于基于运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧；
触发模块1005，用于当下一个同步信号到来时触发目标帧的显示。
在本发明一个实施例中，目标帧生成模块1004，具体用于将原始2D图像划分为多个规则的网格(如将原始2D图像划分为200x100个规则的网格)，并选取网格顶点对应的像素点作为关键像素点，对选取的各关键像素点，基于运动矢量指示的大小和方向进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧。
在本发明的一个实施例中，矢量计算模块1003具体用于利用3D到2D空间变换时矩阵的逆矩阵，计算原始2D图像对应的3D图像各像素点在3D空间中对应的原位置；根据最新的位姿信息以及3D图像对应的位姿信息之间的偏移量，计算出3D空间中各像素点对应的新位置；利用各像素点在3D空间中的原位置与新位置，计算得到3D图像各像素点对应的运动矢量。
在本发明的一个实施例中，矢量计算模块1003用于获取原始2D图像的各像素点的水平位置信息、竖直位置信息以及深度信息，得到原始2D图像的各像素点的位置向量；利用3D到2D空间变换时矩阵的逆矩阵与各像素点的位置向量，计算出原始2D图像对应的3D图像各像素点在3D空间中对应的原位置。
在本发明一个实施例中，目标帧生成模块1004，用于从原始2D图像的像素点中选取部分像素点，得到关键像素点；对选取的各关键像素点，基于运动矢量指示的大小和方向进行位置变换，并在位置变换后出现的空缺区域的像素 点处进行填充，得到目标帧。
在本发明一个实施例中，目标帧生成模块1004用于确定位置变换后网格顶点合围而成的区域中的空缺区域；对空缺区域的像素点利用插值进行填充。
在本发明一个实施例中，采样模块1002，对VR系统的惯性测量单元IMU数据进行采样得到使用者头部最新的位姿信息。
需要说明的是，图10所示的VR系统中图像显示的控制装置是与前述VR系统中图像显示的控制方法相对应的，因而本实施例中对VR系统中图像显示的控制装置实现的功能的举例说明可以参见本发明前述实施例的相关说明，这里不再赘述。
图11是本发明一个实施例的VR头戴设备的结构示意图。如图11所示，该VR头戴设备包括存储器1101和处理器1102，存储器1101和处理器1102之间通过内部总线1103通讯连接，存储器1101存储有能够被处理器1102执行的程序指令，程序指令被处理器1102执行时能够实现上述的VR系统中图像显示的控制方法。
此外，上述的存储器1101中的逻辑指令可以通过软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
本发明的另一个实施例提供一种计算机可读存储介质，计算机可读存储介质存储计算机指令，计算机指令使所述计算机执行上述的方法。
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的 流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图的一个流程或多个流程和/或方框图的一个方框或多个方框中指定的功能的装置。
需要说明的是术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述，仅为本发明的具体实施方式，在本发明的上述教导下，本领域技术人员可以在上述实施例的基础上进行其他的改进或变形。本领域技术人员应该明白，上述的具体描述只是更好的解释本发明的目的，本发明的保护范围以权利要求的保护范围为准

Claims (16)

  1. 一种VR系统中图像显示的控制方法，其特征在于，包括：
    监测VR系统中图像帧的同步信号并获取原始2D图像；
    在下一个同步信号到来之前的预设时间点，对传感器数据进行采样得到被跟踪对象最新的位姿信息，其中，所述位姿信息包括指示被跟踪对象旋转的信息以及指示被跟踪对象平移的信息；
    将所述原始2D图像转换为对应的3D图像，根据最新的位姿信息以及3D图像对应的位姿信息计算3D图像的各像素点对应的运动矢量，
    基于所述运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧；
    当所述下一个同步信号到来时触发所述目标帧的显示。
  2. 根据权利要求1所述的方法，其特征在于，根据最新的位姿信息以及3D图像对应的位姿信息计算3D图像的各像素点对应的运动矢量包括：
    利用3D到2D空间变换时矩阵的逆矩阵，计算原始2D图像对应的3D图像各像素点在3D空间中对应的原位置；
    根据最新的位姿信息以及3D图像对应的位姿信息之间的偏移量，计算出3D空间中各像素点对应的新位置；
    利用各像素点在3D空间中的原位置与新位置，计算得到3D图像各像素点对应的运动矢量。
  3. 根据权利要求2所述的方法，其特征在于，利用3D到2D空间变换时矩阵的逆矩阵，计算原始2D图像对应的3D图像各像素点在3D空间中对应的原位置包括：
    获取原始2D图像的各像素点的水平位置信息、竖直位置信息以及深度信息，得到原始2D图像的各像素点的位置向量；
    利用3D到2D空间变换时矩阵的逆矩阵与各像素点的位置向量，计算出原始2D图像对应的3D图像各像素点在3D空间中对应的原位置。
  4. 根据权利要求1所述的方法，其特征在于，基于所述运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处填充颜色，得到目标帧包括：
    从原始2D图像的像素点中选取部分像素点，得到关键像素点；
    对选取的各关键像素点，基于运动矢量指示的大小和方向进行位置变换， 并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧。
  5. 根据权利要求4所述的方法，其特征在于，所述从原始2D图像的像素点中选取部分关键像素点包括：
    将原始2D图像划分为多个规则的网格，并选取网格顶点对应的像素点作为关键像素点。
  6. 根据权利要求5所述的方法，其特征在于，所述在位置变换后出现的空缺区域的像素点处进行填充包括：
    确定位置变换后网格顶点合围而成的区域中的空缺区域；
    对空缺区域的像素点利用插值进行填充。
  7. 根据权利要求1所述的方法，其特征在于，所述对传感器数据进行采样得到被跟踪对象最新的位姿信息包括：
    对VR系统的惯性测量单元IMU数据进行采样得到使用者头部最新的位姿信息。
  8. 根据权利要求5所述的方法，其特征在于，将原始2D图像划分为200x100个规则的网格。
  9. 一种VR系统中图像显示的控制装置，其特征在于，包括：
    获取模块，用于监测VR系统中图像帧的同步信号并获取原始2D图像；
    采样模块，用于在下一个同步信号到来之前的预设时间点，对传感器数据进行采样得到被跟踪对象最新的位姿信息，其中，所述位姿信息包括指示被跟踪对象旋转的信息以及指示被跟踪对象平移的信息；
    矢量计算模块，用于将所述原始2D图像转换为对应的3D图像，根据最新的位姿信息以及3D图像对应的位姿信息计算3D图像的各像素点对应的运动矢量；
    目标帧生成模块，用于基于所述运动矢量对原始2D图像的像素点进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧；
    触发模块，用于当所述下一个同步信号到来时触发所述目标帧的显示。
  10. 根据权利要求9所述的装置，其特征在于，
    矢量计算模块，具体用于利用3D到2D空间变换时矩阵的逆矩阵，计算原始2D图像对应的3D图像各像素点在3D空间中对应的原位置；根据最新的位姿信息以及3D图像对应的位姿信息之间的偏移量，计算出3D空间中各像素点对应的新位置；利用各像素点在3D空间中的原位置与新位置，计算得到3D图像各像素点对应的运动矢量。
  11. 根据权利要求10所述的装置，其特征在于，
    矢量计算模块，具体用于获取原始2D图像的各像素点的水平位置信息、竖直位置信息以及深度信息，得到原始2D图像的各像素点的位置向量；利用3D到2D空间变换时矩阵的逆矩阵与各像素点的位置向量，计算出原始2D图像对应的3D图像各像素点在3D空间中对应的原位置。
  12. 根据权利要求9所述的装置，其特征在于，
    目标帧生成模块，用于从原始2D图像的像素点中选取部分像素点，得到关键像素点；对选取的各关键像素点，基于运动矢量指示的大小和方向进行位置变换，并在位置变换后出现的空缺区域的像素点处进行填充，得到目标帧。
  13. 根据权利要求9所述的装置，其特征在于，
    目标帧生成模块，用于将原始2D图像划分为多个规则的网格，并选取网格顶点对应的像素点作为关键像素点。
  14. 根据权利要求13所述的装置，其特征在于，
    目标帧生成模块，进一步用于确定位置变换后网格顶点合围而成的区域中的空缺区域；对空缺区域的像素点利用插值进行填充。
  15. 根据权利要求13所述的装置，其特征在于，目标帧生成模块，用于将原始2D图像划分为200x100个规则的网格。
  16. 一种VR头戴设备，其特征在于，包括：存储器和处理器，所述存储器和所述处理器之间通过内部总线通讯连接，所述存储器存储有能够被所述处理器执行的程序指令，所述程序指令被所述处理器执行时能够实现权利要求1-8任一项所述的的方法。
PCT/CN2019/098833 2018-12-29 2019-08-01 Vr系统中图像显示的控制方法、装置及vr头戴设备 Ceased WO2020134085A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/631,136 US20210063735A1 (en) 2018-12-29 2019-08-01 Method and device for controlling image display in a vr system, and vr head mounted device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811646123.7A CN109739356B (zh) 2018-12-29 2018-12-29 Vr系统中图像显示的控制方法、装置及vr头戴设备
CN201811646123.7 2018-12-29

Publications (1)

Publication Number Publication Date
WO2020134085A1 true WO2020134085A1 (zh) 2020-07-02

Family

ID=66362794

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/098833 Ceased WO2020134085A1 (zh) 2018-12-29 2019-08-01 Vr系统中图像显示的控制方法、装置及vr头戴设备

Country Status (3)

Country Link
US (1) US20210063735A1 (zh)
CN (1) CN109739356B (zh)
WO (1) WO2020134085A1 (zh)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109739356B (zh) * 2018-12-29 2020-09-11 歌尔股份有限公司 Vr系统中图像显示的控制方法、装置及vr头戴设备
CN110221690B (zh) * 2019-05-13 2022-01-04 Oppo广东移动通信有限公司 基于ar场景的手势交互方法及装置、存储介质、通信终端
US11363247B2 (en) * 2020-02-14 2022-06-14 Valve Corporation Motion smoothing in a distributed system
CN113467602B (zh) * 2020-03-31 2024-03-19 中国移动通信集团浙江有限公司 Vr显示方法及系统
US11099396B2 (en) * 2020-04-10 2021-08-24 Samsung Electronics Company, Ltd. Depth map re-projection based on image and pose changes
CN112053410A (zh) * 2020-08-24 2020-12-08 海南太美航空股份有限公司 一种基于矢量图形绘制的图像处理方法、系统及电子设备
CN112561962A (zh) * 2020-12-15 2021-03-26 北京伟杰东博信息科技有限公司 一种目标对象的跟踪方法及系统
CN112785530B (zh) * 2021-02-05 2024-05-24 广东九联科技股份有限公司 用于虚拟现实的图像渲染方法、装置、设备及vr设备
CN113031783B (zh) * 2021-05-27 2021-08-31 杭州灵伴科技有限公司 运动轨迹更新方法、头戴式显示设备和计算机可读介质
CN113473105A (zh) * 2021-06-01 2021-10-01 青岛小鸟看看科技有限公司 图像同步方法、图像显示及处理设备、及图像同步系统
CN119148378B (zh) * 2023-06-14 2026-04-03 歌尔股份有限公司 图像显示方法、设备及计算机可读存储介质
WO2025160800A1 (en) * 2024-01-31 2025-08-07 Qualcomm Incorporated General distortion correction for rolling scan displays

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105404393A (zh) * 2015-06-30 2016-03-16 指点无限（美国）有限公司 低延迟虚拟现实显示系统
US20170018121A1 (en) * 2015-06-30 2017-01-19 Ariadne's Thread (Usa), Inc. (Dba Immerex) Predictive virtual reality display system with post rendering correction
US10043318B2 (en) * 2016-12-09 2018-08-07 Qualcomm Incorporated Display synchronized image warping
CN108921951A (zh) * 2018-07-02 2018-11-30 京东方科技集团股份有限公司 虚拟现实图像显示方法及其装置、虚拟现实设备
CN109739356A (zh) * 2018-12-29 2019-05-10 歌尔股份有限公司 Vr系统中图像显示的控制方法、装置及vr头戴设备

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106454322A (zh) * 2016-11-07 2017-02-22 金陵科技学院 Vr的图像处理系统及其方法
CN106782260B (zh) * 2016-12-06 2020-05-29 歌尔科技有限公司 用于虚拟现实运动场景的显示方法及装置
CN107368192B (zh) * 2017-07-18 2021-03-02 歌尔光学科技有限公司 Vr眼镜的实景观测方法及vr眼镜
CN107491173A (zh) * 2017-08-16 2017-12-19 歌尔科技有限公司 一种体感模拟控制方法及设备
US10139899B1 (en) * 2017-11-30 2018-11-27 Disney Enterprises, Inc. Hypercatching in virtual reality (VR) system
CN108170280B (zh) * 2018-01-18 2021-03-26 歌尔光学科技有限公司 一种vr头戴设备及其画面显示方法、系统、存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105404393A (zh) * 2015-06-30 2016-03-16 指点无限（美国）有限公司 低延迟虚拟现实显示系统
US20170018121A1 (en) * 2015-06-30 2017-01-19 Ariadne's Thread (Usa), Inc. (Dba Immerex) Predictive virtual reality display system with post rendering correction
US10043318B2 (en) * 2016-12-09 2018-08-07 Qualcomm Incorporated Display synchronized image warping
CN108921951A (zh) * 2018-07-02 2018-11-30 京东方科技集团股份有限公司 虚拟现实图像显示方法及其装置、虚拟现实设备
CN109739356A (zh) * 2018-12-29 2019-05-10 歌尔股份有限公司 Vr系统中图像显示的控制方法、装置及vr头戴设备

Also Published As

Publication number Publication date
US20210063735A1 (en) 2021-03-04
CN109739356B (zh) 2020-09-11
CN109739356A (zh) 2019-05-10

Similar Documents

Publication Publication Date Title
WO2020134085A1 (zh) Vr系统中图像显示的控制方法、装置及vr头戴设备
JP7442608B2 (ja) 仮想現実および拡張現実ディスプレイシステムのための連続時間ワーピングおよび両眼時間ワーピングおよび方法
JP7262540B2 (ja) 仮想オブジェクトのフィギュア合成方法、装置、電子機器、記憶媒体
EP4036863B1 (en) Human body model reconstruction method and reconstruction system, and storage medium
CN110163943B (zh) 图像的渲染方法和装置、存储介质、电子装置
US10083538B2 (en) Variable resolution virtual reality display system
EP3057066B1 (en) Generation of three-dimensional imagery from a two-dimensional image using a depth map
CN113362442B (zh) 一种虚拟现实图像的渲染方法、存储介质及虚拟现实设备
JP6636163B2 (ja) 画像表示方法、成形そり幕を生成する方法、および頭部装着ディスプレイデバイス
CN109920040B (zh) 显示场景处理方法和装置、存储介质
JP7607759B2 (ja) 画像データ生成装置、表示装置、画像表示システム、画像データ生成方法、および画像表示方法
WO2017003769A1 (en) Low-latency virtual reality display system
JP7353782B2 (ja) 情報処理装置、情報処理方法、及びプログラム
US11461942B2 (en) Generating and signaling transition between panoramic images
WO2023076474A1 (en) Compositor layer extrapolation
JP2001126085A (ja) 画像生成システム、画像表示システム、画像生成プログラムを記録したコンピュータ読み取り可能な記録媒体および画像生成方法
WO2018064287A1 (en) Predictive virtual reality display system with post rendering correction
CN120236005B (zh) 三维头像模型构建方法、系统、装置及存储介质
TW202121344A (zh) 圖像處理方法及裝置、圖像處理設備及儲存媒體
CN115830202B (zh) 一种三维模型渲染方法和装置
Smit et al. An image-warping architecture for vr: Low latency versus image quality
JP4098882B2 (ja) 仮想現実感生成装置及び方法
Hapák et al. Real-time 4D reconstruction of human motion
CN109816765B (zh) 面向动态场景的纹理实时确定方法、装置、设备和介质
CN119420894A (zh) 显示方法、显示装置、头戴显示器及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19906453

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19906453

Country of ref document: EP

Kind code of ref document: A1