CN116958487A - Gesture prediction method, device, apparatus, storage medium, and computer program product - Google Patents

Gesture prediction method, device, apparatus, storage medium, and computer program product Download PDF

Info

Publication number
CN116958487A
CN116958487A CN202211460116.4A CN202211460116A CN116958487A CN 116958487 A CN116958487 A CN 116958487A CN 202211460116 A CN202211460116 A CN 202211460116A CN 116958487 A CN116958487 A CN 116958487A
Authority
CN
China
Prior art keywords
data
gesture
prediction
time
display
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211460116.4A
Other languages
Chinese (zh)
Inventor
欧汉飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202211460116.4A priority Critical patent/CN116958487A/en
Publication of CN116958487A publication Critical patent/CN116958487A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Processing Or Creating Images (AREA)

Abstract

The application discloses a gesture prediction method, a gesture prediction device, gesture prediction equipment, a gesture prediction storage medium and a gesture prediction computer program product, and relates to the technical field of augmented reality. The method includes acquiring object pose data at a first time; converting the object gesture data into the virtual scene to obtain gesture prediction data corresponding to the virtual object and a display time stamp corresponding to the gesture prediction data; storing a correspondence between the gesture prediction data and the display timestamp; acquiring a prediction time stamp at a second moment; and matching the prediction timestamp with the display timestamp in the corresponding relation to obtain target gesture prediction data corresponding to the prediction timestamp, and rendering and displaying the target gesture prediction data, so that excessive delay caused by continuous delay of a subsequent process due to sporadic delay of the process is avoided when the system load is heavy, frame loss is avoided, and probability and influence of frame stuck are reduced.

Description

Gesture prediction method, device, apparatus, storage medium, and computer program product
Technical Field
The present application relates to the field of augmented reality technology, and in particular, to a gesture prediction method, apparatus, device, storage medium, and computer program product.
Background
In an XR (Extended Reality) system, target following refers to establishing an association relationship between an object in the real world and an object in a virtual space. By mapping the object in the real world to a certain position in the virtual space by the target following, when the object in the real world moves or rotates, the object in the virtual space also moves or rotates. To avoid tearing of the display, current XR display systems control the synchronization of XR application rendering and screen refresh with a Vsync (Vertical synchronization, vertical synchronization signal) cadence.
In the related art, an XR application calculates a time stamp to be displayed by an image before rendering a data frame, then transfers the time stamp to an algorithm module through an IPC (Inter-Process Communication ) call of the XR system, predicts the pose of a target object according to the time stamp, then transfers the object pose back to the XR application through the IPC, and after the XR application acquires the predicted pose, continues the drawing of the image of the object and feeds the image into the XR system for composite display.
However, in the above method, XR application is required to transmit a prediction time stamp to the algorithm module through IPC before rendering each frame of data, and then wait for the algorithm module to complete the gesture prediction and return, and then continue the subsequent drawing process. When the system load is heavier, delay happens to the cross-process IPC call, and when the delay exceeds one Vsync period, frame clamping occurs, and frame loss and clamping phenomena are serious.
Disclosure of Invention
The embodiment of the application provides a gesture prediction method, a gesture prediction device, gesture prediction equipment, a storage medium and a computer program product, which can predict the gesture of an entity object acquired by gesture acquisition equipment and the gesture of a virtual object corresponding to the gesture in a virtual scene. The technical scheme is as follows.
In one aspect, a gesture prediction method is provided, the method including:
acquiring object posture data acquired by a posture acquisition device at a first moment, wherein the object posture data is used for linking the posture acquired by the posture acquisition device with the posture of a virtual object in a virtual scene;
converting the object gesture data into the virtual scene to obtain gesture prediction data corresponding to the virtual object and a display time stamp corresponding to the gesture prediction data, wherein the display time stamp is a presentation time of the gesture prediction data obtained by predicting the gesture prediction data based on a first time;
storing a correspondence between the gesture prediction data and the display timestamp;
obtaining a prediction time stamp at a second time, wherein the prediction time stamp is obtained by predicting the rendering time consumption of the virtual scene at the second time;
And matching the prediction timestamp with the display timestamp in the corresponding relation to obtain target gesture prediction data corresponding to the prediction timestamp, and rendering and displaying the target gesture prediction data.
In another aspect, there is provided a posture predicting apparatus, the apparatus including:
the object posture data acquisition module is used for acquiring object posture data acquired by the posture acquisition equipment at a first moment, and the object posture data are used for linking the posture acquired by the posture acquisition equipment with the posture of a virtual object in a virtual scene;
the object posture data conversion module is used for converting the object posture data into the virtual scene to obtain posture prediction data corresponding to the virtual object and a display time stamp corresponding to the posture prediction data, wherein the display time stamp is the presentation time of the posture prediction data obtained by predicting the posture prediction data based on a first time;
the corresponding relation storage module is used for storing the corresponding relation between the gesture prediction data and the display time stamp;
the prediction time stamp obtaining module is used for obtaining a prediction time stamp at a second moment, wherein the prediction time stamp is obtained by predicting the rendering time consumption of the virtual scene at the second moment;
And the predicted gesture data rendering module is used for matching the predicted timestamp with the display timestamp in the corresponding relation, obtaining target gesture predicted data corresponding to the predicted timestamp and rendering and displaying the target gesture predicted data.
In another aspect, a computer device is provided, where the computer device includes a processor and a memory, where the memory stores at least one instruction, at least one program, a set of codes, or a set of instructions, where the at least one instruction, the at least one program, the set of codes, or the set of instructions are loaded and executed by the processor to implement the pose prediction method according to any of the embodiments of the application described above.
In another aspect, a computer readable storage medium is provided, in which at least one instruction, at least one program, a set of codes, or a set of instructions is stored, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by a processor to implement a pose prediction method according to any of the embodiments of the application as described above.
In another aspect, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device performs the posture prediction method described in any of the above embodiments.
The technical scheme provided by the embodiment of the application has the beneficial effects that at least:
the conversion of the object gesture data to the virtual scene is finished before the second moment all the time, the corresponding relation between the obtained gesture prediction data and the display timestamp is stored, the gesture prediction data corresponding to the prediction timestamp can be obtained from the stored corresponding relation all the time at the second moment, further, the gesture prediction data is rendered and displayed, and the prediction storage part and the reading part of the gesture prediction are asynchronously processed, so that the problem that when the system load is heavy, the delay is overlarge due to the continuous delay of the subsequent process caused by the sporadic delay of the process, the frame loss phenomenon is avoided, and the probability and influence of frame clamping are reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a block diagram of gesture prediction provided by an exemplary embodiment of the present application;
FIG. 2 is a timing diagram for gesture prediction provided by an exemplary embodiment of the present application;
FIG. 3 is a schematic drawing of a cadence diagram provided by an exemplary embodiment of the application;
FIG. 4 is a schematic illustration of an implementation environment provided by an exemplary embodiment of the present application;
FIG. 5 is a flowchart of a method for gesture prediction provided by an exemplary embodiment of the present application;
FIG. 6 is a schematic representation of a gesture prediction scenario provided by an exemplary embodiment of the present application;
FIG. 7 is a flowchart of a method for object pose data conversion provided by an exemplary embodiment of the present application;
FIG. 8 is a flowchart of a method for predicting a predicted timestamp according to an exemplary embodiment of the application;
FIG. 9 is a flowchart of a timestamp matching method provided by an exemplary embodiment of the present application;
FIG. 10 is a schematic diagram of a shared memory architecture according to an exemplary embodiment of the present application;
FIG. 11 is a timing diagram for gesture prediction provided by another exemplary embodiment of the present application;
FIG. 12 is a schematic diagram of a gesture prediction algorithm module provided by an exemplary embodiment of the present application;
FIG. 13 is a timing diagram provided by an exemplary embodiment of the present application;
FIG. 14 is a block diagram of a gesture predicting apparatus provided in an exemplary embodiment of the present application;
FIG. 15 is a block diagram of a gesture predicting apparatus according to an exemplary embodiment of the present application;
fig. 16 is a block diagram of a terminal according to an exemplary embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.
It should be understood that, although the terms first, second, etc. may be used in this disclosure to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
XR technology, that is, augmented Reality technology, refers to fusing Reality with Virtual through computer technology and wearable equipment, creating a Virtual environment capable of man-machine interaction, and includes technical features of VR (Virtual Reality), AR (Augmented Reality ) and MR (media Reality), so as to bring immersion feeling of seamless transition between Virtual world and real world to experimenters. VR technology refers to a virtual world that generates a realistic three-dimensional visual, tactile, olfactory, and other sensory experiences by means of a computer, so that a person in the virtual world generates an immersive sensation, and is mostly used in game entertainment scenes, such as VR glasses, VR display, and VR integrated machines; the AR technology is a technology for superposing virtual information to the real world and even realizing beyond reality, is an extension of VR technology to a certain extent, and relatively speaking, the AR equipment product has the characteristics of small volume, light weight, portability and the like; MR technique is VR and AR technique's further development, through presenting virtual scene at real scene, builds the communication closed loop between the user, greatly strengthens user experience and feels. The XR technology comprises the characteristics of the three technologies, has wide application prospect, and can be applied to scenes for realizing science and experiment course remote teaching in education and training, or immersive entertainment scenes in film and television entertainment, such as immersive film watching, games and the like, exhibition activity scenes in concerts, dramas, museums and the like, or 3D home decoration and architectural design scenes in industrial modeling and design, or novel scene consumption, such as cloud shopping, cloud trial fitting and the like.
In XR systems, target following refers to establishing an association of an object in the real world with an object in virtual space. By mapping the object in the real world to a certain position in the virtual space by the target following, when the object in the real world moves or rotates, the object in the virtual space also moves or rotates. To avoid tearing of the display, current XR display systems control the synchronization of XR application drawing and screen refresh with a vertical synchronization signal cadence.
Referring to fig. 1, fig. 1 is a block diagram of gesture prediction according to an exemplary embodiment of the present application, as shown in fig. 1, in the related art, an XR system module 110, an XR application module 120, an algorithm module 130, a display module 140, and a shared memory 150 are included in a target following function module 100. The target following algorithm is operated in the algorithm module 130 to predict the gesture of the entity object to obtain predicted gesture data, the XR system module 110 is responsible for XR system resource management including image synthesis, image display, interaction management and the like, the XR application module 120 is responsible for user scene logic control and application scene rendering, after the XR application module 120 calls the API of the XR system module 110 to obtain the predicted gesture data, the corresponding virtual scene is rendered according to the predicted gesture data, the XR display module 140 is used for displaying the virtual scene, or a vertical synchronization signal is transmitted to the XR system module 110 according to screen refreshing requirements to obtain display content for displaying. Taking the following bare hand gesture as an example, the XR application module 120 sends a gesture prediction request to the XR system module 110, the XR system module 110 sends the gesture prediction request to the XR algorithm module 130 based on interaction management, the XR algorithm module 130 predicts the gesture of the object acquired by the gesture acquisition device, for example, predicts the hand gesture 101 acquired by a data glove with a built-in sensor, etc., so as to obtain predicted gesture data corresponding to the virtual hand gesture 102, and sends the predicted gesture data to the shared memory 150, the XR system module 110 obtains the predicted gesture data from the shared memory 150 based on interaction management, sends a gesture prediction reply including the predicted gesture image to the XR application module 120, the XR application module 120 draws according to the predicted gesture data, sends the drawing content to the XR system module 110, synthesizes the drawing content to obtain display content, and sends the display content to the XR display module 140 for display.
The XR application calculates the time stamp to be displayed by the image before rendering a data frame, then transmits the time stamp to an algorithm module through the IPC call of the XR system, predicts the gesture of the target object according to the time stamp, then transmits the object gesture back to the XR application through the IPC, and after the predicted gesture is acquired by the XR application, the drawing of the image of the object is continued and sent to the XR system for composite display. Referring to fig. 2 schematically, fig. 2 is a schematic diagram of gesture prediction timing provided by an exemplary embodiment of the present application, as shown in fig. 2, an XR application process performs drawing of each frame of data under the rhythm of a vertical synchronization signal, triggers object gesture prediction before drawing, performs step 210, sends a timestamp to an XR system, performs step 220 by the XR system, transfers the timestamp to an algorithm module through IPC call, performs step 230 after receiving the timestamp sent by the XR application, predicts the gesture of the object according to the timestamp, then performs step 240, transfers the predicted gesture data back to the XR system, performs step 250 by the XR system, transfers the predicted gesture data back to the XR application, completes object gesture prediction, performs step 260 after the XR application obtains the predicted gesture, performs drawing based on the predicted gesture data, and finally sends the drawn image to the XR system for synthesis and display.
However, in the above method, XR application is required to transmit a prediction time stamp to the algorithm module through IPC before rendering each frame of data, and then wait for the algorithm module to complete the gesture prediction and return, and then continue the subsequent drawing process. When the system load is heavier, delay happens to the cross-process IPC call, and when the delay exceeds one Vsync period, frame clamping occurs, and frame loss and clamping phenomena are serious. Schematically, fig. 3 is a schematic drawing rhythm provided by an exemplary embodiment of the present application, as shown in fig. 3, an XR display system controls synchronization of XR application drawing and screen refreshing with a vertical synchronization signal rhythm, a part for gesture prediction is indicated by a shaded part of an XR corresponding frame, in an ideal state 310 with a lighter system load, the XR application obtains predicted gesture data obtained by gesture prediction and draws the corresponding frame, and the XR system synthesizes and sends the drawn frame to an XR display for display; in the case of the delay state 320 where the system load is heavy, the posture predicting section 321 corresponding to the frame #2 causes that the frame #2 cannot enter the next process in the original corresponding period, the XR display corresponding period 322 does not acquire the frame #2 for display and still displays the frame #1, and a frame jam or frame loss phenomenon occurs.
According to the gesture prediction method provided by the application, the conversion of the gesture data of the object to the virtual scene is finished before the second moment all the time, the obtained corresponding relation between the gesture prediction data and the display timestamp is stored, so that the gesture prediction data corresponding to the prediction timestamp can be obtained from the stored corresponding relation all the time at the second moment, further, the gesture prediction data is rendered and displayed, and the prediction storage part and the reading part of the gesture prediction are asynchronously processed, so that the problem that the delay is overlarge due to the continuous delay of the subsequent process caused by the sporadic delay of the process when the system load is heavy is avoided, the frame loss phenomenon is avoided, and the probability and influence of frame jamming are reduced.
First, an environment in which the present application is implemented will be described. Referring to fig. 4, a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application is shown, where the implementation environment includes: a terminal 410 and a gesture acquisition device 420.
The terminal 410 and the gesture collection device 420 are connected through a physical link, or can be connected through a communication network in a wireless manner, which is schematically shown that the terminal 410 and the gesture collection device 420 on the gesture collection site are connected through a hardware interface or a physical link in a wired manner, or the terminal 410 and the gesture collection device 420 are connected through a communication network in a Bluetooth or remote communication manner, etc., the gesture collection device 420 collects the gesture 401 of the entity object, and transmits the gesture data of the entity object to the terminal 410 through the wired connection or the communication network to predict, so as to obtain the gesture 402 of the corresponding virtual object of the entity object in the virtual scene.
The terminal 410 includes a system module 411, an application module 412, an algorithm module 413, and a display module 414 for implementing the gesture prediction function.
In some embodiments, the terminal 410 further includes a shared memory 415, and the system module 411 includes a system interaction module 416. The system module 411 acquires the vertical synchronization signal from the display module 414 and periodically synchronizes the vertical synchronization signal and the screen refresh rate to the algorithm module 413. The algorithm module 413 starts the gesture prediction task, completes the recognition and prediction of the physical object gesture 401 under the rhythm of the vertical synchronization signal, obtains the predicted gesture data of the virtual object gesture 402 corresponding to the physical object gesture, and writes the predicted gesture data into the shared memory 415. The application module 412 receives the gesture prediction request sent by the system module 411 to the system module 411 based on the vertical synchronization signal, the system module 411 reads the predicted gesture data from the shared memory 415 through the system interaction module 416 and transmits the predicted gesture data to the response module 412 for drawing, the application module 412 sends the drawn drawing content to the system module 411 for synthesis to obtain the display content, and the system module 411 sends the display content to the display module 414 for display.
The above terminal is optional, and the terminal may be a desktop computer, an XR helmet terminal device, a laptop portable computer, a mobile phone, a tablet computer, an electronic book reader, an MP3 (Moving Picture Experts Group Audio Layer III, dynamic image expert compression standard audio layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image expert compression standard audio layer 4) playing, a smart television, a smart car, or other terminal devices in various forms, which are not limited in this embodiment of the present application.
It should be noted that the server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server that provides cloud services, cloud security, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content distribution network (Content Delivery Network, CDN), and basic cloud computing services such as big data and an artificial intelligence platform.
Cloud Technology (Cloud Technology) refers to a hosting Technology that unifies serial resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data.
In some embodiments, the servers described above may also be implemented as nodes in a blockchain system.
It should be noted that, the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, presented data, etc.), and signals related to the present application are all authorized by the user or are fully authorized by the parties, and the collection, use, and processing of the related data is required to comply with the relevant laws and regulations and standards of the relevant countries and regions.
Next, an application scenario of the gesture prediction method provided by the embodiment of the present application will be described with reference to the above implementation environment.
1. Application in XR game scene
Optionally, in the XR game scene, the player gathers his own object posture data by wearing a posture collection device, such as a handle with a positioning function, a finger ring, or a data glove with a built-in sensor, a arm sleeve, a sock, or an XR helmet with a built-in camera device, an XR glasses, etc., and executes a posture prediction method by a terminal with a connection relation established with the posture collection device, such as an XR helmet with a head display function, glasses, or a display screen with a data processor, a computer, etc., so as to map the object posture data into the XR game scene, obtain posture prediction data, and render and display the posture of a virtual object corresponding to the player in the XR game scene based on the posture prediction data, so that the player controls his own posture change to link with the posture of the virtual object corresponding to the player in the XR game scene in reality, thereby realizing the immersive game experience.
2. Application to virtual shooting scenes
Optionally, in the virtual shooting scene, special-effect actors wear special clothing which is fully covered with various sensors to realize a positioning function, perform in the virtual shooting scene before rendering a background wall with a virtual background and with a part of real scenery, collect overall gesture data of the special-effect actors as object gesture data through the special clothing, execute a gesture prediction method by a terminal device or an XR helmet which is remotely connected with the special clothing, convert and render the object gesture data to obtain corresponding virtual special-effect gestures of the special-effect actors in the virtual shooting scene, display special-effect effects of the virtual special-effect gestures in the virtual shooting scene in an XR helmet worn by a shooting scene on-site demonstration display screen or the special-effect actors, enable the special-effect actors to directly observe special-effect performance effects in real time, adjust performance modes and the like, and improve virtual shooting efficiency.
3. Application in novel consumption scene
Optionally, in the novel consumption scene, a consumer acquires the gesture of the specified object through gesture acquisition equipment such as a mobile phone camera to obtain object gesture data, a terminal device such as a mobile phone or a server executes a gesture prediction method, the object gesture data is mapped to a virtual animation combination of the virtual consumption scene and the commodity to obtain predicted gesture data corresponding to the virtual gesture of the specified object wearing or equipping the commodity, rendering is performed, and the virtual gesture of the specified object wearing or equipping the commodity in the virtual consumption scene is displayed on a display screen of the consumer terminal, so that the consumer can remotely realize shopping experience such as cloud trial assembly.
It should be noted that the above application scenario is merely an illustrative example, and the gesture prediction method provided by the embodiment of the present application may be applied to any scenario where gesture prediction is performed in an XR scenario.
Referring to fig. 5, a flowchart of a gesture predicting method according to an exemplary embodiment of the present application is shown, where the method may be applied to a terminal, a server, or both, and the embodiment of the present application is described by taking the application of the method to the terminal as an example, and as shown in fig. 5, the method includes the following steps:
Step 510, acquiring object posture data acquired by a posture acquisition device at a first moment.
The object gesture data are used for linking the gesture acquired by the gesture acquisition device with the gesture of the virtual object in the virtual scene, and the gesture acquisition device can be implemented as a data glove, an arm sleeve, a handle, a finger ring or virtual data glasses with built-in sensors, or any device with a motion acquisition function, such as a camera device or a pressure sensor, which can acquire the gesture of the entity object in a following way to acquire the object gesture data.
In some embodiments, the object gesture data is used to map the gesture performance of the gesture acquisition device in reality into the virtual scene, so as to achieve the effect that the virtual object in the virtual scene is kept synchronous with the gesture acquired by the gesture acquisition device. Schematically, the gesture collection device is implemented as a glove with built-in sensors, so that in a real scene, a user performs hand grabbing action through the gesture collection device, the gesture collection device collects hand gesture changes of the user and generates object gesture data mapped to a virtual scene, and therefore a hand object model in the virtual scene can simulate hand gestures in reality to execute grabbing animation.
Referring to fig. 6 schematically, fig. 6 is a schematic view of a gesture prediction scenario provided by an exemplary embodiment of the present application, as shown in fig. 6, in an actual application scenario 600, taking hand gesture prediction as an example, gesture collection is performed on hands 611 of a player through a virtual data glasses 601 with a built-in photographing device to obtain object gesture data, and the object gesture data is used to link the gesture collected by the gesture collection device with the gesture of a virtual object 612 in an XR game scenario 610, so that the player views, through the virtual data glasses 601, that hands 611 with the gesture changed in the actual scenario are displayed in the XR game scenario 610 in the gesture of the virtual object 612, and thus, capturing and other game operations are performed in the XR game scenario 610, to realize an immersive game experience. The linkage process is implemented by the gesture predicting part 620, the object gesture data of the object gesture 621 corresponding to the entity object 611 is mapped to the transformed gesture predicting data 622 in the virtual scene 610 based on the preset algorithm, and the virtual gesture 623 corresponding to the virtual object 612 is generated based on the gesture predicting data 622.
And step 520, converting the object gesture data into a virtual scene to obtain gesture prediction data corresponding to the virtual object and a display time stamp corresponding to the gesture prediction data.
Wherein the display time stamp is a presentation time of the posture prediction data predicted from the posture prediction data based on the first time. That is, the display time stamp is a predicted time when the gesture prediction data is represented in the virtual scene after the first time, where the display time stamp is a time stamp predicted based on parameters such as screen rendering time consumption, synchronization period, and the like, and is used to represent a time when the gesture prediction data can be displayed in the interface after the processes such as data transfer, interface rendering, and the like.
In some embodiments, the first time may also be implemented as the current time, i.e. the time at which the object pose data is to be received.
In some embodiments, the terminal converts the object gesture data into a virtual scene through the trained gesture prediction model, inputs the object gesture data into the gesture prediction model, and obtains gesture prediction data output by the model and a display time stamp; or the terminal maps the object gesture data into the virtual scene through a preset algorithm to obtain corresponding gesture prediction data and calculates a display time stamp, wherein the preset algorithm is configured according to the conversion relation between the object gesture data and the gesture prediction data; or the terminal stores a plurality of corresponding relations between the object posture data and the predicted posture data, obtains posture predicted data matched with the object posture data based on the corresponding relations, and calculates a display time stamp.
In step 530, a correspondence between the gesture prediction data and the display timestamp is stored.
The gesture prediction data uniquely corresponding to a certain display time stamp can be found based on the corresponding relation.
In some embodiments, step 530 is implemented as a requirement to store in order the time-sequential relationship of the display timestamps, with the correspondence between the pose prediction data and the display timestamps stored in shared memory.
Schematically, after storing the corresponding relation between the gesture prediction data with the earliest corresponding moment of the display time stamp and the display time stamp as the first group of data in the shared memory, storing the next corresponding relation as the second group of data in the shared memory according to the time sequence of the display time stamp, and so on.
Step 540, obtaining the predicted timestamp at the second time.
The prediction time stamp is obtained by predicting the rendering time consumption of the virtual scene at the second moment.
In some embodiments, the terminal for executing the gesture prediction method includes an XR application, where the XR application can obtain terminal screen parameters based on gesture prediction requirements, where the terminal screen parameters are used to indicate prediction of a prediction timestamp, and illustratively, at a second moment, the XR application predicts rendering time consumption of the virtual scene based on the obtained terminal screen parameters, so as to obtain the prediction timestamp.
And 550, matching the predicted timestamp with the display timestamp in the corresponding relation to obtain target gesture predicted data corresponding to the predicted timestamp, and rendering and displaying the target gesture predicted data.
In some embodiments, based on a preset matching requirement, a display timestamp specified in the corresponding relation is obtained as a display timestamp matched with the predicted timestamp, or based on a preset reading requirement, a specified number of display timestamps are read from the corresponding relation, and matching is performed according to a preset matching condition and the predicted timestamp, or the display timestamp corresponding to the latest storage time in the corresponding relation is directly read to be matched with the predicted timestamp.
In some embodiments, the second time is any time within a specified period after the first time, and when the predicted timestamp is obtained at the second time, the predicted timestamp can match the display timestamp predicted at the first time. That is, after the display time stamp is predicted at the first time and the display time stamp is stored in association with the target posture prediction data, the target posture prediction data corresponding to the prediction time stamp can be determined from the stored association relationship based on the prediction time stamp at the second time.
It should be noted that, in the above embodiment, taking the case that the second time is within the specified period after the first time as an example, in some embodiments, the second time may also be any time before or after the first time, and the target posture prediction data is obtained based on the correspondence between the display timestamp generated at a time before the second time and the target posture prediction data.
In some embodiments, the terminal for executing the gesture prediction method includes an XR application, an XR system and a display screen, illustratively, the reading requirement is preset to sequentially read three display timestamps in the corresponding relation forward from the latest storage time, the matching condition is preset to minimize the difference between the prediction timestamp and the display timestamp, the XR application reads the three display timestamps from the corresponding relation based on the preset reading requirement, matches the prediction timestamp according to the preset matching condition to obtain target gesture prediction data corresponding to the prediction timestamp, renders the target gesture prediction data by the XR application, sends the target gesture prediction data to the XR system to be synthesized, and finally sends the target gesture prediction data to the display screen to be displayed.
In summary, according to the method provided by the embodiment of the application, the conversion of the object gesture data to the virtual scene is completed before the second moment all the time, and the corresponding relation between the obtained gesture prediction data and the display timestamp is stored, so that the gesture prediction data corresponding to the prediction timestamp can be obtained from the stored corresponding relation all the time at the second moment, further, the gesture prediction data is rendered and displayed, and the prediction storage part and the reading part of the gesture prediction are asynchronously processed, so that the problem that when the system load is heavy, the delay is overlarge due to the continuous delay of the subsequent process caused by the sporadic delay of the process, the frame loss phenomenon is avoided, and the probability and influence of frame clamping are reduced.
Referring to fig. 7, fig. 7 is a flowchart of an object posture data conversion method according to an exemplary embodiment of the present application, as shown in fig. 7, in some embodiments, the step 520 is implemented as steps 521 to 523.
And 521, converting the object gesture data into a virtual scene to obtain gesture prediction data corresponding to the virtual object.
Illustratively, as shown in fig. 6, the object pose data of the object pose 621 corresponding to the two hands 611 is input into the trained pose prediction model, and the pose prediction data 622 of the virtual pose 623 corresponding to the virtual object 612 in the XR game scene 610 is output.
At step 522, conversion information is obtained.
The conversion information comprises rendering display related information of a terminal display screen, such as terminal screen parameters, rendering time parameters and the like.
In some embodiments, a terminal for executing the object gesture data conversion method includes an XR application, an XR system and a display screen, where the XR application can start a gesture prediction process based on gesture prediction requirements, the XR system has an interaction function and can connect interactions of each part of the terminal, the display screen corresponds to rendering display related information, the XR application obtains rendering display related information from the XR system through the interaction function of the XR system based on gesture prediction requirements, and the rendering display related information includes terminal screen parameters and rendering duration parameters obtained by the XR system based on display screen refreshing requirements.
Schematically, the XR application obtains a rendering time parameter representing rendering time consumption, a refresh rate parameter representing a screen refresh rate in a terminal screen parameter, and a vertical synchronization signal parameter representing a vertical synchronization signal in the terminal screen parameter from the XR system through an interactive function of the XR system based on gesture prediction requirements.
Step 523, predicting the presentation time of the gesture prediction data based on the conversion information, to obtain a display time stamp.
In some embodiments, when the conversion information includes a terminal screen parameter and a rendering duration parameter, step 523 is implemented as two steps:
first stepThe data processing time is predicted based on terminal screen parameters.
The data processing time is the predicted time for predicting and transmitting based on the gesture prediction data.
In some embodiments, when the terminal screen parameters include a vertical synchronization signal parameter and a refresh rate parameter, the first step is implemented to determine a synchronization period based on the vertical synchronization signal parameter and the refresh rate parameter; the data processing time consumption is determined based on the synchronization period and the time of last reception of the vertical synchronization signal parameter.
In some embodiments, to ensure that the storing of the correspondence is completed before the second time, the data processing time consumption includes an actual data processing time consumption and a reserved buffer duration.
Illustratively, taking a synchronization period duration as a reserved buffer duration, when the refresh rate parameter indicates that the screen refresh rate is 90HZ, the synchronization period T is 11.11ms, based on the synchronization period T and the time T of last receiving the vertical synchronization signal parameter v And a first time (i.e. the current time) T now Determining data processing time DeltaT 1 ，ΔT 1 ＝T+T v +T-T now 。
Second stepThe display timestamp is determined based on the first time, the data processing time consumption, and the rendering duration parameter.
In some embodiments, the second step is implemented to obtain the display timestamp by using the first time as a starting time, and pushing the first time corresponding to the time consumption of the data processing, the second time corresponding to the rendering time parameter, and the third time corresponding to the synchronization period.
Schematically, when the second time corresponding to the rendering time parameter is ΔT, based on the first time T now The first duration delta T corresponding to the data processing time consumption obtained by the first step prediction is shifted backwards 1 The second time length corresponding to the rendering time length parameter is delta T, and the third time length T corresponding to the synchronous period is used for obtaining a display time stamp T 1 ，T 1 ＝T now +ΔT 1 +DeltaT, i.e. T 1 ＝T now +(T+T v +T-T now )+ΔT＝T v +2T+ΔT。
In summary, the method provided by the embodiment of the application defines the object posture data conversion method, predicts the presentation time of the obtained posture prediction data, and obtains the display time stamp, so that the corresponding posture prediction data can be read based on the display time stamp in the asynchronous reading process.
The method provided by the embodiment of the application has the advantages that the method for determining the display time stamp is clarified, the display time stamp is determined based on the first moment, the time consumption of data processing and the rendering time length parameter, the storage is ensured to be completed before the gesture prediction data needs to be read, and the probability of frame blocking is reduced.
The method provided by the embodiment of the application confirms the time consumption determination method of data processing, determines the time consumption of data processing based on the synchronization period and the last time of receiving the vertical synchronization signal parameter, ensures the storage of the corresponding relation before the second time, and prevents the frame loss phenomenon.
Referring to fig. 8, fig. 8 is a flowchart of a prediction method of a prediction timestamp according to an exemplary embodiment of the present application, as shown in fig. 8, in some embodiments, the step 540 is implemented as follows:
step 541, predicting gesture rendering time based on terminal screen parameters.
Wherein the gesture rendering time-consuming is a rendering time-consuming for transferring and rendering based on gesture prediction data.
Illustratively, a refresh rate parameter and a vertical synchronization signal parameter in terminal screen parameters are obtained, a synchronization period T is determined according to the refresh rate parameter, and a time T of last receiving the vertical synchronization signal parameter is determined according to the vertical synchronization signal parameter v Based on the synchronization period T, the time T of last receiving the vertical synchronization signal parameter v And a second time T 0 Predicted pose rendering time-consuming DeltaT 2 ，ΔT 2 ＝T v +T–T 0 。
Step 542, determining a prediction timestamp based on the second time, the gesture rendering time consuming and the rendering duration parameter.
In some embodiments, step 542 is implemented to obtain the prediction timestamp by using the second time as the starting time, and pushing the third duration corresponding to the time consumption of gesture drawing and the second duration corresponding to the rendering duration parameter backward.
Schematically, when the second time corresponding to the rendering time parameter is ΔT, based on the second timeT 0 A fourth duration deltat corresponding to the predicted gesture drawing time consumption predicted in step 541 is shifted backward 2 The second time length corresponding to the rendering time length parameter is delta T, and a prediction time stamp T is obtained 2 ，T 2 ＝T 0 +ΔT 2 +DeltaT, i.e. T 2 ＝T 0 +(T v +T–T 0 )+ΔT＝T v +T+ΔT。
In summary, the method provided by the embodiment of the application provides a method for obtaining a prediction timestamp, which determines the prediction timestamp based on the second time, the gesture drawing time consumption and the rendering time parameter, and is used for obtaining gesture prediction data corresponding to the matched display timestamp in a corresponding relation as reference data for reading the gesture prediction data, so as to obtain accurate gesture prediction data, and improve the accuracy of gesture prediction.
The method provided by the embodiment of the application determines the gesture drawing time consumption determination method, predicts the gesture drawing time consumption based on the terminal screen parameters, and improves the prediction accuracy of the prediction time stamp, thereby improving the accuracy of the prediction time stamp.
Referring to fig. 9, fig. 9 is a flowchart of a timestamp matching method according to an exemplary embodiment of the present application, and as shown in fig. 9, the step 550 is implemented as follows:
step 551, sequentially reading the display time stamps of the designated number forward from the latest storage time in the correspondence.
Schematically, the latest corresponding relation of the latest storage time is the ninth group of data, and the display time stamps of the three groups of data are sequentially read forwards from the latest storage time in the corresponding relation, namely the display time stamps T of the ninth group of data are sequentially read in the corresponding relation 19 Display time stamp T of eighth group data 18 And a display time stamp T of the seventh group of data 17 。
Step 552, the predicted time stamp is matched with the specified number of display time stamps, so as to obtain the target gesture predicted data corresponding to the predicted time stamp.
In some embodiments, step 552 is implemented as two steps:
first, a plurality of differences are determined for the predicted time stamps corresponding to a specified number of display time stamps.
Illustratively, a prediction timestamp T is determined 2 Three differences corresponding to the three display time stamps obtained in step 551, respectively, are Δt 1 、Δt 2 And Deltat 3 ，Δt 1 ＝|T 2 -T 19 |，Δt 2 ＝|T 2 -T 18 |，Δt 3 ＝|T 2 -T 17 |。
And secondly, determining a display time stamp corresponding to the difference value with the smallest numerical value from the plurality of difference values, and obtaining gesture prediction data corresponding to the display time stamp as target gesture prediction data.
Illustratively, the display time stamp corresponding to the difference value with the smallest value is determined from the three difference values, and when the processes corresponding to the steps 520 and 530 do not have serious delay, the difference value with the smallest value is the difference value between the display time stamp and the prediction time stamp in the latest group of corresponding relations corresponding to the latest storage time, namely T 19 Obtain the display time stamp T 19 The corresponding posture prediction data is used as target posture prediction data.
Step 553, rendering and displaying the target gesture prediction data.
Schematically, the target gesture prediction data is rendered by the XR application and sent to the XR system for synthesis, and finally sent to the display screen for display.
In summary, the method provided by the embodiment of the application defines the target gesture prediction data determining method, matches the prediction time stamp with the display time stamps with the designated number to obtain the target gesture prediction data accurately corresponding to the prediction time stamp, and improves the gesture prediction accuracy.
According to the method provided by the embodiment of the application, a prediction timestamp matching method is defined, and the display timestamp corresponding to the difference value with the smallest numerical value is determined from a plurality of difference values corresponding to the prediction timestamp and the specified number of display timestamps, so that gesture prediction data corresponding to the display timestamp is obtained as the target gesture prediction data, the accurate correspondence between the target gesture data and the prediction timestamp is ensured, and the gesture prediction accuracy is improved.
In some embodiments, according to the requirement of sequential storage of display time stamps, the corresponding relation between the gesture prediction data and the display time stamps is stored in a shared memory, and an XR algorithm process in a terminal executing the gesture prediction method and an XR system reading process share one memory, so as to realize cross-process communication. Each process has its own independent space, the address space of the process is a virtual address, and the operating system maps the virtual address into the physical address space according to the actual needs of the process. Each process has its own shared area, and during the running process of the process, the operating system kernel maps the actual shared area address to the physical memory address, so that the operating system can communicate through the shared memory by only mapping the address of the same memory to a different process. The shared memory comprises head data, queue data and tail node tag data, wherein the head data is used for indicating the number of the queue data, the size of the queue data and index data corresponding to the tail node tag data, the queue data comprises a corresponding relation, and the tail node tag data is used for identifying the queue data corresponding to the latest storage moment.
Referring to fig. 10 schematically, fig. 10 is a schematic diagram of a shared memory structure according to an exemplary embodiment of the present application, as shown in fig. 10, a shared memory 1000 uses a circular queue to transmit data, the circular queue length is 8, a header node 1010 stores header data 1011, including the number of queue nodes, the length of a single queue node, tail node tag data and padding characters, nodes 0 to 7 store node data 1020, that is, a correspondence relationship, including position data, direction data and a timestamp in gesture prediction data, and a last written node 5 identifies tail node tag 1030.
In summary, the method provided by the embodiment of the application defines the structure of the shared memory, and can find the corresponding relation of the latest storage time through the tail node label, thereby realizing asynchronous storage and cross-process communication and avoiding the frame losing phenomenon caused by process blocking.
Referring to fig. 11, fig. 11 is a timing diagram for gesture prediction according to another exemplary embodiment of the present application, and as shown in fig. 11, step 1110 is performed by the XR system and algorithm module in the terminal performing the gesture prediction method, to perform a vertical synchronization signal synchronization process. The algorithm module executes step 1120 to acquire object posture data according to the vertical synchronization signal, convert the object posture data to obtain posture prediction data, and write the posture prediction data into the shared memory, and referring to fig. 12, schematically, fig. 12 is a schematic diagram of a posture prediction algorithm module provided by an exemplary embodiment of the present application, and as shown in fig. 12, the algorithm module 1200 receives a synchronization task of the vertical synchronization signal, performs step 1221 posture detection on posture data 1201 acquired by the object acquisition device, performs step 1222 data fusion on other data 1202 acquired by the object acquisition device, and performs 1230 posture prediction based on the results of the posture detection and the data fusion, to obtain posture prediction data. The XR application and the XR system perform step 1130, and the XR application sends a gesture prediction request to the XR system based on the vertical synchronization signal, and the XR system reads gesture prediction data from the shared memory based on the gesture prediction request and returns the gesture prediction data to the XR application, and the XR application draws based on the gesture prediction data. In step 1140, the XR application sends the drawn content to the XR system for synthesis, and sends the obtained display content to the display module for display.
Referring to FIG. 13, FIG. 13 is a timing diagram of an XR prediction algorithm at T during a period 1300, as shown in FIG. 13, according to an exemplary embodiment of the present application v1 The latest vertical synchronous signal is received at the moment, and at the first moment (namely the current moment) T now Begin to process data, which takes a long time delta T 1 Wherein the reserved buffer time length is a synchronization period time length T, and the actual data processing process receives the vertical synchronization signal T at the latest time in the XR application module v2 Before completion, rendering time is delta T, and XR prediction algorithm predicts display time stamp T 1 ＝T now +ΔT 1 +DeltaT, i.e. T 1 ＝T now +(T+T v1 +T-T now )+ΔT＝T v1 +2t+Δt. XR application Module at T v2 The latest vertical synchronous signal is received at the moment, and at the second moment T 0 Gesture drawing is started, and the time consumption delta T of gesture drawing is shortened 2 Prediction timestamp T predicted by XR application module 2 ＝T 0 +ΔT 2 +DeltaT, i.e. T 2 ＝T 0 +(T v +T–T 0 )+ΔT＝T v +T+DeltaT. By the calculation method, the display time stamp and the prediction time stamp are matched with the display time 1301, and the picture corresponding to the prediction attitude data is accurately displayed at the display time 1301.
In summary, the method provided by the embodiment of the application defines the execution time sequence of the gesture prediction, ensures that the algorithm module writes the required gesture prediction data into the shared memory before the second moment when the application module reads the gesture prediction data, reads the correct gesture prediction data from the shared memory based on the display time stamp and the prediction time stamp, reduces the probability of occurrence of frame stuck, and improves the accuracy of gesture prediction.
Fig. 14 is a block diagram showing a configuration of an attitude prediction apparatus according to an exemplary embodiment of the present application, and as shown in fig. 14, the apparatus includes:
the object gesture data acquisition module 1410 is configured to acquire object gesture data acquired by a gesture acquisition device at a first time, where the object gesture data is used to link a gesture acquired by the gesture acquisition device with a gesture of a virtual object in a virtual scene;
an object pose data conversion module 1420, configured to convert the object pose data into the virtual scene, and obtain pose prediction data corresponding to the virtual object and a display timestamp corresponding to the pose prediction data, where the display timestamp is a presentation time of the pose prediction data predicted by the pose prediction data based on a first time;
a correspondence storage module 1430 for storing correspondence between the gesture prediction data and the display time stamp;
the prediction timestamp obtaining module 1440 is configured to obtain a prediction timestamp at a second time, where the prediction timestamp is a timestamp obtained by predicting rendering time consumption of the virtual scene at the second time;
and the target gesture data rendering module 1450 is configured to match the prediction timestamp with the display timestamp in the correspondence, obtain target gesture prediction data corresponding to the prediction timestamp, and perform rendering display.
Referring to fig. 15, fig. 15 is a block diagram illustrating a configuration of a gesture predicting apparatus module according to an exemplary embodiment of the present application, and as shown in fig. 15, in some embodiments, the object gesture data conversion module 1420 includes:
an object gesture data conversion unit 1421, configured to convert the object gesture data into the virtual scene, so as to obtain gesture prediction data corresponding to the virtual object;
a conversion information obtaining unit 1422, configured to obtain conversion information, where the conversion information includes rendering display related information of the terminal display screen;
and a display timestamp prediction unit 1423, configured to predict, based on the conversion information, a presentation time of the gesture prediction data, so as to obtain the display timestamp.
In some embodiments, the conversion information includes a terminal screen parameter and a rendering time length parameter, and the display timestamp prediction unit 1423 includes:
a data processing time-consuming prediction subunit 1424, configured to predict a data processing time consuming based on the terminal screen parameter, where the data processing time consuming is a predicted time consuming for performing prediction and transmission based on the gesture prediction data;
a display timestamp determination prediction subunit 1425, configured to determine the display timestamp based on the first time, the time consumed for data processing, and the rendering duration parameter.
In some embodiments, the screen parameters include a vertical synchronization signal parameter and a refresh rate parameter, the data processing time consuming predictor unit 1424 for determining a synchronization period based on the vertical synchronization signal parameter and the refresh rate parameter; and determining that the data processing is time-consuming based on the synchronization period and the time of last receiving the vertical synchronization signal parameter.
In some embodiments, the display timestamp determining prediction subunit 1425 is configured to use the first time as a starting time, and shift backward a first duration corresponding to the time consumption of the data processing, a second duration corresponding to the rendering duration parameter, and a third duration corresponding to the synchronization period to obtain the display timestamp.
In some embodiments, the prediction timestamp acquisition module 1440 includes:
a gesture drawing time-consuming prediction unit 1441, configured to predict gesture drawing time consumption based on the terminal screen parameter, where the gesture drawing time consumption is drawing time consumption that is transferred and drawn based on the gesture prediction data;
a prediction timestamp prediction unit 1442, configured to determine the prediction timestamp based on the second time, the gesture drawing time consumption, and the rendering duration parameter.
In some embodiments, the prediction timestamp prediction unit 1442 is configured to use the second time as a starting time, and move the fourth duration corresponding to the gesture drawing time consumption and the second duration corresponding to the rendering time parameter backward to obtain the prediction timestamp.
In some embodiments, the correspondence meets a requirement of being stored sequentially in a time sequence relation of the display time stamp, and the target pose data rendering module 1450 includes:
a display time stamp reading unit 1451 for sequentially reading a specified number of display time stamps forward from the latest storage time in the correspondence;
a target pose prediction data obtaining unit 1452, configured to match the prediction timestamp with a specified number of display timestamps, to obtain target pose prediction data corresponding to the prediction timestamp;
and a target posture prediction data rendering unit 1453, configured to render and display the target posture prediction data.
In some embodiments, the target pose prediction data acquisition unit 1452 is configured to determine a plurality of differences corresponding to the predicted time stamps and the specified number of display time stamps; and determining a display time stamp corresponding to the difference value with the smallest numerical value from the plurality of difference values, and obtaining gesture prediction data corresponding to the display time stamp as the target gesture prediction data.
In some embodiments, the correspondence storage module 1430 is configured to store, in a shared memory, correspondence between the gesture prediction data and the display timestamp according to a requirement of sequential storage of the display timestamp, where the shared memory includes header data, queue data and tail node tag data, the header data is used to indicate the number of the queue data, a size of the queue data and index data corresponding to the tail node tag data, the queue data includes the correspondence, and the tail node tag data is used to identify the queue data corresponding to the latest storage time.
In summary, the device provided by the embodiment of the application always completes the conversion of the object gesture data to the virtual scene before the second moment, stores the corresponding relation between the obtained gesture prediction data and the display timestamp, ensures that the gesture prediction data corresponding to the prediction timestamp can be always obtained from the stored corresponding relation at the second moment, further renders and displays the gesture prediction data, and avoids overlarge delay caused by continuous delay of the subsequent process due to sporadic delay of the process when the system load is heavy by asynchronously processing the prediction storage part and the reading part of the gesture prediction, avoids frame loss phenomenon, and reduces the probability and influence of frame clamping.
It should be noted that: in the gesture predicting apparatus provided in the above embodiment, only the division of the above functional modules is used as an example, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to perform all or part of the functions described above.
Fig. 16 shows a block diagram of a terminal 1600 according to an exemplary embodiment of the present application. The terminal 1600 may be: XR helmet terminals, smartphones, tablets, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic image expert compression standard audio plane 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image expert compression standard audio plane 4) players, notebook or desktop computers. Terminal 1600 may also be referred to by other names of user devices, portable terminals, laptop terminals, desktop terminals, etc.
In general, terminal 1600 includes: a processor 1601, and a memory 1602.
Processor 1601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1601 may be implemented in at least one hardware form of a DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor 1601 may also include a host processor, which is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit ); a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1601 may be integrated with a GPU (Graphics Processing Unit, image processor) for use in responsible for rendering and rendering of content to be displayed by the display screen. In some embodiments, the processor 1601 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.
Memory 1602 may include one or more computer-readable storage media, which may be non-transitory. Memory 1602 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 1602 is used to store at least one instruction for execution by processor 1601 to implement the pose prediction method provided by the method embodiments of the present application.
In some embodiments, terminal 1600 also includes other components, and those skilled in the art will appreciate that the structure illustrated in FIG. 16 is not limiting of terminal 1600 and may include more or fewer components than illustrated, or may combine certain components, or employ a different arrangement of components.
The embodiment of the application also provides a computer device which can be implemented as a terminal or a server as shown in fig. 4. The computer device includes a processor and a memory in which at least one instruction, at least one program, code set, or instruction set is stored, the at least one instruction, at least one program, code set, or instruction set being loaded and executed by the processor to implement the gesture prediction method provided by the above method embodiments.
Embodiments of the present application also provide a computer readable storage medium having stored thereon at least one instruction, at least one program, a code set, or an instruction set, where the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the gesture prediction method provided by the above method embodiments.
Embodiments of the present application also provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device performs the posture prediction method described in any of the above embodiments.
Alternatively, the computer-readable storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), solid state disk (SSD, solid State Drives), or optical disk, etc. The random access memory may include resistive random access memory (ReRAM, resistance Random Access Memory) and dynamic random access memory (DRAM, dynamic Random Access Memory), among others. The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The foregoing description of the preferred embodiments of the present application is not intended to limit the application, but rather, the application is to be construed as limited to the appended claims.

Claims (14)

1. A method of gesture prediction, the method comprising:
acquiring object posture data acquired by a posture acquisition device at a first moment, wherein the object posture data is used for linking the posture acquired by the posture acquisition device with the posture of a virtual object in a virtual scene;
converting the object gesture data into the virtual scene to obtain gesture prediction data corresponding to the virtual object and a display time stamp corresponding to the gesture prediction data, wherein the display time stamp is a presentation time of the gesture prediction data obtained by predicting the gesture prediction data based on a first time;
Storing a correspondence between the gesture prediction data and the display timestamp;
obtaining a prediction time stamp at a second time, wherein the prediction time stamp is obtained by predicting the rendering time consumption of the virtual scene at the second time;
and matching the prediction timestamp with the display timestamp in the corresponding relation to obtain target gesture prediction data corresponding to the prediction timestamp, and rendering and displaying the target gesture prediction data.
2. The method of claim 1, wherein said converting the object pose data to the virtual scene to obtain pose prediction data corresponding to the virtual object and a display timestamp corresponding to the pose prediction data, comprises:
converting the object gesture data into the virtual scene to obtain gesture prediction data corresponding to the virtual object;
obtaining conversion information, wherein the conversion information comprises rendering display related information of a terminal display screen;
and predicting the presentation time of the attitude prediction data based on the conversion information to obtain the display time stamp.
3. The method according to claim 2, wherein the conversion information includes a terminal screen parameter and a rendering duration parameter;
Predicting the presentation time of the gesture prediction data based on the conversion information to obtain the display timestamp, including:
predicting data processing time consumption based on the terminal screen parameters, wherein the data processing time consumption is predicted time consumption for predicting and transmitting based on the gesture prediction data;
the display time stamp is determined based on the first time, the data processing time consumption, and the rendering duration parameter.
4. A method according to claim 3, wherein the screen parameters include a vertical synchronization signal parameter and a refresh rate parameter;
the method for predicting the data processing time consumption based on the terminal screen parameters comprises the following steps:
determining a synchronization period based on the vertical synchronization signal parameter and the refresh rate parameter;
and determining that the data processing is time-consuming based on the synchronization period and the time of last receiving the vertical synchronization signal parameter.
5. The method of claim 4, wherein the determining the display timestamp based on the first time, the data processing time consuming, the rendering duration parameter comprises:
and taking the first moment as a starting moment, pushing the first time length corresponding to the time consumption of the data processing, the second time length corresponding to the rendering time length parameter and the third time length corresponding to the synchronous period backwards to obtain the display time stamp.
6. The method of claim 5, wherein the obtaining the predicted timestamp at the second time comprises:
predicting gesture drawing time consumption based on the terminal screen parameters, wherein the gesture drawing time consumption is drawing time consumption for transmitting and drawing based on the gesture prediction data;
and determining the prediction time stamp based on the second moment, the gesture drawing time consumption and the rendering time length parameter.
7. The method of claim 6, wherein the determining the predicted timestamp based on the second time, the pose rendering time consuming, and the rendering duration parameter comprises:
and taking the second moment as a starting moment, pushing the fourth time length corresponding to the gesture drawing time consumption and the second time length corresponding to the rendering time length parameter backwards to obtain the prediction time stamp.
8. The method according to any one of claims 1 to 7, wherein the correspondence conforms to a requirement of sequential storage in a time-series relationship of the display time stamps;
the step of matching the prediction timestamp with the display timestamp in the corresponding relation to obtain target gesture prediction data corresponding to the prediction timestamp and rendering and displaying the target gesture prediction data, comprises the following steps:
Sequentially and forwards reading a designated number of display time stamps from the latest storage time in the corresponding relation;
matching the predicted time stamps with a specified number of display time stamps to obtain target gesture predicted data corresponding to the predicted time stamps;
and rendering and displaying the target gesture prediction data.
9. The method of claim 8, wherein matching the predicted time stamp with a specified number of display time stamps to obtain target pose prediction data corresponding to the predicted time stamp comprises:
determining a plurality of differences corresponding to the predicted time stamps and the specified number of display time stamps;
and determining a display time stamp corresponding to the difference value with the smallest numerical value from the plurality of difference values, and obtaining gesture prediction data corresponding to the display time stamp as the target gesture prediction data.
10. The method of claim 8, wherein the storing the correspondence between the pose prediction data and the display timestamp comprises:
storing the corresponding relation between the gesture prediction data and the display time stamp in a shared memory according to the requirement of sequential storage of the display time stamp, wherein the shared memory comprises head data, queue data and tail node tag data, the head data is used for indicating the number of the queue data, the size of the queue data and index data corresponding to the tail node tag data, the queue data comprises the corresponding relation, and the tail node tag data is used for identifying the queue data corresponding to the latest storage time.
11. A posture predicting device, characterized in that the device comprises:
the object posture data acquisition module is used for acquiring object posture data acquired by the posture acquisition equipment at a first moment, and the object posture data are used for linking the posture acquired by the posture acquisition equipment with the posture of a virtual object in a virtual scene;
the object posture data conversion module is used for converting the object posture data into the virtual scene to obtain posture prediction data corresponding to the virtual object and a display time stamp corresponding to the posture prediction data, wherein the display time stamp is the presentation time of the posture prediction data obtained by predicting the posture prediction data based on a first time;
the corresponding relation storage module is used for storing the corresponding relation between the gesture prediction data and the display time stamp;
the prediction time stamp obtaining module is used for obtaining a prediction time stamp at a second moment, wherein the prediction time stamp is obtained by predicting the rendering time consumption of the virtual scene at the second moment;
and the predicted gesture data rendering module is used for matching the predicted timestamp with the display timestamp in the corresponding relation, obtaining target gesture predicted data corresponding to the predicted timestamp and rendering and displaying the target gesture predicted data.
12. A computer device comprising a processor and a memory, wherein the memory has stored therein at least one program that is loaded and executed by the processor to implement the pose prediction method according to any of claims 1 to 10.
13. A computer readable storage medium having stored therein at least one program loaded and executed by a processor to implement the pose prediction method according to any of claims 1 to 10.
14. A computer program product comprising a computer program which when executed by a processor implements the pose prediction method according to any of claims 1 to 10.
CN202211460116.4A 2022-11-17 2022-11-17 Gesture prediction method, device, apparatus, storage medium, and computer program product Pending CN116958487A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211460116.4A CN116958487A (en) 2022-11-17 2022-11-17 Gesture prediction method, device, apparatus, storage medium, and computer program product

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211460116.4A CN116958487A (en) 2022-11-17 2022-11-17 Gesture prediction method, device, apparatus, storage medium, and computer program product

Publications (1)

Publication Number Publication Date
CN116958487A true CN116958487A (en) 2023-10-27

Family

ID=88453601

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211460116.4A Pending CN116958487A (en) 2022-11-17 2022-11-17 Gesture prediction method, device, apparatus, storage medium, and computer program product

Country Status (1)

Country Link
CN (1) CN116958487A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117294832A (en) * 2023-11-22 2023-12-26 湖北星纪魅族集团有限公司 Data processing method, device, electronic equipment and computer readable storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117294832A (en) * 2023-11-22 2023-12-26 湖北星纪魅族集团有限公司 Data processing method, device, electronic equipment and computer readable storage medium
CN117294832B (en) * 2023-11-22 2024-03-26 湖北星纪魅族集团有限公司 Data processing method, device, electronic equipment and computer readable storage medium

Similar Documents

Publication Publication Date Title
CN112614202B (en) GUI rendering display method, terminal, server, electronic equipment and storage medium
CN110716645A (en) Augmented reality data presentation method and device, electronic equipment and storage medium
US20130010071A1 (en) Methods and systems for mapping pointing device on depth map
US20220148279A1 (en) Virtual object processing method and apparatus, and storage medium and electronic device
CN112070906A (en) Augmented reality system and augmented reality data generation method and device
US20230306694A1 (en) Ranking list information display method and apparatus, and electronic device and storage medium
JP2014531693A (en) Motion-controlled list scrolling
CN112788583B (en) Equipment searching method and device, storage medium and electronic equipment
Alshaal et al. Enhancing virtual reality systems with smart wearable devices
CN113052078A (en) Aerial writing track recognition method and device, storage medium and electronic equipment
CN116958487A (en) Gesture prediction method, device, apparatus, storage medium, and computer program product
CN110192169A (en) Menu treating method, device and storage medium in virtual scene
CN109559370A (en) A kind of three-dimensional modeling method and device
US20150352442A1 (en) Game having a Plurality of Engines
CN116152416A (en) Picture rendering method and device based on augmented reality and storage medium
CN111064981B (en) System and method for video streaming
CN108829329B (en) Operation object display method and device and readable medium
KR101481103B1 (en) System of supplying fusion contents multimedia with image based for user participating
KR102292420B1 (en) Apparatus, system and method for producing virtual reality contents
CN112884906A (en) System and method for realizing multi-person mixed virtual and augmented reality interaction
CN114579806B (en) Video detection method, storage medium and processor
US20230154126A1 (en) Creating a virtual object response to a user input
AU2018279783B2 (en) Systems and methods for displaying and interacting with a dynamic real-world environment
CN111065053B (en) System and method for video streaming
CN116958247A (en) Object posture prediction method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40100469

Country of ref document: HK