CN115767206A - Data processing method and system based on extended reality - Google Patents

Abstract

The embodiment of the specification provides a data processing method and a system based on augmented reality, wherein the data processing method based on augmented reality comprises the following steps: in an extended reality scene, audio data generated by a target virtual user in a preset time interval are collected, virtual actions of the target virtual user corresponding to the audio data are determined, animation data of the target virtual user are determined according to the virtual actions, corresponding time stamps are added in coding results of the audio data according to the preset time interval to generate target coding results, the corresponding time stamps are added in the animation data to generate target animation data, the target coding results are sent to a client through an audio transmission channel, the target animation data are sent to the client through an animation data transmission channel, so that the client decodes the target coding results, and the generated decoding results and the animation data are synchronously played according to the time stamps.

Description

Data processing method and system based on extended reality

technical field

The embodiments of this specification relate to the technical field of extended reality, and in particular, to a data processing method based on the extended reality.

Background technique

Extended reality technology is a brand-new human-computer interaction technology. This technology can simulate real scenes, allowing users to feel things in the objective physical world through the virtual reality system, and can break through space, time and other objective constraints. Feel the experience that you can't experience in the real world to "seamlessly integrate" the real world information and the virtual world information.

Currently common extended reality scenarios include virtual game scenarios and virtual live broadcast scenarios. However, the current extended reality scenario is usually implemented by sending multimedia data to the user terminal, and the user terminal separates the corresponding video data and audio data from the multimedia data, displays the video data on the display, and plays the audio data through the speaker. , so as to realize the playback of multimedia data.

However, in this processing method, since the processing time of video data is generally longer than the processing time of audio data, the video playback will lag behind the audio, resulting in the phenomenon that audio and video are not synchronized. Therefore, there is an urgent need for an effective method to solve such problems.

Contents of the invention

In view of this, the embodiment of this specification provides a data processing method based on extended reality. One or more embodiments of this specification also relate to an extended reality-based data processing device, an extended reality-based data processing system, a computing device, a computer-readable storage medium, and a computer program to solve existing Technical defects in the technology.

According to the first aspect of the embodiments of this specification, an extended reality-based data processing method is provided, including:

Collect audio data generated by the target virtual user within a preset time interval in the extended reality scene;

determining a virtual action of the target virtual user corresponding to the audio data, and determining animation data of the target virtual user according to the virtual action;

According to the preset time interval, adding a corresponding time stamp to the encoding result of the audio data to generate a target encoding result, and adding a corresponding time stamp to the animation data to generate target animation data;

sending the target encoding result to the client through an audio transmission channel, and sending the target animation data to the client through an animation data transmission channel, so that the client can decode the target encoding result, and According to the time stamp, the generated decoding result and the animation data are played synchronously.

According to the second aspect of the embodiments of this specification, an extended reality-based data processing device is provided, including:

The collection module is configured to collect audio data generated by the target virtual user within a preset time interval in the extended reality scene;

A determination module configured to determine a virtual action of the target virtual user corresponding to the audio data, and determine animation data of the target virtual user according to the virtual action;

The adding module is configured to add a corresponding time stamp to the encoding result of the audio data according to the preset time interval to generate a target encoding result, and add a corresponding time stamp to the animation data to generate a target animation data;

The sending module is configured to send the target encoding result to the client through the audio transmission channel, and send the target animation data to the client through the animation data transmission channel, so that the client can understand the target The encoding result is decoded, and the generated decoding result and the animation data are played synchronously according to the time stamp.

According to the third aspect of the embodiments of this specification, another extended reality-based data processing method is provided, including:

receiving animation data generated by the target virtual user within a preset time interval in the extended reality scene through the animation data transmission channel, wherein the animation data includes a first time stamp;

receiving an encoding result of audio data generated by the target virtual user within a preset time interval in the extended reality scenario through an audio transmission channel, wherein the encoding result includes a second timestamp;

Decoding the encoding result, and performing animation rendering based on the animation data;

According to the first time stamp and the second time stamp, the generated decoding result and animation rendering result are played synchronously in the extended reality scene.

According to the fourth aspect of the embodiments of this specification, another data processing device based on extended reality is provided, including:

The first receiving module is configured to receive the animation data generated by the target virtual user within a preset time interval in the extended reality scene through the animation data transmission channel, wherein the animation data includes a first time stamp;

The second receiving module is configured to receive an encoding result of audio data generated by the target virtual user within a preset time interval in the extended reality scene through an audio transmission channel, wherein the encoding result includes a second time stamp;

A processing module configured to decode the encoding result, and perform animation rendering based on the animation data;

The playing module is configured to synchronously play the generated decoding result and animation rendering result in the extended reality scene according to the first time stamp and the second time stamp.

According to the fifth aspect of the embodiments of this specification, another data processing system based on extended reality is provided, including:

a first client, a cloud server, and at least one second client;

The first client is configured to collect audio data generated by the target virtual user within a preset time interval in the extended reality scene, determine the virtual action of the target virtual user corresponding to the audio data, and The virtual action determines the animation data of the target virtual user, and according to the preset time interval, adds a corresponding time stamp to the encoding result of the audio data to generate a target encoding result, and adds a corresponding time stamp to the animation data. timestamp, generate target animation data, and send the target encoding result and the target animation data to the cloud server;

The cloud server is configured to transmit the target encoding result to the at least one second client through an audio transmission channel, and transmit the target animation data to the at least one second client through an animation data transmission channel ;

The at least one second client is configured to decode the target encoding result, and perform animation rendering based on the animation data, and synchronously play the generated decoding result and animation rendering result according to the timestamp .

According to a sixth aspect of the embodiments of this specification, a computing device is provided, including:

memory and processor;

The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement any one of the steps of the extended reality-based data processing method.

According to the seventh aspect of the embodiments of this specification, there is provided a computer-readable storage medium, which stores computer-executable instructions, and when the instructions are executed by a processor, any one of the steps of the extended reality-based data processing method is implemented. .

According to an eighth aspect of the embodiments of the present specification, a computer program is provided, wherein, when the computer program is executed in a computer, it causes the computer to execute the steps of the above-mentioned extended reality-based data processing method.

In one embodiment of this specification, by collecting the audio data generated by the target virtual user within a preset time interval in the extended reality scene, the virtual action of the target virtual user corresponding to the audio data is determined, and according to the virtual action Determine the animation data of the target virtual user, add a corresponding time stamp to the encoding result of the audio data according to the preset time interval, generate a target encoding result, and add the corresponding time stamp to the animation data , generate target animation data, send the target encoding result to the client through the audio transmission channel, and send the target animation data to the client through the animation data transmission channel, so that the client can understand the target The encoding result is decoded, and the generated decoding result and the animation data are played synchronously according to the time stamp.

The embodiment of this specification adds time stamps to the audio data and animation data, so that the client can play the animation data and audio data synchronously based on the time stamps, which is conducive to improving the synchronous playback effect of audio and video; in addition, when performing data transmission , using independent transmission channels to transmit the encoding results of animation data and audio data respectively, so that the two do not interfere with each other during the transmission process, which is conducive to ensuring the real-time performance of data transmission, thereby ensuring the efficiency of data processing; in addition, when transmitting animation data , without compressing and encoding it, it is beneficial to avoid data loss, so that the client performs animation rendering based on the animation data, and the rendering result can meet expectations. Based on the animation rendering result and the decoding result of the audio data, the audio and video are played synchronously, which is beneficial Improve user viewing experience.

Description of drawings

FIG. 1 is an architecture diagram of an extended reality-based data processing system provided by an embodiment of this specification;

Fig. 2 is a flow chart of an extended reality-based data processing method provided by an embodiment of this specification;

Fig. 3 is a flow chart of an extended reality-based data processing method provided by an embodiment of this specification;

Fig. 4 is a flowchart of an extended reality-based data processing method provided by an embodiment of this specification;

Fig. 5a is a flowchart of an extended reality-based data processing method provided by an embodiment of this specification;

Fig. 5b is a flowchart of an extended reality-based data processing method provided by an embodiment of this specification;

Fig. 6 is a schematic structural diagram of an extended reality-based data processing device provided by an embodiment of this specification;

Fig. 7 is a flow chart of another data processing method based on extended reality provided by an embodiment of this specification;

Fig. 8 is a flowchart of a processing process of an extended reality-based data processing method provided by an embodiment of this specification;

Fig. 9 is a schematic structural diagram of another data processing device based on extended reality provided by an embodiment of this specification;

Fig. 10 is a structural block diagram of a computing device provided by an embodiment of this specification.

Detailed ways

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the specification. However, this specification can be implemented in many other ways different from those described here, and those skilled in the art can make similar extensions without violating the connotation of this specification, so this specification is not limited by the specific implementations disclosed below.

Terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only, and are not intended to limit one or more embodiments of this specification. As used in one or more embodiments of this specification and the appended claims, the singular forms "a", "the", and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and/or" used in one or more embodiments of the present specification refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present specification, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, the first may also be referred to as the second, and similarly, the second may also be referred to as the first without departing from the scope of one or more embodiments of the present specification. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

First, terms and terms involved in one or more embodiments of this specification are explained.

Virtual user (virtual human, Avatar): A three-dimensional model synthesized through digital technology, which exists in the non-physical world in digital form, and has human appearance behavior perception and anthropomorphic interaction ability dependence.

Audio data: Sound data collected by the microphone.

Animation (action) data: refers to key point data generated during virtual user interaction, such as natural facial expressions such as lip, blink and eyebrow movements, head movement, and key point data of body bone movements. It is necessary to maintain the consistency and non-modification of the animation data, and the data cannot be encoded and lossy compressed.

Taking the lip expression data as an example, it consists of floating point data of 15 visual phonemes, through which the corresponding lip animation is rendered. If there is an error in the data value between the receiving end and the sending end, the rendering effect will be abnormal.

Synchronization: the processing method of audio data and animation data in the production-transmission-reception link, and finally achieve the consistent display effect of "sound-animation".

In this specification, a data processing method based on extended reality is provided. This specification also relates to a data processing device based on extended reality, a data processing system based on extended reality, a computing device, and a computer-readable storage medium. And a computer program, which will be described in detail one by one in the following embodiments.

Fig. 1 shows an architecture diagram of an extended reality-based data processing system provided according to an embodiment of this specification, including:

A first client 102, a cloud server 104 and at least one second client 106;

The first client 102 is configured to collect audio data generated by the target virtual user within a preset time interval in the extended reality scene, determine the virtual action of the target virtual user corresponding to the audio data, and The virtual action determines the animation data of the target virtual user, and according to the preset time interval, a corresponding time stamp is added to the encoding result of the audio data to generate a target encoding result, and is added to the animation data Generate target animation data corresponding to the timestamp, and send the target encoding result and the target animation data to the cloud server 104;

The cloud server 104 is configured to transmit the target encoding result to the at least one second client 106 through an audio transmission channel, and transmit the target animation data to the at least one second client 106 through an animation data transmission channel. Client 106;

The at least one second client 106 is configured to decode the target encoding result, perform animation rendering based on the animation data, and synchronize the generated decoding result and animation rendering result according to the timestamp play.

Specifically, the first client collects the audio data generated by the target virtual user within a preset time interval in the extended reality scene, determines the virtual lip movement and/or virtual body movement of the target virtual user corresponding to the audio data, and The animation data of the target virtual user is determined according to the virtual lip motion and/or the virtual body motion.

The first client encodes the audio data, and according to a preset time interval, adds a corresponding time stamp to the encoding result of the audio data and the animation data, and then sends the encoding result with the time stamp and the animation data to the cloud server.

The cloud server transmits the time-stamped encoding result to the second client through the audio transmission channel, and transmits the time-stamped animation data to the second client through the animation data transmission channel.

The second client decodes the encoding result with the added timestamp, performs animation rendering based on the animation data, and then plays the generated decoding result and animation rendering result synchronously according to the encoding result and the timestamp contained in the animation data.

The embodiment of this specification adds time stamps to the audio data and animation data, so that the client can play the animation data and audio data synchronously based on the time stamps, which is conducive to improving the synchronous playback effect of audio and video; in addition, when performing data transmission , using independent transmission channels to transmit the encoding results of animation data and audio data respectively, so that the two do not interfere with each other during the transmission process, which is conducive to ensuring the real-time performance of data transmission, thereby ensuring the efficiency of data processing; in addition, when transmitting animation data , without compressing and encoding it, it is beneficial to avoid data loss, so that the client performs animation rendering based on the animation data, and the rendering result can meet expectations. Based on the animation rendering result and the decoding result of the audio data, the audio and video are played synchronously, which is beneficial Improve user viewing experience.

The foregoing is a schematic solution of an extended reality-based data processing system in this embodiment. It should be noted that the technical solution of the extended reality-based data processing system and the following technical solution of the extended reality-based data processing method belong to the same concept, and the details of the technical solution based on the extended reality data processing system are not described in detail , can refer to the following description of the technical solution of the data processing method based on extended reality.

Fig. 2 shows a flow chart of an extended reality-based data processing method according to an embodiment of the present specification, which specifically includes the following steps.

Step 202, collecting audio data generated by the target virtual user within a preset time interval in the extended reality scene.

Specifically, the extended reality-based data processing method provided in the embodiment of this specification is applied to the first client.

Extended Reality (XR) refers to the creation of real and virtual combined digital environments and new human-computer interaction methods through modern high-tech means with computers as the core, bringing seamless transition between the virtual world and the real world to the experiencers. The immersion of AR (Augmented Reality, Augmented Reality), VR (Virtual Reality, Virtual Reality), MR (Mediated Reality, Mediated Reality) and other technologies collectively.

The extended reality scene in the embodiment of this specification can be a virtual conference scene or a virtual live broadcast scene based on extended reality. The virtual user who is speaking.

In the extended reality scenario, it is necessary to synchronize the audio data and animation data generated by any virtual user in real time on the user terminals of other virtual users. Therefore, in order to ensure the consistency of the audio and picture of each user terminal, the embodiments of this specification can separately collect Audio data and animation data of the target virtual user, and time stamps are added to the audio data and animation data, so that after the audio data and animation data are sent to each user terminal, each user terminal can use the time stamp as a reference to transfer the animation data and audio The data is played synchronously.

Step 204, determine the virtual action of the target virtual user corresponding to the audio data, and determine the animation data of the target virtual user according to the virtual action.

Specifically, the virtual action is the facial or body action of the virtual user, such as smiling, greeting, and the like.

The purpose of the embodiments of this specification is to synchronously play the audio data and animation data of the target virtual user on each user terminal. Therefore, the audio data and animation data of the virtual user need to be collected separately. However, in practical applications, due to the extended reality scene, The speaking content of the virtual user and the virtual actions made by the virtual user during the speaking process can be set in advance. Therefore, the embodiment of this specification can first collect the audio data generated by the target virtual user in the preset time interval in the extended reality scene, and then determine the corresponding virtual action according to the audio data, and determine the animation data of the target virtual user according to the virtual action .

For example, if the audio data of the target virtual user is: Hi, everyone. According to the audio data, it is determined that the target virtual user is greeting everyone, and then it can be determined that its virtual action includes the waving motion of the limbs. In this case, the waving animation can be determined as the animation data of the target virtual user.

During specific implementation, the audio data generated by the target virtual user within the preset time interval in the extended reality scene is collected, including:

Collecting audio data generated by the target virtual user within at least one first preset time interval in the extended reality scene;

Correspondingly, the determining the animation data of the target virtual user according to the virtual action includes:

According to the time sequence of the at least one first preset time interval, at least one virtual action corresponding to the audio data is integrated to generate animation data of the target virtual user in the second preset time interval, wherein the The second preset time interval is composed of the at least one first preset time interval.

Further, according to the time sequence of the at least one first preset time interval, at least one virtual action corresponding to the audio data is integrated to generate animation data of the target virtual user within the second preset time interval, include:

Determine the first time length corresponding to the second preset time interval according to the preset frame rate;

When it is determined that the first time length is greater than a second time length corresponding to the first preset time interval, determine at least one virtual action corresponding to the second preset time interval;

sorting the at least one virtual action according to the time sequence of the at least one first preset time interval, and determining the target virtual action according to the sorting result;

The target virtual action is integrated to generate animation data of the target virtual user within the second preset time interval.

Specifically, since in the extended reality scene, the target virtual user's speech content and facial or body movements change in real time during the speech process, therefore, in order to ensure the accuracy of the data collection results, the embodiment of this specification can be based on the first Preset time intervals to collect the audio data of the target virtual user in segments. For example, the time length corresponding to the first preset time interval can be 10ms, then the first preset time interval can be 0-10ms, 10ms-20ms, etc., so Within 1 second, 100 audio data of the target virtual user can be collected.

Further, after collecting and obtaining the audio data generated by the target virtual user in at least one first preset time interval, the virtual actions corresponding to each audio data can be integrated according to the chronological order of the first preset time interval to generate Animation data of the target virtual user within the second preset time interval.

Specifically, you can first determine the preset frame rate of the animation, and determine the first time length corresponding to the second preset time interval according to the preset frame rate, and then determine that the first time length is greater than the second time corresponding to the first preset time interval In the case of length, determine at least one virtual action corresponding to the second preset time interval, and sort the virtual actions according to the chronological order of each first preset time interval, and then determine the target virtual action according to the sorting result, and then The target virtual action is integrated to generate animation data of the target virtual user within the second preset time interval.

For example, if the preset frame rate of the animation is 30fps, it means that the playback speed of the animation is to play 30 frames of images per second, which is equivalent to playing one frame of images in 33ms, that is, the time length corresponding to the second preset time interval is 33ms, but The time length of the first preset time interval of the audio data is 10ms, that is, the first time length corresponding to the second preset time interval is greater than the second time length corresponding to the first preset time interval, and every first preset If the audio data in the time interval corresponds to one virtual action, then a second preset time interval will correspond to three virtual actions of the same type. A target virtual action is selected, and then different types of target virtual actions are integrated to generate animation data of the target virtual user within a second preset time interval.

Or, according to the time sequence of the at least one first preset time interval, at least one virtual action corresponding to the audio data is integrated to generate animation data of the target virtual user within the second preset time interval, and The first time length corresponding to the second preset time interval can be determined according to the preset frame rate, and the second preset time can be determined when the first time length is determined to be greater than the second time length corresponding to the first preset time interval At least one virtual action corresponding to the interval, and at least one virtual action is deduplicated, and then according to the preset frame rate and the number of remaining virtual actions after deduplication, determine whether to add or delete the remaining virtual actions, so that the remaining virtual actions The number of actions is equal to the preset frame rate, and the remaining virtual actions are integrated according to the time sequence of each first preset time interval to generate animation data of the target virtual user in the second preset time interval.

During specific implementation, after the audio data of the target virtual user is collected, the virtual action of the target virtual user is determined based on the audio data, and when the animation data is determined according to the virtual action, the virtual action of the target virtual user corresponding to the audio data is determined. The process of action can be realized in the following ways:

Acquire the preset virtual action type of the target virtual user, and determine the visual phoneme corresponding to the target virtual action type;

determining a phoneme value corresponding to a target visual phoneme in the visual phonemes according to the audio data;

A virtual action of the target virtual action type corresponding to the target virtual user is determined according to the phoneme value corresponding to the target visual phoneme.

Specifically, the preset types of virtual movements can include virtual lip movements, virtual eye movements, and virtual body movements, etc. Different virtual movement types can correspond to different numbers of visual phonemes, and different visual phonemes can represent a type A virtual action, the phoneme value (0 or 1) corresponding to the visual phoneme can represent whether the virtual user makes the virtual action.

For example, virtual lip movements can correspond to 15 visual phonemes, namely sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, ih, oh, ou, which correspond to 15 mouth shape. Each visual phoneme represents a virtual movement (mouth shape) of the lips.

Among them, sil means that the virtual user's lips do not make any movement, and when the phoneme value of sil is 1, it means that the virtual user's mouth shape is the mouth shape when no sound is made; PP means that the virtual user's lips are moving. When starting the action of "b, p or m" in English, when the phoneme value of PP is 1, it means that the mouth shape of the virtual user is the mouth shape when sending the English "b, p or m"; FF means the lip shape of the virtual user When the phoneme value of FF is 1, it means that the mouth shape of the virtual user is the mouth shape when the Chinese "f or v" is pronounced; TH means that the lip shape of the virtual user is For the action of starting the English "th", when the phoneme value of TH is 1, it means that the mouth shape of the virtual user is the mouth shape when the English "th" is pronounced.

DD means that the virtual user’s lips make an action of pronouncing the English “t or d”. When the phoneme value of DD is 1, it means that the virtual user’s mouth shape is the mouth shape when pronouncing the English “t or d”; kk is It represents the movement of the virtual user's lips to pronounce the English "k or g". The user's lips make the movement of speaking English "tS, dZ or S", and when the phoneme value of CH is 1, it means that the virtual user's mouth shape is the mouth shape when speaking English "tS, dZ or S".

SS means that the virtual user's lips make an action of "s or z" in English. When the phoneme value of SS is 1, it means that the mouth shape of the virtual user is the mouth shape of the English "s or z"; nn means Represents the virtual user's lips making the movement of "n or l" in English. When the phoneme value of nn is 1, it means that the mouth shape of the virtual user is the mouth shape when the English "n or l" is pronounced; RR stands for virtual The user's lips make an action of pronouncing the English "r", and when the phoneme value of RR is 1, it means that the virtual user's mouth shape is the same as when the English "r" is pronounced; aa means that the virtual user's lips make For the action of pronouncing English "a", when the phoneme value of aa is 1, it means that the virtual user's mouth shape is the same as when pronouncing English "a".

E means that the virtual user's lips make an action of pronouncing the English "e". When the phoneme value of E is 1, it means that the virtual user's mouth shape is the mouth shape when the English "e" is pronounced; ih means the virtual user's mouth shape The lips make the movement of speaking English "a". When the phoneme value of ih is 1, it means that the virtual user's mouth shape is the mouth shape when speaking English "a"; oh means that the virtual user's lips make the pronunciation of English For the action of "oh", when the phoneme value of oh is 1, it means that the mouth shape of the virtual user is the mouth shape of the English "oh"; When the phoneme value of ou is 1, it means that the mouth shape of the virtual user is the mouth shape of the English "ou".

Therefore, the embodiment of this specification can obtain the preset virtual action type of the target virtual user, and determine the visual phoneme corresponding to the target virtual action type, and then determine the phoneme value corresponding to the target visual phoneme in the visual phoneme according to the audio data, and according to the target visual phoneme The phoneme value corresponding to the phoneme determines the virtual action of the target virtual action type corresponding to the target virtual user.

For example, if the target virtual action type is virtual lip action, the corresponding visual phonemes are sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, ih, oh, and ou. Then, according to the audio data, it is determined that the target virtual user has pronounced the Chinese "f" within the first preset time interval, then the phoneme value of the visual phoneme FF can be set to 1, and the target virtual user corresponding to the target virtual user can be determined. The virtual action of the action type is the action of the lips making the Chinese "f" sound, and then the animation data of the target virtual user can be determined according to this action.

Step 206: According to the preset time interval, add a corresponding time stamp to the encoding result of the audio data to generate a target encoding result, and add a corresponding time stamp to the animation data to generate target animation data.

Specifically, in the embodiment of this specification, after the audio data and animation data are sent to the client, when the client plays the audio data and animation data, the audio and video synchronization can be realized, and the audio data can be compressed and encoded before the data is sent. And add the corresponding time stamp to the compressed encoding result to generate the target encoding result. As for the animation data, after the animation data is sent to the client, the client needs to perform animation rendering based on the animation data. If the animation data is compressed and encoded before the animation data is sent, the client will After the result, it needs to be decoded. This process of compression encoding and decoding may lead to the loss of part of the animation data, resulting in the failure of the animation rendering effect to meet expectations. Perform compression, directly add a time stamp to the animation data, and send the target animation data with the time stamp and the target encoding result of the audio data to the client.

During specific implementation, according to the preset time interval, a corresponding time stamp is added to the encoding result of the audio data to generate a target encoding result, and a corresponding time stamp is added to the animation data to generate target animation data, include:

According to the first preset time interval, adding a corresponding time stamp to the encoding result of the audio data to generate a target encoding result; and,

According to the second preset time interval, a corresponding time stamp is added to the animation data to generate target animation data.

Specifically, as mentioned above, since the time lengths corresponding to the first preset time interval and the second preset time interval may be different, in order to ensure the accuracy of the timestamp adding results, the first preset time interval Add a corresponding time stamp to the encoding result of the audio data, and add a corresponding time stamp to the animation data according to the second preset time interval, the time stamp can be the first preset time interval or the second preset time interval start time or end time.

Step 208: Send the target encoding result to the client through the audio transmission channel, and send the target animation data to the client through the animation data transmission channel, so that the client performs the target encoding result on the client. Decoding, and synchronously playing the generated decoding result and the animation data according to the time stamp.

Specifically, in the embodiment of this specification, the target encoding result of audio data and the target animation data can be transmitted through independent data transmission channels, so that the audio data and animation data do not interfere with each other during the transmission process, thereby ensuring real-time data transmission. sex.

After sending the target encoding result of the audio data to the client through the audio transmission channel, and sending the target animation data to the client through the animation data transmission channel, the client can decode the target encoding result of the audio data, and The time stamp added to the result and the target animation data, and the generated decoding result and animation data are played synchronously to achieve the audio and video synchronization effect of the client.

During specific implementation, the target encoding result is sent to the client through the audio transmission channel, and the target animation data is sent to the client through the animation data transmission channel, including:

Send the target encoding result and the target animation data to the cloud server, so that the cloud server sends the target encoding result to the client through the audio transmission channel, and transmits the target animation data through the animation data transmission channel sent to the client.

Specifically, the embodiment of this specification can realize data transmission by means of a cloud server, that is, when the target encoding result of audio data and target animation data need to be transmitted to the client, the target encoding result of audio data and target animation data can be sent to the The cloud server sends the target encoding result to the client through the audio transmission channel, and sends the target animation data to the client through the animation data transmission channel.

In addition, before sending the encoding result of the audio data and the animation data to the cloud server or the client, the target encoding result can also be added to the audio data sending queue according to the timestamp in the target encoding result; and,

Add the target animation data to an animation data sending queue according to the time stamp in the target animation data.

Further, sending the target encoding result to the client through an audio transmission channel, and sending the target animation data to the client through an animation data transmission channel, including:

Sending the target encoding result in the audio data sending queue to the cloud server according to a first preset time period; and,

Send the target animation data in the animation data sending queue to the cloud server according to the second preset time period, so that the cloud server will send the target encoding result to the client through the audio transmission channel, and send the target animation data to the client through the audio transmission channel. The target animation data is sent to the client through the animation data transmission channel.

Specifically, after collecting audio data and animation data for a period of time, the audio data can be preprocessed and added to the audio data sending queue. The preprocessing includes compression encoding and adding a time stamp VTS, etc. Add the time stamp ATS to the animation data, and keep the form of binary data, and add it to the animation data sending queue.

For the target encoding results in the audio data sending queue and the target animation data in the animation data sending queue, they can be regularly taken out from the sending queue according to the same or different preset time periods, and sent to the cloud server, and the cloud server will The target encoding result is sent to the client through the audio transmission channel, and the target animation data is sent to the client through the animation data transmission channel.

Based on this, a schematic diagram of a data processing process based on an extended reality scene provided by the embodiment of this specification is shown in FIG. 3 . In Fig. 3, the sending end (first client) first generates audio data and animation data within a period of time, for example: 10ms audio data, 33ms animation data, wherein the animation data can include lip movements related to audio, Natural facial expressions such as blinking and eyebrow movements, head movements, body bone movements, and more.

Then preprocess the audio data and add it to the audio data sending queue. The preprocessing includes compression, encoding and adding time stamp VTS, etc.; add the time stamp ATS to the animation data, and keep the form of binary data, and add it to the animation data sending queue. And periodically take out the data in the sending queue and send it to the cloud server.

The audio transmission channel and the animation data transmission channel on the cloud server side are independent services, which are respectively responsible for forwarding data to the receiving end (second client). An independent animation data transmission channel can send large-capacity data without interfering with audio data.

The receiving end receives audio data and animation data, adds them to the buffer after processing, and then synchronizes the animation data and audio data. When playing, the audio data is played and the animation is rendered in the current frame.

In practical applications, there are audio data processing threads and animation data processing threads at the receiving end (second client). The schematic diagram of the audio-video synchronous playback process provided by the embodiment of this specification is shown in FIG. 4 .

In Fig. 4, for the audio data processing thread, the audio data and the time stamp VTS of the buffer can be read first, and then the time stamp CTS of the current cursor can be updated to make CTS=VTS, and the audio can be played. For the animation data processing thread, the time stamp CTS can be updated first. Read the animation data and timestamp ATS of the buffer, and judge whether the ATS matches the timestamp CTS of the current cursor. The matching conditions can be customized, for example, the difference between the two is less than 20ms. If it matches, perform animation rendering and display based on the animation data; if it does not match, then judge whether (ATS-CTS) is less than DeltaTS, if (ATS-CTS)<DeltaTS, then discard the animation data; if (ATS-CTS)≥ DeltaTS means that the rendering time of the animation data has not been reached. In this case, wait for the CTS update and delay the rendering. Among them, DeltaTS represents the synchronization delay threshold, which can be customized according to the actual situation, for example, set to 50ms.

In addition, a schematic diagram of an extended reality-based data processing process applied to a virtual live broadcast scene provided by the embodiment of this specification is shown in Figure 5a. In Figure 5a, the anchor terminal collects the anchor's audio data and animation data such as body expressions, and generates Synchronize time stamps, add time stamps to audio data and animation data, and then transmit audio data and animation data such as body expressions to the audience terminal through the cloud server, and the audience terminal performs animation rendering based on the animation data, and performs audio and animation Play synchronously.

A schematic diagram of an extended reality-based data processing process applied to a virtual meeting scene provided by the embodiment of this specification is shown in Figure 5b. In Figure 5b, the speaker terminal collects the audio data of the conference speaker and animation data such as body expressions, Generate synchronous timestamps, and add timestamps to audio data and animation data respectively, then transmit audio data and animation data such as body expressions to the cloud server, and perform animation rendering based on the animation data through the cloud rendering engine service, and convert the audio data and animation rendering results are sent to the audience terminal for synchronous audio and animation playback on the audience terminal, so as to achieve the effect of live video broadcasting. In addition, the user can submit interactive instructions with the speaker through the audience terminal, and the cloud rendering engine service will execute the interactive instructions. processing, and send the processing result to the audience terminal to display the corresponding interactive effect.

In one embodiment of this specification, by collecting the audio data generated by the target virtual user within a preset time interval in the extended reality scene, the virtual action of the target virtual user corresponding to the audio data is determined, and according to the virtual action Determine the animation data of the target virtual user, add a corresponding time stamp to the encoding result of the audio data according to the preset time interval, generate a target encoding result, and add the corresponding time stamp to the animation data , generate target animation data, send the target encoding result to the client through the audio transmission channel, and send the target animation data to the client through the animation data transmission channel, so that the client can understand the target The encoding result is decoded, and the generated decoding result and the animation data are played synchronously according to the time stamp.

The embodiment of this specification adds time stamps to the audio data and animation data, so that the client can play the animation data and audio data synchronously based on the time stamps, which is conducive to improving the synchronous playback effect of audio and video; in addition, when performing data transmission , using independent transmission channels to transmit the target animation data and audio data target encoding results respectively, so that the two do not interfere with each other during the transmission process, which is conducive to ensuring the real-time performance of data transmission, thereby ensuring the efficiency of data processing; in addition, when transmitting animation data, it is not compressed and encoded, which is beneficial to avoid data loss, so that the client can perform animation rendering based on the animation data, and the rendering result can meet expectations. It is beneficial to improve the viewing experience of the user.

Corresponding to the foregoing method embodiments, this specification also provides an embodiment of an extended reality-based data processing device. FIG. 6 shows a schematic structural diagram of an extended reality-based data processing device provided by an embodiment of this specification. As shown in Figure 6, the device includes:

The collection module 602 is configured to collect audio data generated by the target virtual user within a preset time interval in the extended reality scene;

The determination module 604 is configured to determine the virtual action of the target virtual user corresponding to the audio data, and determine the animation data of the target virtual user according to the virtual action;

The adding module 606 is configured to add a corresponding time stamp to the encoding result of the audio data according to the preset time interval to generate a target encoding result, and add a corresponding time stamp to the animation data to generate a target animation data;

The sending module 608 is configured to send the target encoding result to the client through an audio transmission channel, and send the target animation data to the client through an animation data transmission channel, so that the client can Decoding the target encoding result, and synchronously playing the generated decoding result and the animation data according to the timestamp.

Optionally, the collection module 602 is further configured to:

Collecting audio data generated by the target virtual user within at least one first preset time interval in the extended reality scene;

Correspondingly, the determining module 604 is further configured to:

According to the time sequence of the at least one first preset time interval, at least one virtual action corresponding to the audio data is integrated to generate animation data of the target virtual user in the second preset time interval, wherein the The second preset time interval is composed of the at least one first preset time interval.

Optionally, the determining module 604 is further configured to:

Determine the first time length corresponding to the second preset time interval according to the preset frame rate;

When it is determined that the first time length is greater than a second time length corresponding to the first preset time interval, determine at least one virtual action corresponding to the second preset time interval;

sorting the at least one virtual action according to the time sequence of the at least one first preset time interval, and determining the target virtual action according to the sorting result;

The target virtual action is integrated to generate animation data of the target virtual user within the second preset time interval.

Optionally, the adding module 606 is further configured to:

According to the first preset time interval, adding a corresponding time stamp to the encoding result of the audio data to generate a target encoding result; and,

According to the second preset time interval, a corresponding time stamp is added to the animation data to generate target animation data.

Optionally, the determining module 604 is further configured to:

Acquire the preset virtual action type of the target virtual user, and determine the visual phoneme corresponding to the target virtual action type;

determining a phoneme value corresponding to a target visual phoneme in the visual phonemes according to the audio data;

A virtual action of the target virtual action type corresponding to the target virtual user is determined according to the phoneme value corresponding to the target visual phoneme.

Optionally, the sending module 608 is further configured to:

Send the target encoding result and the target animation data to the cloud server, so that the cloud server sends the target encoding result to the client through the audio transmission channel, and transmits the target animation data through the animation data transmission channel sent to the client.

Optionally, the extended reality-based data processing device further includes a processing module configured to:

adding the target encoding result to an audio data sending queue according to the timestamp in the target encoding result; and,

Add the target animation data to an animation data sending queue according to the time stamp in the target animation data.

Optionally, the sending module 608. is further configured to:

Sending the target encoding result in the audio data sending queue to the cloud server according to a first preset time period; and,

Send the target animation data in the animation data sending queue to the cloud server according to the second preset time period, so that the cloud server will send the target encoding result to the client through the audio transmission channel, and send the target animation data to the client through the audio transmission channel. The target animation data is sent to the client through the animation data transmission channel.

The foregoing is a schematic solution of an extended reality-based data processing device in this embodiment. It should be noted that the technical solution of the data processing device based on extended reality and the technical solution of the data processing method based on extended reality belong to the same idea, and the details of the technical solution based on the data processing device of extended reality are not described in detail. For details, please refer to the above description of the technical solution of the data processing method based on extended reality.

Fig. 7 shows a flow chart of another extended reality-based data processing method according to an embodiment of the present specification, which specifically includes the following steps.

Step 702: Receive animation data generated by the target virtual user within a preset time interval in the extended reality scene through the animation data transmission channel, wherein the animation data includes a first time stamp.

Step 704: Receive an encoding result of audio data generated by the target virtual user within a preset time interval in the extended reality scenario through an audio transmission channel, wherein the encoding result includes a second time stamp.

Step 706, decode the encoding result, and perform animation rendering based on the animation data.

Step 708, according to the first time stamp and the second time stamp, synchronously play the generated decoding result and animation rendering result in the extended reality scene.

During specific implementation, the synchronous playback of the generated decoding result and animation rendering result in the extended reality scene according to the first timestamp and the second timestamp includes:

updating a reference time according to the first time stamp, and determining whether the time difference between the second time stamp and the reference time is less than a first preset time threshold;

If so, in the extended reality scene, the generated decoding result and animation rendering result are played synchronously.

In addition, when it is determined that the time difference between the second time stamp and the reference time is greater than a first preset time threshold, in the extended reality scene, play the generated decoding result;

judging whether the time difference between the second time stamp and the reference time is less than a second preset time threshold;

If so, then delete the animation data;

If not, execute the step of updating the reference time according to the first timestamp.

Specifically, the extended reality-based data processing method provided by the embodiment of this specification is applied to the second client, and the second client receives the encoding result of the audio data and the animation data respectively through independent channels. In addition, the second client has Audio data processing thread and animation data processing thread, for audio data processing thread, can read the audio data and timestamp VTS of the buffer first, then update the timestamp CTS of the current cursor, make CTS=VTS, and play audio, for animation The data processing thread can first read the animation data in the buffer and the timestamp ATS, and judge whether the ATS matches the timestamp CTS of the current cursor. The matching conditions can be customized, for example, the difference between the two is less than 20ms. If it matches, perform animation rendering and display based on the animation data; if it does not match, then judge whether (ATS-CTS) is less than DeltaTS, if (ATS-CTS)<DeltaTS, then discard the animation data; if (ATS-CTS)≥ DeltaTS means that the rendering time of the animation data has not been reached. In this case, wait for the CTS update and delay the rendering. Among them, DeltaTS represents the synchronization delay threshold, which can be customized according to the actual situation, for example, set to 50ms.

The embodiment of this specification adds time stamps to the audio data and animation data, so that the client can play the animation data and audio data synchronously based on the time stamps, which is conducive to improving the synchronous playback effect of audio and video; in addition, when performing data transmission , using independent transmission channels to transmit the encoding results of animation data and audio data respectively, so that the two do not interfere with each other during the transmission process, which is conducive to ensuring the real-time performance of data transmission, thereby ensuring the efficiency of data processing; in addition, when transmitting animation data , without compressing and encoding it, it is beneficial to avoid data loss, so that the client performs animation rendering based on the animation data, and the rendering result can meet expectations. Based on the animation rendering result and the decoding result of the audio data, the audio and video are played synchronously, which is beneficial Improve user viewing experience.

The foregoing is a schematic solution of another data processing method based on extended reality in this embodiment. It should be noted that the technical solution of another extended reality-based data processing method is the same concept as the above-mentioned technical solution of an extended reality-based data processing method, and the technical solution of an extended reality-based data processing device is not described in detail. For the details of the description, please refer to the description of the technical solution of the above-mentioned extended reality-based data processing method.

The following describes the data processing method based on extended reality by taking the application of the data processing method based on extended reality provided in this specification in a virtual conference scene as an example in conjunction with FIG. 8 . Wherein, FIG. 8 shows a flowchart of a processing process of an extended reality-based data processing method provided by an embodiment of this specification, which specifically includes the following steps.

Step 802, the first client collects the audio data generated by the speaking user within a preset time interval in the virtual conference scene based on the extended reality, and determines the virtual lip movement and/or virtual body movement of the speaking user corresponding to the audio data , and determine the animation data of the speaking user according to the virtual lip movement and/or the virtual body movement.

Step 804, the first client encodes the audio data, and adds a corresponding time stamp to the encoding result of the audio data and the animation data according to the preset time interval.

Step 806, the first client sends the coding result and animation data with time stamp added to the cloud server.

Step 808, the cloud server transmits the encoding result with the time stamp added to the second client through the audio transmission channel.

Step 810, the cloud server transmits the animation data with time stamp added to the second client through the animation data transmission channel.

Step 812, the second client decodes the encoded result with the time stamp added, and performs animation rendering based on the animation data.

Step 814, the second client synchronously plays the generated decoding result and animation rendering result according to the encoding result with added time stamp and the time stamp included in the animation data.

The embodiment of this specification adds time stamps to the audio data and animation data, so that the client can play the animation data and audio data synchronously based on the time stamps, which is conducive to improving the synchronous playback effect of audio and video; in addition, when performing data transmission , using independent transmission channels to transmit the encoding results of animation data and audio data respectively, so that the two do not interfere with each other during the transmission process, which is conducive to ensuring the real-time performance of data transmission, thereby ensuring the efficiency of data processing; in addition, when transmitting animation data , without compressing and encoding it, it is beneficial to avoid data loss, so that the client performs animation rendering based on the animation data, and the rendering result can meet expectations. Based on the animation rendering result and the decoding result of the audio data, the audio and video are played synchronously, which is beneficial Improve user viewing experience.

Corresponding to the above-mentioned method embodiments, this specification also provides another embodiment of an extended reality-based data processing device. FIG. 9 shows the structure of another extended reality-based data processing device provided by an embodiment of this specification. schematic diagram. As shown in Figure 9, the device includes:

The first receiving module 902 is configured to receive the animation data generated by the target virtual user within a preset time interval in the extended reality scene through the animation data transmission channel, wherein the animation data includes a first time stamp;

The second receiving module 904 is configured to receive an encoding result of the audio data generated by the target virtual user within a preset time interval in the extended reality scene through an audio transmission channel, wherein the encoding result includes a second time stamp ;

The processing module 906 is configured to decode the encoding result, and perform animation rendering based on the animation data;

The playing module 908 is configured to synchronously play the generated decoding result and animation rendering result in the extended reality scene according to the first time stamp and the second time stamp.

Optionally, the playback module 908 is further configured to:

updating a reference time according to the first time stamp, and determining whether the time difference between the second time stamp and the reference time is less than a first preset time threshold;

If so, in the extended reality scene, the generated decoding result and animation rendering result are played synchronously.

Optionally, the playback module 908 is further configured to:

When it is determined that the time difference between the second time stamp and the reference time is greater than a first preset time threshold, in the extended reality scene, play the generated decoding result;

judging whether the time difference between the second time stamp and the reference time is less than a second preset time threshold;

If so, then delete the animation data;

If not, execute the step of updating the reference time according to the first timestamp.

The foregoing is a schematic solution of another data processing device based on extended reality in this embodiment. It should be noted that the technical solution of another data processing device based on extended reality belongs to the same idea as the technical solution of another data processing method based on extended reality, and the technical solution of the data processing device based on extended reality is not For details of the detailed description, refer to the description of the technical solution of another extended reality-based data processing method above.

Fig. 10 shows a structural block diagram of a computing device 1000 provided according to an embodiment of this specification. Components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020 . The processor 1020 is connected to the memory 1010 through the bus 1030, and the database 1050 is used for storing data.

Computing device 1000 also includes access device 1040 that enables computing device 1000 to communicate via one or more networks 1060 . Examples of these networks include the Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or a combination of communication networks such as the Internet. Access device 1040 may include one or more of any type of network interface (e.g., a network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, Worldwide Interoperability for Microwave Access ( Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, etc.

In an embodiment of the present specification, the above-mentioned components of the computing device 1000 and other components not shown in FIG. 10 may also be connected to each other, for example, through a bus. It should be understood that the structural block diagram of the computing device shown in FIG. 10 is only for the purpose of illustration, rather than limiting the scope of this description. Those skilled in the art can add or replace other components as needed.

Computing device 1000 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile telephones (e.g., smartphones), ), wearable computing devices (eg, smart watches, smart glasses, etc.), or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. Computing device 1000 may also be a mobile or stationary server.

Wherein, the processor 1020 is configured to execute the following computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned extended reality-based data processing method are implemented.

The foregoing is a schematic solution of a computing device in this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned extended reality-based data processing method belong to the same concept. Description of the technical solution of the method.

An embodiment of the present specification also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the above-mentioned extended reality-based data processing method are implemented.

The foregoing is a schematic solution of a computer-readable storage medium in this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above-mentioned extended reality-based data processing method belong to the same idea, and details not described in detail in the technical solution of the storage medium can be found in the above-mentioned data processing based on extended reality Description of the technical solution of the method.

An embodiment of the present specification also provides a computer program, wherein, when the computer program is executed in a computer, the computer is made to execute the steps of the above-mentioned extended reality-based data processing method.

The foregoing is a schematic solution of a computer program in this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above-mentioned extended reality-based data processing method belong to the same idea, and details not described in detail in the technical solution of the computer program can be found in the above-mentioned data processing based on extended reality Description of the technical solution of the method.

The foregoing describes specific embodiments of this specification. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

The computer instructions include computer program code, which may be in source code form, object code form, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-OnlyMemory), Random access memory (RAM, RandomAccessMemory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, computer-readable media Excludes electrical carrier signals and telecommunication signals.

It should be noted that, for the sake of simplicity of description, the above-mentioned method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the embodiments of this specification are not limited by the described action sequence. Because according to the embodiment of the present specification, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the specification.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are only for helping to explain the present specification. Alternative embodiments are not exhaustive in all detail, nor are the inventions limited to specific implementations described. Obviously, many modifications and changes can be made according to the contents of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can well understand and use this specification. This specification is to be limited only by the claims, along with their full scope and equivalents.

Claims (14)

1. A data processing method based on extended reality, comprising: Collect audio data generated by the target virtual user within a preset time interval in the extended reality scene; determining a virtual action of the target virtual user corresponding to the audio data, and determining animation data of the target virtual user according to the virtual action; According to the preset time interval, adding a corresponding time stamp to the encoding result of the audio data to generate a target encoding result, and adding a corresponding time stamp to the animation data to generate target animation data; sending the target encoding result to the client through an audio transmission channel, and sending the target animation data to the client through an animation data transmission channel, so that the client can decode the target encoding result, and According to the time stamp, the generated decoding result and the animation data are played synchronously. 2. The data processing method based on extended reality according to claim 1, wherein in the collection of the extended reality scene, the audio data generated by the target virtual user within a preset time interval includes: Collecting audio data generated by the target virtual user within at least one first preset time interval in the extended reality scene; Correspondingly, the determining the animation data of the target virtual user according to the virtual action includes: According to the time sequence of the at least one first preset time interval, at least one virtual action corresponding to the audio data is integrated to generate animation data of the target virtual user in the second preset time interval, wherein the The second preset time interval is composed of the at least one first preset time interval. 3. The data processing method based on extended reality according to claim 2, wherein the at least one virtual action corresponding to the audio data is integrated according to the time sequence of the at least one first preset time interval to generate the The animation data of the target virtual user in the second preset time interval includes: Determine the first time length corresponding to the second preset time interval according to the preset frame rate; When it is determined that the first time length is greater than a second time length corresponding to the first preset time interval, determine at least one virtual action corresponding to the second preset time interval; sorting the at least one virtual action according to the time sequence of the at least one first preset time interval, and determining the target virtual action according to the sorting result; The target virtual action is integrated to generate animation data of the target virtual user within the second preset time interval. 4. The data processing method based on extended reality according to claim 2 or 3, wherein according to the preset time interval, a corresponding time stamp is added to the encoding result of the audio data to generate a target encoding result, and Add corresponding timestamps to the animation data to generate target animation data, including: According to the first preset time interval, adding a corresponding time stamp to the encoding result of the audio data to generate a target encoding result; and, According to the second preset time interval, a corresponding time stamp is added to the animation data to generate target animation data. 5. The data processing method based on extended reality according to claim 1, said determining the virtual action of said target virtual user corresponding to said audio data, comprising: Acquire the preset virtual action type of the target virtual user, and determine the visual phoneme corresponding to the target virtual action type; determining a phoneme value corresponding to a target visual phoneme in the visual phonemes according to the audio data; A virtual action of the target virtual action type corresponding to the target virtual user is determined according to the phoneme value corresponding to the target visual phoneme. 6. The data processing method based on extended reality according to claim 1, wherein the target encoding result is sent to the client through an audio transmission channel, and the target animation data is sent to the client through an animation data transmission channel. Clients, including: Send the target encoding result and the target animation data to the cloud server, so that the cloud server sends the target encoding result to the client through the audio transmission channel, and transmits the target animation data through the animation data transmission channel sent to the client. 7. The data processing method based on extended reality according to claim 1, further comprising: adding the target encoding result to an audio data sending queue according to the timestamp in the target encoding result; and, Add the target animation data to an animation data sending queue according to the time stamp in the target animation data. 8. The data processing method based on extended reality according to claim 7, wherein the target encoding result is sent to the client through an audio transmission channel, and the target animation data is sent to the client through an animation data transmission channel. Clients, including: Sending the target encoding result in the audio data sending queue to the cloud server according to a first preset time period; and, Send the target animation data in the animation data sending queue to the cloud server according to the second preset time period, so that the cloud server will send the target encoding result to the client through the audio transmission channel, and send the target animation data to the client through the audio transmission channel. The target animation data is sent to the client through the animation data transmission channel. 9. A data processing method based on extended reality, comprising: receiving animation data generated by the target virtual user within a preset time interval in the extended reality scene through the animation data transmission channel, wherein the animation data includes a first time stamp; receiving an encoding result of audio data generated by the target virtual user within a preset time interval in the extended reality scenario through an audio transmission channel, wherein the encoding result includes a second timestamp; Decoding the encoding result, and performing animation rendering based on the animation data; According to the first time stamp and the second time stamp, the generated decoding result and animation rendering result are played synchronously in the extended reality scene. 10. The data processing method based on extended reality according to claim 9, according to the first timestamp and the second timestamp, the generated decoding result and animation rendering result in the extended reality scene Play synchronously, including: updating a reference time according to the first time stamp, and determining whether the time difference between the second time stamp and the reference time is less than a first preset time threshold; If so, in the extended reality scene, the generated decoding result and animation rendering result are played synchronously. 11. The data processing method based on extended reality according to claim 10, further comprising: When it is determined that the time difference between the second time stamp and the reference time is greater than a first preset time threshold, in the extended reality scene, play the generated decoding result; judging whether the time difference between the second time stamp and the reference time is less than a second preset time threshold; If so, then delete the animation data; If not, execute the step of updating the reference time according to the first timestamp. 12. A data processing system based on extended reality, comprising: a first client, a cloud server, and at least one second client; The first client is configured to collect audio data generated by the target virtual user within a preset time interval in the extended reality scene, determine the virtual action of the target virtual user corresponding to the audio data, and The virtual action determines the animation data of the target virtual user, and according to the preset time interval, adds a corresponding time stamp to the encoding result of the audio data to generate a target encoding result, and adds a corresponding time stamp to the animation data. timestamp, generate target animation data, and send the target encoding result and the target animation data to the cloud server; The cloud server is configured to transmit the target encoding result to the at least one second client through an audio transmission channel, and transmit the target animation data to the at least one second client through an animation data transmission channel ; The at least one second client is configured to decode the target encoding result, and perform animation rendering based on the animation data, and synchronously play the generated decoding result and animation rendering result according to the timestamp . 13. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the extended reality-based data described in any one of claims 1 to 11 is realized. The steps of the processing method. 14. A computer-readable storage medium storing computer-executable instructions, which implement the steps of the extended reality-based data processing method according to any one of claims 1 to 11 when the computer-executable instructions are executed by a processor.

Images