CN118314255A - Display method, device, equipment, readable storage medium and computer program product - Google Patents

Abstract

The present application discloses a display method, an apparatus, a device, a readable storage medium and a computer program product. The method of an embodiment of the present application includes: obtaining the conversation content between a user and a virtual object and the video data of the conversation between the user and the virtual object; identifying the conversation content to obtain a conversation recognition result, and identifying the video data to obtain the first skeleton key point data of the user; determining the second skeleton key point data according to the conversation recognition result and the first skeleton key point data; and controlling the display of the virtual object according to the second skeleton key point data.

Description

Display method, device, equipment, readable storage medium and computer program product

Technical Field

The present application belongs to the field of communication technology, and specifically relates to a display method, device, equipment, readable storage medium and computer program product.

Background technique

Currently, when experiencing the metaverse scene, most virtual objects have no body movements or simple movements when talking to users, giving users a feeling that it is not a real-scene conversation or chat, resulting in poor interactive immersion and a poor user experience.

Summary of the invention

The embodiments of the present application provide a display method, apparatus, device, readable storage medium and computer program product, which can solve the problems of poor immersion and poor user experience in the existing interaction between virtual objects and users.

In a first aspect, a display method is provided, comprising:

Acquire the conversation content between the user and the virtual object and the video data of the conversation between the user and the virtual object;

Recognize the conversation content to obtain a conversation recognition result, and recognize the video data to obtain first skeleton key point data of the user;

Determine second skeleton key point data according to the dialogue recognition result and the first skeleton key point data;

The display of the virtual object is controlled according to the second skeleton key point data.

In some implementations, the step of identifying the video data to obtain first skeleton key point data includes:

Determining an image frame sequence according to the video data;

Performing posture estimation on the image frame sequence to obtain two-dimensional key point data;

The two-dimensional key point data is filtered to obtain the first skeleton key point data.

In some implementations, performing posture estimation on the image frame sequence to obtain two-dimensional key point data includes:

Determine N image frames according to the image frame sequence;

Inputting the N image frames into a posture estimation model in sequence to obtain N two-dimensional key point data, wherein the posture estimation model is a convolutional neural network model;

Wherein, N is a positive integer greater than 1.

In some implementations, filtering the two-dimensional key point data to obtain the first skeleton key point data includes:

Subtract adjacent frames from each other to obtain N-1 key point differential data.

Determine adaptive stable time series differential data according to the N two-dimensional key point data and the N-1 key point differential data;

Inputting the adaptive stable time-series difference data and the N two-dimensional key point data into a filtering model to obtain the first skeleton key point data;

Among them, the network structure of the filtering model includes a fully connected residual layer.

In some implementations, determining the adaptive stable temporal difference data according to the N two-dimensional key point data and the N-1 key point difference data includes:

Determine a key point data change rate between two adjacent frames according to the N two-dimensional key point data and the N-1 key point differential data;

Determine a mean value of the key point data change rate according to the key point data change rate of the two adjacent frames;

The adaptive stable time series differential data is determined according to the key point data change rate of the two adjacent frames, the key point data change rate average and the N-1 key point differential data.

In some implementations, determining second skeleton key point data according to the dialogue recognition result and the first skeleton key point data includes:

According to the dialogue recognition result, a plurality of first matching results are determined in a virtual object corpus in a virtual object database; wherein the dialogue recognition result includes an emotion value and a dialogue scene, the virtual object database includes a virtual object corpus and a virtual object action library, the virtual object corpus includes a plurality of pre-stored text contents, the virtual object action library includes a plurality of pre-stored skeleton key point data, and each of the pre-stored skeleton key point data has a corresponding emotion value and a dialogue scene;

Calculating the similarity between each of the first matching results and the conversation recognition result;

In the case where the similarities between all the first matching results and the dialogue recognition results exceed a preset similarity range, determining the first pre-stored skeleton key point data corresponding to the emotion value in the dialogue recognition result in the virtual object action library as the second skeleton key point data;

In the case where there is a first matching result whose similarity is within a preset similarity range among the plurality of first matching results, determining at least one second pre-stored skeleton key point data in the virtual object action library according to the dialogue scene in the dialogue recognition result;

According to the dialogue scene in the dialogue recognition result and the first skeleton key point data, the second skeleton key point data is determined in the at least one second pre-stored skeleton key point data.

In some implementations, determining the second skeleton key point data in the at least one second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result and the first skeleton key point data includes:

Calculate a first similarity of each of the second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result, the first weight, and each of the second pre-stored skeleton key point data;

Calculate the second similarity of each of the second pre-stored skeleton key point data according to the first skeleton key point data, the second weight and each of the second pre-stored skeleton key point data;

Determine a third similarity of each of the second pre-stored skeleton key point data according to the first similarity and the second similarity;

The second pre-stored skeleton key point data satisfying a preset condition according to the third similarity is determined as the second skeleton key point data.

In a second aspect, a display device is provided, comprising:

An acquisition module, used to acquire the conversation content between the user and the virtual object and the video data of the conversation between the user and the virtual object;

A recognition module, used for recognizing the conversation content to obtain a conversation recognition result, and recognizing the video data to obtain the first skeleton key point data of the user;

A determination module, used for determining second skeleton key point data according to the dialogue recognition result and the first skeleton key point data;

A display module is used to control the display of the virtual object according to the second skeleton key point data.

In some embodiments, the identification module is used to:

Determining an image frame sequence according to the video data;

Performing posture estimation on the image frame sequence to obtain two-dimensional key point data;

The two-dimensional key point data is filtered to obtain the first skeleton key point data.

In some embodiments, the identification module is used to:

Determine N image frames according to the image frame sequence;

Inputting the N image frames into a posture estimation model in sequence to obtain N two-dimensional key point data, wherein the posture estimation model is a convolutional neural network model;

Wherein, N is a positive integer greater than 1.

In some embodiments, the identification module is used to:

Subtract adjacent frames from each other to obtain N-1 key point differential data.

Determine adaptive stable time series differential data according to the N two-dimensional key point data and the N-1 key point differential data;

Inputting the adaptive stable time-series difference data and the N two-dimensional key point data into a filtering model to obtain the first skeleton key point data;

Among them, the network structure of the filtering model includes a fully connected residual layer.

In some embodiments, the identification module is used to:

Determine a key point data change rate between two adjacent frames according to the N two-dimensional key point data and the N-1 key point differential data;

Determine a mean value of the key point data change rate according to the key point data change rate of the two adjacent frames;

The adaptive stable time series differential data is determined according to the key point data change rate of the two adjacent frames, the key point data change rate average and the N-1 key point differential data.

In some embodiments, the determining module is used to:

According to the dialogue recognition result, a plurality of first matching results are determined in a virtual object corpus in a virtual object database; wherein the dialogue recognition result includes an emotion value and a dialogue scene, the virtual object database includes a virtual object corpus and a virtual object action library, the virtual object corpus includes a plurality of pre-stored text contents, the virtual object action library includes a plurality of pre-stored skeleton key point data, and each of the pre-stored skeleton key point data has a corresponding emotion value and a dialogue scene;

Calculating the similarity between each of the first matching results and the conversation recognition result;

In the case where the similarities between all the first matching results and the dialogue recognition results exceed a preset similarity range, determining the first pre-stored skeleton key point data corresponding to the emotion value in the dialogue recognition result in the virtual object action library as the second skeleton key point data;

In the case where there is a first matching result whose similarity is within a preset similarity range among the plurality of first matching results, determining at least one second pre-stored skeleton key point data in the virtual object action library according to the dialogue scene in the dialogue recognition result;

According to the dialogue scene in the dialogue recognition result and the first skeleton key point data, the second skeleton key point data is determined in the at least one second pre-stored skeleton key point data.

In some embodiments, the determining module is used to:

Calculate a first similarity of each of the second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result, the first weight, and each of the second pre-stored skeleton key point data;

Calculate the second similarity of each of the second pre-stored skeleton key point data according to the first skeleton key point data, the second weight and each of the second pre-stored skeleton key point data;

Determine a third similarity of each of the second pre-stored skeleton key point data according to the first similarity and the second similarity;

The second pre-stored skeleton key point data satisfying a preset condition according to the third similarity is determined as the second skeleton key point data.

In a third aspect, a device is provided, which terminal includes a processor and a memory, wherein the memory stores a program or instruction that can be executed on the processor, and when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

In a fourth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

In a fifth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

In a sixth aspect, a computer program/program product is provided. The computer program/program product is stored in a storage medium, and the program/program product is executed by at least one processor to implement the method as described in the first aspect.

In an embodiment of the present application, a dialogue recognition result is obtained based on the recognition of the dialogue content between the user and the virtual object, and the first skeleton key point data of the user is obtained based on the recognition of the video data during the dialogue between the user and the virtual object. The second skeleton key point data is determined based on the dialogue recognition result and the first skeleton key point data, and the display of the virtual object is determined based on the second skeleton key point data. In this way, in the process of interaction between the user and the virtual object, the skeleton key point data of the virtual object can be matched in real time based on the user's behavior, and the virtual object can be driven to perform corresponding actions based on the matched skeleton key point data, so that the virtual object can perform corresponding feedback behaviors based on the user's behavior, enhance the immersion of the user's interaction with the virtual object, and improve the user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of a display method provided in an embodiment of the present application;

FIG. 2a is one of the application schematic diagrams of the display method provided in the embodiment of the present application;

FIG2b is a second application schematic diagram of the display method provided in an embodiment of the present application;

FIG2c is a third application schematic diagram of the display method provided in an embodiment of the present application;

FIG2d is a fourth application schematic diagram of the display method provided in an embodiment of the present application;

FIG2e is a fifth application schematic diagram of the display method provided in an embodiment of the present application;

FIG2f is a sixth application schematic diagram of the display method provided in an embodiment of the present application;

FIG2g is a seventh application schematic diagram of the display method provided in an embodiment of the present application;

FIG2h is an eighth application schematic diagram of the display method provided in an embodiment of the present application;

FIG2i is a ninth application schematic diagram of the display method provided in an embodiment of the present application;

FIG2j is a tenth application schematic diagram of the display method provided in an embodiment of the present application;

FIG2k is an eleventh application schematic diagram of the display method provided in an embodiment of the present application;

FIG21 is a twelfth schematic diagram of an application of a display method provided in an embodiment of the present application;

FIG2m is a thirteenth application schematic diagram of the display method provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of a display device provided in an embodiment of the present application;

FIG4 is one of the structural schematic diagrams of the device provided in the embodiment of the present application;

FIG. 5 is a second schematic diagram of the structure of the device provided in an embodiment of the present application.

Detailed ways

The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "and/or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A and excluding B; Scheme 2: including B and excluding A; Scheme 3: including both A and B. The character "/" generally indicates that the objects associated with each other are in an "or" relationship.

The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

The display method provided in the embodiment of the present application is described in detail below through some embodiments and their application scenarios in combination with the accompanying drawings.

Referring to FIG. 1 , an embodiment of the present application provides a display method, the method comprising:

Step 101: Acquire the conversation content between the user and the virtual object and the video data of the conversation between the user and the virtual object;

Step 102: Recognize the conversation content to obtain a conversation recognition result, and recognize the video data to obtain the first skeleton key point data of the user;

Step 103: Determine the second skeleton key point data according to the dialogue recognition result and the first skeleton key point data;

Step 104: Control the display of the virtual object according to the second skeleton key point data.

The application scenarios targeted by the embodiments of the present application may be scenarios in which users interact with digital humans in the metaverse, or scenarios in which users interact with virtual objects of other categories, such as scenarios in which users interact with non-player characters (NPCs) in a virtual game scene. The embodiments of the present application do not limit the application scenarios of the solutions and the specific categories of virtual objects, as long as the scenarios in which users interact with virtual objects and can obtain the relevant data involved in the solutions of the present application are met.

The conversation content between the user and the virtual object can be determined based on the conversation content input by the user. For example, it can be text information input by the user, and the conversation content with the virtual object is extracted based on the text information, or it can be voice information input by the user, and the conversation content with the virtual object is extracted based on the voice recognition function. The specific implementation can be achieved based on the existing conversation content extraction method.

The video data of the above-mentioned user's conversation with the virtual object refers to the video data captured by a video shooting device during the user's conversation with the virtual object. The video shooting device can be an external camera, or a smart device worn by the user, such as a VR device. The embodiment of the present application does not limit the specific implementation method for obtaining the video data of the user's conversation with the virtual object, and it can be achieved by using the known existing method for obtaining the video data of the user's conversation with the virtual object.

In an embodiment of the present application, a dialogue recognition result is obtained based on the recognition of the dialogue content between the user and the virtual object, and the first skeleton key point data of the user is obtained based on the recognition of the video data during the dialogue between the user and the virtual object. The second skeleton key point data is determined based on the dialogue recognition result and the first skeleton key point data, and the display of the virtual object is determined based on the second skeleton key point data. In this way, in the process of interaction between the user and the virtual object, the skeleton key point data of the virtual object can be matched in real time based on the user's behavior, and the virtual object can be driven to perform corresponding actions based on the matched skeleton key point data, so that the virtual object can perform corresponding feedback behaviors based on the user's behavior, enhance the immersion of the user's interaction with the virtual object, and improve the user experience.

It should be noted that the above-mentioned recognition of the conversation content to obtain the conversation recognition result can be specifically implemented based on the existing semantic recognition technology.

In some implementations, identifying the video data to obtain the first skeleton key point data includes:

(1) Determine an image frame sequence based on video data;

(2) Perform pose estimation on the image frame sequence to obtain two-dimensional key point data;

(3) Filter the two-dimensional key point data to obtain the first skeleton key point data.

In an embodiment of the present application, the recognition processing of video data is specifically: dividing the video data into a sequence of multiple images, and then performing posture estimation processing on the image sequence. Specifically, the 2D key points of the human body can be predicted through a human posture estimation network. Considering the 2D key points of the human body predicted by the human posture estimation network, directly applying the deep learning model to output the key points will cause jitter, and then the model output results need to be filtered and preprocessed to improve the accuracy of data processing.

In some implementations, performing posture estimation on an image frame sequence to obtain two-dimensional key point data includes:

(1) Determine N image frames according to the image frame sequence;

(2) Inputting N image frames into the posture estimation model in sequence to obtain N two-dimensional key point data, the posture estimation model is a convolutional neural network model;

Wherein, N is a positive integer greater than 1.

In an embodiment of the present application, a number N of image frames that need to be filtered is set, and the N image frames are sequentially input into a posture estimation model to obtain N two-dimensional key point data.

It can be understood that the above N can be equal to the total number of image frames in the image frame sequence, that is, posture estimation is performed on all image frames, or the above N can be less than the total number of image frames in the image frame sequence, that is, posture estimation is performed on some image frames in all image frames, and the specific setting can be flexibly based on actual technical requirements.

In some implementations, filtering the two-dimensional key point data to obtain first skeleton key point data includes:

(1) Subtract adjacent frames of N two-dimensional key point data to obtain N-1 key point differential data;

(2) Determine adaptive stable time series difference data based on N two-dimensional key point data and N-1 key point difference data;

(3) inputting the adaptive stable time difference data and N two-dimensional key point data into the filtering model to obtain the first skeleton key point data;

Among them, the network structure of the filtering model includes a fully connected residual layer.

In an embodiment of the present application, adaptive stable temporal difference data is determined based on N two-dimensional key point data and N-1 key point differential data between adjacent frames, and then the first skeleton key point data is predicted by a filtering model with a fully connected residual layer.

The above-mentioned adaptive stable time series difference data can effectively stabilize N two-dimensional key point data. When a key point data jumps greatly, the residual value is adaptively reduced to reduce the jump range of the prediction result.

In some implementations, determining adaptive stable temporal difference data based on N two-dimensional key point data and N-1 key point difference data includes:

(1) Determine the key point data change rate between two adjacent frames based on N two-dimensional key point data and N-1 key point differential data;

(2) Determine the average value of the key point data change rate based on the key point data change rate of two adjacent frames;

(3) Determine the adaptive stable time series differential data based on the key point data change rate of two adjacent frames, the average value of the key point data change rate and N-1 key point differential data.

In the embodiment of the present application, the average value of the key point data change rate of N two-dimensional key point data is calculated and the key point data change rate of two adjacent frames is calculated, and then the adaptive stable time series difference data is calculated by conversion, and finally the calculated adaptive stable time series difference data is used as the branch input of the filtering network.

In some implementations, determining the second skeleton key point data according to the dialogue recognition result and the first skeleton key point data includes:

(1) determining a plurality of first matching results in a virtual object corpus in a virtual object database according to the dialogue recognition result;

The dialogue recognition result includes an emotion value (for example, indicating that the user's dialogue emotion is positive, negative or neutral) and a dialogue scene, the virtual object database includes a virtual object corpus and a virtual object action library, the virtual object corpus includes a plurality of pre-stored text contents, the virtual object action library includes a plurality of pre-stored skeleton key point data, and each pre-stored skeleton key point data has a corresponding emotion value and a dialogue scene;

For example, the NPC database includes the NPC human action library and the NPC corpus. The creation of the database is mainly based on a large amount of movie dialogue scene data and short video dialogue scene data. The collected data is batch converted into dialogue text content feature vector conversion and human action video into human skeleton drive data. The basic action library in the NPC human action library mainly includes common actions such as waving, bowing, blessing, etc.

It should be noted that the establishment of the above-mentioned NPC human motion library and NPC corpus can be implemented based on existing technologies, or the above-mentioned user dialogue recognition results and the recognition and processing method of the first skeletal key point data can be used, that is, the dialogue content materials of the NPC are collected from a large amount of movie dialogue scene data and short video dialogue scene data in the same way as the user dialogue recognition, and the NPC's skeletal key point data are collected from a large amount of movie dialogue scene data and short video dialogue scene data in the same way as the user skeleton key point data recognition.

It is understandable that when establishing a virtual object database, each pre-stored skeletal key point data in the virtual object action library has a corresponding emotion value and dialogue scene, so that the corresponding skeletal key point data can be matched in the virtual object action library based on the user's dialogue recognition results.

(2) Calculate the similarity between each first matching result and the dialog recognition result; execute (3) or execute (4) or (5) according to the result;

First, the conversation recognition result obtained by identifying the user conversation content is matched with the virtual object corpus to obtain multiple groups of similar text matching results, and then the similarity between each first matching result and the conversation recognition result is calculated respectively, wherein the specific similarity calculation needs to be comprehensively calculated in combination with the sentiment value and the conversation scene. The specific calculation method may adopt a technical similarity calculation method, such as a Euclidean distance similarity calculation method. The embodiment of the present application does not specifically limit the similarity calculation method. For the sake of clarity, the following description of the scheme is uniformly described using the Euclidean distance similarity calculation method as an example, wherein the smaller the similarity calculated based on the Euclidean distance similarity calculation method, the higher the degree of similarity between the two.

(3) when the similarities between all the first matching results and the dialogue recognition results exceed the preset similarity range, the first pre-stored skeleton key point data corresponding to the emotion value in the dialogue recognition result in the virtual object action library is determined as the second skeleton key point data;

The preset similarity range can be a similarity threshold set manually, which is used to filter the first matching result. If the similarity calculation results of all the first matching results are greater than the threshold value by Euclidean distance similarity calculation, it means that the overall similarity of the first matching result is poor. At this time, the first pre-stored skeleton key point data is matched in the virtual object action library by the emotion value as the second skeleton key point data.

(4) when there is a first matching result whose similarity is within a preset similarity range among the multiple first matching results, determining at least one second pre-stored skeleton key point data in the virtual object action library according to the dialogue scene in the dialogue recognition result;

(5) Determine second skeleton key point data from at least one second pre-stored skeleton key point data based on the dialogue scene in the dialogue recognition result and the first skeleton key point data.

Using the Euclidean distance similarity calculation, if the similarity calculation results of the first matching results are all less than the threshold value, it means that at least some of the first matching results have a relatively high similarity. At this time, the second skeleton key point data is further matched and selected in combination with the dialogue scene and the first skeleton key point data. Specifically, at least one second pre-stored skeleton key point data is first determined in the virtual object action library with reference to the dialogue scene, and then the second skeleton key point data is filtered out in combination with the dialogue scene and the first skeleton key point data.

In some implementations, determining second skeleton key point data from at least one second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result and the first skeleton key point data includes:

(1) calculating a first similarity of each second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result, the first weight, and each second pre-stored skeleton key point data;

The similarity between the dialogue scene and the second pre-stored skeleton key point data is calculated, and a corresponding weight is added; it is understandable that the specific weight can be set based on an empirical value.

(2) calculating the second similarity of each second pre-stored skeleton key point data according to the first skeleton key point data, the second weight and each second pre-stored skeleton key point data;

The similarity between the first skeleton key point data and the second pre-stored skeleton key point data is calculated, and corresponding weights are added; it is understandable that the specific weights can be set based on empirical values.

(3) determining a third similarity of each second pre-stored skeleton key point data according to the first similarity and the second similarity;

After calculating the similarity between the dialogue scene and the second pre-stored skeleton key point data, and the similarity between the first skeleton key point data and the second pre-stored skeleton key point data, further weighted calculation is performed based on the weights of the two to obtain the final similarity result.

(4) The second pre-stored skeleton key point data that meets the preset conditions according to the third similarity is determined as the second skeleton key point data.

The second skeleton key point data is determined using the final similarity result. The above preset condition may be a similarity threshold, or may be directly selecting the one with the best similarity.

The following is a further description of the technical solution of this application:

The overall flow chart of this scheme is shown in Figure 2a:

The main technical steps are:

user terminal:

Step 1: Obtain the conversation content between the user and the NPC and identify the conversation scenario based on semantic analysis;

Step 2: Obtain the user's video stream data and identify the key points of human body motion skeletons through the skeleton drive of this solution;

Step 3: Quantify and weight the results of step 1 and step 2, and use the results for NPC action matching;

NPC side:

Step 1: Build an NPC database (including NPC human motion library and NPC corpus);

Step 2: Match the user-side recognition result with the NPC action library;

Step 3: Obtain the matching result and drive the NPC to take action;

The detailed steps are as follows:

user terminal

Step 1: Semantic analysis of user conversation content

The semantic analysis module process is mainly shown in Figure 2b:

(1) First, we create a corpus by collecting batches of conversation scene data (text, video);

(2) Perform data preprocessing on the training text data, including word segmentation and stop word removal;

(3) Perform feature engineering on the preprocessed data, such as TF-IDF, Word2Vec, and Bert models;

(4) Training model (sentiment classification model, Word2Vec feature vector model);

(5) Input the user conversation text and process it in steps (2)-(3), and input the processed results into the trained model;

(6) Output results, which include user sentiment analysis (mainly divided into three categories: positive, negative, and neutral in this solution) and the most similar n sets of data (text, video) in the matching dialogue scene library;

This module can use existing technology. This step is to encode the text corpus into feature vectors and then match the user's conversation content with the text corpus. Some of the technologies are described as follows (taking Word2Vec as an example):

The Word2Vec model includes two methods for training word vectors: continuous bag of words (CBOW) and skip-gram. The basic idea of CBOW is to predict the keyword through the words in the context of the keyword. The basic idea of skip-gram is the opposite of CBOW, which is to predict the words in the context through the keyword. As shown in Figure 2c:

The Word2Vec training process is as follows:

(1) CBOW:

Step 1: First, the context words of the keyword are one-hot encoded. The length of the obtained vector is the length of the vocabulary (the total number of different words in all training corpora), and the shape is a 1×V vector. The position of the word is 1, and the other positions are 0.

Step 2: Multiply the unique hot encoding of each context word by a V×N weight matrix W, and each word is multiplied by the same weight matrix W. Multiple 1×N vectors are obtained.

Step 3: For so many 1×N vectors, use the method of adding and averaging to integrate them into a 1×N vector.

Step 4: Then add this 1×N vector to the N×V matrix W′ corresponding to the keyword to get a 1×V vector.

Step 5: This 1×V vector needs to be normalized through the softmax layer to obtain the predicted vector of the keyword. This vector is not one-hot encoded, but composed of many floating-point value probabilities. The position with the highest probability is the position where the keyword is one-hot encoded as 1.

Step 6: Perform an error calculation between the predicted keyword vector and the label vector (one-hot encoded vector), usually using cross entropy.

Step 7: The error is propagated back to the neuron. After each forward calculation, the error is propagated back to achieve the purpose of adjusting the weight matrices W and W′, which is the same as the BP neural network.

When the loss reaches the optimal value, the training ends and we get the weight matrix we need. Through this weight matrix, we can form a word vector based on the input one-hot vector.

(2) Skip-gram

Step 1: First, perform one-hot encoding on the keywords.

Step 2: Multiply the unique hot encoding of each keyword by a V×N weight matrix W.

Step 3: Then multiply the 1×N vector of this keyword with the N×V matrix W′ of the context word vector of the keyword (share one) to obtain multiple 1×V vectors.

Step 4: These 1×V vectors need to be normalized through the softmax layer to obtain the predicted vectors of the keyword context words.

Step 5: Perform an error calculation between the predicted vector of the keyword context words and the label vector (one-hot encoded vector), usually using cross entropy, and sum up the multiple cross entropy losses obtained.

Step 6: The error is propagated back to the neuron. After each forward calculation, the error is propagated back to achieve the purpose of adjusting the weight matrices W and W′.

Step 7: Finally, the weight matrix that forms the word vector is W.

Step 2: Obtain video stream data and output human skeleton key points according to the skeleton drive module

This step mainly describes the skeleton drive module. This module proposes an innovative key point filtering method and adds an adaptive residual mechanism to effectively make the human motion drive more stable. The technical implementation process is shown in Figure 2d, and the corresponding human 2D key points are shown in Figure 2e. The main implementation steps are as follows:

Step 1: Read the video stream and divide it into image frame sequences

Step 2: Image preprocessing

The human body area is located, the center point of the human body area is expanded outward by 1.2 times and the image is cut out according to the aspect ratio of 16:9. The cutout image is reset to the size specified by the model input for training and prediction.

The image preprocessing example is shown in the figure below in Figure 2f. The detection frame of the human body area is expanded outward by 1.2 times based on the center point (as shown in the large dotted box in the figure), and then cut out according to the aspect ratio of 16:9 and input into the model for training. If the human body detection frame is detected as shown in the upper right corner of Figure 2f, it is expanded outward according to 16:9, and the excess area is filled with 0. The 16:9 cutout processing of the human body detection frame is mainly due to the size of the skeleton key point model input setting of 16:9. In order to ensure that the image after cutout is resized to the specified size of the model without deformation, the shape ratio of the human body is guaranteed to be consistent with the original video.

Step 3: The human body posture estimation network predicts the 2D key points of the human body

The human posture estimation network model in this scheme mainly adopts the MobilenetV2 network structure (a lightweight convolutional neural network that uses deep separable convolution). The flow chart of the human 2D key point model is shown in Figure 2g below, and the MobilenetV2 skeleton network structure is shown in Figure 2h and Figure 2i:

Step 4: Key point filtering preprocessing

The 2D key points of the human body predicted by the human posture estimation network are obtained from Step 3. Directly applying the deep learning model to output the key points will cause jitter, and then the model output results need to be filtered and preprocessed. The main steps of filtering and preprocessing of the 2D key points of the human body in this scheme are:

(1) Setting the number of image frames N that need to be filtered, and sequentially inputting them into the human posture estimation network model to obtain the predicted human 2D key point data;

(2) Subtracting adjacent frames of the 2D key point data of the human body corresponding to the N groups of image frames to obtain N-1 key point differential data; this step is shown in Figures 2j and 2k.

(3) The processed data in (2) is input into the human key point filtering model proposed in this scheme for training and prediction. The model structure is shown in Figure 21. The input of the human key point filtering model is N frames of 2D key point data and N-1 frames of adaptively stable time series difference data of key points; the output of the model is the filtered N frames of 2D key point data; the function of the human key point filtering model is to add N frames of 2D key points of human key points and N-1 frames of adaptively stable time series difference data of key points as input for feature combination and then predict the filtered N frames of 2D key point data. The filtered N frames of 2D key point data effectively prevent the key points from jumping and jittering. N frames of 2D key point data: For each frame of data, it contains 14 skeletal key points, and the coordinates (x, y) of each key point represent the position of the key point in this frame of image. The right branch of the network receives N consecutive frames of data at the same time, with a dimension of 14*2*N. N-1 frames of key point adaptively preprocessed time series difference data: In order to make the prediction of each frame of key points more stable, this scheme calculates the mean rate of change of the key point position in the N frames of image. And calculate the change rate of key points of two adjacent frames of images α _i,n , and use the conversion formula resd _i,n of the key point change rate proposed in this scheme to realize the calculation of adaptive stable time series difference data. Finally, the calculated adaptive stable time series difference data is used as the left branch input of the network, and the data dimension is 14*2*(N-1).

The adaptive and stable time series difference data calculation formula is as follows:

Difference of key points between two adjacent frames: d _i,n = pi _,n - _pi,n-1

The change rate of key points in two adjacent frames

The average change rate of key point positions in N frames:

Adaptive stable time series difference data:

The network loss function optimizes the error at the same time during training as follows:

The error is defined as

The total loss function is: Loss = L _P

Where N is the number of sequence frames, which is selected as 8 in this paper. i represents the key point index, pi _,n is the coordinate predicted by the human posture network of the ith key point in the nth frame, pi _,n-1 is the coordinate predicted by the human posture network of the ith key point in the nth-1th frame; _ki,n is the coordinate predicted by the filter network structure of the ith key point in the nth frame; _gi,n is the real coordinate of the ith key point in the nth frame. The adaptive stable time series difference data designed in this scheme effectively stabilizes the coordinate data of 14 human skeleton key points. When a key point jumps greatly, the residual value is adaptively reduced to reduce the jump range of the prediction result.

The training data is generated in two parts. One part is obtained by 2D human posture network, such as openpose. The other part is generated by adding random noise disturbance to the real value of key points. The disturbance method is:

sp _i,n =gi _,n *rand(0.8，1.2)

sp _i,n represents the generated noisy input, and rand(0.8, 1.2) represents generating a random number between 0.8 and 1.2. Mixing the two types of data makes the training data more diverse and improves the generalization ability of the model.

When the video stream frame number F <= N, it is not processed. When F >= N, the N frames of data [F-N, F] are passed into the network to obtain the filtered result.

Step 3: NPC action matching

Through steps one and two, we obtain data such as sentiment analysis after the user semantic analysis module, dialogue scene matching results, and the user's human body movement skeletal key points; in order to effectively match the corresponding body movements of the NPC, step three is mainly used to match the optimal NPC body movements.

The NPC human motion matching flow chart is shown in Figure 2m:

Perform semantic analysis in step 1 according to the user conversation content, match it with the NPC corpus and output n groups of similar text matching results;

Set the similarity threshold to filter the matching results of (1);

If the filtered results are all outside the threshold range, the user's emotion value (positive, negative or neutral) is output based on the emotion analysis result. The basic action library in the NPC action library pre-creates the corresponding emotion label and then matches the user's emotion so that the NPC generates the corresponding action drive.

Determine whether the matching result after filtering is greater than 1 group. If greater than 1, calculate the similarity between the user's human skeleton key points and the skeleton key points vector of the dialogue scene to obtain S1, record the text similarity as S2, and calculate the weighted value S0 = (1-p)*S1+p*S2, where p is an empirical value, which is set to 0.6 in this solution. Sort S0 in reverse order and output the best matching result. If there is only one matching result, output the matching result as the best match.

NPC side

Step 1: Build an NPC database

NPC includes NPC human action library and NPC corpus. The creation of the database is mainly based on a large amount of movie dialogue scene data and short video dialogue scene data. The collected data is batch converted into dialogue text content feature vector conversion (the implementation steps are the same as the user-side step 1) and human action video is converted into human skeleton drive data (the implementation steps are the same as the user-side step 2). The basic action library in the NPC human action library mainly includes common actions such as waving, bowing, blessing, etc.

Step 2: Match the user-side recognition results with the NPC action library

The matching mainly uses the Euclidean distance similarity calculation method, and the detailed steps have been described in step three on the user side.

Step 3: Get the matching result and drive the NPC to take action

The NPC human motion drive mainly uses the human skeleton drive module in the second step of the user side, outputs the corresponding human skeleton motion data and enables the NPC to perform limb drive.

Referring to FIG. 3 , an embodiment of the present application provides a display device, the device comprising:

An acquisition module 301 is used to acquire the conversation content between the user and the virtual object and the video data of the conversation between the user and the virtual object;

The recognition module 302 is used to recognize the conversation content to obtain a conversation recognition result, and to recognize the video data to obtain the first skeleton key point data of the user;

A determination module 303 is used to determine second skeleton key point data according to the dialogue recognition result and the first skeleton key point data;

The display module 304 is used to control the display of the virtual object according to the second skeleton key point data.

In some embodiments, the identification module is used to:

Determining an image frame sequence according to the video data;

Performing posture estimation on the image frame sequence to obtain two-dimensional key point data;

The two-dimensional key point data is filtered to obtain the first skeleton key point data.

In some embodiments, the identification module is used to:

Determine N image frames according to the image frame sequence;

Inputting the N image frames into a posture estimation model in sequence to obtain N two-dimensional key point data, wherein the posture estimation model is a convolutional neural network model;

Wherein, N is a positive integer greater than 1.

In some embodiments, the identification module is used to:

Subtract adjacent frames from each other to obtain N-1 key point differential data.

Determine adaptive stable time series differential data according to the N two-dimensional key point data and the N-1 key point differential data;

Inputting the adaptive stable time-series difference data and the N two-dimensional key point data into a filtering model to obtain the first skeleton key point data;

Among them, the network structure of the filtering model includes a fully connected residual layer.

In some embodiments, the identification module is used to:

Determine a key point data change rate between two adjacent frames according to the N two-dimensional key point data and the N-1 key point differential data;

Determine a mean value of the key point data change rate according to the key point data change rate of the two adjacent frames;

The adaptive stable time series differential data is determined according to the key point data change rate of the two adjacent frames, the key point data change rate average and the N-1 key point differential data.

In some embodiments, the determining module is used to:

According to the dialogue recognition result, a plurality of first matching results are determined in a virtual object corpus in a virtual object database; wherein the dialogue recognition result includes an emotion value and a dialogue scene, the virtual object database includes a virtual object corpus and a virtual object action library, the virtual object corpus includes a plurality of pre-stored text contents, the virtual object action library includes a plurality of pre-stored skeleton key point data, and each of the pre-stored skeleton key point data has a corresponding emotion value and a dialogue scene;

Calculating the similarity between each of the first matching results and the conversation recognition result;

In the case where the similarities between all the first matching results and the dialogue recognition results exceed a preset similarity range, determining the first pre-stored skeleton key point data corresponding to the emotion value in the dialogue recognition result in the virtual object action library as the second skeleton key point data;

In the case where there is a first matching result whose similarity is within a preset similarity range among the plurality of first matching results, determining at least one second pre-stored skeleton key point data in the virtual object action library according to the dialogue scene in the dialogue recognition result;

According to the dialogue scene in the dialogue recognition result and the first skeleton key point data, the second skeleton key point data is determined in the at least one second pre-stored skeleton key point data.

In some embodiments, the determining module is used to:

Calculate a first similarity of each of the second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result, the first weight, and each of the second pre-stored skeleton key point data;

Calculate the second similarity of each of the second pre-stored skeleton key point data according to the first skeleton key point data, the second weight and each of the second pre-stored skeleton key point data;

Determine a third similarity of each of the second pre-stored skeleton key point data according to the first similarity and the second similarity;

The second pre-stored skeleton key point data satisfying a preset condition according to the third similarity is determined as the second skeleton key point data.

The display device in the embodiment of the present application may be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices other than a terminal. For example, the terminal may include but is not limited to the types of terminals 11 listed above, and other devices may be servers, network attached storage (NAS), etc., which are not specifically limited in the embodiment of the present application.

The display device provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 1 to 2 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

As shown in Figure 4, an embodiment of the present application also provides a device 400, including a processor 401 and a memory 402, and the memory 402 stores a program or instruction that can be executed on the processor 401. When the program or instruction is executed by the processor 401, the various steps of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The embodiment of the present application also provides a device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method embodiment. The device embodiment corresponds to the above-mentioned device method embodiment, and each implementation process and implementation method of the above-mentioned method embodiment can be applied to the device embodiment and can achieve the same technical effect.

Specifically, an embodiment of the present application also provides a device. As shown in FIG5 , the device 500 includes: an antenna 51, a radio frequency device 52, a baseband device 53, a processor 54, and a memory 55. The antenna 51 is connected to the radio frequency device 52. In the uplink direction, the radio frequency device 52 receives information through the antenna 51 and sends the received information to the baseband device 53 for processing. In the downlink direction, the baseband device 53 processes the information to be sent and sends it to the radio frequency device 52. The radio frequency device 52 processes the received information and sends it out through the antenna 51.

The method executed by the device in the above embodiment may be implemented in a baseband device 53, which includes a baseband processor.

The baseband device 53 may include, for example, at least one baseband board, on which a plurality of chips are arranged, as shown in FIG5 , wherein one of the chips is, for example, a baseband processor, which is connected to the memory 55 through a bus interface to call a program in the memory 55 and execute the network device operations shown in the above method embodiment.

The device may further include a network interface 56, such as a Common Public Radio Interface (CPRI).

Specifically, the device 500 of the embodiment of the present application also includes: instructions or programs stored in the memory 55 and executable on the processor 54. The processor 54 calls the instructions or programs in the memory 55 to execute the methods executed by the modules shown in Figure 2 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

The embodiments of the present application further provide a computer program/program product, which is stored in a storage medium and is executed by at least one processor to implement the various processes of the above-mentioned method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described here.

It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, it can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable the terminal or network side device to execute the method described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims (11)

1. A display method, comprising: Acquire the conversation content between the user and the virtual object and the video data of the conversation between the user and the virtual object; Recognize the conversation content to obtain a conversation recognition result, and recognize the video data to obtain first skeleton key point data of the user; Determine second skeleton key point data according to the dialogue recognition result and the first skeleton key point data; The display of the virtual object is controlled according to the second skeleton key point data. 2. The method according to claim 1, characterized in that the step of identifying the video data to obtain the first skeleton key point data comprises: Determining an image frame sequence according to the video data; Performing posture estimation on the image frame sequence to obtain two-dimensional key point data; The two-dimensional key point data is filtered to obtain the first skeleton key point data. 3. The method according to claim 2, characterized in that the step of performing posture estimation on the image frame sequence to obtain two-dimensional key point data comprises: Determine N image frames according to the image frame sequence; Inputting the N image frames into a posture estimation model in sequence to obtain N two-dimensional key point data, wherein the posture estimation model is a convolutional neural network model; Wherein, N is a positive integer greater than 1. 4. The method according to claim 3, characterized in that the filtering process on the two-dimensional key point data to obtain the first skeleton key point data comprises: Subtract adjacent frames from each other to obtain N-1 key point differential data. Determine adaptive stable time series differential data according to the N two-dimensional key point data and the N-1 key point differential data; Inputting the adaptive stable time-series difference data and the N two-dimensional key point data into a filtering model to obtain the first skeleton key point data; Among them, the network structure of the filtering model includes a fully connected residual layer. 5. The method according to claim 4, characterized in that the step of determining the adaptive stable time series differential data based on the N two-dimensional key point data and the N-1 key point differential data comprises: Determine a key point data change rate between two adjacent frames according to the N two-dimensional key point data and the N-1 key point differential data; Determine a mean value of the key point data change rate according to the key point data change rate of the two adjacent frames; The adaptive stable time series differential data is determined according to the key point data change rate of the two adjacent frames, the key point data change rate average and the N-1 key point differential data. 6. The method according to claim 1, characterized in that Determining second skeleton key point data according to the dialogue recognition result and the first skeleton key point data includes: According to the dialogue recognition result, a plurality of first matching results are determined in a virtual object corpus in a virtual object database; wherein the dialogue recognition result includes an emotion value and a dialogue scene, the virtual object database includes a virtual object corpus and a virtual object action library, the virtual object corpus includes a plurality of pre-stored text contents, the virtual object action library includes a plurality of pre-stored skeleton key point data, and each of the pre-stored skeleton key point data has a corresponding emotion value and a dialogue scene; Calculating the similarity between each of the first matching results and the conversation recognition result; In the case where the similarities between all the first matching results and the dialogue recognition results exceed a preset similarity range, determining the first pre-stored skeleton key point data corresponding to the emotion value in the dialogue recognition result in the virtual object action library as the second skeleton key point data; In the case that there is a first matching result whose similarity is within a preset similarity range among the multiple first matching results, at least one second pre-stored skeleton key point data is determined in the virtual object action library according to the dialogue scene in the dialogue recognition result; and the second skeleton key point data is determined in the at least one second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result and the first skeleton key point data. 7. The method according to claim 6, characterized in that the determining the second skeleton key point data in the at least one second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result and the first skeleton key point data comprises: Calculate a first similarity of each of the second pre-stored skeleton key point data according to the dialogue scene in the dialogue recognition result, the first weight, and each of the second pre-stored skeleton key point data; Calculate the second similarity of each of the second pre-stored skeleton key point data according to the first skeleton key point data, the second weight and each of the second pre-stored skeleton key point data; Determine a third similarity of each of the second pre-stored skeleton key point data according to the first similarity and the second similarity; The second pre-stored skeleton key point data satisfying a preset condition according to the third similarity is determined as the second skeleton key point data. 8. A display device, comprising: An acquisition module, used to acquire the conversation content between the user and the virtual object and the video data of the conversation between the user and the virtual object; A recognition module, used for recognizing the conversation content to obtain a conversation recognition result, and recognizing the video data to obtain the first skeleton key point data of the user; A determination module, used for determining second skeleton key point data according to the dialogue recognition result and the first skeleton key point data; A display module is used to control the display of the virtual object according to the second skeleton key point data. 9. A device, characterized in that it comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the display method according to any one of claims 1 to 7 are implemented. 10. A readable storage medium, characterized in that the readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the display method according to any one of claims 1 to 7 are implemented. 11. A computer program product, characterized in that it comprises computer instructions, and when the computer instructions are executed by a processor, the steps of the display method according to any one of claims 1 to 7 are implemented.