CN118377384B - Virtual scene interaction method and system - Google Patents

Abstract

The present invention belongs to the field of virtual interaction technology, and discloses a virtual scene interaction method and system; the method comprises: obtaining current user interaction instruction data and relevant environment data, preprocessing the current user interaction instruction data, and obtaining a fused interaction feature sequence; inputting the fused interaction feature sequence into a pre-built time prediction model based on deep reinforcement learning, and obtaining a user interaction sequence in a future time period; obtaining n future virtual scene states according to relevant environment data and the user interaction sequence, using a pre-built probability graph model; rendering corresponding virtual scene pictures based on the n future virtual scene states, and transmitting the virtual scene pictures to a user-end device, thereby greatly improving the user's interaction experience.

Description

A virtual scene interaction method and system

Technical Field

The present invention relates to the field of virtual interaction technology, and more specifically, to a virtual scene interaction method and system.

Background Art

The patent application with the publication number CN115525149A discloses a method, device and system for data interaction between virtual scenes and real scenes, including a real scene data acquisition module, a virtual scene construction module and a voice fusion module. The real scene data acquisition module identifies the gesture operation of the real user to obtain gesture interaction data; the virtual scene construction module constructs a virtual building scene based on the gesture interaction data and the control interaction data; the voice fusion module fuses the voice information of the real user into the virtual building scene to obtain the final virtual building scene. By fusing the gesture interaction data and the voice information to construct the virtual building scene, the problem of insufficient data for interaction between the existing data interaction devices of the virtual scene and the real scene is solved.

However, it is still impossible to fully utilize rich multimodal interaction information such as user gestures, which is particularly important for virtual scene interaction, because gestures are an important way for users to express their intentions. Since it is impossible to fully capture the user's interaction intentions, it is impossible to accurately predict the user's behavior, which affects the intelligence of the interaction. Secondly, the existing virtual scene interaction methods usually only render the virtual scene according to the user's current interaction instructions, and cannot predict the user's future interaction behavior sequence in advance; this means that the scene cannot make corresponding changes in time, and the interaction experience will inevitably appear not smooth and natural enough. In addition, the existing methods lack the ability to model the context of the virtual scene, and cannot make full use of important information such as the position of objects in the scene and lighting conditions to predict scene changes. This limits the accuracy of the interaction and cannot truly achieve natural interaction between users and virtual scenes.

In view of this, the present invention proposes a virtual scene interaction method and system to solve the above problems.

Summary of the invention

In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: A virtual scene interaction method, comprising: S1, obtaining user current interaction instruction data and related environment data, preprocessing the user current interaction instruction data, and obtaining a fused interaction feature sequence;

S2. Input the fused interaction feature sequence into the pre-built time prediction model based on deep reinforcement learning to obtain a user interaction sequence in a future time period;

S3. According to the relevant environmental data and the user interaction sequence, the pre-built probabilistic graph model is used to obtain the future n virtual scene states; based on the future n virtual scene states, the corresponding virtual scene images are rendered, and the virtual scene images are transmitted to the user terminal device.

Furthermore, the user's current interaction instruction data includes user input device operation data and biometric data;

User input device operation data includes key data, gesture data, and device displacement data; biometric data includes eye movement data and limb movement data;

The relevant environment data includes the user's virtual position data, viewing direction data and the user's environment state data; the user's environment state data includes scene object position data and lighting data; the key data includes the timestamp, key code and scan code of the key event;

The method for acquiring the gesture data includes:

A depth camera is used to continuously collect point cloud data of the user's hand area at a fixed frame rate to form a point cloud data sequence;

Extracting accurate hand point cloud data from the point cloud data sequence, downsampling the hand area point cloud data to obtain complete hand point cloud data;

The complete hand point cloud data is subjected to feature extraction to obtain the palm normal vector, fingertip position vector, finger base position vector and inter-finger angle vector; the palm normal vector, fingertip position vector, finger base position vector and inter-finger angle vector are concatenated to obtain the gesture feature vector;

The obtained gesture feature vectors are arranged into a gesture feature vector sequence in chronological order, the gesture feature vector sequence is input into a pre-trained gesture recognition model, the probability distribution of the corresponding gesture type is output, and the encoding of the gesture type with the largest probability in the probability distribution is taken as the gesture data.

Furthermore, the downsampling method includes:

Insert the point cloud data sequence into the pre-built octree in an orderly manner, and then traverse the octree at a fixed spatial resolution to obtain the complete hand point cloud data;

The octree is constructed in the following manner:

The three-dimensional space where the point cloud data sequence is located is divided into regular cubic units. According to the coordinates of the points in the point cloud data sequence, each point is inserted into the corresponding cubic unit. For a cubic unit containing m points, it is further recursively divided into 8 sub-units until there is at most one point in each cubic unit, that is, the construction of the octree is preliminarily completed;

The method of traversing the octree includes:

Set a distance threshold d, start from the root node of the octree, traverse each node of the octree in depth-first or breadth-first order, and for each non-empty node, calculate the distance from all points in the node to the center of the node, and select the point closest to the center of the node as the representative point of the node;

Initialize an empty list L, traverse each non-empty node X of the octree, calculate the distance d(c,p) between the center point c of X and the representative point p, if d(c,p) is greater than or equal to d, then add p to the list L, and finally the points in the list L are the complete hand point cloud data;

The calculation formula of the distance d(c,p) is:

Among them, (c _x , _cy ,c _z ) are the three-dimensional coordinates of the center point c, and (p _x , _py ,p _z ) are the three-dimensional coordinates of the representative point p.

Furthermore, the method of performing feature extraction includes:

Perform principal component analysis on the complete hand point cloud data to obtain the main direction vector Palm normal vector for The vector product of the negative z-axis direction of the three-dimensional coordinate system where the hand is located, that is, in, is the unit vector of the z-axis;

The complete hand point cloud data is constructed into a set P, and a complete convex hull CH(P) is constructed based on the set P;

Perform hierarchical clustering on the complete CH(P) to obtain k+1 levels, denoted as L0, L1, ..., Lk-1, Lk; where Lk-1 is the fingertip point set;

For each point vj in Lk-1, traverse all vertices on CH(P), calculate the Euclidean distance from vj to all vertices and take the minimum value dj. After calculating the dj of all vj, take the point corresponding to the minimum value of dj, which is the fingertip position. Vectorize the fingertip position to get the fingertip position vector Where t is the index of the finger;

Calculate the position vector of each fingertip Distance _dt to the palm plane, along the normal to the palm plane Direction from Shift back α to get the corresponding root position vector Among them, α is the empirical coefficient;

For any two adjacent fingertip position vectors and calculate Vector and Vector in, is the finger root position vector corresponding to the t+1 finger;

Based on vector and Calculate the finger angle The inter-finger angle is vectorized to obtain the inter-finger angle vector.

Furthermore, the construction method of the convex shell CH(P) includes:

Initially, the convex hull CH(P) is a tetrahedron formed by any four non-coplanar points. For a point b in the set P, if b is in CH(P), then b is ignored. If b is on CH(P), then b is added to the vertex set of CH(P). If b is outside CH(P), then b is used to traverse and divide CH(P) to generate a new CH(P), and finally a complete CH(P) is obtained.

The method of performing hierarchical clustering includes:

Initialize L0 equal to CH(P). For each level Lr, r belongs to the interval [0, k], and k is a positive integer greater than 1; calculate the Euclidean distance d(vp, vq) of all point pairs (vp, vq) in Lr, extract the point pair (pi, pj) corresponding to the largest Euclidean distance, merge (pi, pj) into a new cluster, remove pi and pj, and form a new point set Lr+1 with the remaining points and the new cluster; repeat until Lr+1 is an empty set or contains only one cluster, and the hierarchical clustering of the complete CH(P) is completed.

Furthermore, the training method of the gesture recognition model includes:

Collect gesture feature vector sequences of M groups of gesture actions and the encoding of their corresponding gesture types to form a training data set; define a temporal convolutional network as the backbone network of the gesture recognition model, the temporal convolutional network consists of h residual temporal convolutional blocks, each residual temporal convolutional block includes a temporal convolutional layer and a dropout layer, and residual connections are added to the temporal convolutional layer and the dropout layer;

The calculation formula of the temporal convolution layer is:

Among them, y _s is the output feature of the temporal convolution layer at time step s, k is the size of the convolution kernel, i is the offset of the convolution kernel in the time dimension, x _si is the input feature of the si-th time step, Θ represents the convolution operation, _wi is the convolution kernel coefficient, B is the bias term, f() is the ReLU function, and f(x _si ) is the result of transforming the input feature x _si through the ReLU function;

The output of the last residual temporal convolution block is a context feature sequence. A classification head is added to the backbone network, and the context feature sequence is input into the classification head. The features of each time step in the context feature sequence are classified using a fully connected layer to obtain the probability distribution of the gesture type.

The classification formula is: P(y _s |c _s ) = Softmax(W _ql ·c _s +B _ql ); where W _ql is the weight of the fully connected layer, B _ql is the bias of the fully connected layer, and cs is the feature of the context feature sequence at time step s;

Define the loss function of the gesture recognition model Among them, S is the total length of the context feature sequence, is the encoding of the actual gesture type at the s-th time step;

The gesture recognition model is trained using the training data set, the loss function is minimized in the process, and the parameters of the gesture recognition model are continuously optimized to obtain a trained gesture recognition model.

Furthermore, the method of preprocessing the user's current interaction instruction data includes:

Normalizing the data of different modes in the user's current interaction instruction data to obtain normalized instruction data;

Align the normalized instruction data in time series to obtain the corresponding multimodal sequence; splice the multimodal sequence in time dimension to form a feature time series data with g feature channels;

The feature time series data is constructed as a three-dimensional tensor, where two dimensions represent time and feature channels, and the third dimension is used to represent the corresponding data.

Furthermore, the method of constructing the time prediction model based on deep reinforcement learning includes:

The time prediction model includes a sequence modeling network and a reinforcement learning network. The gated recurrent unit network is used as the basic framework of the sequence modeling network. The gated recurrent unit network includes a network input layer, a GRU hidden layer, and a state representation output layer.

Collect interaction sequence data of group N1, each interaction sequence data contains time step feature representation and corresponding label;

The interactive sequence data is gradually input into the gated recurrent unit network in batches to obtain the final state vector of the corresponding batch, and the final state vector of the corresponding batch is input into the preset classifier to obtain the distribution of the predicted label probability; the cross entropy loss between the label probability and the label is calculated, and the trainable parameters of the gated recurrent unit network and the classifier are gradient updated using an optimizer, and the cross entropy loss is minimized to obtain a sequence modeling network;

Define the reinforcement learning network as a semi-Markov decision process, and define a tuple (S1, N2, P1, R1) for the semi-Markov decision process;

The semi-Markov decision process is divided into the strategy of the high-level manager and the strategy of the low-level manager. The goal of the semi-Markov decision process is defined as learning an optimal strategy of the high-level manager and Q optimal strategies of the low-level managers. The final state vector of the corresponding batch obtained by the gated recurrent unit network of the interaction sequence data is used as the state of the semi-Markov decision process. The semi-Markov decision process is trained, and the policy gradient algorithm is used to optimize the strategies of the high-level manager and the low-level manager respectively. The expected cumulative reward calculated in the semi-Markov decision process is maximized. The constructed reinforcement learning network is obtained.

Furthermore, the construction method of the probability graph model includes:

Define random variables of the probabilistic graph model, where the random variables are data in the relevant environmental data; construct an undirected graph structure based on the mutual influence relationship between various objects in the virtual scene; the nodes in the undirected graph structure represent random variables, and the edges in the undirected graph structure represent the mutual dependence relationship between random variables; define a node potential function for each node, and define an edge potential function for each edge;

Collect historical data of group M1, which is the time series data of virtual scene status; initialize the parameters of node potential function and edge potential function, use the time series data of virtual scene status to train the probability graph model, and obtain the parameters of node potential function and edge potential function; the training method is the maximum likelihood estimation method; thus, the construction of the probability graph model is completed;

The method of obtaining the future n virtual scene states includes:

Define the transition probability of the probabilistic graphical model, and based on the transition probability of the probabilistic graphical model, infer the next n time steps to obtain the probability distribution of each object in the virtual scene at each time step. During the inference process, input the user interaction sequence into the probabilistic graphical model in sequence according to the time step;

For each time step, the state of the scene objects is sampled or selected according to the probability distribution to construct the virtual scene state at that time step, and then the future n virtual scene states are obtained.

A virtual scene interaction system, used to implement the virtual scene interaction method, comprising: a data acquisition and processing module, used to obtain the user's current interaction instruction data and related environment data, and pre-process the user's current interaction instruction data to obtain a fused interaction feature sequence;

The prediction module is used to input the fused interaction feature sequence into a pre-built time prediction model based on deep reinforcement learning to obtain a user interaction sequence in a future time period;

The virtual environment control module is used to obtain n future virtual scene states based on relevant environmental data and user interaction sequences using a pre-built probabilistic graph model; render corresponding virtual scene images based on the n future virtual scene states, and transmit the virtual scene images to the user-end device. The modules are connected via wired and/or wireless means.

Technical effects and advantages of a virtual scene interaction method and system of the present invention:

The present invention can integrate multiple user interaction data, such as buttons, gestures, eye movements, etc., especially for virtual interaction, the accuracy of gesture data is more important, which can fully capture the user's interaction intention, thereby more accurately predicting the user's behavior and improving the intelligence of the interaction. Secondly, the time prediction model based on deep reinforcement learning can predict the user's future interaction behavior sequence in advance, and drive the probabilistic graph model to generate the state changes of future virtual scenes, which greatly enhances the fluency and naturalness of the interaction. In addition, the probabilistic graph model is used to model the contextual environment of the virtual scene, such as object position, lighting conditions, etc., to better predict the changes in the scene, thereby further improving the accuracy of the interaction. In general, through multimodal fusion, time prediction modeling and probabilistic graph modeling, the user's interaction experience is greatly improved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a virtual scene interaction method of the present invention;

FIG. 2 is a schematic diagram of a virtual scene interaction system of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Example 1

Referring to FIG. 1 , a virtual scene interaction method according to this embodiment includes:

S1. Obtain the user's current interaction instruction data and related environment data, pre-process the user's current interaction instruction data, and obtain a fused interaction feature sequence;

S2. Input the fused interaction feature sequence into the pre-built time prediction model based on deep reinforcement learning to obtain a user interaction sequence in a future time period;

S3. According to the relevant environmental data and the user interaction sequence, the pre-built probabilistic graph model is used to obtain the future n virtual scene states; based on the future n virtual scene states, the corresponding virtual scene images are rendered, and the virtual scene images are transmitted to the user terminal device.

Furthermore, the user's current interaction instruction data includes user input device operation data and biometric data;

User input device operation data includes key data, gesture data, and device displacement data; biometric data includes eye movement data and limb movement data;

The relevant environment data includes the user's virtual position data, viewing direction data and the user's environment state data; the user's environment state data includes scene object position data and lighting data;

The key data includes the timestamp, key code and scan code of the key event. It should be noted that the timestamp is obtained by the operating system kernel or driver of the input device when it receives a hardware interrupt. For example, when the user presses a key, the keyboard hardware triggers an interrupt signal. The operating system kernel or keyboard driver receives the interrupt, and the kernel or driver calls the system time API to obtain the current system time, accurate to the microsecond or nanosecond level. The obtained time value is the timestamp of the key event.

The key code is obtained by the keyboard driver of the input device by querying the key mapping table according to the scan code; the scan code comes from the keyboard hardware circuit of the input device. Specifically, the keyboard hardware circuit is composed of multiple key matrices, and each key corresponds to a row and column intersection in the matrix. When a key is pressed, the corresponding row and column intersection will form a circuit path. The keyboard hardware circuit detects this path and generates a hardware scan code, i.e., the scan code, according to the row and column positions.

The timestamp, key code, and scan code together fully describe a key event. For example, when the "A" key is pressed, the data [12:30:45.782, 65, 30] is recorded.

The methods for obtaining gesture data include:

A depth camera is used to continuously collect point cloud data of the user's hand area at a fixed frame rate (such as 30fps) to form a point cloud data sequence; the point cloud data of the hand area collected by the depth camera does not only contain the hand but also some background areas;

The precise point cloud data of the hand is extracted from the point cloud data sequence, and the point cloud data of the hand area is downsampled to reduce the data volume, improve the subsequent processing efficiency, and obtain complete point cloud data of the hand.

Ways to downsample include:

The point cloud data sequence is inserted into the pre-built octree in an orderly manner, and then the octree is traversed at a fixed spatial resolution to obtain the complete hand point cloud data.

The construction of the octree includes:

The three-dimensional space where the point cloud data sequence is located is divided into regular cubic units. According to the coordinates of the points in the point cloud data sequence, each point is inserted into the corresponding cubic unit. For a cubic unit containing m points, it is further recursively divided into 8 sub-units until there is at most one point in each cubic unit, that is, the construction of the octree is preliminarily completed.

The ways to traverse the octree include:

Set a distance threshold d, which represents the minimum distance between two adjacent sampled points. The larger the value of d, the sparser the sampled points are, and the smaller the value of d, the denser the sampled points are.

Starting from the root node of the octree, traverse each node of the octree in a depth-first or breadth-first manner. For each non-empty node (including point cloud points), calculate the distance from all points in the node to the center of the node, and select the point closest to the center of the node as the representative point of the node.

In an octree, depth-first traversal starts from the root node, selects a child node, and continues to traverse the child nodes of the child node downward until a leaf node or an empty node is encountered, then backtracks to the previous node and continues to traverse other child nodes.

In an octree, breadth-first access starts from the root node, visits the root node first, then visits all first-layer nodes in sequence, then visits all second-layer nodes, and so on.

The node center refers to the geometric center of the cubic space area. When traversing the octree for downsampling, for each non-empty node (including point cloud points), it is necessary to calculate the distance from all point cloud points in the node to the node center, and then select the point cloud point with the nearest distance as the representative point of the node; the purpose of this is to retain the most representative point cloud points for the area during downsampling.

Initialize an empty list L to store sampling points, traverse each non-empty node X of the octree, calculate the distance d(c,p) between the center point c of X and the representative point p, if d(c,p) is greater than or equal to d, then add p to the list L, and finally the points in the list L are the complete hand point cloud data;

Among them, the calculation formula of distance d(c,p) is:

Among them, (c _x , _cy ,c _z ) are the three-dimensional coordinates of the center point c, and (p _x , _py ,p _z ) are the three-dimensional coordinates of the representative point p.

The complete hand point cloud data is subjected to feature extraction to obtain the palm normal vector, fingertip position vector, finger base position vector and inter-finger angle vector; the palm normal vector, fingertip position vector, finger base position vector and inter-finger angle vector are concatenated to obtain the gesture feature vector.

Methods for feature extraction include:

Perform principal component analysis (PCA) on the complete hand point cloud data to obtain the main direction vector Palm normal vector for The vector product of the negative z-axis direction of the three-dimensional coordinate system where the hand is located, that is, in, is the unit vector of the z-axis (0, 0, 1).

The complete hand point cloud data is constructed into a set P, and a complete convex hull CH(P) is constructed based on the set P;

The construction methods of convex hull CH(P) include:

Initially, the convex hull CH(P) is a tetrahedron composed of any four non-coplanar points. For a point b in the set P, if b is in CH(P), then b is ignored; if b is on CH(P), then b is added to the vertex set of CH(P); if b is outside CH(P), then b is used to traverse and divide CH(P) to generate a new CH(P), and finally a complete CH(P) is obtained.

Perform hierarchical clustering on the complete CH(P) to obtain k+1 levels, denoted as L0, L1, ..., Lk-1, Lk; where Lk-1 is the fingertip point set;

For each point vj in Lk-1, traverse all vertices on CH(P), calculate the Euclidean distance from vj to all vertices and take the minimum value dj. After calculating the dj of all vj, take the point corresponding to the minimum value of dj, which is the fingertip position. Vectorize the fingertip position to get the fingertip position vector Where t is the index of the finger.

Ways to perform hierarchical clustering include:

Initialize L0 equal to CH(P). For each level Lr, r belongs to the interval [0, k], and k is a positive integer greater than 1. Calculate the Euclidean distance d(vp, vq) of all point pairs (vp, vq) in Lr, extract the point pair (pi, pj) corresponding to the largest Euclidean distance, merge (pi, pj) into a new cluster, remove pi and pj, and form a new point set Lr+1 with the remaining points and the new cluster. Repeat until Lr+1 is an empty set or contains only one cluster, and the hierarchical clustering of the complete CH(P) is completed.

Calculate the position vector of each fingertip Distance _dt to the palm plane, along the normal to the palm plane Direction from Shift back α to get the corresponding root position vector Among them, α is an empirical coefficient, usually a decimal between 0 and 1, which reflects the proportion of finger length to the entire hand length.

For any two adjacent fingertip position vectors and calculate Vector and Vector in, is the finger root position vector corresponding to the t+1 finger;

Based on vector and Calculate the finger angle The inter-finger angle is vectorized to obtain the inter-finger angle vector.

The obtained gesture feature vectors are arranged into a gesture feature vector sequence in chronological order, and the gesture feature vector sequence is input into a pre-trained gesture recognition model, and the probability distribution of the corresponding gesture type is output. The code of the gesture type with the largest probability in the probability distribution is taken as the gesture data, such as code 6 represents the "waving" gesture.

The training methods of gesture recognition model include:

Collect gesture feature vector sequences of M groups of gesture actions and the encoding of their corresponding gesture types to form a training data set; define a temporal convolutional network as the backbone network of the gesture recognition model. The temporal convolutional network consists of h residual temporal convolutional blocks. Each residual temporal convolutional block includes a temporal convolutional layer and a dropout layer, and residual connections are added to the temporal convolutional layer and the dropout layer. Adding a dropout layer prevents overfitting, and adding a residual connection helps gradient propagation.

The calculation formula of the temporal convolution layer is:

Among them, _ys is the output feature of the temporal convolution layer at time step s, k is the size of the convolution kernel, i is the offset of the convolution kernel in the time dimension, ranging from 0 to k-1, _xsi is the input feature of the si-th time step, Θ represents the convolution operation, _wi is the convolution kernel coefficient, B is the bias term, f() is the ReLU function, and f( _xsi ) is the result of transforming the input feature _xsi through the ReLU function;

The output of the last residual temporal convolution block is a context feature sequence. A classification head is added to the backbone network, and the context feature sequence is input into the classification head. The features of each time step in the context feature sequence are classified using a fully connected layer to obtain the probability distribution of the gesture type.

The classification formula is: P(y _s |c _s ) = Soft max(W _ql ·c _s +B _ql ); where W _ql is the weight of the fully connected layer, B _ql is the bias of the fully connected layer, and c _s is the feature of the context feature sequence at time step s;

Define the loss function of the gesture recognition model Among them, S is the total length of the context feature sequence, that is, the total number of time steps, is the encoding of the actual gesture type at the s-th time step;

The gesture recognition model is trained using the training data set, the loss function is minimized in the process, and the parameters of the gesture recognition model are continuously optimized to obtain a trained gesture recognition model.

The motion state of the device is detected by the built-in acceleration sensor or gyroscope sensor of the device. The motion state includes displacement and rotation. The device displacement data is obtained by fusing the data.

Eye movement data is obtained through the front camera of the device in combination with an eye tracking algorithm. The eye tracking algorithm includes an image processing-based algorithm or a 3D model-based algorithm. The image processing algorithm or the 3D model-based algorithm can detect the user's eye movement trajectory in real time.

Limb motion data is obtained through the device's depth camera or RGB camera combined with a skeleton tracking algorithm; the skeleton tracking algorithm is based on a machine learning model that detects and tracks key points of the human body from images or videos to obtain limb motion trajectory data.

The relevant environmental data is obtained through the three-dimensional model data of the virtual scene. Specifically, when constructing the virtual scene, the characters and objects in the scene are three-dimensionally modeled, and their precise position coordinates, the user's virtual position data and viewing direction data are recorded, and the lighting data is obtained by setting the lighting parameters of the virtual scene.

It should be noted that the above data are all quantified to make them suitable for subsequent data processing.

Furthermore, the method of preprocessing the user's current interaction instruction data includes:

The data of different modes in the user's current interaction command data are normalized to obtain normalized command data so that their numerical ranges are at the same order of magnitude; the normalization methods include Min-Max normalization and Z-Score normalization.

The normalized instruction data is aligned in time series so that different modes have corresponding data at the same time point, and the corresponding multimodal sequence is obtained. Interpolation, upsampling or downsampling is used to achieve alignment.

The multimodal sequences are spliced in the time dimension to form a feature time series data with g feature channels; for example, key data, gesture data, and eye movement data are parallel to form multiple feature channels. This splicing and fusion method retains the original time dynamics of each modality and is easy for subsequent modeling;

The feature time series data is constructed as a three-dimensional tensor, in which two dimensions represent time and feature channels, and the third dimension is used to represent the corresponding data; it is convenient to input into the model that can process multi-channel data.

Effectively fuse the quantified multimodal data to prepare data for subsequent feature extraction and modeling processes.

Furthermore, the construction methods of the time prediction model based on deep reinforcement learning include:

The time prediction model is defined to include a sequence modeling network and a reinforcement learning network. The gated recurrent unit network is used as the basic framework of the sequence modeling network, and the gated recurrent unit network includes a network input layer, a GRU hidden layer, and a state representation output layer.

Collect interaction sequence data of group N1. Each interaction sequence data contains time step feature representation and corresponding label (the user's final behavior).

The interaction sequence data is gradually input into the gated recurrent unit network in batches to obtain the final state vector of the corresponding batch, and the final state vector of the corresponding batch is input into the preset classifier (fully connected layer + softmax function) to obtain the distribution of predicted label probabilities; the cross entropy loss between label probabilities and labels is calculated, and the trainable parameters of the gated recurrent unit network and classifier are gradient updated using an optimizer (such as Adam), and the cross entropy loss is minimized to obtain a trained gated recurrent unit network.

The network input layer is used to receive the fused interaction feature sequence as input. The fused interaction feature sequence is a vector sequence with a length of Y, and each vector in the vector sequence corresponds to a time step;

The GRU hidden layer consists of F GRU units, each of which corresponds to a time step in the processing vector sequence; for the a-th time step, an update gate controller vector is calculated using the input vector of the current time step and the hidden state vector of the previous time step. Each element value of the update gate controller vector is between 0 and 1, indicating the degree to which the corresponding hidden state element should retain information from the previous state.

Similarly, a reset gate controller vector is calculated using the input vector of the current time step and the hidden state vector of the previous time step; each element value of the reset gate controller vector is between 0 and 1, indicating the degree to which the corresponding hidden state element should discard the information from the previous state;

Based on the current input vector, the reset gate controller vector and the hidden state vector of the previous time step, a candidate new hidden state vector is calculated; the candidate new hidden state vector integrates the information from the current input and the previous state.

The final hidden state vector of the current time step is calculated using the update gate controller vector, the hidden state vector of the previous time step, and the candidate new hidden state vector; the final hidden state vector is a fusion of the previous hidden state and the candidate new hidden state, and the update gate controller determines the fusion ratio of the two.

The state representation output layer is recursively calculated for Y time steps to obtain the hidden state vector of the last time step, which is used as the encoding representation of the entire fused interaction feature sequence, that is, the final state vector.

It should be explained that the hidden state vector is a vector saved by the GRU unit at each time step, which is used to encode information from the input sequence of the previous time step. Specifically, at the ath time step, the GRU unit will calculate the update gate controller vector z(a) based on the input vector x(a) of the current time step and the hidden state vector h(a-1) of the previous time step. The value of each element of the updated gate controller vector is between 0 and 1, which is used to control the extent to which the hidden state vector h(a) of this time step should retain information from the previous hidden state h(a-1). Similarly, the reset gate controller vector, candidate new hidden state vector and final state vector are also obtained in this way.

The update gate controller vector z(a) is a function of the previous hidden state h(a-1) and the current input x(a). They are affixed using the learned weight matrix W(z) and bias b(z), and then activated using the sigmoid activation function to obtain a value between 0 and 1.

The value of each element of z(a) is close to 0, which means that the element should retain more information from the previous hidden state h(a-1) in the new hidden state h(a); conversely, the value is close to 1, which means that the element should incorporate more new information from the current input x(a).

Therefore, the hidden state vector h(a-1) not only carries the sequence information of the previous time step, but also controls how the new hidden state h(a) should be updated by updating the gate controller vector z(a), thereby encoding the dynamic information of the entire input sequence.

The reinforcement learning network is constructed in the following ways:

Define the reinforcement learning network as a semi-Markov decision process, and define a tuple (S1, N2, P1, R1) for the semi-Markov decision process; where S1 is the state space of the decision of the low-level manager, N2 is the subtask space that the high-level manager can choose, P1 is the state transition function of the decision of the low-level manager, P(s'|s1, n1, π_n) represents the probability of transferring to state s' after executing the strategy π_n corresponding to subtask n1 in state s1, R1 is the reward function of the decision of the low-level manager, and R(s1, n1, π_n) represents the reward obtained by executing the strategy π_n corresponding to subtask n1 in state s1.

The semi-Markov decision process is divided into the strategy of the high-level manager and the strategy of the low-level manager; the strategy π_m(n1|s1) of the high-level manager represents the probability of selecting subtask n1 in state s1, and the strategy π_n(a1|s1) of the low-level manager represents the probability of selecting action a1 when executing subtask n1 in state s1.

The goal of defining a semi-Markov decision process is to learn an optimal high-level manager strategy and Q optimal low-level manager strategies. The final state vector of the corresponding batch obtained by the gated recurrent unit network through the interaction sequence data is used as the state of the semi-Markov decision process. The semi-Markov decision process is trained, and the policy gradient algorithm is used to optimize the high-level manager strategy and the low-level manager strategy respectively; the expected cumulative reward obtained in the semi-Markov decision process is maximized; and the reinforcement learning network is constructed.

The calculation formula for expected cumulative rewards is:

Where J(π _m ,{π _n }) represents the expected cumulative reward when given the strategy π _m of the high-level manager and the set {π _n } of the strategies of the low-level managers, which is the objective function to be maximized; represents the expected value of the trajectory τ, where the trajectory τ is generated according to the set {π _n } consisting of the strategies of the high-level manager π _m and the strategies of the low-level managers, CD is the number of time steps of the final state vector, cd is the time step index of the final state vector, γ ^cd is the discount factor at time step cd, Indicates the strategy of the lower-level manager corresponding to the execution of subtask n _cd in state s _cd The immediate reward obtained, s _cd is the state at time step cd, n _cd represents the subtask executed at time step cd, which is selected by the strategy of the high-level manager, Indicates the strategy of the lower-level manager used when executing subtask n _cd .

It should be noted that in the semi-Markov decision process, there are two levels: high-level managers (high-level strategies) and low-level managers (low-level strategies). The high-level manager decides which subtask to execute, while the low-level manager decides how to execute the subtask (which specific actions to perform). The generation of trajectories includes:

Initialize the initial state s0, which is represented by the final state vector obtained by the gated recurrent unit network; select a subtask nt according to the current state and the strategy of the high-level manager. The specific method is to sample a subtask from the probability distribution of the strategy; according to the current state and the selected subtask nt, use the corresponding low-level manager's strategy to execute the subtask, that is, continuously sample the action sequence according to the probability distribution of the low-level manager's strategy until the subtask is completed or the maximum step limit is reached; during the execution of the subtask, record the state sequence experienced, the action sequence executed, and the immediate reward obtained; after the subtask is executed, transfer from the current state to the next state according to the state transfer function; repeat until the termination condition (such as the maximum trajectory length) is reached; through the above process, a complete trajectory τ is generated, which includes the state sequence, subtask sequence, action sequence and reward sequence.

The user interaction process is decomposed into multiple subtasks. The high-level manager selects which subtask to execute, while the low-level manager executes the specific action sequence of the corresponding subtask.

The hierarchical approach can better capture the hierarchical structure in the interaction process and improve sample efficiency. The high-level manager focuses on the higher-level subtask selection, while the low-level manager focuses on the specific action execution, and the two collaborate with each other.

Specifically, the input of the time prediction model is a fused interaction feature sequence. The current state representation is first obtained through the sequence modeling network, and then the predicted action for the next time step (i.e., the predicted user interaction) is output based on the reinforcement learning network. Executing this action will produce a new state representation, and so on, until the interaction sequence for the entire future time period is predicted.

During the training process, the temporal prediction model continuously adjusts its parameters to improve the accuracy of the prediction by maximizing the fit with the real interaction sequence.

Furthermore, the construction methods of the probabilistic graphical model include:

Define the random variables of the probabilistic graph model. The random variables are the data in the relevant environmental data, such as object position, lighting conditions, user position and viewing angle. Each random variable takes a finite number of discrete states or continuous states. Construct an undirected graph structure based on the mutual influence relationship between objects in the virtual scene. The nodes in the undirected graph structure represent random variables, and the edges in the undirected graph structure represent the mutual dependence relationship between random variables. Usually, there are edge connections between neighboring nodes, and there may be no direct connection between distant nodes.

A node potential function is defined for each node to describe the probability distribution of the node taking different states. The node potential function is a parameterized function, such as a Gaussian function or a logistic function. An edge potential function is defined for each edge to describe the interaction between adjacent nodes. The edge potential function depends on the state of the adjacent nodes and is fitted by the compatibility or incompatibility between the encoding nodes.

Collect historical data of the M1 group, including time series data of virtual scene status and time series data of user interaction behavior.

Initialize the parameters of the node potential function and the edge potential function, use the time series data of the virtual scene state to train the probabilistic graph model, and obtain the parameters of the node potential function and the edge potential function; the training method is the maximum likelihood estimation method; that is, the construction of the probabilistic graph model is completed.

The methods for obtaining the future n virtual scene states include:

The user interaction sequence is used as the control input of the probabilistic graphical model to drive the change of the scene, and the transition probability of the probabilistic graphical model is defined. Based on the transition probability of the probabilistic graphical model, the future n time steps are inferred to obtain the probability distribution of each object in the virtual scene at each time step. During the inference process, the user interaction sequence is input in sequence according to the time step to simulate the user's interactive behavior in the virtual scene.

For each time step, the state of the scene objects is sampled or selected according to the probability distribution to construct the virtual scene state at that time step, and then the future n virtual scene states are obtained; the scene state includes information on object position, lighting conditions, user position, and viewing angle, etc.

It should be noted that the transition probability describes the probability distribution of the scene transferring to the next state given the current state and user interaction behavior. Specifically, the current time step is e, and the user's interaction behavior is represented as u(e). u(e) is used as the input of the probabilistic graph model, which will affect the transition probability of each node. At time step e, the joint probability distribution of the probabilistic graph model is expressed as the product of the node potential function and the edge potential function. The edge potential function is related to the user interaction u(e). Next, the transition probability from time step e to time step e+1 is defined, which is equal to the product of the conditional probability of each node transferring to the next state based on the state of its neighbor nodes and the user interaction u(t). The transition probability of a single node is obtained through learning, which includes the impact of user interaction on the node's transition.

After defining the transition probability of the probabilistic graphical model, in the reasoning process, the transition probability model is used, combined with the user's interaction sequence as input, to perform rolling reasoning for the next n time steps. At each time step, the probability distribution of the state of each object in the virtual scene is obtained, and then the specific state of the object is selected from the probability distribution through sampling and other methods, so that the complete virtual scene state at that time step can be constructed.

The rendering methods of virtual scene images include:

Parse the relevant environmental data of each scene at each moment from the future n virtual scene states, load the required three-dimensional model resources based on the parsed relevant environmental data, and use the loaded three-dimensional model resources to build a three-dimensional virtual scene; adjust the specific rendering engine, codec and transmission protocol, and at the same time, consider factors such as real-time, low latency, and bandwidth utilization to ensure the smooth presentation of the virtual scene, and efficiently and realistically render it into the corresponding virtual scene picture, providing users with a better interactive experience.

This embodiment can integrate multiple user interaction data, such as buttons, gestures, eye movements, etc., especially for virtual interaction, the accuracy of gesture data is more important, which can fully capture the user's interaction intention, thereby more accurately predicting the user's behavior and improving the intelligence of the interaction. Secondly, the time prediction model based on deep reinforcement learning can predict the user's future interaction behavior sequence in advance, and drive the probabilistic graph model to generate the state changes of the future virtual scene, which greatly enhances the fluency and naturalness of the interaction. In addition, the probabilistic graph model is also used to model the context of the virtual scene, such as object position, lighting conditions, etc., to better predict the changes in the scene, thereby further improving the accuracy of the interaction. In general, through multimodal fusion, time prediction modeling and probabilistic graph modeling, the user's interaction experience is greatly improved.

Example 2

Please refer to FIG. 2 . For the parts not described in detail in this embodiment, please refer to the description of Embodiment 1. A virtual scene interaction system is provided, including: a data acquisition and processing module, which is used to obtain the user's current interaction instruction data and related environment data, and pre-process the user's current interaction instruction data to obtain a fused interaction feature sequence;

The prediction module is used to input the fused interaction feature sequence into a pre-built time prediction model based on deep reinforcement learning to obtain a user interaction sequence in a future time period;

The virtual environment control module is used to obtain n future virtual scene states based on relevant environmental data and user interaction sequences using a pre-built probabilistic graph model; render the corresponding virtual scene images based on the n future virtual scene states, and transmit the virtual scene images to the user-end device. The modules are connected by wired and/or wireless means to achieve data transmission between modules.

Example 3

This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, an operation mode of a virtual scene interaction method provided above is implemented.

Since the electronic device introduced in this embodiment is an electronic device used to implement a virtual scene interaction method in the embodiment of the present application, based on the virtual scene interaction method introduced in the embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements an electronic device used in a virtual scene interaction method in the embodiment of the present application, it belongs to the scope of protection of this application.

The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims (6)

1. A virtual scene interaction method, characterized by comprising: S1, obtaining user current interaction instruction data and related environment data, preprocessing the user current interaction instruction data, and obtaining a fused interaction feature sequence; S2. Input the fused interaction feature sequence into the pre-built time prediction model based on deep reinforcement learning to obtain a user interaction sequence in a future time period; S3. According to the relevant environmental data and the user interaction sequence, the pre-built probabilistic graph model is used to obtain n future virtual scene states; based on the n future virtual scene states, the corresponding virtual scene images are rendered, and the virtual scene images are transmitted to the user terminal device; The user's current interaction instruction data includes user input device operation data and biometric data; User input device operation data includes key data, gesture data, and device displacement data; biometric data includes eye movement data and limb movement data; The relevant environment data includes the user's virtual position data, viewing direction data and the user's environment state data; the user's environment state data includes scene object position data and lighting data; the key data includes the timestamp, key code and scan code of the key event; The method for acquiring the gesture data includes: A depth camera is used to continuously collect point cloud data of the user's hand area at a fixed frame rate to form a point cloud data sequence; Extracting accurate hand point cloud data from the point cloud data sequence, downsampling the hand area point cloud data to obtain complete hand point cloud data; The complete hand point cloud data is subjected to feature extraction to obtain the palm normal vector, fingertip position vector, finger base position vector and inter-finger angle vector; the palm normal vector, fingertip position vector, finger base position vector and inter-finger angle vector are concatenated to obtain the gesture feature vector; Arrange the acquired gesture feature vectors into a gesture feature vector sequence in chronological order, input the gesture feature vector sequence into a pre-trained gesture recognition model, output the probability distribution of the corresponding gesture type, and take the code of the gesture type with the largest probability in the probability distribution as the gesture data; The method of preprocessing the user's current interaction instruction data includes: Normalizing the data of different modes in the user's current interaction instruction data to obtain normalized instruction data; Align the normalized instruction data in time series to obtain the corresponding multimodal sequence; splice the multimodal sequence in time dimension to form a feature time series data with g feature channels; The feature time series data is constructed as a three-dimensional tensor, where two dimensions represent time and feature channels, and the third dimension is used to represent the corresponding data; The method for constructing the time prediction model based on deep reinforcement learning includes: The time prediction model includes a sequence modeling network and a reinforcement learning network. The gated recurrent unit network is used as the basic framework of the sequence modeling network. The gated recurrent unit network includes a network input layer, a GRU hidden layer, and a state representation output layer. Collect interaction sequence data of group N1, each interaction sequence data contains time step feature representation and corresponding label; The interactive sequence data is gradually input into the gated recurrent unit network in batches to obtain the final state vector of the corresponding batch, and the final state vector of the corresponding batch is input into the preset classifier to obtain the distribution of the predicted label probability; the cross entropy loss between the label probability and the label is calculated, and the trainable parameters of the gated recurrent unit network and the classifier are gradient updated using an optimizer, and the cross entropy loss is minimized to obtain a sequence modeling network; Define the reinforcement learning network as a semi-Markov decision process, and define a tuple (S1, N2, P1, R1) for the semi-Markov decision process; where S1 is the state space of the decision of the low-level manager, N2 is the subtask space that the high-level manager can choose, P1 is the state transition function of the decision of the low-level manager, and R1 is the reward function of the decision of the low-level manager; The semi-Markov decision process is divided into the strategy of the high-level manager and the strategy of the low-level manager; the goal of the semi-Markov decision process is defined as learning an optimal strategy of the high-level manager and Q optimal strategies of the low-level managers, the final state vector of the corresponding batch obtained by the gated recurrent unit network of the interaction sequence data is used as the state of the semi-Markov decision process, the semi-Markov decision process is trained, and the strategy of the high-level manager and the strategy of the low-level manager are optimized respectively using the policy gradient algorithm; the expected cumulative reward calculated in the semi-Markov decision process is maximized; and the reinforcement learning network is constructed; The construction method of the probability graph model includes: Define random variables of the probabilistic graph model, where the random variables are data in the relevant environmental data; construct an undirected graph structure based on the mutual influence relationship between various objects in the virtual scene; the nodes in the undirected graph structure represent random variables, and the edges in the undirected graph structure represent the mutual dependence relationship between random variables; define a node potential function for each node, and define an edge potential function for each edge; Collect historical data of group M1, which is the time series data of virtual scene status; initialize the parameters of node potential function and edge potential function, use the time series data of virtual scene status to train the probability graph model, and obtain the parameters of node potential function and edge potential function; the training method is the maximum likelihood estimation method; thus, the construction of the probability graph model is completed; The method of obtaining the future n virtual scene states includes: Define the transition probability of the probabilistic graphical model, and based on the transition probability of the probabilistic graphical model, infer the next n time steps to obtain the probability distribution of each object in the virtual scene at each time step. During the inference process, input the user interaction sequence into the probabilistic graphical model in sequence according to the time step; For each time step, the state of the scene objects is sampled or selected according to the probability distribution to construct the virtual scene state at that time step, and then the future n virtual scene states are obtained. 2. The virtual scene interaction method according to claim 1, characterized in that the downsampling method comprises: Insert the point cloud data sequence into the pre-built octree in an orderly manner, and then traverse the octree at a fixed spatial resolution to obtain the complete hand point cloud data; The octree is constructed in the following manner: The three-dimensional space where the point cloud data sequence is located is divided into regular cubic units. According to the coordinates of the points in the point cloud data sequence, each point is inserted into the corresponding cubic unit. For a cubic unit containing m points, it is further recursively divided into 8 sub-units until there is at most one point in each cubic unit, that is, the construction of the octree is preliminarily completed; The method of traversing the octree includes: Set a distance threshold d, start from the root node of the octree, traverse each node of the octree in depth-first or breadth-first order, and for each non-empty node, calculate the distance from all points in the node to the center of the node, and select the point closest to the center of the node as the representative point of the node; Initialize an empty list L, traverse each non-empty node X of the octree, calculate the distance d(c,p) between the center point c of X and the representative point p, if d(c,p) is greater than or equal to d, then add p to the list L, and finally the points in the list L are the complete hand point cloud data; The calculation formula of the distance d(c,p) is: Among them, (c _x , _cy ,c _z ) are the three-dimensional coordinates of the center point c, and (p _x , _py ,p _z ) are the three-dimensional coordinates of the representative point p. 3. The virtual scene interaction method according to claim 2, characterized in that the feature extraction method comprises: Perform principal component analysis on the complete hand point cloud data to obtain the main direction vector Palm normal vector for The vector product of the negative z-axis direction of the three-dimensional coordinate system where the hand is located, that is, in, is the unit vector of the z-axis; The complete hand point cloud data is constructed into a set P, and a complete convex hull CH(P) is constructed based on the set P; Perform hierarchical clustering on the complete CH(P) to obtain k+1 levels, denoted as L0, L1, ..., Lk-1, Lk; where Lk-1 is the fingertip point set; For each point vj in Lk-1, traverse all vertices on CH(P), calculate the Euclidean distance from vj to all vertices and take the minimum value dj. After calculating the dj of all vj, take the point corresponding to the minimum value of dj, which is the fingertip position. Vectorize the fingertip position to get the fingertip position vector Where t is the index of the finger; Calculate the position vector of each fingertip Distance _dt to the palm plane, along the normal to the palm plane Direction from Shift back α to get the corresponding root position vector Among them, α is the empirical coefficient; For any two adjacent fingertip position vectors and calculate Vector and Vector in, is the finger root position vector corresponding to the t+1 finger; Based on vector and Calculate the finger angle The inter-finger angle is vectorized to obtain the inter-finger angle vector. 4. The virtual scene interaction method according to claim 3, characterized in that the construction method of the convex hull CH(P) includes: Initially, the convex hull CH(P) is a tetrahedron formed by any four non-coplanar points. For a point b in the set P, if b is in CH(P), then b is ignored. If b is on CH(P), then b is added to the vertex set of CH(P). If b is outside CH(P), then b is used to traverse and divide CH(P) to generate a new CH(P), and finally a complete CH(P) is obtained. The method of performing hierarchical clustering includes: Initialize L0 equal to CH(P). For each level Lr, r belongs to the interval [0, k], and k is a positive integer greater than 1; calculate the Euclidean distance d(vp, vq) of all point pairs (vp, vq) in Lr, extract the point pair (pi, pj) corresponding to the largest Euclidean distance, merge (pi, pj) into a new cluster, remove pi and pj, and form a new point set Lr+1 with the remaining points and the new cluster; repeat until Lr+1 is an empty set or contains only one cluster, and the hierarchical clustering of the complete CH(P) is completed. 5. The virtual scene interaction method according to claim 4, characterized in that the training method of the gesture recognition model includes: Collect gesture feature vector sequences of M groups of gesture actions and the encoding of their corresponding gesture types to form a training data set; define a temporal convolutional network as the backbone network of the gesture recognition model, the temporal convolutional network consists of h residual temporal convolutional blocks, each residual temporal convolutional block includes a temporal convolutional layer and a dropout layer, and residual connections are added to the temporal convolutional layer and the dropout layer; The calculation formula of the temporal convolution layer is: Among them, y _s is the output feature of the temporal convolution layer at time step s, k is the size of the convolution kernel, i is the offset of the convolution kernel in the time dimension, x _si is the input feature of the si-th time step, Θ represents the convolution operation, _wi is the convolution kernel coefficient, B is the bias term, f() is the ReLU function, and f(x _si ) is the result of transforming the input feature x _si through the ReLU function; The output of the last residual temporal convolution block is a context feature sequence. A classification head is added to the backbone network, and the context feature sequence is input into the classification head. The features of each time step in the context feature sequence are classified using a fully connected layer to obtain the probability distribution of the gesture type. The classification formula is: P( _{yscs )} = Softmax( _Wql · _cs + _Bql ); where _Wql is the weight of the fully connected layer, _Bql is the bias of the fully connected layer, and cs is the feature of the context feature sequence at time step s; Define the loss function of the gesture recognition model Among them, S is the total length of the context feature sequence, is the encoding of the actual gesture type at the s-th time step; The gesture recognition model is trained using the training data set, the loss function is minimized in the process, and the parameters of the gesture recognition model are continuously optimized to obtain a trained gesture recognition model. 6. A virtual scene interaction system, used to implement the virtual scene interaction method according to any one of claims 1 to 5, characterized in that it comprises: a data acquisition and processing module, used to obtain the user's current interaction instruction data and related environmental data, and pre-process the user's current interaction instruction data to obtain a fused interaction feature sequence; The prediction module is used to input the fused interaction feature sequence into a pre-built time prediction model based on deep reinforcement learning to obtain a user interaction sequence in a future time period; The virtual environment control module is used to obtain n future virtual scene states based on relevant environmental data and user interaction sequences using a pre-built probabilistic graph model; render corresponding virtual scene images based on the n future virtual scene states, and transmit the virtual scene images to the user-end device. The modules are connected via wired and/or wireless means.