CN114092639A - Upper body posture reconstruction method and device, electronic equipment and storage medium - Google Patents

Abstract

The application discloses a method for reconstructing upper body postures, which comprises the following steps: acquiring 6-DoF attitude information acquired by inertial measurement unit sensors at the head and two hand positions of a user; calculating the 6-DoF posture information based on the deep neural network model to obtain a plurality of posture information of the upper body joint positions of the user; and carrying out three-dimensional reconstruction calculation on the upper body posture of the user according to the plurality of posture information to obtain the upper body human model of the user. The method can effectively track the 6-DoF posture information of the head position and the two hand positions of the user, complete the reconstruction of the upper body human body model of the user and simplify the reconstruction process of the human body model.

Description

Upper body posture reconstruction method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of computer vision, and more particularly, to a method and an apparatus for reconstructing an upper body posture, an electronic device, and a storage medium.

Background

The method for capturing and reconstructing the motion posture of the human body comprises the steps of accurately measuring the motion information of a moving object in a three-dimensional space by utilizing optical fibers, videos, geomagnetism, inertial sensors and the like, further calculating and processing the collected motion data, and reconstructing a motion model of the human body in a computer virtual world, thereby realizing the reproduction of the motion of the human body. The core technology mainly comprises motion capture and virtual reality. With the development of computer science and technology, especially the gradual maturity of real-time three-dimensional graph generation and display technology, human motion posture capture and reconstruction show great application value in more and more fields of motion human science research, movie and television production, game design, medical health and the like.

In the application of virtual display, the whole body human motion model of a user is usually reconstructed to realize scene interaction, the prior related technology can not directly reconstruct the human motion model based on equipment of virtual reality, augmented reality and mixed reality, the model is mainly reconstructed by mechanical, acoustic and optical posture capture, the mechanical device has good capture effect only when the action rate is slow due to the limitation of the structure of the mechanical device, and a tracking module is additionally added at different parts of a human body, the user experience is poor, the acoustic type is sensitive to parameters such as temperature, relative humidity and atmospheric pressure in the scene, the capture precision is influenced by echo caused by a large shielding object in the environment, the optical type has higher requirement on the irradiation condition of light in the environment and can only be used in a specific scene calibrated for a camera, therefore, the method for capturing the motion posture of the human body has many defects, thereby having an influence on the reconstruction of the motion model.

Disclosure of Invention

The embodiment of the application provides an upper body posture reconstruction method and device, electronic equipment and a storage medium. The method aims to directly utilize the existing virtual reality, augmented reality and mixed reality system to construct the upper body posture model of the user without additionally arranging acquisition equipment of extra posture information, so that the process of capturing the motion posture of the human body is simplified, and the posture reconstruction efficiency is improved.

In a first aspect, some embodiments of the present application provide a method for reconstructing an upper body pose, the method including: the method comprises the steps of obtaining 6-DoF posture information collected by an inertial measurement unit sensor on the head and two hands of a user, calculating the 6-DoF posture information based on a deep neural network model to obtain a plurality of posture information of the upper body joint positions of the user, and performing three-dimensional reconstruction calculation on the upper body posture of the user according to the plurality of posture information to obtain an upper body human body model of the user.

In a second aspect, some embodiments of the present application further provide an upper body posture reconstruction device, including: the device comprises an information acquisition module, a posture calculation module and a posture reconstruction module, wherein the information acquisition module is used for acquiring 6-DoF posture information acquired by an inertial measurement unit sensor on the head and two hand positions of a user, the posture calculation module is used for calculating the 6-DoF posture information based on a deep neural network model to obtain a plurality of posture information of the upper body joint positions of the user, and the posture reconstruction module is used for performing three-dimensional reconstruction calculation on the upper body posture of the user according to the plurality of posture information to obtain an upper body human body model of the user.

In a third aspect, some embodiments of the present application further provide an electronic device, including a display screen, a processor, and a memory, where the memory stores computer program instructions, and the computer program instructions, when called by the processor, execute the above-mentioned upper body posture reconstruction method.

In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium storing program codes, where the program codes are executed by a processor to perform the above-mentioned upper body posture reconstruction method.

According to the upper body posture reconstruction method, the 6-DoF posture information collected by the inertial measurement unit sensors at the head and two hand positions of the user can be obtained, the 6-DoF posture information is calculated based on the deep neural network model, a plurality of posture information of the upper body joint positions of the user is obtained, and further, the three-dimensional reconstruction calculation is carried out on the upper body posture of the user according to the plurality of posture information, and the upper body manikin of the user is obtained. Therefore, the reconstruction of the upper body human model of the user can be completed only by tracking the 6-DoF posture information of the head position and the two hand positions of the user, so that the use of equipment for capturing the motion posture of the human body is reduced, and the reconstruction process of the human model is simplified.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 shows an architectural diagram of an upper body posture reconstruction system provided in an embodiment of the present application.

Fig. 2 shows a schematic flow chart of an upper body posture reconstruction method provided in an embodiment of the present application.

Fig. 3 is a schematic flow chart illustrating another upper body posture reconstruction method according to an embodiment of the present disclosure.

FIG. 4 shows a flow diagram of the steps of training the deep neural network model of FIG. 3.

FIG. 5 shows a schematic diagram of a single-layer LSTM-RNN structure provided by the embodiments of the present application.

Fig. 6 shows a schematic flow chart of the step of deriving joint coordinates of fig. 3.

Fig. 7 shows an upper body human skeleton hierarchy tree according to an embodiment of the present application.

Fig. 8 shows a framework diagram of upper body posture reconstruction software provided in an embodiment of the present application.

Fig. 9 shows a block diagram of an upper body posture reconstruction device according to an embodiment of the present application.

Fig. 10 is a block diagram of an electronic device according to an embodiment of the present disclosure.

Fig. 11 is a block diagram of a computer-readable storage medium according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.

In order to make the technical solutions of the present application better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

With the rapid development of virtual reality technology, virtual reality has become more and more well known. In order to pursue a stronger immersion feeling, almost in all virtual reality, human-computer interaction of both single people and multiple people has a requirement for reconstructing human body gestures in real time, and actions captured by peripheral devices (such as a camera and a laser radar) are generally utilized to recognize action gestures in a user interaction process and generate a 3D human body model so as to improve the experience feeling of a user in the virtual reality.

Because the current mainstream 3D human model reconstruction technology has strict requirements on input data, for example, human posture data or human whole body image data of 6 different parts of a body need to be acquired, and these data cannot be acquired in the existing vr (visual reality)/ar (augmented reality) products. Therefore, the reconstruction process of the 3D human body model requires the addition and subtraction of peripheral devices to capture the motion gesture of the user, thereby increasing the use cost and complicating the reconstruction process of the human body model.

Generally, existing VR/AR headsets are equipped with 6-degree-of-freedom (DoF) positioning and dual-controller 6-DoF positioning, such as Facebook Oculus headsets, Microsoft hollenes headsets, Mixed reality handles, etc., which cannot directly provide body posture data or body whole-body image data, and even if the image data can be provided, the user cannot be found due to the wearing of the user, so that the existing human body motion posture capturing and reconstructing schemes cannot be directly applied to the current virtual reality products.

In order to solve the above problems, the inventor has made a long-term study and provides an upper body posture reconstruction method provided in an embodiment of the present application, which includes directly obtaining 6-DoF posture information acquired by an inertial measurement unit sensor at a head and both hands position of a user by using a virtual device in a mode of sacrificing model reconstruction accuracy, calculating the 6-DoF posture information based on a deep neural network model to obtain a plurality of posture information of upper body joint positions of the user, and further performing three-dimensional reconstruction calculation on the upper body posture of the user according to the plurality of posture information to obtain an upper body human body model of the user. Therefore, under the condition of reducing the use of peripheral equipment, the construction of the upper body human body model can be realized by utilizing virtual reality, augmented reality and mixed reality, no additional capturing/collecting equipment is needed, and the process of capturing and reconstructing the motion posture of the human body is simplified.

Referring to fig. 1, fig. 1 is a schematic diagram of an upper body posture reconstruction system according to an embodiment of the present disclosure, and in some embodiments, the upper body posture reconstruction system 400 may include: an acquisition module 401, a processing module 403, and a display module 405. Optionally, the acquisition module 401 is configured to acquire 6-DoF pose information, and the acquisition module 401 may be a virtual reality product: inertial Measurement Unit (IMU) sensors mounted in the head display 401A, left handle 401B, and right handle 401C. The IMU is a sensor mainly used for detecting and measuring acceleration and rotational motion, and can acquire 6-DoF posture information on the positions of the head and the hands of the user through the IMU, where the 6-DoF posture information may include at least position information, rotational information, and inertial information of the head and the hands of the user, and is not limited herein.

Optionally, the processing module 403 is configured to process the acquired 6-DoF pose information to obtain a plurality of pose information of positions of upper body joints of the user, perform three-dimensional reconstruction calculation on the upper body pose of the user based on the plurality of pose information to obtain an upper body human model of the user, further, the display module 405 is configured to display the upper body human model of the user, and optionally, the display module 405 may perform operations such as rendering on the upper body human model of the user, so that the upper body human model more truly restores the pose of the upper body human of the user. For example, the upper body posture reconstruction system 400 can be used in a virtual reality game scene, after a user wears a head display and two-hand handles, the upper body posture reconstruction system 400 can collect the 6-DoF posture information of the user on the head and two-hand positions in real time by using the head display and the two-hand handles, and then generate and display the upper body manikin of the user in the game scene.

It should be noted that the framework diagram of the upper body posture reconstruction system shown in fig. 1 is only an example, the architecture and the application scenario of the upper body posture reconstruction system described in the embodiment of the present invention are for more clearly illustrating the technical solution of the embodiment of the present invention, and do not form a limitation on the technical solution provided in the embodiment of the present invention, and it can be known by those skilled in the art that the technical solution provided in the embodiment of the present invention is also applicable to similar technical problems along with the evolution of the architecture of the upper body posture reconstruction system and the appearance of a new application scenario.

Embodiments in the present application will be described in detail below with reference to the accompanying drawings.

Referring to fig. 2, fig. 2 illustrates an upper body posture reconstruction method provided in an embodiment of the present application, where the upper body posture reconstruction method may include the following steps S110 to S130.

Step S110: and 6-DoF attitude information acquired by the inertial measurement unit sensors on the head and the two hand positions of the user is acquired.

In the embodiment of the application, the 6-DoF attitude information refers to the 6-DoF attitude information collected by the inertial measurement unit sensor on the head and two-hand positions of the user. In consideration of the fact that different motion capture systems are different in built human body models and different in motion requirements of human bodies to be captured, the number and specific binding positions of IMU sensors required by the motion capture systems are different, motion attitude data of the upper body of the human body needs to be captured, and in order to reduce the use of additional equipment, the IMU sensors in virtual reality products can be directly used for acquiring 6-DoF attitude information.

As an embodiment, the 6-DoF pose information on the head and hands positions of the user may be collected using IMU sensors mounted in the head display and the hands grips. For example, during a user's experience with a game using a head display and a two-handed handle, which may be a virtual reality product of the Pico Neo series, the 6-DoF pose information is acquired in real time through IMU sensors in the body of the head display and the two-handed handle.

As another implementation mode, the IMU sensors can be directly bound to the head of the human body and the wrists of the two hands respectively according to the specific requirement of capturing the 6-DoF posture information. Considering that the relevant muscles and skin deform along with the movement of the human body, the bound sensor and the limb generate relative displacement or rotation movement, and great error influence is brought to the posture measurement result of the IMU sensor.

In order to avoid as much as possible measurement errors due to relative movements between the sensor and the limb, we choose to bind the sensor to a place where the muscles and skin do not deform or deform less when the respective limb moves. For example, an IMU sensor that acquires 6-DoF pose information for the user's forearm is bound to the side of the forearm at an intermediate location approximately 3cm from the elbow.

Illustratively, a wireless transmission mode can be adopted, and a data communication protocol based on PCOMM between the sensor wireless receiver and a PC end upper computer program BMR (waiting Motion receiver) is defined. The upper computer program sends a command to the wireless receiver through the RS232 serial port, each IMU sensor transmits the collected 6-DoF attitude information to the wireless receiver through a wireless signal, the wireless receiver transmits the attitude information to the upper computer program through the serial port, and the upper computer program completes the analysis task of the data packet. Wherein, the data packet structure may be:

step S120: and calculating the 6-DoF posture information based on the deep neural network model to obtain a plurality of posture information of the upper body joint positions of the user.

In the prior art, the motion posture of the whole body is derived by calculating collected 6-DoF posture information of some joint activities of a user based on a reverse kinematics principle. However, since many joints have a high degree of freedom, it is difficult to accurately define the calculation parameters of each joint in the calculation process, and in the embodiment of the present application, the 6-DoF posture information may be calculated by using a deep neural network model to obtain a plurality of posture information of the positions of the joints of the upper body of the user.

Among them, Deep Neural Network (DNN) is also called Multi-Layer Perceptron (MLP). The DNN model is a simplified model that simulates the way the human brain processes information, and works by simulating a large number of interconnected processing units in an abstract form similar to neurons, and consists of three parts: an input layer, wherein elements represent input fields; one or more hidden layers; an output layer with one or more elements representing the object field, the elements being connected by a variable connection strength (weight). The posture information refers to information of the motion posture of the main joint position of the upper body of the user, and the posture information is also 6-DoF information.

In some embodiments, the acquired 6-DoF pose information can be input into a trained deep neural network model as input data, and the deep neural network can output a plurality of pose information of the upper body joint position of the user through calculation. Optionally, the DNN model may be a Recurrent Neural Network (RNN), an output layer of which may output probability values using a Sigmoid activation function, and may be trained using a cross entropy loss function when the model is trained. Considering that different modes of model training exist, different calculation modes can be utilized to calculate the 6-DoF posture information.

As an embodiment, a DNN model for calculating the posture information can be trained in advance, and then the posture information can be calculated off-line for the 6-DoF posture information. As another embodiment, the parameters of the DNN model can be updated in real time based on the acquired 6-DoF posture information, and the updated DNN model is obtained to calculate the posture information on line.

According to the method and the device, reverse derivation formulas of all joints are not required to be defined explicitly, model parameters are learned automatically based on a neural network, optimal mapping from input data to output data is trained and optimized, posture information is obtained according to the optimal mapping and 6-DoF posture information, the posture information can be understood as data expression of vertexes of a three-dimensional space corresponding to the posture of the upper body of a user, and then the upper body manikin which can be conveniently fused into virtual reality equipment and application is generated based on the posture information.

Step S130: and performing three-dimensional reconstruction calculation on the upper body posture of the user according to the plurality of posture information to obtain the upper body human model of the user.

Wherein the upper body mannequin refers to a 3D model of the upper body of the user. After the 6-DoF posture information is obtained, the intuitive and easily understood 6-DoF posture information needs to be converted into a data format which is easy to understand and calculate by a computer system, and then the upper body human body model can be generated. In some embodiments, coordinate transformation may be performed on the plurality of posture information to obtain joint coordinates for building a skeleton model of the upper body of the user, and then reconstruction of the posture of the upper body is completed according to the joint coordinates to obtain the upper body manikin of the user.

In the embodiment of the application, the 6-DoF posture information collected by the inertial measurement unit sensors at the head and two hand positions of the user can be obtained, the 6-DoF posture information is calculated based on the deep neural network model to obtain a plurality of posture information of the joint positions of the upper body of the user, and further, the three-dimensional reconstruction calculation is carried out on the posture of the upper body of the user according to the plurality of posture information to obtain the human body model of the upper body of the user. The reconstruction of the upper body human model of the user is completed by tracking the 6-DoF posture information of the head position and the two hand positions of the user and calculating the 6-DoF posture information based on the deep neural network, so that the use of equipment for capturing the motion posture of the human body is greatly reduced, and the reconstruction process of the human body model is simplified.

Referring to fig. 3, fig. 3 illustrates another upper body posture reconstruction method provided in the embodiment of the present application, which may include steps S210 to S280.

Step S210: and 6-DoF attitude information acquired by the inertial measurement unit sensors on the head and the two hand positions of the user is acquired.

In this embodiment, the specific implementation of step S210 may refer to the description of step S110 provided in the above embodiments, and is not described herein again.

Step S220: a deep neural network model for determining a plurality of posture information of the upper body joint position of the user is trained based on the historical 6-DoF posture information and the historical posture information.

The historical 6-DoF posture information and the historical posture information refer to a large amount of collected 6-DoF posture information and posture information before the deep neural network model is trained, and the historical 6-DoF posture information and the historical posture information are used for training the deep neural network model. Specifically, referring to fig. 4, step S220 may include steps S221 to S223.

Step S221: historical 6-DoF pose information and historical pose information are obtained.

Step S222: and taking the historical posture information as a label, and taking the historical 6-DoF posture information and the historical posture information as a training set.

As an implementation mode, a subject can wear a head display and two handles and can move according to a designed action, historical 6-DoF posture information and historical posture information are further collected in the moving process of the subject, tags corresponding to various postures are further added to the historical posture information, and the historical 6-DoF posture information and the historical posture information are used as a training set. Considering that the speed and amplitude of the head and hand movements of the subject are different in the course of action, optionally, after obtaining the historical 6-DoF posture information and the historical posture information of the user, the historical 6-DoF posture information and the historical posture information may be normalized.

Illustratively, the pose information and the pose information of the 6-DoF are taken for 300 subjects wearing a head display and a two-handed handle (IMU sensor) who moved left and right for 90 minutes. Considering the insufficient data diversity, a Model with sufficient generalization capability cannot be trained, and therefore, more 6-DoF posture information for Model training can be synthesized, for example, the 6-DoF posture information can be synthesized by using a virtual IMU sensor and a Skinned Multi-Person Linear Model. In order to make the deep neural network model have robustness to prediction errors of the attitude information, Gaussian noise is added into the 6-DoF attitude information in the training process, and then a data set formed by historical 6-DoF attitude information and historical attitude information is divided into a training set and a test set of the model training according to a certain proportion.

Step S223: a deep neural network model for determining a plurality of posture information of the main joint positions of the upper body of the user is trained based on a training set.

As one embodiment, a Long Short Term Memory (LSTM) unit may be used to construct a recurrent neural network (LSTM-RNN) as a model for pose information generation. Specifically, as shown in FIG. 5, FIG. 5 illustrates a single-layer LSTM-RNN structure, wherein the initial states include an initial hidden state and an initial cellular state, typically initialized with a zero vector. The input data is data formed by historical 6-DoF attitude information and historical attitude information. The output layer can be a softmax layer which can convert the output numerical values of the multiple classifications into relative probabilities, and the output of the output layer is a multi-dimensional vector which respectively represents the probabilities of the corresponding multiple kinds of posture information. Optionally, a cross entropy loss function is set.

Further, after the LSTM-RNN model is built, training the model by using a training set, specifically, each training cycle comprises initializing an initial hidden state and an initial cellular state, inputting data in the training set, carrying out forward propagation and calculating loss, carrying out backward propagation and calculating gradient, updating model weight, observing loss values in the training process, changing learning rate and continuing training when loss is not reduced any more until the test effect of the model in the test set reaches the optimum, and obtaining the well-trained LSTM-RNN model.

Step S230: and calculating the 6-DoF posture information based on the deep neural network model to obtain a plurality of posture information of the upper body joint positions of the user.

As an implementation mode, the collected 6-DoF posture information can be input into an LSTM-RNN model for calculation, and a plurality of posture information of the upper body joint positions of the user can be obtained.

As another implementation mode, image information of the upper body of the user can be obtained, and the 6-DoF posture information and the image information are calculated by using the deep neural network model to obtain a plurality of posture information of the main joint positions of the upper body of the user. For example, after the image information of the upper body of the user during movement is collected, the Visual Inertial VINet network (Visual Inertial interferometry) can be used for fusing the image information and the 6-DoF posture information to learn the posture information of the upper body of the user.

Step S240: and matching the plurality of posture information with preset posture information to obtain a matching result, and judging whether error posture information exists in the plurality of posture information according to the matching result.

Considering that the deep neural network model can have wrong parameters in the training process, wrong attitude information occurs in the output of the model. The preset posture information refers to posture information which is set in advance according to a human body kinematics principle and moves under a normal posture condition, and wrong posture information is found by comparing the posture information output by the model with the preset posture information, so that the wrong posture information output by the model is adjusted.

After acquiring a plurality of posture information output by the deep neural network model, performing matching calculation on the posture information and preset posture information. For example, whether the pose information and the preset pose information are matched or not is determined by calculating Cosine similarity (Cosine similarity) of the pose information and the preset pose information, and when the two are not matched, the pose information output by the model can be determined as error pose information.

Step S250: and if the wrong posture information exists, determining deviation between the plurality of posture information and preset posture information, and adjusting the plurality of posture information of the upper body joint position of the user according to the deviation.

The deviation amount is obtained by judging the degree of difference between the posture information output by the model and the preset posture information, and optionally, the deviation amount can be obtained by subtracting the posture information from the preset posture information.

As an embodiment, when the error posture information occurs in the output of the model, an absolute value of a difference between the error posture information and the preset posture information may be calculated, and further, the error posture information may be adjusted to the corresponding preset posture information according to a magnitude of the absolute value of the difference. For example, when the user has a bent posture between the lower arm and the upper arm, if the posture information of the two arms output by the model is different from the preset posture information, the posture information of the two arms output by the model may be replaced by the corresponding preset posture information, or the output posture information of the two arms may be adjusted to be close to the corresponding preset posture information.

Step S260: and performing coordinate conversion on the plurality of posture information according to a coordinate conversion rule to obtain joint coordinates for establishing a skeleton model of the upper body of the user.

The coordinate conversion rule refers to a joint coordinate conversion relation between a self coordinate system of the IMU sensor and a corresponding human skeleton coordinate system in the three-dimensional human skeleton model. The posture information of the human body movement is obtained from the 6-DoF posture information calculation through a deep neural network model, and the posture information needs to be converted into a data format which is easy to understand and calculate by a computer. Therefore, the posture information of each part of the human body moving in real time is subjected to coordinate conversion and then is used for driving the established three-dimensional human skeleton model of the virtual world, and the purpose is to drive the human skeleton model to simulate human actions and finish reconstruction of the posture of the upper body of the human body. Specifically, referring to fig. 6, step S260 may include step S261 and step S262.

Step S261: and determining coordinate conversion rules of a plurality of coordinate systems corresponding to the posture information and coordinate systems corresponding to the upper body skeleton model of the user.

As an embodiment, the IMU sensor coordinate system and the bone coordinate system may be inter-transformed using a synthetic transformation based on rotational and translational transformations of coordinates. The coordinate rotation transformation is realized by using rotation matrix multiplication of the initial coordinate corresponding to the Euler angle rotation axis. And the coordinate translation transformation is realized by adding and subtracting corresponding rotation matrixes, so that the coordinate transformation of the plurality of posture information is completed.

Step S262: and converting the plurality of posture information into joint coordinates of a coordinate system corresponding to the upper body skeleton model of the user based on a coordinate conversion rule.

The pose information pertains to the coordinates of the IMU sensor's own coordinate system relative to the geomagnetic coordinate system, and therefore, it is necessary to convert this pose information to its corresponding skeletal coordinate system in the upper body skeletal model during motion pose capture.

As an implementation manner, based on a coordinate transformation rule, a rotation matrix between bones of the upper body can be calculated, specifically, all the bone trees in the three-dimensional upper body human body bone hierarchy structure tree are traversed in sequence, and a posture value, that is, a joint coordinate, of each bone on the upper body is calculated.

For example, referring to fig. 7, fig. 7 shows a hierarchical structure tree of upper body human skeleton, wherein 201 is a root node

Is a quaternion corresponding to the rotation relation of an IMU sensor coordinate system corresponding to the ith skeleton in the upper body human skeleton tree relative to a geomagnetic coordinate system, i belongs to [1, 12 ]]，

The quaternion corresponding to the rotation relation between the ith bone in the human body bone tree and the corresponding sensor coordinate system is already obtained during the initial calibration, so the quaternion Xi under the self bone coordinate system is calculated and disclosed as follows:

if the father skeleton of the i +1 th skeleton in the human skeleton tree is the ith skeleton, the rotation relation quaternion from the i +1 th skeleton to the father skeleton is as follows:

substituting the formula (1) into the formula (2) to obtain:

according to the formula (1), the posture coordinate of the waist root node is firstly obtained, and then according to the formula (3), all skeleton trees in the human skeleton hierarchical structure tree are traversed in sequence, and the joint coordinate of each skeleton is calculated.

Step S270: and generating an upper body skeleton model of the user according to the joint coordinates.

Step S280: rendering the upper body skeleton model based on preset rendering data to generate an upper body human body three-dimensional model of the user.

The coordinate system of the model itself is called the local coordinate system. The model is built in the local coordinate system of the object itself without considering the relation such as the distance, the size ratio, the orientation and the like with respect to other models, and thus the upper body skeleton model of the user can be built in the set local coordinate system. Wherein the upper body skeleton model is a virtual 3D skeleton model generated based on joint coordinates. The preset rendering data refers to attribute values for rendering the upper body skeleton model, and may include skin attributes, clothing attributes, gender attributes, and the like, which are not limited herein.

As an embodiment, after the upper body skeleton model of the user is drawn based on the joint coordinates, the upper body skeleton model may be rendered according to preset rendering data to generate an upper body three-dimensional model of the user. For example, in Direct3D, the user's upper body three-dimensional model of the human body may be described by a mesh. A grid is actually a set of collections with rendering properties. The complex three-dimensional model of the upper body can be made by professional three-dimensional modeling software, such as 3D Max, Maya and the like, and the relevant information of the model is saved in a specific model file, and then the model file is loaded into a mesh model object in an application program, and the display of the three-dimensional object model is controlled by the object.

In the embodiment of the application, the 6-DoF attitude information collected by the inertial measurement unit sensors on the head and the two hands of the user can be acquired, training a deep neural network model of a plurality of posture information for determining the positions of the upper body joints of the user based on the historical 6-DoF posture information and the historical posture information, further calculating the 6-DoF posture information based on the deep neural network model to obtain a plurality of posture information of the positions of the upper body joints of the user, matching the plurality of attitude information with preset attitude information to obtain a matching result, judging whether error attitude information exists in the plurality of attitude information according to the matching result, if so, determining a deviation amount between the plurality of posture information and preset posture information, and adjusting the plurality of posture information of the upper body joint position of the user according to the deviation amount. Thus, the erroneously estimated attitude information is corrected.

Further, according to a coordinate conversion rule, coordinate conversion is carried out on the plurality of posture information, joint coordinates used for building a skeleton model of the upper body of the user are obtained, the upper body skeleton model of the user is generated according to the joint coordinates, the upper body skeleton model is rendered based on preset rendering data, a three-dimensional model of the upper body of the user is generated, and therefore the upper body posture of the user is reconstructed in real time by directly utilizing an IMU sensor of a virtual reality product.

Referring to fig. 8, fig. 8 is a schematic diagram illustrating a framework of upper body posture reconstruction software 300 according to an embodiment of the present application. The upper body posture reconstruction software 300 may include a system common module, a system configuration module, a data storage module, a graphic rendering module, a data analysis module, and a device communication module.

In some embodiments, a system generic module may contain the generic constructs on which the various parts of the system rely, primarily the motion capture data structures and their underlying operations, generic enumeration and constant data, system function calls, and the like.

Optionally, the system configuration module may include functional constructs for operating various configurable portions of the system, and may initially configure the sensor by defining the skeletal structure of the human body. The data storage module may include a memory for historical 6-DoF pose information, historical pose information, pre-set rendering data, etc. for rendering and animation generation of the upper body skeletal model, etc. The graphic rendering module comprises a graphic rendering function based on Managed Direct3D, provides a rendering function of an upper body human body three-dimensional model, can perform single-person behavior rendering and group behavior rendering, and can adopt NET Framework as a Managed code programming model.

The device communication module may include functional packaging of a system hardware device communication mechanism, and basic operations such as unpacking, decoding, formatting, and data acquisition of raw data, for example, acquiring data required for upper body posture reconstruction calculation, such as 6-DoF posture information acquired by an IMU sensor, through communication protocols such as Wi-Fi, Bluetooth, or ZigBee. The user interaction module integrates various system functions based on the upper-layer structure of the module, provides a user-friendly graphical interface and access operation, and provides support for a user in different virtual reality application scenes. The data analysis module can perform real-time and off-line analysis on the acquired 6-DoF attitude information and estimate the attitude information based on a deep neural network model.

Referring to fig. 9, a block diagram of an upper body posture reconstructing device 500 according to an embodiment of the present application is shown. The upper body posture reconstructing apparatus 500 includes: the information acquisition module 510 is used for acquiring 6-DoF attitude information acquired by the inertial measurement unit sensors at the head and two hand positions of the user; the posture calculation module 520 is used for calculating the 6-DoF posture information based on the deep neural network model to obtain a plurality of posture information of the upper body joint positions of the user; and the posture reconstruction module 530 is configured to perform three-dimensional reconstruction calculation on the upper body posture of the user according to the plurality of posture information to obtain an upper body human body model of the user.

In some embodiments, the upper body pose reconstruction device 500 may further include: and the model training module is used for training a deep neural network model of a plurality of posture information for determining the positions of the upper body joints of the user based on the historical 6-DoF posture information and the historical posture information.

In some embodiments, the model training module may be specifically configured to obtain historical 6-DoF pose information and historical pose information; taking historical attitude information as a label, and taking historical 6-DoF attitude information and historical attitude information as a training set; a deep neural network model for determining a plurality of posture information of the main joint positions of the upper body of the user is trained based on a training set.

In some embodiments, the upper body pose reconstruction device 500 may further include: the matching module is used for matching the plurality of posture information with preset posture information to obtain a matching result; the judging module is used for judging whether error posture information exists in the plurality of posture information or not according to the matching result; the determining module is used for determining the deviation amount between the plurality of attitude information and the preset attitude information if the error attitude information exists; and the adjusting module is used for adjusting the plurality of posture information of the upper body joint positions of the user according to the deviation amount.

In some embodiments, the pose reconstruction module 530 may include: the conversion unit is used for carrying out coordinate conversion on the plurality of posture information according to a coordinate conversion rule to obtain joint coordinates for establishing a skeleton model of the upper body of the user; a generation unit for generating an upper body skeleton model of the user based on the joint coordinates; and the rendering unit is used for rendering the upper body skeleton model based on preset rendering data to generate and obtain the upper body human body three-dimensional model of the user.

In some embodiments, the conversion unit may be specifically configured to determine a plurality of coordinate conversion rules of a coordinate system corresponding to the posture information and a coordinate system corresponding to the upper body skeleton model of the user; and converting the plurality of posture information into joint coordinates of a coordinate system corresponding to the upper body skeleton model of the user based on a coordinate conversion rule.

In some embodiments, the pose computation module 520 may be further specifically configured to: acquiring image information of the upper part of a user; and calculating the 6-DoF posture information and the image information by using the deep neural network model to obtain a plurality of posture information of the main joint positions of the upper body of the user.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and modules may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, the coupling between the modules may be electrical, mechanical or other type of coupling.

In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.

According to the scheme, the 6-DoF posture information collected by the inertial measurement unit sensors on the head and two hand positions of the user can be obtained, the 6-DoF posture information is calculated based on the deep neural network model, a plurality of posture information of the upper body joint positions of the user are obtained, further, the three-dimensional reconstruction calculation is carried out on the upper body posture of the user according to the plurality of posture information, and the upper body manikin of the user is obtained. The reconstruction of the upper body human model of the user is completed by tracking the 6-DoF posture information of the head position and the two hand positions of the user and calculating the 6-DoF posture information based on the deep neural network, so that the use of equipment for capturing the motion posture of the human body is greatly reduced, and the reconstruction process of the human body model is simplified.

As shown in fig. 10, an embodiment of the present application further provides an electronic device 600, where the electronic device 600 includes a processor 610, a memory 620, and a display screen 630, where the memory 620 stores computer program instructions, and the computer program instructions are invoked by the processor 610 to execute the above-mentioned upper body posture reconstruction method.

The processor 610 may include one or more processing cores. The processor 610 interfaces with various components throughout the battery management system using various interfaces and lines to perform various functions of the battery management system and to process data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 620 and invoking data stored in the memory 620. Alternatively, the processor 610 may be implemented in hardware using at least one of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 610 may integrate one or more of a Central Processing Unit (CPU) 610, a Graphics Processing Unit (GPU) 610, a modem, and the like. Wherein, the CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing display content; the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 610, but may be implemented by a communication chip.

The Memory 620 may include a Random Access Memory (RAM) 620, and may also include a Read-Only Memory (Read-Only Memory) 620. The memory 620 may be used to store instructions, programs, code, sets of codes, or sets of instructions. The memory 620 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing various method embodiments described below, and the like. The storage data area can also store data (such as a phone book, audio and video data, chatting record data) created by the electronic device map in use and the like.

As shown in fig. 11, an embodiment of the present application further provides a computer-readable storage medium 700, where the computer-readable storage medium 700 has stored therein computer program instructions 710, and the computer program instructions 710 can be called by a processor to execute the method described in the foregoing embodiment.

The computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium includes a non-volatile computer-readable storage medium. The computer readable storage medium 700 has storage space for program code for performing any of the method steps of the method described above. The program code can be read from or written to one or more computer program products. The program code may be compressed, for example, in a suitable form.

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various alternative implementations provided by the embodiments described above.

Although the present application has been described with reference to the preferred embodiments, it is to be understood that the present application is not limited to the disclosed embodiments, but rather, the present application is intended to cover various modifications, equivalents and alternatives falling within the spirit and scope of the present application.

Claims (10)

1. A method of upper body pose reconstruction, the method comprising:

acquiring 6-DoF attitude information acquired by inertial measurement unit sensors at the head and two hand positions of a user;

calculating the 6-DoF posture information based on a deep neural network model to obtain a plurality of posture information of the upper body joint positions of the user;

and carrying out three-dimensional reconstruction calculation on the upper body posture of the user according to the plurality of posture information to obtain the upper body human model of the user.

2. The method of claim 1, wherein prior to the computing the 6-DoF pose information based on the deep neural network model to derive a plurality of pose information for the user's upper body joint positions, the method further comprises:

training a deep neural network model of a plurality of posture information for determining the position of the upper body joint of the user based on the historical 6-DoF posture information and the historical posture information.

3. The method of claim 2, wherein training a deep neural network model for determining a plurality of pose information for the user's upper body joint position based on historical 6-DoF pose information and historical pose information comprises:

acquiring historical 6-DoF attitude information and historical attitude information;

taking the historical posture information as a label, and taking the historical 6-DoF posture information and the historical posture information as a training set;

training a deep neural network model for determining a plurality of posture information of the main joint positions of the upper body of the user based on the training set.

4. The method of claim 1, wherein after computing the 6-DoF pose information based on a deep neural network model to derive a plurality of pose information for the user's upper body joint positions, the method further comprises:

matching the plurality of attitude information with preset attitude information to obtain a matching result;

judging whether error posture information exists in the plurality of posture information or not according to the matching result;

if the error attitude information exists, determining the deviation amount between the plurality of attitude information and the preset attitude information;

and adjusting a plurality of posture information of the upper body joint position of the user according to the deviation amount.

5. The method of claim 1, wherein the performing a three-dimensional reconstruction calculation on the upper body pose of the user according to the plurality of pose information to obtain an upper body human model of the user comprises:

according to a coordinate conversion rule, carrying out coordinate conversion on the plurality of posture information to obtain joint coordinates for establishing a skeleton model of the upper body of the user;

generating an upper body skeleton model of the user according to the joint coordinates;

rendering the upper body skeleton model based on preset rendering data to generate an upper body human body three-dimensional model of the user.

6. The method of claim 5, wherein the coordinate transforming the plurality of pose information according to a coordinate transformation rule to derive joint coordinates for building a skeletal model of the user's upper body comprises:

determining coordinate conversion rules of a plurality of coordinate systems corresponding to the posture information and coordinate systems corresponding to the upper body skeleton model of the user;

and converting the posture information into joint coordinates of a coordinate system corresponding to the upper body skeleton model of the user based on the coordinate conversion rule.

7. The method of claim 1, wherein the computing the 6-DoF pose information based on a deep neural network model yields a plurality of pose information for the user's upper body joint positions, further comprising:

acquiring image information of the upper part of the body of the user;

and calculating the 6-DoF posture information and the image information by using a deep neural network model to obtain a plurality of posture information of the main joint positions of the upper body of the user.

8. An upper body posture reconstruction device, characterized in that the device comprises:

the information acquisition module is used for acquiring 6-DoF attitude information acquired by the inertial measurement unit sensors at the head and two hand positions of a user;

the gesture calculation module is used for calculating the 6-DoF gesture information based on a deep neural network model to obtain a plurality of gesture information of the upper body joint position of the user;

and the posture reconstruction module is used for carrying out three-dimensional reconstruction calculation on the upper body posture of the user according to the plurality of posture information to obtain the upper body human body model of the user.

9. An electronic device, comprising:

a memory;

one or more processors coupled with the memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to perform the method of any of claims 1-7.

10. A computer-readable storage medium, having stored thereon program code that can be invoked by a processor to perform the method according to any one of claims 1 to 7.

Images