WO2019192172A1 - 一种姿态预测方法、装置和电子设备 - Google Patents

一种姿态预测方法、装置和电子设备 Download PDF

Info

Publication number
WO2019192172A1
WO2019192172A1 PCT/CN2018/112854 CN2018112854W WO2019192172A1 WO 2019192172 A1 WO2019192172 A1 WO 2019192172A1 CN 2018112854 W CN2018112854 W CN 2018112854W WO 2019192172 A1 WO2019192172 A1 WO 2019192172A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
network model
training
attitude
prediction
Prior art date
Application number
PCT/CN2018/112854
Other languages
English (en)
French (fr)
Inventor
朱育革
戴天荣
蔡磊
Original Assignee
歌尔股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 歌尔股份有限公司 filed Critical 歌尔股份有限公司
Publication of WO2019192172A1 publication Critical patent/WO2019192172A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks

Definitions

  • the present invention relates to the field of head-mounted display devices, and in particular, to an attitude prediction method, apparatus, and electronic device.
  • VR Virtual Reality
  • AR Augmented Reality
  • the controller can be located in an HMD (Head Mounted Display) or in a handle. Excessive delay of the controller will destroy the immersion. Especially for the device with head-eye motion feedback such as HMD, too high delay will cause serious discomfort and motion sickness of the user, affecting the user's physical health.
  • a complete VR/AR system usually requires a combination of optimizations to reduce controller latency.
  • the VR/AR controller is usually integrated with an attitude sensor such as an IMU (Inertial Measurement Unit), and uses the attitude sensor to acquire the user's posture data, and predicts the user's motion posture according to the attitude data to reduce the user's perception. The delay to the system.
  • IMU Inertial Measurement Unit
  • the existing attitude prediction based on the attitude sensor has a large error, the accuracy is not up to the requirement, or the calculation complexity is high, the implementation is very difficult, the actual demand cannot be met, and the user experience is not good.
  • the embodiment of the invention provides an attitude prediction method, device and electronic device.
  • an attitude prediction method comprising:
  • the posture data generated by the posture sensor of the user during motion is sampled to obtain the original data set
  • the acquired attitude data in a predetermined time period is input into the neural network model after the training is completed, and the attitude data corresponding to the target time after the current time output of the trained neural network model is obtained;
  • the predetermined time period is a time period determined by the current time and a predetermined time before the current time.
  • an attitude prediction apparatus for use in a head mounted display device, comprising:
  • a sampling unit configured to sample the posture data generated by the posture sensor when the user moves according to a preset sampling frequency, to obtain an original data set
  • a training unit configured to determine a layer of the initial neural network model and a number of nodes of each layer according to the original data set and the specified prediction duration, and use the original data set to train the initial neural network model to obtain the trained nerve Network model
  • a prediction unit configured to input the posture data in the predetermined time period of the acquisition into the neural network model after the training is completed in a posture prediction process, and obtain the target time corresponding to the current time after the training of the completed neural network model output Attitude data;
  • the predetermined time period is a time period determined by the current time and a predetermined time before the current time.
  • an electronic device includes a memory and a processor.
  • the memory and the processor are communicably connected by an internal bus.
  • the memory stores program instructions executable by the processor, and the program instructions are processed by the processor.
  • the above-described attitude prediction method can be implemented at the time of execution.
  • the attitude prediction method and apparatus obtains a raw data set by sampling posture data generated by a posture sensor of a user during exercise, and training a neural network model by using the original data set, and in the attitude prediction process, the past reservation is made.
  • the attitude data in the time period is input into the neural network model, accurate and accurate attitude data corresponding to a certain moment in the future (ie, the target time) can be predicted, and the accuracy of the predicted attitude data is high, so that the head mounted display device can
  • the scene image to be presented is calculated in advance according to the predicted posture data, and the virtual image viewed by the user is matched with the movement of the user's head or the hand, thereby facilitating the head-mounted display device to reduce the delay and improve the user experience.
  • the technical solution of the embodiment of the present invention utilizes a neural network model of machine learning to predict a user's motion posture, and the neural network algorithm is suitable for the time series of the posture data based on the user's motion pattern during use of the head-mounted display device.
  • 1 is a comparison diagram of prediction results using a prior art polynomial fitting method
  • FIG. 2 is a flow chart of an attitude prediction method according to an embodiment of the present invention.
  • FIG. 3 is a comparison diagram of prediction results of an attitude prediction method to which an embodiment of the present invention is applied;
  • FIG. 4 is a block diagram of an attitude prediction apparatus according to an embodiment of the present invention.
  • FIG. 5 is a block diagram of an electronic device in accordance with one embodiment of the present invention.
  • HMD Head Mounted Display
  • IMU/Optic attitude sensor
  • the current attitude prediction methods mainly include:
  • the algorithm substitutes the current (ie, time axis coordinate 0) attitude data (position, orientation, velocity, acceleration, etc.) into the Newton kinematics formula to derive the attitude at the future time point 0+ ⁇ T.
  • the algorithm assumes that the attitude data in the [0, ⁇ T] time interval is the same as the attitude data at the zero point.
  • the speed and angular velocity of the attitude sensor suddenly change or even reversing in the [0, ⁇ T] interval, the attitude predicted by the algorithm at 0+ ⁇ T will have a serious deviation.
  • An important indicator of the attitude prediction algorithm is the error size and error distribution. If the error is too large and the user is aware, it will more seriously aggravate the user's discomfort. This situation is worse than not making the attitude prediction.
  • the sliding window algorithm can be used to slide forward from 0 o'clock and Combine multiple polynomials, and finally weight-average multiple curves by weight.
  • the prediction effect of the polynomial fitting method and its variants is improved compared with the Newtonian kinematics method.
  • the polynomial fitting method utilizes the trend of the attitude curve in history, so when the attitude changes slowly, it is higher than the Newton kinematics method. The accuracy, but the degree of improvement is limited.
  • the disadvantage of the polynomial fitting method is that the prediction error of the polynomial fitting method is very large when the attitude changes drastically, such as when the sensor rapidly rotates to a certain point and immediately rotates in the opposite direction, or where the speed direction changes.
  • Figure 1 is a comparison of prediction results using a prior art polynomial fitting method; see Figure 1, the horizontal axis represents time, the vertical axis represents the Euler angle roll value calculated from the IMU raw data, the unit (degree), and 101 represents the polynomial
  • the attitude prediction value obtained by the fitting method, 102 is the actual value of the attitude generated by the attitude sensor; as can be seen from Fig. 1, the attitude data predicted by the polynomial fitting method is very different from the actual attitude data.
  • the reason may be that the polynomial fitting method simply fits the historical data curve, so that the trend of the fitted curve rises or falls at a future time point in the case where the trend direction is constant.
  • the historical trend of the movement is also in line with the trend of the next few moments.
  • the trend of the fitted curve rising or falling at the future time point is the same as the historical trend direction, but it is opposite to the actual trend direction, resulting in a large prediction error.
  • the polynomial fitting method has a high computational complexity (especially the weighted average variant method). It can be said that the polynomial fitting is a complicated and time-consuming process.
  • the weighted average variant method according to the number N of slidings, Need to do a polynomial fitting N times, which is a very time consuming process.
  • the algorithm is very difficult to implement and is not conducive to large-scale implementation of applications.
  • the embodiment of the present invention proposes a new scheme for predicting the attitude of the attitude sensor of the head mounted display device at a certain moment in the future to improve the prediction accuracy and reduce the computational complexity.
  • the inventor of the embodiment of the present invention believes that since the controller of the head-mounted display device is moving following the head or the hand of the user, the movement of the head and the hand of the person is restricted by the muscles and joints, and the movement is The attitude curve in space will form some specific patterns. In the case where the attitude sensor sampling rate is sufficiently high, these specific modes can be captured by the attitude sensor and embodied in the attitude data returned by the attitude sensor.
  • an embodiment of the present invention proposes a novel attitude prediction method, which introduces machine learning and deep neural network, and collects a large number of sensors to train the neural network in the actual posture data in the VR/AR application, so that the neural network can recognize the neural network.
  • the pattern in the attitude change curve so as to make a high-precision prediction of the posture at a certain moment in the future.
  • an attitude prediction method includes:
  • Step S201 sampling the posture data generated by the posture sensor of the user during the motion according to the preset sampling frequency to obtain the original data set; where the original data set is, for example, the posture data within a certain period of time before the current time, for example, [- The corresponding time on the time axis in the interval ⁇ T,0] is from the attitude data arranged after going to the rear, 0 means the current time (at the 0 point of the time axis), and the minus sign "-" represents the direction, indicating the time before the current time. , ⁇ T represents the length of time, such as 6 seconds.
  • Step S202 determining the number of layers of the initial neural network model and the number of nodes of each layer according to the original data set and the specified prediction duration, and training the initial neural network model using the original data set to obtain the neural network model after the training is completed. ;
  • the initial neural network model based on the complexity of the original data set and the demand for the prediction duration (for example, if you want to predict the posture after 5 milliseconds, then the 5 milliseconds here is the prediction duration), and then use the original data.
  • the initial neural network model is trained to obtain a neural network model that is finally used for pose prediction.
  • the requirements for the prediction duration are different. For example, in one case, the posture after 5 milliseconds is predicted, and in the other case, the posture after 2 milliseconds is predicted.
  • the requirements for predicting duration are different.
  • the initial neural network model is different (mainly the number of layers of the neural network model and the number of nodes in each layer).
  • the original data set is used to train the neural network model, since the original data set represents a time interval determined in the past period of time (ie, taking a certain time as a starting point and ending at a certain time in the negative direction of the time axis).
  • the attitude data of the corresponding attitude data is such that not only the attitude data at a certain moment is utilized in the prediction, but also the deeper motion information contained in the original data set is utilized, and the accuracy of the attitude prediction is improved.
  • Step S203 in the attitude prediction process, inputting the posture data in the predetermined time period to the neural network model after the training is completed, and obtaining the posture corresponding to the target time after the current time output of the neural network model after the training is completed. data;
  • the predetermined time period is a time period determined by the current time and a predetermined time before the current time.
  • the attitude prediction method in the embodiment of the present invention uses machine learning to train a neural network model, thereby transferring a highly complex optimization process to offline model training, and ensuring low computational complexity in actual prediction. Meet the efficiency requirements and achieve simplicity.
  • some movement modes may occur frequently during the use of the head-mounted display device, such as the user's head acceleration rotation, rapid rotation to a certain position and then suddenly stop, etc., these movement modes are in the time series of the attitude data.
  • the neural network model can capture the tiny motion patterns in the historical pose data, so as to train based on the tiny motion patterns in the captured historical pose data, even if the user's posture changes drastically, the neural network model still Compared with the prediction accuracy of the polynomial fitting method, the polynomial fitting method simply fits the historical motion curve, so that when the direction of the motion trend changes drastically, the curve is fitted in the future. Although the trend of rising or falling time points is the same as the historical trend direction, it is opposite to the actual trend direction, and the prediction error is large.
  • the neural network model of machine learning is used to accurately predict the posture, which facilitates the head-mounted display device to calculate the image of the head-mounted display device in advance according to the predicted higher-precision posture data, so that the displayed picture and the user's motion are compared. Matching, which reduces latency and enhances the user's immersive experience.
  • the attitude prediction method includes three parts, namely: (I) acquiring attitude data, (II) training a neural network model, and (III) using a trained neural network model to implement attitude prediction, the following respectively Be explained.
  • the pose data is collected.
  • the attitude data generated by the posture sensor of the user during motion is sampled according to the preset sampling frequency, and the original data set is obtained.
  • the posture data generated by the posture sensor under normal use state is sampled, and the time series of the posture data is obtained, which is saved as the original data set; the time of the posture data.
  • the sequence indicates one or more of the following user head or hand movements: smooth rotation, accelerated rotation, deceleration rotation, predetermined time rotation to a preset position, stop, predetermined time rotation to a predetermined position, and back rotation.
  • the predetermined time here is stopped after rotating to the preset position, for example, the head is kept at the angle for 45 minutes after the head is turned from the initial position to the right 45 degrees.
  • the predetermined time is rotated to the predetermined position and then rotated back. For example, within 3 seconds, the head is turned from the initial position to the right 45 degrees and then the head is turned to the initial position within 2 seconds.
  • the sampling frequency of the attitude data is, for example, 1000 Hz, then the original data set is a time series obtained by sampling once every 1 ms.
  • the initial neural network model layer and the number of nodes in each layer are determined according to the original data set obtained by the sampling and the specified prediction duration, and the initial neural network is trained by using the original data set, which specifically includes: The set is collated, and a mapping relationship set indicating a mapping relationship between the posture data in the predetermined time period and the posture data in the target time is generated; the data in the mapping relationship is input into the initial neural network model, and the initial nerve is trained by supervised learning. Network model.
  • a deep neural network is first designed. It can be understood that the number of layers of the neural network and the number of nodes in each layer are determined by the actual situation of the complexity of the data set, the length of the historical time interval, and the length of the prediction time. This embodiment does not limit this.
  • the neural network performs supervised learning, so that after the training is finished, a neural network model for attitude prediction can be obtained.
  • a neural network model for attitude prediction can be obtained.
  • an arbitrary array of poses of length ⁇ T_1 is input to the neural network model.
  • the time of the last element of the array is Z, then the neural network model can output the predicted posture at Z+ ⁇ T_2.
  • the function can be transplanted to various platforms for implementation. Specifically, after the neural network model training is completed, the specified parameters of the neural network model after training are extracted and outputted (the number of layers of the neural network model, the number of nodes in each layer, and activation). The value of the function, loss function, etc.) to run the neural network model on the specified terminal platform.
  • the specified terminal platform here includes, but is not limited to, a PC platform, a mobile platform, a single chip microcomputer, etc., and it can be seen that the method of the embodiment has low requirements on the platform and is convenient for large-scale promotion and application.
  • the specified parameters of the neural network model are adjusted according to the training result; the specified parameters include: the number of layers of the neural network model, the number of nodes in each layer, and activation. Function and loss function. Extract the parameters of the trained model and implement the corresponding prediction function on the required platform.
  • the reason why the specified parameters of the neural network model are adjusted according to the training results is mainly to determine whether the prediction result meets the application error requirement according to the test result, and whether the neural network model has produced over-fitting or under-fitting, thereby improving the prediction accuracy. , reduce the error.
  • the accuracy of the training set and the test set should increase. (Despite the existence of the fitting error, the accuracy of the test set has no training set. The accuracy is so high).
  • the neural network can be well implemented when fitting the training set. The loss function is small and the accuracy is very large.
  • the loss function is large and the accuracy fluctuates within a relatively low range. That is, the training set has a good prediction effect, and the test set has a poor prediction effect.
  • the data of the nodes in the middle layer or the middle layer of the neural network model is reduced, or dropout is added, and training is performed again. And test, repeated adjustments until the error meets the requirements to end the training.
  • the dropout here refers to the temporary discarding of the neural network unit from the network for a certain probability in the training process of the deep learning neural network.
  • the loss function of the neural network model is set to the mean square error of the collected actual pose data value and the predicted pose data value according to the training result. That is, the loss function is the mean square error between the actual pose value and the predicted pose value, thus ensuring that the difference between the predicted value and the actual value is the smallest, and thus the error is small.
  • the loss function here is also called the cost function, which is the objective function of neural network optimization.
  • the process of neural network training or optimization is the process of minimizing the loss function. The smaller the loss function value, the corresponding prediction result. The closer the value to the actual result.
  • the prediction process is: inputting the attitude data in the acquired predetermined time period into the neural network model, and obtaining the attitude data corresponding to the target time (for example, 0+ ⁇ T_2) after the current time output by the neural network model.
  • the predetermined time period is such as [- ⁇ T_1, 0] (the negative sign here indicates the direction, which is the time period determined by the current time (such as 0 point) and the predetermined time (23 points) before the current time.
  • the time series stored in the time interval is the time series of the above-mentioned posture data, which are arranged in chronological order on the time axis (arranged in time order from the time axis according to the time axis), and each piece of data represents the posture value obtained by one sampling, before and after
  • the time interval between the pieces of data is the sampling period (the reciprocal of the sampling frequency).
  • the attitude prediction method reads and saves the original data returned by the attitude controller such as the HMD (including the data collected by the gyroscope, the accelerometer, etc.) through the data collection module in the controller. ). Then, the collected data is sorted to generate a series of mapping relationship data such as [- ⁇ T_1, 0] ⁇ ⁇ T_2. Next, the generated mapping relationship data is substituted into the neural network model for training. The structure of the neural network is adjusted according to the training result, and the accuracy of the prediction is optimized until the prediction accuracy satisfies the requirements.
  • the main adjustment parameters include the number of layers of the neural network, Batch Size (the total number of samples in a Batch), Epoch (complete data set), the size of the training set and the size of the test set.
  • the neural network model after the completion of training is applied to achieve accurate prediction.
  • the attitude prediction method of the present embodiment achieves the following beneficial effects: based on the machine learning neural network prediction method, the computational complexity at the time of prediction is low, which is between the Newtonian kinematics method and the polynomial fitting method.
  • the neural network model trained by machine learning is represented by a series of vector matrix multiplications and additions. The specific complexity depends mainly on the size of the matrix and the vector and the number of layers.
  • the complexity of the neural network during training is relatively high, in this embodiment, the function fitting with the largest amount of computation is performed at the time of offline training, so that the computational complexity in real-time prediction is low, and to some extent, it can be considered as Transfer the more complex optimization process to offline training.
  • the prediction method based on the neural network model of the present embodiment has a high prediction accuracy when the attitude direction changes drastically.
  • the reason is that the neural network can capture the tiny motion patterns in the historical posture during training, such as the smooth rotation of the head, the acceleration rotation, the deceleration rotation, the rapid rotation to a certain position and then suddenly stop, quickly rotate to a certain position and then turn back. Etc. These motion patterns, which may occur frequently during the use of the device, have their own characteristics in the time series of the attitude data.
  • the neural network can perform more accurate and unified data prediction based on these features, thus ensuring the implementation.
  • the prediction accuracy of the example is a high prediction accuracy when the attitude direction changes drastically.
  • FIG. 3 is a comparison diagram of prediction results of an attitude prediction method according to an embodiment of the present invention.
  • the horizontal axis represents time
  • the vertical axis represents Euler angle roll values calculated according to IMU raw data
  • the unit is degrees
  • 302 represents The actual attitude value of the attitude sensor 301 indicates the predicted value of the attitude prediction method of the present embodiment.
  • the predicted value is very close to the actual value of the attitude, which also indicates that the prediction accuracy of the attitude prediction method of the present embodiment is high and satisfies
  • the actual application requirements are beneficial to the head-mounted display device to reduce the delay and improve the user experience.
  • FIG. 4 is a block diagram of an attitude prediction apparatus according to an embodiment of the present invention.
  • the attitude prediction apparatus 400 is applied to a head mounted display device, including:
  • the sampling unit 401 is configured to sample the posture data generated by the posture sensor when the user moves according to the preset sampling frequency, to obtain the original data set;
  • the training unit 402 is configured to determine the number of layers of the initial neural network model and the number of nodes of each layer according to the original data set and the specified prediction duration, and use the original data set to train the initial neural network model to obtain the training completion.
  • Neural network model
  • the prediction unit 403 is configured to input the posture data in the predetermined time period of the acquisition into the neural network model after the training is completed in the one-time attitude prediction process, and obtain the target time corresponding to the current time after the training of the completed neural network model output.
  • the predetermined time period is a time period determined by the current time and a predetermined time before the current time.
  • the training unit 402 is configured to: organize the original data set, and generate a mapping relationship set indicating a mapping relationship between the posture data in the predetermined time period and the posture data in the target time;
  • the data in the mapping relationship set is input into the initial neural network model, and the initial neural network model is trained using supervised learning.
  • the attitude prediction apparatus 400 further includes:
  • the training optimization unit is used to adjust the specified parameters of the neural network model according to the training result in the process of using the supervised learning to train the neural network model;
  • the specified parameters include: the number of layers of the neural network model, the number of nodes of each layer, the activation function, and the loss. function.
  • the training optimization unit is specifically configured to set the loss function of the neural network model to the mean square error of the collected actual pose data value and the predicted pose data value according to the training result.
  • the attitude prediction apparatus 400 further includes: a platform extension unit configured to extract and output a value of a specified parameter of the trained neural network model to run the trained nerve on the designated terminal platform Network model.
  • the specified parameters of the neural network model after the training is completed include: the number of layers of the neural network model, the number of nodes of each layer, the activation function, and the loss function.
  • the sampling unit is specifically configured to: according to a preset sampling frequency, sample the posture data generated by the posture sensor in a normal use state when the user moves, and obtain a time series of the posture data as the original data set.
  • the time series of the attitude data indicates one or more of the following user's head or hand movements: smooth rotation, acceleration rotation, deceleration rotation, predetermined time rotation to the preset position, stop, predetermined time rotation to the predetermined position Turn back again.
  • the posture prediction apparatus of the present embodiment is corresponding to the foregoing posture prediction method. Therefore, for the content that is not described in the embodiment of the present invention, reference may be made to the description in the foregoing method embodiments, and details are not described herein again.
  • the electronic device includes a memory 501 and a processor 502.
  • the memory 501 and the processor 502 are communicably connected by an internal bus 503.
  • the memory 501 stores Program instructions executable by the processor 502, which when executed by the processor 502, are capable of implementing the steps of the pose prediction method of the previous embodiments.
  • embodiments of the present invention can be provided as a method, system, or computer program product.
  • the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware.
  • the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

Abstract

一种姿态预测方法、装置和电子设备，方法包括：按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样(S201)，得到原始数据集；根据原始数据集确定神经网络的层数以及各层节点数，得到训练完成的神经网络模型(S202)；在一次姿态预测过程中，将采集的预定时间段内的姿态数据输入到神经网络模型中，得到神经网络模型输出的当前时刻之后目标时刻对应的姿态数据(S203)；其中，预定时间段是由当前时刻以及当前时刻之前的预定时刻确定的时间段。可见，本技术方案利用机器学习训练出的神经网络模型进行姿态预测不仅满足了效率要求，降低了计算复杂度。而且，能够捕捉历史姿态中的微小的运动模式，预测的精度高，提升了用户的沉浸感等体验。

Description

一种姿态预测方法、装置和电子设备 技术领域
本发明涉及头戴显示设备技术领域，具体涉及一种姿态预测方法、装置和电子设备。
发明背景
VR(Virtual Reality，虚拟现实)和AR(Augmented Reality，增强现实)技术通过控制器让用户获得可交互的沉浸式体验。控制器可位于HMD(Head Mounted Display，头戴显示设备)中，也可位于手柄中。控制器的延迟过高会破坏沉浸感，尤其对于HMD这种头-眼运动反馈的设备来说，过高的延迟会引发用户严重的不适感和晕动症，影响用户的生理健康。一个完善的VR/AR系统通常需要结合多种优化手段以降低控制器的延迟。VR/AR的控制器中通常集成有IMU(Inertial Measurement Unit，惯性测量单元简称)等姿态传感器，并利用姿态传感器获取用户的姿态数据，根据姿态数据对用户的运动姿态进行预测来降低用户所察觉到的系统延迟。
但是现有基于姿态传感器进行姿态预测的方案要么误差较大，精度达不到要求，要么计算复杂度高，实现起来非常困难，不能满足实际需求，用户体验不佳。
发明内容
为了解决现有虚拟现实或增强现实技术对用户姿态进行预测时误差较大，计算复杂度高，用户体验不佳的技术问题，本发明实施例提供了一种姿态预测方法、装置和电子设备。
根据本发明的一个方面，提供了一种姿态预测方法，包括：
按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样，得到原始数据集；
根据原始数据集和指定预测时长确定初始神经网络模型的层数以及各层的节点数，并使用所述原始数据集对初始神经网络模型进行训练，以得到训练完成后的神经网络模型；
在一次姿态预测过程中，将采集的预定时间段内的姿态数据输入到训练完成后的神经网络模型中，得到训练完成后的神经网络模型输出的当前时刻之后 目标时刻对应的姿态数据；
其中，预定时间段是由当前时刻以及当前时刻之前的预定时刻确定的时间段。
根据本发明的另一个方面，提供了一种姿态预测装置，应用于头戴显示设备，包括：
采样单元，用于按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样，得到原始数据集；
训练单元，用于根据原始数据集和指定预测时长确定初始神经网络模型的层数以及各层的节点数，并使用所述原始数据集对初始神经网络模型进行训练，以得到训练完成后的神经网络模型；
预测单元，用于在一次姿态预测过程中，将采集的预定时间段内的姿态数据输入到训练完成后的神经网络模型中，得到训练完成后的神经网络模型输出的当前时刻之后目标时刻对应的姿态数据；
其中，预定时间段是由当前时刻以及当前时刻之前的预定时刻确定的时间段。
根据本发明的再一个方面，提供了一种电子设备，包括存储器和处理器，存储器和处理器之间通过内部总线通讯连接，存储器存储有能够被处理器执行的程序指令，程序指令被处理器执行时能够实现上述姿态预测方法。
有益效果：本发明实施例的姿态预测方法和装置，通过对用户运动时姿态传感器生成的姿态数据进行采样得到原始数据集，利用原始数据集训练神经网络模型，在姿态预测过程中，将过去预定时间段内的姿态数据输入至该神经网络模型中，即可预测出未来某一时刻(即目标时刻)对应的、精确的姿态数据，预测的姿态数据的精确度高，使得头戴显示设备能够根据预测出的姿态数据提前计算将要呈现的场景画面，保证用户观看到的虚拟图像与用户头部或手部的运动相匹配，从而方便头戴显示设备降低延迟，提高用户体验。更重要的是，本发明实施例的技术方案利用机器学习的神经网络模型来预测用户运动姿态，神经网络算法，适合基于头戴显示设备使用过程中用户的运动模式在姿态数据的时间序列上的特征，对未来的姿态做出高精度的预测，并且通过机器学习训练出的神经网络模型，在计算上表现为一系列的向量矩阵乘法与加法，具体的复杂度主要取决于矩阵与向量的大小以及层数，而运算量最大的函数拟合可放 在离线训练时进行，这保证了姿态预测时的计算复杂度较低，从而兼顾了运动姿态预测的计算复杂度和预测精度两项指标，解决了现有姿态预测算法要么误差较大，达不到精度要求，要么计算复杂度高实现困难的技术问题，提高了头戴显示设备的竞争力。
附图简要说明
图1是应用现有技术多项式拟合法的预测结果对比图；
图2本发明一个实施例的姿态预测方法的流程图；
图3是应用本发明一个实施例的姿态预测方法的预测结果对比图；
图4是本发明一个实施例的姿态预测装置的框图；
图5是本发明一个实施例的电子设备的框图。
具体实施方式
业界普遍认为HMD(Head Mounted Display，头戴显示设备)的延迟应当低于20ms，甚至低于15ms或10ms才能保证流畅的体验。但这些指标对现有的VR/AR软硬件系统是一个很高的挑战，在VR/AR软件层面，一种重要的优化手段是对控制器上所集成的姿态传感器(IMU/Optic)未来一段时间的姿态进行预测，以此提前进行响应和准备来降低用户所能察觉到的系统延迟。
目前的姿态预测方法主要有：
(一)牛顿运动学法。该算法将当前(即时间轴坐标为0处)的姿态数据(位置、朝向、速度、加速度等)代入到牛顿运动学公式中，以推算出未来时间点0+ΔT处的姿态。该算法假定了[0，ΔT]时间区间内的姿态数据与0点处的姿态数据相同。实际中，若姿态传感器的速度和角速度在[0，ΔT]区间内突然发生变化甚至发生方向折返时，该算法预测出的0+ΔT处的姿态将会有严重的偏差。而姿态预测算法的一个重要指标是误差大小以及误差分布。若误差过大引起用户的觉察，将会更加严重的加剧用户的不适感，这种情况比不做姿态预测还要糟糕。
由上可知，牛顿运动学法的主要缺点是计算精度非常差，在预测时只利用了当前时间点0处的姿态数据，忽略了历史数据对未来数据的影响，使得预测的精度很差。而实际上在0点之前一段时间，即[-ΔT，0]内的运动状态对0点之后的运动状态有很大的影响。
(二)多项式拟合法。接前述，[-ΔT，0]区间内的数据包含了更深层次的 运动信息，比如，加速度的导数等，这些额外的信息可用来提高预测的准确性。因此一些方案提出了采用多项式拟合法，将[-ΔT，0]区间内的姿态变动曲线拟合出高阶的多项式，随后利用得到的多项式推算0+ΔT处的姿态值。多项式拟合法还有一些变种，比如，认为0点之前的历史数据中，越靠近0点的数据对0点之后的运动状态影响更大，因此可以采用滑动窗算法从0点往前滑动并拟合出多个多项式，最后再按权重对多条曲线进行加权平均。
多项式拟合法及其变种的预测效果比牛顿运动学法有所提升，多项式拟合法利用了姿态曲线在历史上的变化趋势，因此在姿态变化较缓慢的时候，相比牛顿运动学法有较高的准确度，但提高的程度有限。多项式拟合法缺点主要是在姿态发生剧烈变化时，比如传感器快速的转动到某一点并马上向反方向转动时，或者在速度方向发生变化的地方，多项式拟合法的预测误差非常大。
图1是应用现有技术多项式拟合法的预测结果对比图；参见图1，横轴表示时间，纵轴表示根据IMU原始数据计算出的欧拉角roll值，单位(度)，101表示经多项式拟合法获取的姿态预测值，102是姿态传感器生成的姿态实际值；由图1可知，多项式拟合法预测的姿态数据与实际的姿态数据误差非常大。
本申请的发明人分析认为原因可能是，多项式拟合法只是简单的对历史数据曲线进行拟合，这样在趋势方向不变的情况下，拟合出的曲线在未来时间点上升或下降的趋势大致符合历史的运动趋势，也符合未来一小段时间的变化趋势。但在方向剧烈变化时，拟合出的曲线在未来时间点上升或下降的趋势虽与历史趋势方向相同，但却与实际的趋势方向相反，从而造成预测误差较大。
同时，多项式拟合法计算复杂度很高(尤其是加权平均的变种方法)，可以说，多项式拟合是一个复杂且耗时的过程，对于加权平均的变种方法来说，根据滑动的次数N，需要做N次多项式拟合，这是一个非常耗时的过程。在移动平台或单片机等计算能力较弱的平台上，该算法实现起来非常困难，不利于大规模实施应用。
对此，本发明实施例提出一种新的对头戴显示设备的姿态传感器未来某一时刻的姿态进行预测的方案，以提升预测精度并降低计算复杂度。本发明实施例的发明人认为，由于头戴显示设备的控制器是跟随用户的头部或手部进行运动的，而人的头部和手部的运动受肌肉、关节的限制，运动时在空间中的姿态曲线会形成一些特定的模式。在姿态传感器采样率足够高的情况下，这些特定的模式能够被姿态传感器所捕捉到，并体现在姿态传感器所返回的姿态数据中。
基于此，本发明实施例提出一种全新的姿态预测方法，引入机器学习以及深度神经网络，并采集大量传感器在VR/AR应用中的实际姿态数据对神经网络进行训练，使该神经网络能够识别姿态变化曲线中的模式，从而对未来某个时刻的姿态做出高精度的预测。
为使本发明的目的、技术方案和优点更加清楚，下面将结合附图对本发明实施方式作进一步地详细描述。
图2是本发明一个实施例的姿态预测方法的流程图，参见图2，本发明实施例的姿态预测方法，包括：
步骤S201，按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样，得到原始数据集；这里的原始数据集例如是当前时刻之前某个时间段内的姿态数据，比如，[-ΔT，0]区间内对应的按时间轴上的时间从前往后排列的姿态数据，0表示当前时刻(时间坐标轴的0点处)，负号“-”代表方向，表示当前时刻之前的时刻，ΔT表示时间长度，比如6秒。
步骤S202，根据原始数据集和指定预测时长确定初始神经网络模型的层数以及各层的节点数，并使用所述原始数据集对初始神经网络模型进行训练，以得到训练完成后的神经网络模型；
这里，先根据原始数据集的复杂程度，以及对预测时长的需求(比如，想要预测5毫秒后的姿态，那么这里的5毫秒即为预测时长)确定一个初始神经网络模型，随后使用原始数据集对该初始神经网络模型进行训练，得到最终用于进行姿态预测的神经网络模型。
需要说明的是，实际应用中，预测时长的需求不同，比如，一种情况下，要预测5毫秒后的姿态，另一种情况下，要预测2毫秒后的姿态。预测时长的需求不同，在原始数据集相同的情况下，构建的初始神经网络模型也不同(主要是神经网络模型的层数以及各层节点数不同)。
本实施例中，利用原始数据集来训练神经网络模型，由于原始数据集代表了过去一段时间内(即，以某时刻为起点，以时间轴的负方向上某一时刻为终点确定的时间区间对应的姿态数据)的姿态数据，这样在预测时不仅利用了某时刻的姿态数据，而且利用了原始数据集包含的更深层次的运动信息，提高了姿态预测的准确度。
步骤S203，在一次姿态预测过程中，将采集的预定时间段内的姿态数据输入到训练完成后的神经网络模型中，得到该训练完成后的神经网络模型输出的 当前时刻之后目标时刻对应的姿态数据；
其中，预定时间段是由当前时刻以及当前时刻之前的预定时刻确定的时间段。
由图2所示可知，本发明实施例的姿态预测方法，利用机器学习训练出神经网络模型，从而将复杂度较大的优化过程转移到了离线模型训练中，保证实际预测时计算复杂度低，满足效率要求，实现简单。另外，在使用头戴显示设备的过程中可能会频繁出现一些运动模式，如用户头部加速转动、快速转动到某位置然后突然停止等等模式，这些运动模式在姿态数据的时间序列上都有各自的特征，而神经网络模型能够捕捉历史姿态数据中的微小的运动模式，从而基于捕捉到的历史姿态数据中的微小的运动模式进行训练，即使用户的姿态方向发生剧烈变化，神经网络模型仍然能够保持较高的预测精度，与多项式拟合法的预测精度相比，多项式拟合法只是简单的对历史运动变化曲线进行拟合，这样在运动趋势方向发生剧烈变化时，拟合出的曲线在未来时间点上升或下降的趋势虽与历史趋势方向相同，但却与实际的趋势方向相反，从而预测误差较大。如此，利用机器学习的神经网络模型来精确预测姿态，方便了头戴显示设备根据预测出的精确度较高的姿态数据来提前计算头戴显示设备的画面，使显示的画面与用户的运动相匹配，从而降低延迟，提升用户的沉浸感体验。
在本发明的一个实施例中，姿态预测方法包括三部分，分别为：(I)采集姿态数据，(II)训练神经网络模型，(III)利用训练出的神经网络模型实现姿态预测，以下分别进行说明。
首先，采集姿态数据。
本实施例中，按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样，得到原始数据集。具体包括：按照预设采样频率，对用户运动时姿态传感器在正常使用状态下生成的姿态数据进行采样，得到姿态数据的时间序列(Time Series)，保存下来，作为原始数据集；姿态数据的时间序列指示下列用户头部或手部运动中的一项或多项：平滑转动、加速转动、减速转动、预定时间转动到预设位置后停止、预定时间转动到预定位置再向回转动。这里的预定时间转动到预设位置后停止，比如是3秒内头部从初始位置转到右边45度后保持在该角度一定时间。预定时间转动到预定位置再向回转动，比如是，3秒内头部从初始位置转到右边45度后又在2秒内将头部转到初始位置。其中，姿态数据的采样频率例如是1000Hz，那么原始数据集即为每隔1ms进行一次采样 得到的时间序列。
其次，训练神经网络。
本实施例中，根据前述采样得到的原始数据集和指定预测时长确定初始神经网络模型的层数以及各层的节点数，并使用原始数据集对初始神经网络进行训练，具体包括：对原始数据集进行整理，生成指示预定时间段内的姿态数据以及目标时刻的姿态数据的映射关系的映射关系集；将映射关系集中的数据输入到初始神经网络模型中，并采用监督式学习训练该初始神经网络模型。
实际应用中，先设计一个深度神经网络。可以理解，神经网络的层数以及每一层的节点数量由数据集的复杂程度、历史时间区间的长短、预测时间的长短等实际情况而定，本实施例对此不作限制。
接着利用搜集到的数据集，让神经网络进行监督式学习，这样训练结束后，可以得到一个用于姿态预测的神经网络模型。在实际预测时对该神经网络模型输入任意的长度为ΔT_1的姿态数组，比如该数组最后一个元素所在时间为Z，那么，该神经网络模型即可输出Z+ΔT_2处所预测的姿态。
本实施例中，经过以上步骤，训练得到下列函数：
Z+T_2＝f([Z-ΔT_1,Z])
该函数可移植到各个平台上进行实现，具体的，在神经网络模型训练完成后，提取并输出训练完成后的神经网络模型的指定参数(神经网络模型的层数、各层的节点数、激活函数、损失函数等)的值，从而在指定终端平台上运行神经网络模型。这里的指定终端平台包括但不限于PC平台、移动平台、单片机等，可知，本实施例的方法对平台要求低，方便大规模推广应用。
需要强调的是，本实施例在采用监督式学习训练神经网络模型过程中，会根据训练结果调整神经网络模型的指定参数；指定参数包括：神经网络模型的层数、各层的节点数、激活函数以及损失函数。提取训练出的模型的参数，在所需的平台上实现相应的预测函数。这里之所以根据训练结果调整神经网络模型的指定参数，主要是根据测试的结果确定预测结果是否达到了应用的误差要求，以及神经网络模型是否产生了过拟合或欠拟合，从而提高预测精度，减小误差。比如，理论上，在采用监督学习训练神经网络时，随着迭代的增加，训练集和测试集的精确度均应该上升(尽管有拟合的误差的存在，测试集的精确度没有训练集的精确度那么高)。但实际中，神经网络拟合训练集时可以很好地实现，损失函数很小，精确度很大；但是，在拟合测试集时损失函数很大，精 确度在一个比较低的范围内波动，即训练集预测效果好，测试集预测效果差，这样当初始的训练和测试结果表明发生了过拟合时，减少神经网络模型中间层或中间层上节点的数据，或加入dropout，再次训练并测试，反复调整直到误差满足要求即可结束训练。这里的dropout是指在深度学习神经网络的训练过程中，对于神经网络单元，按照一定的概率将其暂时从网络中丢弃。
需要强调的是，本实施例中，根据训练结果将神经网络模型的损失函数设置为采集的实际姿态数据值与预测姿态数据值的均方差。即，损失函数为实际姿态值与预测姿态值的均方差，这样确保了预测值与实际值间的差值最小，从而误差较小。这里的损失函数(loss function)也叫代价函数(cost function)，是神经网络优化的目标函数，神经网络训练或者优化的过程就是最小化损失函数的过程，损失函数值越小，对应预测的结果和真实结果的值就越接近。
最后，利用训练出的模型实现姿态预测。
预测过程是：将采集的预定时间段内的姿态数据输入到神经网络模型中，得到神经网络模型输出的当前时刻之后目标时刻(比如0+ΔT_2)对应的姿态数据。预定时间段比如[-ΔT_1,0](这里的负号表示方向，是由当前时刻(比如0点)以及当前时刻之前的预定时刻(23点)确定的时间段。这里[-ΔT_1,0]时间区间内存放的即为前述姿态数据的时间序列，是按时间轴上时间先后顺序排列(按时间轴从前往后的时间顺序排列)，每一条数据代表一次采样所得到的姿态数值，前后两条数据之间的时间间隔即为采样周期(采样频率的倒数)。
至此，在本发明的一个实施例中，该姿态预测方法，通过控制器中的数据收集模块，读取并保存HMD等姿态控制器所返回的原始数据(包括陀螺仪、加速度计等采集的数据)。然后，对收集到的数据进行整理，生成一系列形如[-ΔT_1,0]→ΔT_2的映射关系数据。接着，将上述生成的映射关系数据代入到神经网络模型中进行训练。根据训练结果调整神经网络的结构，优化预测的准确性，直到预测精度满足要求。训练过程中，主要调整的参数包括神经网络的层数、Batch Size(一个Batch中的样本总数)、Epoch(完整的数据集)、训练集与测试集的大小等重要参数。最后，应用训练完成后的神经网络模型实现准确预测。
需要说明的是，在神经网络中，当一个完整的数据集通过了神经网络一次并且返回了一次，这个过程称为一个epoch。而在不能将数据一次性通过神经网络时，就需要将数据集分成几个batch。
本实施例的姿态预测方法实现了以下有益效果：基于机器学习神经网络的预测方法，预测时的计算复杂度较低，介于牛顿运动学法与多项式拟合法之间。机器学习训练出的神经网络模型，在计算上表现为一系列的向量矩阵乘法与加法，具体的复杂度主要取决于矩阵与向量的大小以及层数。虽然神经网络在训练时的复杂度较高，但本实施例中将运算量最大的函数拟合放在离线训练时进行，这样实时预测时的计算复杂度较低，某种程度上可认为是将复杂度较大的优化过程转移到了离线训练中。
此外，计算精度较多项式拟合法更好。本实施例的基于神经网络模型的预测方法在姿态方向发生剧烈变化时，预测的精度仍然较高。原因是神经网络在训练时能够捕捉历史姿态中的微小的运动模式，如头部的平滑转动、加速转动、减速转动、快速转动到某位置然后突然停止、快速转动到某个位置再向回转动等等，这些在使用设备的过程中可能频繁出现的运动模式，在姿态数据的时间序列上都有各自的特征，神经网络能够根据这些特征进行更准确、统一的数据预测，从而保证了本实施例的预测精度。
图3是应用本发明一个实施例的姿态预测方法的预测结果对比图，在图3中横轴表示时间，纵轴表示根据IMU原始数据计算出的欧拉角roll值，单位为度，302表示姿态传感器实际的姿态值，301表示本实施例姿态预测方法的预测值，由图3可知，预测值与姿态实际值非常接近，这也说明了本实施例的姿态预测方法预测精度较高，满足了实际应用需求，有利于头戴显示设备降低延迟，提升了用户体验。
图4是本发明一个实施例的姿态预测装置的框图，参见图4，姿态预测装置400应用于头戴显示设备，包括：
采样单元401，用于按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样，得到原始数据集；
训练单元402，用于根据原始数据集和指定预测时长确定初始神经网络模型的层数以及各层的节点数，并使用所述原始数据集对初始神经网络模型进行训练，以得到训练完成后的神经网络模型；
预测单元403，用于在一次姿态预测过程中，将采集的预定时间段内的姿态数据输入到训练完成后的神经网络模型中，得到训练完成后的神经网络模型输出的当前时刻之后目标时刻对应的姿态数据；
其中，预定时间段是由当前时刻以及当前时刻之前的预定时刻确定的时间 段。
在本发明的一个实施例中，训练单元402，具体用于对原始数据集进行整理，生成指示预定时间段内的姿态数据以及目标时刻的姿态数据的映射关系的映射关系集；
将映射关系集中的数据输入到初始神经网络模型中，并采用监督式学习训练该初始神经网络模型。
在本发明的一个实施例中，姿态预测装置400还包括：
训练优化单元，用于在采用监督式学习训练神经网络模型过程中，根据训练结果调整神经网络模型的指定参数；指定参数包括：神经网络模型的层数、各层的节点数、激活函数以及损失函数。
在本发明的一个实施例中，训练优化单元具体用于，根据训练结果将神经网络模型的损失函数设置为采集的实际姿态数据值与预测姿态数据值的均方差。
在本发明的一个实施例中，姿态预测装置400还包括：平台扩展单元，用于提取并输出训练完成后的神经网络模型的指定参数的值，以在指定终端平台上运行训练完成后的神经网络模型。训练完成后的神经网络模型的指定参数包括：神经网络模型的层数、各层的节点数、激活函数以及损失函数。
在本发明的一个实施例中，采样单元具体用于，按照预设采样频率，对用户运动时姿态传感器在正常使用状态下生成的姿态数据进行采样，得到姿态数据的时间序列，作为原始数据集；其中，姿态数据的时间序列指示下列用户头部或手部运动中的一项或多项：平滑转动、加速转动、减速转动、预定时间转动到预设位置后停止、预定时间转动到预定位置再向回转动。
需要说明的是，本实施例的姿态预测装置是和前述姿态预测方法相对应的，因而本实施例中对姿态预测装置没有描述的内容可参见前述方法实施例中的说明，这里不再赘述。
图5是本发明一个实施例的电子设备的框图，电子设备包括：存储器501和处理器502，所述存储器501和所述处理器502之间通过内部总线503通讯连接，所述存储器501存储有能够被所述处理器502执行的程序指令，所述程序指令被所述处理器502执行时能够实现前述实施例中姿态预测方法的步骤。
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结 合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
以上所述，仅为本发明的具体实施方式，在本发明的上述教导下，本领域技术人员可以在上述实施例的基础上进行其他的改进或变形。本领域技术人员应该明白，上述的具体描述只是更好的解释本发明的目的，本发明的保护范围应以权利要求的保护范围为准。

Claims (15)

  1. 一种姿态预测方法，其中，包括：
    按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样，得到原始数据集；
    根据所述原始数据集和指定预测时长确定初始神经网络模型的层数以及各层的节点数，并使用所述原始数据集对初始神经网络模型进行训练，以得到训练完成后的神经网络模型；
    在一次姿态预测过程中，将采集的预定时间段内的姿态数据输入到所述训练完成后的神经网络模型中，得到所述训练完成后的神经网络模型输出的当前时刻之后的目标时刻对应的姿态数据；
    其中，所述预定时间段是由当前时刻以及当前时刻之前的预定时刻确定的时间段。
  2. 根据权利要求1所述的方法，其中，根据所述原始数据集和指定预测时长确定初始神经网络模型的层数以及各层的节点数，并使用所述原始数据集对初始神经网络进行训练包括：
    对所述原始数据集进行整理，生成指示所述预定时间段内的姿态数据以及目标时刻的姿态数据的映射关系的映射关系集；
    将所述映射关系集中的数据输入到初始神经网络模型中，并采用监督式学习训练该初始神经网络模型。
  3. 根据权利要求2所述的方法，其中，还包括：
    在采用监督式学习训练神经网络模型过程中，根据训练结果调整神经网络模型的指定参数；
    所述指定参数包括：神经网络模型的层数、各层的节点数、激活函数以及损失函数。
  4. 根据权利要求3所述的方法，其中，所述根据训练结果调整神经网络模型的指定参数包括：
    根据训练结果将神经网络模型的损失函数设置为采集的实际姿态数据值与预测姿态数据值的均方差。
  5. 根据权利要求1所述的方法，其中，还包括：
    在神经网络模型训练完成后，提取并输出训练完成后的神经网络模型的指定参数的值，以在指定终端平台上运行所述神经网络模型。
  6. 根据权利要求5所述的方法，其中，所述训练完成后的神经网络模型的指定参数包括：神经网络模型的层数、各层的节点数、激活函数以及损失函数。
  7. 根据权利要求1所述的方法，其中，按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样，得到原始数据集包括：
    按照预设采样频率，对用户运动时姿态传感器在正常使用状态下生成的姿态数据进行采样，得到姿态数据的时间序列，作为原始数据集；
    所述姿态数据的时间序列指示下列用户头部或手部运动中的一项或多项：
    平滑转动、加速转动、减速转动、预定时间转动到预设位置后停止、预定时间转动到预定位置再向回转动。
  8. 一种姿态预测装置，其中，应用于头戴显示设备，包括：
    采样单元，用于按照预设采样频率，对用户运动时姿态传感器生成的姿态数据进行采样，得到原始数据集；
    训练单元，用于根据所述原始数据集和指定预测时长确定初始神经网络模型的层数以及各层的节点数，并使用所述原始数据集对初始神经网络模型进行训练，以得到训练完成后的神经网络模型；
    预测单元，用于在一次姿态预测过程中，将采集的预定时间段内的姿态数据输入到所述训练完成后的神经网络模型中，得到所述训练完成后的神经网络模型输出的当前时刻之后的目标时刻对应的姿态数据；
    其中，所述预定时间段是由当前时刻以及当前时刻之前的预定时刻确定的时间段。
  9. 根据权利要求8所述的装置，其中，
    所述训练单元，具体用于对所述原始数据集进行整理，生成指示所述预定时间段内的姿态数据以及目标时刻的姿态数据的映射关系的映射关系集；
    将所述映射关系集中的数据输入到初始神经网络模型中，并采用监督式学习训练该初始神经网络模型。
  10. 根据权利要求9所述的装置，其中，还包括：
    训练优化单元，用于在采用监督式学习训练神经网络模型过程中，根据训练结果调整神经网络模型的指定参数；
    所述指定参数包括：神经网络模型的层数、各层的节点数、激活函数以及损失函数。
  11. 根据权利要求10所述的装置，其中，所述训练优化单元具体用于，根据训练结果将神经网络模型的损失函数设置为采集的实际姿态数据值与预测姿 态数据值的均方差；
  12. 根据权利要求8所述的装置，其中，所述装置还包括：
    平台扩展单元，用于在神经网络模型训练完成后，提取并输出训练完成后的神经网络模型的指定参数的值，以在指定终端平台上运行所述训练完成后的神经网络模型。
  13. 根据权利要求12所述的装置，其中，所述训练完成后的神经网络模型的指定参数包括：神经网络模型的层数、各层的节点数、激活函数以及损失函数。
  14. 根据权利要求8所述的装置，其中，
    所述采样单元，具体用于按照预设采样频率，对用户运动时姿态传感器在正常使用状态下生成的姿态数据进行采样，得到姿态数据的时间序列，作为原始数据集；其中，所述姿态数据的时间序列指示下列用户头部或手部运动中的一项或多项：平滑转动、加速转动、减速转动、预定时间转动到预设位置后停止、预定时间转动到预定位置再向回转动。
  15. 一种电子设备，包括存储器和处理器，所述存储器和所述处理器之间通过内部总线通讯连接，所述存储器存储有能够被所述处理器执行的程序指令，所述程序指令被所述处理器执行时能够实现权利要求1-7中任一项所述的姿态预测方法。
PCT/CN2018/112854 2018-04-04 2018-10-31 一种姿态预测方法、装置和电子设备 WO2019192172A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810300681.1 2018-04-04
CN201810300681.1A CN108664122A (zh) 2018-04-04 2018-04-04 一种姿态预测方法和装置

Publications (1)

Publication Number Publication Date
WO2019192172A1 true WO2019192172A1 (zh) 2019-10-10

Family

ID=63782107

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/112854 WO2019192172A1 (zh) 2018-04-04 2018-10-31 一种姿态预测方法、装置和电子设备

Country Status (2)

Country Link
CN (1) CN108664122A (zh)
WO (1) WO2019192172A1 (zh)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111027692A (zh) * 2019-11-04 2020-04-17 中国电子科技集团公司第十四研究所 一种目标运动态势预测方法及装置
CN111311714A (zh) * 2020-03-31 2020-06-19 北京慧夜科技有限公司 一种三维动画的姿态预测方法和系统
CN111343340A (zh) * 2020-03-11 2020-06-26 杭州十域科技有限公司 一种基于大数据的低端硬件设备的姿态判别方法
CN112116656A (zh) * 2020-08-03 2020-12-22 歌尔股份有限公司 同步定位与地图构建slam中的增量建图方法和装置
CN112486318A (zh) * 2020-11-26 2021-03-12 北京字跳网络技术有限公司 图像显示方法、装置、可读介质及电子设备
CN113221681A (zh) * 2021-04-26 2021-08-06 平安科技（深圳）有限公司 模型训练方法、动作姿态生成方法、装置、设备及介质
CN113268859A (zh) * 2021-04-25 2021-08-17 北京控制工程研究所 航天器在轨博弈的仿真模拟方法、系统及存储介质
CN113569327A (zh) * 2021-08-10 2021-10-29 同济大学 一种盾构机姿态优化方法、系统及模型训练方法、系统
CN113806172A (zh) * 2021-09-14 2021-12-17 北京百度网讯科技有限公司 一种处理设备状态参数的方法、装置、电子设备及存储介质
CN113919236A (zh) * 2021-10-29 2022-01-11 中国国家铁路集团有限公司 预测轨道不平顺的模型生成方法和轨道不平顺的预测方法
CN114096010A (zh) * 2021-11-01 2022-02-25 江苏科技大学 一种面向监测的水下传感器网络能量持续管理方法
CN114187360A (zh) * 2021-12-14 2022-03-15 西安交通大学 基于深度学习和四元数的头部位姿估计方法
CN114296539A (zh) * 2020-10-07 2022-04-08 财团法人工业技术研究院 方向预测方法、虚拟实境装置及非暂态计算机可读取媒体
CN116321286A (zh) * 2023-03-13 2023-06-23 朱林清 应用智能算法的数据偏差鉴定系统
CN117560638A (zh) * 2024-01-10 2024-02-13 山东派蒙机电技术有限公司 应用于移动端通信系统的融合通信方法、装置及设备

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108664122A (zh) * 2018-04-04 2018-10-16 歌尔股份有限公司 一种姿态预测方法和装置
CN109794935B (zh) * 2019-01-09 2022-01-14 中国工程物理研究院激光聚变研究中心 基于神经网络分析的装配过程力与力矩预测系统及方法
KR102292277B1 (ko) * 2019-01-30 2021-08-20 바이두닷컴 타임즈 테크놀로지(베이징) 컴퍼니 리미티드 자율 주행 차량에서 3d cnn 네트워크를 사용하여 솔루션을 추론하는 lidar 위치 추정
CN110068326B (zh) * 2019-04-29 2021-11-30 京东方科技集团股份有限公司 姿态计算方法、装置、电子设备以及存储介质
CN110248178B (zh) * 2019-06-18 2021-11-23 深圳大学 利用物体跟踪和历史轨迹全景视频的视口预测方法及系统
CN112649815B (zh) * 2019-10-10 2023-04-11 华为技术有限公司 处理数据的方法和装置
WO2021190421A1 (zh) * 2020-03-27 2021-09-30 海信视像科技股份有限公司 基于虚拟现实的控制器光球追踪方法和虚拟现实设备
CN113616494B (zh) * 2020-05-08 2023-07-04 未来穿戴技术有限公司 按摩控制方法、装置、计算机设备和计算机可读存储介质
CN112100841B (zh) * 2020-09-09 2024-04-19 中铁二十局集团有限公司 盾构机姿态预测方法、装置、终端设备以及存储介质
US11892562B2 (en) * 2020-10-30 2024-02-06 KaiKuTek Inc. Impulse-like gesture recognition method, and impulse-like gesture recognition system
CN112947180B (zh) * 2021-02-04 2022-06-24 中国地质大学（武汉） 重型机械作业状态识别预测方法、装置、设备及存储介质
CN112818898B (zh) * 2021-02-20 2024-02-20 北京字跳网络技术有限公司 模型训练方法、装置和电子设备
CN113537489B (zh) * 2021-07-09 2024-03-19 厦门大学 一种手肘角度预测方法、终端设备及存储介质
CN113486214A (zh) * 2021-07-23 2021-10-08 广州酷狗计算机科技有限公司 音乐匹配方法、装置、计算机设备及存储介质
CN113792484B (zh) * 2021-09-07 2024-02-27 珠海格力电器股份有限公司 货物冲击姿态模型构建方法和平台
CN113836795A (zh) * 2021-09-07 2021-12-24 珠海格力电器股份有限公司 货物初始放置姿态模型构建方法和平台

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103984416A (zh) * 2014-06-10 2014-08-13 北京邮电大学 一种基于加速度传感器的手势识别方法
US20160132773A1 (en) * 2014-11-06 2016-05-12 International Business Machines Corporation Method for Automatic Near-Real-Time Prediction, Classification, and Notification of Events in Natural Language Systems
CN106682733A (zh) * 2016-11-07 2017-05-17 上海资誉电子科技有限公司 无人机运动状态分析方法和装置
CN108664122A (zh) * 2018-04-04 2018-10-16 歌尔股份有限公司 一种姿态预测方法和装置

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107403440B (zh) * 2016-05-18 2020-09-08 株式会社理光 用于确定对象的姿态的方法和装置
EP3475883A1 (en) * 2016-06-27 2019-05-01 Robin Young Dynamically managing artificial neural networks
CN106778579B (zh) * 2016-12-07 2020-08-11 电子科技大学 一种基于累计属性的头部姿态估计方法
CN107688773A (zh) * 2017-07-07 2018-02-13 北京联合大学 一种基于深度学习的手势识别方法
CN107766844A (zh) * 2017-11-13 2018-03-06 杭州有盾网络科技有限公司 一种网纹照人脸识别的方法、装置、设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103984416A (zh) * 2014-06-10 2014-08-13 北京邮电大学 一种基于加速度传感器的手势识别方法
US20160132773A1 (en) * 2014-11-06 2016-05-12 International Business Machines Corporation Method for Automatic Near-Real-Time Prediction, Classification, and Notification of Events in Natural Language Systems
CN106682733A (zh) * 2016-11-07 2017-05-17 上海资誉电子科技有限公司 无人机运动状态分析方法和装置
CN108664122A (zh) * 2018-04-04 2018-10-16 歌尔股份有限公司 一种姿态预测方法和装置

Cited By (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111027692A (zh) * 2019-11-04 2020-04-17 中国电子科技集团公司第十四研究所 一种目标运动态势预测方法及装置
CN111343340A (zh) * 2020-03-11 2020-06-26 杭州十域科技有限公司 一种基于大数据的低端硬件设备的姿态判别方法
CN111311714A (zh) * 2020-03-31 2020-06-19 北京慧夜科技有限公司 一种三维动画的姿态预测方法和系统
CN112116656A (zh) * 2020-08-03 2020-12-22 歌尔股份有限公司 同步定位与地图构建slam中的增量建图方法和装置
CN114296539B (zh) * 2020-10-07 2024-01-23 财团法人工业技术研究院 方向预测方法、虚拟实境装置及非暂态计算机可读取媒体
CN114296539A (zh) * 2020-10-07 2022-04-08 财团法人工业技术研究院 方向预测方法、虚拟实境装置及非暂态计算机可读取媒体
CN112486318A (zh) * 2020-11-26 2021-03-12 北京字跳网络技术有限公司 图像显示方法、装置、可读介质及电子设备
CN113268859A (zh) * 2021-04-25 2021-08-17 北京控制工程研究所 航天器在轨博弈的仿真模拟方法、系统及存储介质
CN113268859B (zh) * 2021-04-25 2023-07-14 北京控制工程研究所 航天器在轨博弈的仿真模拟方法、系统及存储介质
CN113221681B (zh) * 2021-04-26 2023-09-26 平安科技（深圳）有限公司 模型训练方法、动作姿态生成方法、装置、设备及介质
CN113221681A (zh) * 2021-04-26 2021-08-06 平安科技（深圳）有限公司 模型训练方法、动作姿态生成方法、装置、设备及介质
WO2022227208A1 (zh) * 2021-04-26 2022-11-03 平安科技（深圳）有限公司 模型训练方法、动作姿态生成方法、装置、设备及介质
CN113569327A (zh) * 2021-08-10 2021-10-29 同济大学 一种盾构机姿态优化方法、系统及模型训练方法、系统
CN113569327B (zh) * 2021-08-10 2024-02-09 同济大学 一种盾构机姿态优化方法、系统及模型训练方法、系统
CN113806172A (zh) * 2021-09-14 2021-12-17 北京百度网讯科技有限公司 一种处理设备状态参数的方法、装置、电子设备及存储介质
CN113806172B (zh) * 2021-09-14 2024-02-06 北京百度网讯科技有限公司 一种处理设备状态参数的方法、装置、电子设备及存储介质
CN113919236A (zh) * 2021-10-29 2022-01-11 中国国家铁路集团有限公司 预测轨道不平顺的模型生成方法和轨道不平顺的预测方法
CN114096010A (zh) * 2021-11-01 2022-02-25 江苏科技大学 一种面向监测的水下传感器网络能量持续管理方法
CN114096010B (zh) * 2021-11-01 2024-05-07 江苏科技大学 一种面向监测的水下传感器网络能量持续管理方法
CN114187360B (zh) * 2021-12-14 2024-02-06 西安交通大学 基于深度学习和四元数的头部位姿估计方法
CN114187360A (zh) * 2021-12-14 2022-03-15 西安交通大学 基于深度学习和四元数的头部位姿估计方法
CN116321286B (zh) * 2023-03-13 2023-09-22 湖北华中电力科技开发有限责任公司 应用智能算法的数据偏差鉴定系统
CN116321286A (zh) * 2023-03-13 2023-06-23 朱林清 应用智能算法的数据偏差鉴定系统
CN117560638A (zh) * 2024-01-10 2024-02-13 山东派蒙机电技术有限公司 应用于移动端通信系统的融合通信方法、装置及设备
CN117560638B (zh) * 2024-01-10 2024-03-22 山东派蒙机电技术有限公司 应用于移动端通信系统的融合通信方法、装置及设备

Also Published As

Publication number Publication date
CN108664122A (zh) 2018-10-16

Similar Documents

Publication Publication Date Title
WO2019192172A1 (zh) 一种姿态预测方法、装置和电子设备
US10860091B2 (en) Motion predictions of overlapping kinematic chains of a skeleton model used to control a computer system
CN109716361B (zh) 执行触摸运动预测的深度机器学习
CN108161882B (zh) 一种基于增强现实的机器人示教再现方法及装置
CN110880189B (zh) 联合标定方法及其联合标定装置和电子设备
US11100314B2 (en) Device, system and method for improving motion estimation using a human motion model
CN106527709B (zh) 一种虚拟场景调整方法及头戴式智能设备
CN106780608A (zh) 位姿信息估计方法、装置和可移动设备
KR102241414B1 (ko) 머신 러닝 모델을 이용한 특정 움직임에 대한 피드백을 제공하는 전자 장치 및 그 동작 방법
CN110068326B (zh) 姿态计算方法、装置、电子设备以及存储介质
CN105892658A (zh) 基于头戴显示设备预测头部姿态的方法和头戴显示设备
WO2020009715A2 (en) Tracking user movements to control a skeleton model in a computer system
CN108051001B (zh) 一种机器人移动控制方法、系统及惯性传感控制装置
WO2019080879A1 (zh) 数据处理方法、计算机设备和存储介质
CN107145706B (zh) 虚拟现实vr设备融合算法性能参数的评估方法及装置
CN111275769A (zh) 一种单目视觉参数的校正方法及装置
WO2023226156A1 (zh) 时间戳校正方法、装置、设备、介质及计算机程序产品
JP2018015436A (ja) 打具の挙動の解析装置
CN113916223B (zh) 定位方法及装置、设备、存储介质
US20230011082A1 (en) Combine Orientation Tracking Techniques of Different Data Rates to Generate Inputs to a Computing System
CN114296539B (zh) 方向预测方法、虚拟实境装置及非暂态计算机可读取媒体
CN113450903B (zh) 一种人体动作映射方法、装置、计算机设备和存储介质
US11454646B2 (en) Initiation of calibration of multiple sensor modules related to an orientation of a user of the sensor modules
CN115727871A (zh) 一种轨迹质量检测方法、装置、电子设备和存储介质
CN112486326B (zh) 手势控制运动的位置坐标预测方法、智能终端及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18913378

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18913378

Country of ref document: EP

Kind code of ref document: A1