CN109829439B

CN109829439B - Method and device for calibrating predicted value of head motion track

Info

Publication number: CN109829439B
Application number: CN201910107574.1A
Authority: CN
Inventors: 闫桂新; 楚明磊; 王云奇; 张�浩; 陈丽莉; 苗京花; 孙剑; 田文红; 董瑞君
Original assignee: BOE Technology Group Co Ltd; Beijing BOE Optoelectronics Technology Co Ltd
Current assignee: BOE Technology Group Co Ltd; Beijing BOE Optoelectronics Technology Co Ltd
Priority date: 2019-02-02
Filing date: 2019-02-02
Publication date: 2020-12-29
Anticipated expiration: 2039-02-02
Also published as: CN109829439A

Abstract

The invention discloses a method and a device for calibrating a predicted value of a head motion track, which relate to the technical field of head attitude estimation and can effectively calibrate the predicted value of the head motion track, and the main technical scheme of the invention is as follows: in the process of shooting a head motion video, extracting continuous N frame images from a received video stream image, wherein N is a positive integer and is more than 1; determining a head movement track formed by continuous N frame images and a movement state corresponding to the head movement track; predicting the head position in the shooting area at the next moment according to the head motion track, wherein the next moment is the moment corresponding to the capture of the (N + 1) th frame of image; searching whether a motion state sample matched with a motion state corresponding to the head motion track exists in a pre-established motion state set; and if so, calibrating the head position predicted in the shooting area at the next moment according to the motion state sample matched with the motion state corresponding to the head motion track.

Description

Method and device for calibrating predicted value of head motion track

Technical Field

The invention relates to the technical field of head posture estimation, in particular to a method and a device for calibrating a predicted value of a head motion track.

Background

In recent years, head pose has become a major research hotspot in the field of computer vision as a key technology in the research of motivation detection, face recognition and the like, and has attracted more and more attention of research workers. In the context of computer vision, head pose estimation is a process of inferring human head positioning from digital images, requiring the use of two-dimensional images to derive three-dimensional information.

At present, under the trend that the requirement of head posture estimation on real-time performance is higher and higher, the frame rate can be improved through a prediction method, and then the movement track of the head displayed by a computer is more timely and smooth. However, no matter which method is used to predict the head motion trajectory, certain errors exist, especially in the application scenario of increasing the frame rate, which brings with it the jitter of data, which will greatly reduce the user experience.

Disclosure of Invention

In view of this, the present invention provides a method and an apparatus for calibrating a predicted value of a head motion trajectory, and mainly aims to enable the predicted value of the head motion trajectory to approach a true value of the head motion trajectory as much as possible by calibrating the predicted value of the head motion trajectory, so as to avoid data jitter caused by increasing a frame rate, and improve user experience.

In order to solve the above problems, the present invention mainly provides the following technical solutions:

in one aspect, the present invention provides a method for calibrating a predicted value of a head motion trajectory, the method including:

in the process of shooting a head motion video, extracting continuous N frame images from a received video stream image, wherein N is a positive integer and is more than 1;

determining a head movement track formed by the continuous N frame images and a movement state corresponding to the head movement track;

predicting the head position in the shooting area at the next moment according to the head motion track, wherein the next moment is the moment corresponding to the capture of the (N + 1) th frame of image;

searching whether a motion state sample matched with a motion state corresponding to the head motion track exists in a pre-created motion state set or not, wherein a plurality of head motion track samples and a plurality of motion state samples corresponding to each head motion track sample are stored in the motion state set, and head positions at the next moment predicted according to one head motion track are stored in motion state data corresponding to the motion state samples;

and if so, calibrating the head position in the shooting area at the next predicted moment according to the motion state sample matched with the motion state corresponding to the head motion track.

Optionally, before the extracting N consecutive frame images from the received video stream image, the method further includes:

capturing a plurality of head motion track samples in the process of shooting a head motion video, wherein continuous M head positions are recorded in the head motion track samples, the head positions are determined by coordinate positions of x, y and z axis dimensions in space, and M is a positive integer and is greater than 1;

determining coordinate positions corresponding to the M head positions on one axis dimension;

acquiring time information corresponding to the positions of the M heads respectively;

determining a motion state sample corresponding to the head motion track sample and motion state data corresponding to the motion state sample on the axis dimension according to the coordinate position and the time information;

in the axis dimension, motion state data corresponding to the same motion state sample is summarized by counting motion state samples corresponding to a plurality of head motion track samples, and 1 or more groups of motion state data corresponding to the same motion state sample are obtained;

on the axis dimension, the motion state sample is taken as a query dimension, a motion state set sample is created, and a mapping relation between the motion state sample and motion state data is stored in the motion state set sample;

and summarizing the motion state set samples in three axis dimensions to create a motion state set.

Optionally, the determining, according to the coordinate position and the time information, a motion state sample corresponding to the head motion trajectory sample and motion state data corresponding to the motion state sample in the axis dimension includes:

on one axis dimension, forming a head motion track sequence by the coordinate positions corresponding to the M head positions respectively;

forming time sequences by using the time information respectively corresponding to the M head positions;

calculating the average speed between every two adjacent coordinate positions according to the coordinate positions and the time information respectively corresponding to the M head positions to obtain a plurality of corresponding average speeds to form a speed sequence;

determining a motion state corresponding to the head motion track sequence according to the average speed between every two adjacent coordinate positions, and taking the motion state as a motion state sample corresponding to the head motion track sample on the axis dimension;

obtaining a state value corresponding to the motion state by performing data processing on the motion state

Processing data information corresponding to the head motion track sequence and data information corresponding to the time sequence by utilizing curve fitting to obtain a predicted coordinate position of the (M + 1) th head position corresponding to the next moment in the axis dimension;

acquiring the real coordinate position of the (M + 1) th head position corresponding to the next moment in the axis dimension;

setting a calibration mode and a calibration weight corresponding to the calibration mode, wherein the calibration weight is used for indicating the deviation degree between the predicted coordinate position and the real coordinate position;

summarizing the head motion track sequence, the time sequence, the speed sequence, the state value corresponding to the motion state, the predicted coordinate position of the M +1 th head position, the real coordinate position of the M +1 th head position, the calibration mode and the calibration weight to serve as the motion state data corresponding to the motion state sample.

Optionally, the determining, according to the average speed between each two adjacent coordinate positions, a motion state corresponding to the head motion trajectory sequence, as a motion state sample corresponding to the head motion trajectory sample in the axis dimension, includes:

optionally selecting three continuous coordinate positions from the coordinate positions respectively corresponding to the M head positions;

respectively acquiring the average speed between the front two coordinate positions and the average speed between the rear two coordinate positions in the three coordinate positions;

if the corresponding numerical value between the first two coordinate positions is judged to be smaller, the head is determined to do accelerated motion;

if the corresponding numerical value between the two former coordinate positions is larger, determining that the head does deceleration movement;

if the average speed between the first two coordinate positions is judged to be equal to the average speed between the second two coordinate positions, the head is determined to do uniform motion;

and counting the motion states corresponding to any continuous three coordinate positions in the M head positions to obtain the motion states corresponding to the head motion track sequence, and taking the motion states as motion state samples.

Optionally, if N is equal to M, the searching whether a motion state sample matching the motion state corresponding to the head motion trajectory exists in the pre-created motion state set includes:

determining a succession of N head positions from a head motion trajectory consisting of said succession of N frame images, said head positions being determined by coordinate positions in spatial x, y, z axis dimensions;

determining coordinate positions corresponding to the N head positions on one axis dimension;

acquiring time information corresponding to the N head positions respectively;

determining a motion state corresponding to the head motion track on the axis dimension according to the coordinate position and the time information;

and querying a motion state set sample corresponding to the axis dimension, and if a motion state sample identical to the motion state exists, determining that a motion state sample matched with the motion state corresponding to the head motion track exists in the motion state set.

Optionally, the calibrating, according to the motion state sample matched with the motion state corresponding to the head motion trajectory, the head position in the shooting area at the next time that is predicted includes:

acquiring time information corresponding to the N head positions respectively;

calculating the average speed between every two adjacent coordinate positions according to the coordinate positions and the time information respectively corresponding to the N head positions to obtain a plurality of corresponding average speeds to form a speed sequence;

searching corresponding motion state data from the motion state set according to a motion state corresponding to a head motion track consisting of the continuous N frame images;

if the searched motion state data are a plurality of groups, extracting a speed sequence stored in each group of motion state data;

comparing the similarity of the speed sequence stored in each group of motion state data with the speed sequence corresponding to the head motion track consisting of the N head positions, and selecting the speed sequence meeting the preset condition from a plurality of groups of corresponding speed sequences;

and extracting a calibration mode and a calibration weight from the motion state data corresponding to the selected speed sequence, and calibrating the head prediction coordinate position corresponding to the (N + 1) th frame image corresponding to the next moment.

Optionally, the method further includes:

in the process of shooting a head motion video, after acquiring a head real position corresponding to the (N + 1) th frame image, marking a head motion track formed by the continuous N frame images as a newly added head motion track sample;

marking the motion state corresponding to the head motion track as a newly-added motion state sample;

obtaining motion state data corresponding to the newly added motion state sample according to the newly added head motion trail sample and the newly added motion state sample;

and adding the newly added motion state sample and the corresponding motion state data into a motion state set.

In another aspect, the present invention further provides a device for calibrating a predicted value of a head motion trajectory, the device including:

an extraction unit, configured to extract consecutive N frame images from a received video stream image in a process of capturing a head motion video, where N is a positive integer and is greater than 1;

a determination unit configured to determine a head movement trajectory composed of the N consecutive frame images extracted by the extraction unit and a movement state corresponding to the head movement trajectory;

a prediction unit, configured to predict a head position within the shooting area at a next time according to the head movement trajectory determined by the determination unit, where the next time is a time corresponding to capturing of the (N + 1) th frame of image;

a searching unit, configured to search whether a motion state sample matching a motion state corresponding to the head motion trajectory determined by the determining unit exists in a pre-created motion state set, where the motion state set stores a plurality of head motion trajectory samples and a plurality of motion state samples corresponding to each head motion trajectory sample, and motion state data corresponding to the motion state samples stores a head position at a next time predicted according to one head motion trajectory;

and if the head motion trajectory exists, calibrating the head position predicted at the next moment in the shooting area according to the motion state sample matched with the motion state corresponding to the head motion trajectory.

Optionally, the apparatus further comprises:

the head motion video shooting device comprises a capturing unit, a processing unit and a processing unit, wherein the capturing unit is used for capturing a plurality of head motion track samples, continuous M head positions are recorded in the head motion track samples, the head positions are determined by coordinate positions of x, y and z axis dimensions in space, and M is a positive integer and is more than 1;

the determining unit is further configured to determine, in one axis dimension, coordinate positions corresponding to the M head positions, respectively;

an obtaining unit, configured to obtain time information corresponding to the M head positions, respectively;

the determining unit is further configured to determine, according to the coordinate position and the time information, a motion state sample corresponding to the head motion trajectory sample and motion state data corresponding to the motion state sample in the axis dimension;

the summarizing unit is used for summarizing motion state data corresponding to the same motion state sample by counting the motion state samples corresponding to the head motion track samples on the axis dimension to obtain 1 or more groups of motion state data corresponding to the same motion state sample;

the creating unit is used for creating a motion state set sample by taking the motion state sample as a query dimension on the axis dimension, wherein the motion state set sample stores a mapping relation between the motion state sample and motion state data;

the creating unit is further configured to aggregate the motion state set samples in three axis dimensions, and create a motion state set.

Optionally, the determining unit includes:

the composition module is used for composing a head motion track sequence by the coordinate positions corresponding to the M head positions on one axis dimension;

the composition module is further configured to compose a time sequence from the time information corresponding to the positions of the M heads;

the calculation module is used for calculating the average speed between every two adjacent coordinate positions according to the coordinate positions and the time information respectively corresponding to the M head positions to obtain a plurality of corresponding average speeds and form a speed sequence;

a determining module, configured to determine, according to an average speed between each two adjacent coordinate positions, a motion state corresponding to the head motion trajectory sequence, as a motion state sample corresponding to the head motion trajectory sample in the axis dimension;

a processing module for obtaining a state value corresponding to the motion state by performing data processing on the motion state

The prediction module is used for processing the data information corresponding to the head motion track sequence and the data information corresponding to the time sequence by utilizing curve fitting to obtain a predicted coordinate position of the (M + 1) th head position corresponding to the next time in the axis dimension;

an obtaining module, configured to obtain, in the axis dimension, a real coordinate position of an M +1 th head position corresponding to the next time;

the setting module is used for setting a calibration mode and a calibration weight corresponding to the calibration mode, and the calibration weight is used for indicating the deviation degree between the predicted coordinate position and the real coordinate position;

and the summarizing module is used for summarizing the head motion track sequence, the time sequence, the speed sequence, the state value corresponding to the motion state, the predicted coordinate position of the M +1 th head position, the real coordinate position of the M +1 th head position, the calibration mode and the calibration weight, and taking the collected data as the motion state data corresponding to the motion state sample.

Optionally, the determining module includes:

a selection submodule for selecting optionally continuous three coordinate positions from the coordinate positions respectively corresponding to the M head positions;

the obtaining submodule is used for respectively obtaining the average speed between the front two coordinate positions and the average speed between the rear two coordinate positions in the three coordinate positions;

the determining submodule is used for determining that the head does accelerated motion if the corresponding numerical value between the two former coordinate positions is judged to be smaller;

the determining submodule is further used for determining that the head does deceleration motion if the corresponding numerical value between the two former coordinate positions is judged to be larger;

the determining submodule is further used for determining that the head does uniform motion if the average speed between the two former coordinate positions is equal to the average speed between the two latter coordinate positions;

and the counting submodule is used for counting the motion states corresponding to any continuous three coordinate positions in the M head positions to obtain the motion states corresponding to the head motion track sequence and using the motion states as motion state samples.

Optionally, if N is equal to M, the search unit includes:

a determining module for determining a series of N head positions from a head motion trajectory composed of the series of N frame images, the head positions being determined by coordinate positions in spatial x, y, z-axis dimensions;

the determining module is further configured to determine, in one axis dimension, coordinate positions corresponding to the N head positions, respectively;

an obtaining module, configured to obtain time information corresponding to the N head positions respectively;

the determining module is further configured to determine a motion state corresponding to the head motion trajectory in the axis dimension according to the coordinate position and the time information;

and the query module is used for querying the motion state set sample corresponding to the axis dimension, and if the motion state sample same as the motion state exists, determining that the motion state sample matched with the motion state corresponding to the head motion track exists in the motion state set.

Optionally, the prediction unit includes:

the calculation module is used for calculating the average speed between every two adjacent coordinate positions according to the coordinate positions and the time information respectively corresponding to the N head positions to obtain a plurality of corresponding average speeds and form a speed sequence;

the searching module is used for searching corresponding motion state data from the motion state set according to the motion state corresponding to the head motion track formed by the continuous N frame images;

the extracting module is used for extracting the speed sequence stored in each group of motion state data if the searched motion state data are a plurality of groups;

the selection module is used for comparing the similarity of the speed sequences stored in each group of motion state data with the speed sequences corresponding to the head motion track consisting of the N head positions, and selecting the speed sequences meeting the preset conditions from the multiple groups of corresponding speed sequences;

and the calibration module is used for extracting a calibration mode and a calibration weight from the motion state data corresponding to the selected speed sequence and calibrating the head prediction coordinate position corresponding to the (N + 1) th frame image corresponding to the next moment.

Optionally, the apparatus further comprises:

the marking unit is used for marking a head motion track formed by the continuous N frame images as a newly added head motion track sample after acquiring the head real position corresponding to the (N + 1) th frame image in the process of shooting the head motion video;

the marking unit is further configured to mark a motion state corresponding to the head motion trajectory as a newly added motion state sample;

the determining unit is further configured to obtain motion state data corresponding to the newly added motion state sample according to the newly added head motion trajectory sample and the newly added motion state sample;

and the adding unit is used for adding the newly added motion state sample and the corresponding motion state data to a motion state set.

By the technical scheme, the technical scheme provided by the invention at least has the following advantages:

the invention provides a method and a device for calibrating a predicted value of a head motion track, the method and the device calibrate the predicted value of the head motion track by utilizing a pre-established motion state set, and because head motion track samples which have executed a task of predicting the position of a head are stored in the motion state set, if the predicted value corresponding to the head motion track in the same motion state is calibrated by utilizing the head motion track samples, the method is more favorable for improving the accuracy of calibration.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 is a flowchart of a method for calibrating a predicted value of a head motion trajectory according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a face feature point according to an embodiment of the present invention;

fig. 3 is a flowchart of another method for calibrating a predicted value of a head movement trajectory according to an embodiment of the present invention;

fig. 4 is a block diagram of a calibration apparatus for a predicted value of a head motion trajectory according to an embodiment of the present invention;

fig. 5 is a block diagram of another calibration apparatus for a predicted value of a head movement trajectory according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

The embodiment of the present invention provides a calibration method for a predicted value of a head motion trajectory, as shown in fig. 1, the method calibrates the predicted value of the head motion trajectory by using a pre-created motion state set, and makes the predicted value of the head motion trajectory approach to an actual value of the head motion trajectory as much as possible, and for this, the embodiment of the present invention provides the following specific steps:

101. in the process of capturing a head motion video, consecutive N frame images are extracted from the received video stream images.

Wherein N is a positive integer and greater than 1.

In the embodiment of the invention, the moving video of the head can be shot through the camera, the frame rate, the image size and the color mode of the camera can be preset before shooting, and in the shooting process, the computer connected with the camera receives the video stream image corresponding to the moving video of the head in real time and displays the video stream image to a user according to the preset frame rate of the camera. For the embodiment of the invention, continuous N frame images can be selected for predicting the head position at the next moment after the head track according to the head motion track formed by the N frame images before the computer does not calculate the accurate head position at the next moment according to the received video stream images, thereby being beneficial to improving the real-time performance of the computer for displaying the head motion track in the man-machine interaction process.

102. And determining a head motion track formed by continuous N frame images and a motion state corresponding to the head motion track.

In the embodiment of the present invention, the specific steps of determining the head motion trajectory composed of consecutive N frame images may be as follows:

in the first step, the first frame image is selected from the continuous N frame images and is marked as I (x, y), and the x and the y represent coordinates in the image I, and the image is subjected to Gaussian filtering to reduce image noise.

Secondly, using a haar feature classifier to perform face detection on the image I (x, y), and using an ASM algorithm to perform feature point detection on the face in the image I (x, y), wherein the total number of 68 feature points is shown in FIG. 2. Since the head posture is obtained by calculation by inputting 68 feature points as vectors into the pnp algorithm, the position of the head in space can be obtained and expressed as P_t(x,y,z,t), which indicates the head position at time t.

And thirdly, according to the two steps, the head positions corresponding to the N frame images respectively can be calculated, and then the head positions are combined to obtain the head motion track formed by the continuous N frame images.

In the embodiment of the present invention, after the head positions corresponding to the N frame images respectively can be calculated, since the capturing times corresponding to the N frame images respectively are known, the motion state of the head motion trajectory composed of consecutive N frame images can be derived.

103. According to the head movement trajectory, the head position in the shooting area at the next time is predicted.

The next time is the time corresponding to the capture of the (N + 1) th frame image.

In the embodiment of the present invention, a curve fitting method may be used to predict the head position in the shooting area at the next time, since the head position is determined by the coordinate position of the x, y, and z axis dimensions in space, for the convenience of describing this step clearly, taking the x axis dimension and selecting N as 5 as an example, continuing the above step 102 as follows:

adding the above-mentioned P_t(x, y, z, t) is decomposed into a set of three directions:

recording said P in the x-axis dimension_t(x, y, z, t) as follows:

wherein j represents the j-th track sequence of the head movement in the x direction, t_j1To t_j5Representing the 5 time instants belonging to the j-th sequence.

Sequence calculation by quadratic polynomial fitting method

Of the quadratic function f^j(t) from a quadratic function f^j(t) calculating t_j6Predicted value f of time^j(t_j6)。

It should be noted that, for the embodiment of the present invention, the time of predicting the head position should be as close to the time of acquiring the next frame image as possible, so as to ensure that the head position is predicted in time before acquiring the next frame image, and to help improve the accuracy of predicting the head position, for the specific method of determining the prediction time, the following steps are performed:

t_j6＝t_j5+ Δ t (equation 3)

Where α is used to adjust the temporal position of prediction, a minimum value α of 0 indicates no prediction, and a maximum value α of 1 indicates that the prediction time is approximately equal to the time of next frame image acquisition.

104. And searching whether a motion state sample matched with the motion state corresponding to the head motion track exists in the pre-created motion state set.

The head motion state collection comprises a plurality of head motion track samples and a plurality of motion state samples corresponding to each head motion track sample, and the motion state data corresponding to the motion state samples comprises head positions at the next moment predicted according to one head motion track.

In the embodiment of the invention, the head position in the camera visual field is shot, and in the visual field, the head motion track set is limited, so that the related content of the head motion track which is subjected to the prediction operation before can be put into a set, the set is a motion state set, the head motion tracks stored in the set are samples, each head motion track sample corresponds to a motion state and is called a motion state sample, and correspondingly, the motion state sample also corresponds to motion state data. Since two head movement trajectories are compared, if the corresponding movement states of the two head movement trajectories are the same and the corresponding movement state data are also similar, the two head movement trajectories are also similar to a large extent. For this reason, if the head movement trajectory to be subjected to prediction is very similar to the already-performed head movement trajectory stored in the movement state set, the known data related to the already-performed head movement trajectory also helps to calibrate the predicted value obtained subsequently to the head movement trajectory to be subjected to prediction.

105. And if so, calibrating the head position predicted in the shooting area at the next moment according to the motion state sample matched with the motion state corresponding to the head motion track.

In the embodiment of the present invention, if a motion state sample matching the motion state corresponding to the head motion trajectory exists in the motion state set created in advance, the head position in the shooting area at the next moment is predicted according to the motion state sample matching the motion state corresponding to the head motion trajectory, that is, the current predicted value is calibrated by using the relevant content of the head motion trajectory for which prediction has been performed in the motion state set.

In order to describe the above embodiments in more detail, another calibration method for a head motion trajectory predicted value is provided in an embodiment of the present invention, as shown in fig. 3, the method is to create a motion state set by using a known head motion trajectory before calibrating the predicted value of the head motion trajectory by using a pre-created motion state set, so as to cover multiple motion states as much as possible, so as to facilitate calibration of the predicted value obtained by a task of predicting the head motion trajectory in the same motion state, and the following specific steps are provided in an embodiment of the present invention:

201. a set of motion states is created.

It should be noted that, in the embodiment of the present invention, before performing prediction on the head motion trajectory, a motion state set should be preferentially created, and specific steps are as follows:

firstly, in the process of shooting a head movement video, capturing a plurality of head movement track samples, wherein continuous M head positions are recorded in the head movement track samples, M is a positive integer and is greater than 1, as the head positions are determined by coordinate positions of x, y and z axis dimensions in space, taking one axis dimension as an example, stating that a movement state set sample is created, and correspondingly, summarizing the movement state set samples respectively corresponding to the three axis dimensions to obtain a movement state set.

Secondly, taking an axis dimension as an example, the step of creating a motion state set sample comprises: firstly, determining coordinate positions corresponding to M head positions respectively on one axis dimension, and acquiring time information corresponding to the M head positions respectively; secondly, determining a motion state sample corresponding to the head motion track sample and motion state data corresponding to the motion state sample on an axis dimension according to the coordinate position and the time information; and thirdly, on the axis dimension, summarizing the motion state data corresponding to the same motion state sample by counting the motion state samples corresponding to the plurality of head motion track samples to obtain 1 or more groups of motion state data corresponding to the same motion state sample.

For example, for a head movement locus, taking the x-axis direction as an example, the values of the head position x are recorded for 5 times in succession, and the time t is obtained₁，t₂，t₃，t₄，t₅Value x of₁，x₂，x₃，x₄，x₅. An average speed v is generated between two adjacent positions, and v is obtained₁，v₂，v₃，v₄Wherein v is_i＝(x_i+1-x_i)/(t_i+1-t_i) And i is 1,2,3, 4. There are three situations between two adjacent speeds:

(1)v_i<v_i+1at this time, the head makes an accelerated motion, which is represented by numeral 2;

(2)v_i>vi +1, at this time, the head performs deceleration motion, which is represented by the numeral 1;

(3) vi +1, where the head moves at a constant speed, indicated by the numeral 0.

Thus, 3 state values s1, s2 and s3 are generated among v1, v2, v3 and v4 respectively, each state value has values of 0,1 and 2, and s₁，s₂，s₃Indicating the state of motion of the head during these several moments. Binary corresponding 0,1,2 are 00,01,10, respectively, in a computer, and an 8-bit symbol is used to represent these three values: 00s₁s₂s₃. According to s₁，s₂，s₃There may be 27 different combinations of values of (a), so the motion state set created by the present invention includes 27 different motion states:

State＝{S₁,S₂，...,S₂₇}. (formula 5)

The 27 state includes the following x₁，x₂，x₃，x₄，x₅Any newly-appearing motion state can be added to the set for all possibilities of basic head motion states, so that for the embodiment of the present invention, the same head motion trajectory can correspond to multiple sets of motion state data according to different motion states.

Further, a detailed statement is made on the method for determining the motion state sample corresponding to the head motion trajectory sample and the motion state data corresponding to the motion state sample in the axis dimension according to the coordinate position and the time information, taking one axis dimension as an example, and the specific steps may be as follows:

firstly, on one axis dimension, forming a head motion track sequence by using coordinate positions corresponding to M head positions respectively, forming a time sequence by using time information corresponding to M head positions respectively, and calculating an average speed between every two adjacent coordinate positions according to the coordinate positions corresponding to M head positions respectively and the time information to obtain a plurality of corresponding average speeds to form a speed sequence.

For example, following the above example, for one head movement trajectory, taking the x-axis direction as an example, the following data information is obtained by recording the values of the head position x for 5 consecutive times:

head motion trajectory sequence X_real＝{x₁，x₂，x₃，x₄，x₅}

Time sequence T_real＝{t₁，t₂，t₃，t₄，t₅}

Velocity sequence V ═ V₁，v₂，v₃，v₄}

Secondly, determining a motion state corresponding to the head motion track sequence according to the average speed between every two adjacent coordinate positions, and using the motion state as a motion state sample corresponding to the head motion track sample in the axis dimension, wherein the motion state sample comprises: optionally selecting continuous three coordinate positions from the coordinate positions corresponding to the M head positions respectively, acquiring the average speed between the first two coordinate positions and the average speed between the second two coordinate positions from the three coordinate positions respectively, determining that the head does accelerated motion if the corresponding numerical value between the first two coordinate positions is smaller, determining that the head does decelerated motion if the corresponding numerical value between the first two coordinate positions is larger, determining that the head does uniform motion if the average speed between the first two coordinate positions is equal to the average speed between the second two coordinate positions, and counting the motion states corresponding to any continuous three coordinate positions in the M head positions to obtain the motion states corresponding to the head motion track sequence as a motion state sample.

For example, following the above example, for a head movement track, taking the x-axis direction as an example, 5 consecutive head position x values are recorded, and there are three movement states between two adjacent speeds.

(2)v_i>v_i+1at this time, the head performs deceleration movement, which is represented by numeral 1;

(3)v_i＝v_i+1at this time, the head makes a uniform motion, which is represented by a numeral 0.

Then, processing the data information corresponding to the head motion trajectory sequence and the data information corresponding to the time sequence by using curve fitting to obtain a predicted coordinate position of the (M + 1) th head position corresponding to the next time in the axis dimension, and specifically obtaining the predicted coordinate position of the head position, please refer to step 103, which is not described herein again. And acquiring the real coordinate position of the (M + 1) th head position corresponding to the next moment in the axis dimension.

Thirdly, setting a calibration mode and a calibration weight corresponding to the calibration mode, wherein the calibration weight is used for indicating the deviation degree between the predicted coordinate position and the real coordinate position.

For example, the calibration mode style ═ {0,1}, and the weight alpha is calibrated.

(1) When style is 0, the predicted value tends to be stable, and at this time, the predicted value tends to be stable

alpha＝(f(t₆)-x₆)/f(t₆)，alpha∈[-1,1](formula 6)

(2) When style is 1, it indicates that the deviation between the predicted value and the true value is large, and at this time, the style indicates that the deviation is large

alpha＝f(t₆)-x₆(formula 7)

Fourthly, summarizing the head motion track sequence, the time sequence, the speed sequence, the state value corresponding to the motion state, the predicted coordinate position of the (M + 1) th head position, the real coordinate position of the (M + 1) th head position, the calibration mode and the calibration weight to be used as the motion state data corresponding to the motion state sample.

202. In the process of shooting a head motion video, continuous N frame images are extracted from received video stream images, wherein N is a positive integer and is more than 1.

In the embodiment of the present invention, please refer to step 101 for the statement of this step, which is not described herein again.

203. And determining a head motion track formed by continuous N frame images and a motion state corresponding to the head motion track.

In the embodiment of the present invention, please refer to step 102 for the statement of this step, which is not described herein again.

204. According to the head movement trajectory, the head position in the shooting area at the next time is predicted.

In the embodiment of the present invention, please refer to step 103 for the statement of this step, which is not described herein again.

205. And searching whether a motion state sample matched with the motion state corresponding to the head motion track exists in a pre-created motion state set.

In the embodiment of the present invention, it should be noted that, when calibrating the head position prediction value corresponding to the currently executed predicted head motion trajectory by using the related data content of the head motion trajectory samples in the motion state set, it should be ensured that the length of the head motion trajectory samples and the length of the currently executed predicted head motion trajectory samples should be the same, so N is defined to be equal to M in the embodiment of the present invention.

Specifically, the specific method for searching whether a motion state sample matched with the motion state corresponding to the head motion trajectory exists in a pre-created motion state set includes: determining continuous N head positions from a head motion track consisting of continuous N frame images, wherein the head positions are determined by coordinate positions of x, y and z axis dimensions in space, determining the coordinate positions corresponding to the N head positions respectively in one axis dimension, acquiring time information corresponding to the N head positions respectively, determining a motion state corresponding to the head motion track in the axis dimension according to the coordinate positions and the time information, inquiring a motion state set sample corresponding to the axis dimension, and determining that a motion state sample matched with the motion state corresponding to the head motion track exists in the motion state set if the motion state sample same as the motion state exists.

For example, following the above example, for one head movement trajectory, taking the x-axis direction as an example, the values of the head position x are recorded 5 times in succession, and the calculation is performed

The state values of the sequence, find the matching entry in the motion state set.

206. And if so, calibrating the head position predicted in the shooting area at the next moment according to the motion state sample matched with the motion state corresponding to the head motion track.

In the embodiment of the present invention, the method in this step may include the following specific steps:

firstly, determining continuous N head positions from a head motion track consisting of continuous N frame images, wherein the head positions are determined by coordinate positions of x, y and z axis dimensions in space, determining the coordinate positions corresponding to the N head positions respectively in one axis dimension, acquiring time information corresponding to the N head positions respectively, calculating the average speed between every two adjacent coordinate positions according to the coordinate positions and the time information corresponding to the N head positions respectively, obtaining a plurality of corresponding average speeds, and forming a speed sequence,

secondly, searching corresponding motion state data according to the motion state corresponding to the head motion track consisting of continuous N frame images from the motion state set, if the searched motion state data are multiple groups, extracting the speed sequence stored in each group of motion state data, selecting the speed sequence meeting the preset condition from the plurality of groups of corresponding speed sequences, wherein the speed sequence meeting the preset condition is that the speed sequence which is most similar to the speed sequence of the head motion track consisting of N frame images is searched from the plurality of groups of speed sequences, this is because, on the premise that the motion states are the same, if the velocity sequences are more similar, the head positions predicted to the head motion trajectory to a great extent may also be more similar, furthermore, the predicted value of the head position is calibrated by using the motion state data corresponding to the speed sequence, which is more beneficial to improving the accuracy of calibration.

For example, following the above example, when two motion states are similar, they have similar velocity sequences, and thus traverse

Find a match condition for each motion state in (1)

State of motion of

Wherein

To represent

Q th under motion_jThe speed values of the group data. If it is

If the predicted value does not exist, the predicted value is not processed, and if the predicted value exists, calibration is carried out.

Thirdly, extracting a calibration mode and a calibration weight from the motion state data corresponding to the selected speed sequence, and calibrating the head prediction coordinate position corresponding to the (N + 1) th frame image corresponding to the next moment.

For example, obtain

In the calibration mode

When in use

When the calibration value is

f^j(t_j6)′＝f^j(t_j6)-alpha^qj*f^j(t_j6) (ii) a (formula 9)

When in use

When the calibration value is

Further, in the process of shooting the head motion video, after the head real position corresponding to the (N + 1) th frame image is obtained, the motion state set is updated, a head motion track formed by continuous N frame images is marked as a new head motion track sample, a motion state corresponding to the head motion track is marked as a new motion state sample, motion state data corresponding to the new motion state sample is obtained according to the new head motion track sample and the new motion state sample, and the new motion state sample and the corresponding motion state data are added into the motion state set. However, to avoid storing redundant motion state data, an upper limit may be set, and if the number of sets of motion state data reaches this upper limit, it may be necessary to delete duplicate or similar data.

In the embodiment of the present invention, after the calibration task is completed, the head motion sequence may be updated, and then the head motion trajectory at the next time is predicted, for example, in N consecutive frame images, the first frame image is deleted, the acquired N +1 th frame image is added, and the head position at the next time of the head motion trajectory is predicted.

Further, as an implementation of the method shown in fig. 1 and fig. 2, an embodiment of the present invention provides a calibration apparatus for a predicted value of a head movement trajectory. The embodiment of the apparatus corresponds to the embodiment of the method, and for convenience of reading, details in the embodiment of the apparatus are not repeated one by one, but it should be clear that the apparatus in the embodiment can correspondingly implement all the contents in the embodiment of the method. The device is applied to the calibration of the predicted value of the head motion track, and particularly as shown in fig. 4, the device comprises:

an extracting unit 301, configured to extract N consecutive frame images from a received video stream image in a process of capturing a head motion video, where N is a positive integer and is greater than 1;

a determining unit 302, configured to determine a head motion trajectory composed of the consecutive N frame images extracted by the extracting unit 301 and a motion state corresponding to the head motion trajectory;

a predicting unit 303, configured to predict a head position in the shooting area at a next time according to the head motion trajectory determined by the determining unit 302, where the next time is a time corresponding to capturing of the N +1 th frame image;

a searching unit 304, configured to search whether a motion state sample matching the motion state corresponding to the head motion trajectory determined by the determining unit 302 exists in a pre-created motion state set, where a plurality of head motion trajectory samples and a plurality of motion state samples corresponding to each head motion trajectory sample are stored in the motion state set, and a head position at a next time predicted according to one head motion trajectory is stored in motion state data corresponding to the motion state sample;

a calibration unit 305, configured to calibrate, if there is any, a head position within the shooting area at the predicted next time according to a motion state sample matching a motion state corresponding to the head motion trajectory.

Further, as shown in fig. 5, the apparatus further includes:

a capturing unit 306, configured to capture a plurality of head motion trajectory samples in the process of capturing a head motion video, where the head motion trajectory samples have M continuous head positions recorded therein, the head positions being determined by coordinate positions of x, y, and z axis dimensions in space, and M being a positive integer and greater than 1;

the determining unit 302 is further configured to determine, in one axis dimension, coordinate positions corresponding to the M head positions, respectively;

an obtaining unit 307, configured to obtain time information corresponding to the M head positions respectively;

the determining unit 302 is further configured to determine, according to the coordinate position and the time information, a motion state sample corresponding to the head motion trajectory sample and motion state data corresponding to the motion state sample in the axis dimension;

a summarizing unit 308, configured to summarize motion state data corresponding to the same motion state sample by counting motion state samples corresponding to a plurality of head motion trajectory samples in the axis dimension, so as to obtain 1 or more sets of motion state data corresponding to the same motion state sample;

a creating unit 309, configured to create a motion state set sample by using the motion state sample as a query dimension in the axis dimension, where a mapping relationship between the motion state sample and motion state data is stored in the motion state set sample;

the creating unit 309 is further configured to aggregate the motion state set samples in three axis dimensions, and create a motion state set.

Further, as shown in fig. 5, the determining unit 302 includes:

a composing module 3021, configured to compose a head motion trajectory sequence from coordinate positions corresponding to the M head positions in one axis dimension;

the composing module 3021 is further configured to compose time sequences by using the time information corresponding to the M head positions respectively;

a calculating module 3022, configured to calculate an average speed between every two adjacent coordinate positions according to the coordinate positions and the time information respectively corresponding to the M head positions, to obtain a plurality of corresponding average speeds, and form a speed sequence;

a determining module 3023, configured to determine, according to the average speed between each two adjacent coordinate positions, a motion state corresponding to the head motion trajectory sequence, as a motion state sample corresponding to the head motion trajectory sample in the axis dimension;

a processing module 3024, configured to perform data processing on the motion state to obtain a state value corresponding to the motion state

A prediction module 3025, configured to process, by using curve fitting, data information corresponding to the head movement trajectory sequence and data information corresponding to the time sequence, to obtain a predicted coordinate position of an M +1 th head position corresponding to a time immediately after the axis dimension;

an obtaining module 3026, configured to obtain, in the axis dimension, a real coordinate position of an M +1 th head position corresponding to the next time;

a setting module 3027, configured to set a calibration manner and a calibration weight corresponding to the calibration manner, where the calibration weight is used to indicate a degree of deviation between the predicted coordinate position and the real coordinate position;

a summarizing module 3028, configured to summarize the head motion trajectory sequence, the time sequence, the speed sequence, the state value corresponding to the motion state, the predicted coordinate position of the M +1 th head position, the real coordinate position of the M +1 th head position, the calibration manner, and the calibration weight, as the motion state data corresponding to the motion state sample.

Further, as shown in fig. 5, the determining module 3023 includes:

a selecting submodule 30231 for selecting optionally continuous three coordinate positions among the coordinate positions corresponding to the M head positions, respectively;

an obtaining submodule 30232 configured to obtain, in the three coordinate positions, an average velocity between the first two coordinate positions and an average velocity between the second two coordinate positions, respectively;

a determining submodule 30233, configured to determine that the head performs an accelerated motion if it is determined that the corresponding value between the two previous coordinate positions is small;

the determining submodule 30233 is further configured to determine that the head performs deceleration motion if the corresponding value between the two previous coordinate positions is determined to be larger;

the determining submodule 30233 is further configured to determine that the head does uniform motion if it is determined that the average speed between the first two coordinate positions is equal to the average speed between the second two coordinate positions;

and the counting submodule 30234 is configured to count motion states corresponding to any three consecutive coordinate positions in the M head positions, and obtain a motion state corresponding to the head motion trajectory sequence, which is used as a motion state sample.

Further, as shown in fig. 5, if N is equal to M, the lookup unit 304 includes:

a determining module 3041, configured to determine, from a head motion trajectory composed of the consecutive N frame images, consecutive N head positions, where the head positions are determined by coordinate positions of spatial x, y, and z-axis dimensions;

the determining module 3041 is further configured to determine, in an axis dimension, coordinate positions corresponding to the N head positions, respectively;

an obtaining module 3042, configured to obtain time information corresponding to the N head positions respectively;

the determining module 3041 is further configured to determine, according to the coordinate position and the time information, a motion state corresponding to the head motion trajectory in the axis dimension;

the query module 3043 is configured to query the motion state set sample corresponding to the axis dimension, and if there is a motion state sample that is the same as the motion state, determine that there is a motion state sample in the motion state set that matches the motion state corresponding to the head motion trajectory.

Further, as shown in fig. 5, the prediction unit 303 includes:

a determining module 3031, configured to determine, from a head motion trajectory composed of the consecutive N frame images, consecutive N head positions, where the head positions are determined by coordinate positions of spatial x, y, and z-axis dimensions;

the determining module 3031 is further configured to determine, in one axis dimension, coordinate positions corresponding to the N head positions respectively;

an obtaining module 3032, configured to obtain time information corresponding to the respective positions of the N headers;

a calculating module 3033, configured to calculate an average speed between every two adjacent coordinate positions according to the coordinate positions and the time information corresponding to the N head positions, respectively, to obtain a plurality of corresponding average speeds, and form a speed sequence;

a searching module 3034, configured to search, from the motion state set, corresponding motion state data according to a motion state corresponding to a head motion trajectory formed by the consecutive N frame images;

an extracting module 3035, configured to extract a speed sequence stored in each set of motion state data if the searched motion state data is multiple sets;

a selecting module 3036, configured to select a speed sequence meeting a preset condition from multiple sets of corresponding speed sequences by performing similarity comparison between the speed sequence stored in each set of motion state data and the speed sequence corresponding to the head motion trajectory composed of the N head positions;

a calibration module 3037, configured to extract a calibration mode and a calibration weight from the motion state data corresponding to the selected velocity sequence, and calibrate the head prediction coordinate position corresponding to the N +1 th frame image corresponding to the next time.

Further, as shown in fig. 5, the apparatus further includes:

a marking unit 310, configured to mark a head motion trajectory formed by the N +1 th frame images as a new head motion trajectory sample after acquiring a head real position corresponding to the N +1 th frame image in a process of shooting a head motion video;

the marking unit 310 is further configured to mark the motion state corresponding to the head motion trajectory as a newly added motion state sample;

the determining unit 302 is further configured to obtain motion state data corresponding to the newly added motion state sample according to the newly added head motion trajectory sample and the newly added motion state sample;

an adding unit 311, configured to add the newly added motion state sample and the corresponding motion state data to a motion state set.

In summary, the present invention provides a method and an apparatus for calibrating a predicted value of a head motion trajectory, in which a pre-created motion state set is used to calibrate the predicted value of the head motion trajectory, and because a head motion trajectory sample that has already executed a task of predicting a head position is stored in the motion state set, if the predicted value corresponding to the head motion trajectory in the same motion state is calibrated by using the head motion trajectory sample, the accuracy of calibration is further improved. In addition, before the predicted value of the head motion track is calibrated by using the pre-created motion state set, the motion state set is created by using the known head motion track, and multiple motion states are covered as much as possible, so that the predicted value obtained by the head motion track predicting task in the same motion state can be calibrated.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory computer readable media (trahsity media) such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for calibrating a predicted value of a head movement track, the method comprising:

summarizing motion state set samples on three axis dimensions, and creating a motion state set;

2. The method according to claim 1, wherein the determining motion state samples corresponding to the head motion trail samples and the motion state data corresponding to the motion state samples in the axis dimension according to the coordinate position and the time information comprises:

obtaining a state value corresponding to the motion state by performing data processing on the motion state;

3. The method according to claim 2, wherein the determining the motion state corresponding to the head motion trail sequence according to the average speed between each two adjacent coordinate positions as the motion state sample corresponding to the head motion trail sample in the axis dimension comprises:

4. The method of claim 1, wherein if N is equal to M, said searching whether there is a motion state sample in a pre-created motion state set that matches a motion state corresponding to the head motion trajectory comprises:

acquiring time information corresponding to the N head positions respectively;

5. The method according to claim 2, wherein the calibrating the predicted head position within the capture area at the next time according to the motion state sample matched with the motion state corresponding to the head motion trajectory comprises:

acquiring time information corresponding to the N head positions respectively;

6. The method of claim 1, further comprising:

7. An apparatus for calibrating a predicted value of a head movement locus, the apparatus comprising:

a determining unit, configured to determine, in one axis dimension, coordinate positions corresponding to the M head positions, respectively;

the creating unit is further configured to collect motion state set samples in three axis dimensions and create a motion state set;

8. The apparatus of claim 7, wherein the determining unit comprises:

the processing module is used for performing data processing on the motion state to obtain a state value corresponding to the motion state;

9. The apparatus of claim 8, wherein the determining module comprises:

10. The apparatus of claim 7, wherein if N is equal to M, the lookup unit comprises:

11. The apparatus of claim 8, wherein the prediction unit comprises:

12. The apparatus of claim 7, further comprising:

13. An electronic device, comprising: a memory, a processor and a program stored on the memory and executable on the processor, the processor implementing a method of calibrating a predicted value of a head motion trajectory according to any one of claims 1 to 6 when executing the program.

14. A computer-readable storage medium, on which a computer program is stored, which when executed implements a method of calibrating a predicted value of a head movement trajectory according to any one of claims 1 to 6.